DatasetMetadata

pydantic model openapi_client.models.dataset_metadata.DatasetMetadata

DatasetMetadata

Fields:

• advanced_config (openapi_client.models.advanced_config.AdvancedConfig | None)

• are_ai_services_enabled (openapi_client.models.dataset_metadata_skip_file_header.DatasetMetadataSkipFileHeader | None)

• are_auto_constraint_suggestions_enabled (bool | None)

• assume_role (str | None)

• auto_refresh (bool | None)

• catalog_id (str | None)

• column_table_constraints (Dict[str, Any] | None)

• cors_dataset (str | None)

• cost_tags (List[Dict[str, Any]] | None)

• data_category (str | None)

• data_category_config (openapi_client.models.dataset_metadata_data_category_config.DatasetMetadataDataCategoryConfig | None)

• data_classification (List[str] | None)

• data_cleanup_duration (int | None)

• data_conversion_params (Dict[str, Any] | None)

• data_metrics_collection_options (openapi_client.models.data_metrics_collection_options.DataMetricsCollectionOptions | None)

• dataset_description (str | None)

• dataset_key_options (Dict[str, Any] | None)

• dataset_name (str)

• dataset_s3_path (str | None)

• dataset_schema (List[openapi_client.models.column_name_and_type.ColumnNameAndType] | None)

• dataset_type (str)

• datasource_id (str | None)

• datasource_type (str | None)

• directory_path (str | None)

• display_name (str | None)

• dist_key (str | None)

• dist_type (str | None)

• domain (str)

• dq_constraints (List[Dict[str, Any]] | None)

• encoding (str | None)

• file_delimiter (str | None)

• file_name (str | None)

• file_type (str | None)

• global_secondary_indexes (List[Dict[str, Any]] | None)

• hudi_table_options (Dict[str, Any] | None)

• iceberg_table_options (Dict[str, Any] | None)

• identity_attribute_values (List[str] | None)

• ingestion_type (str | None)

• input_regex (str | None)

• is_auto_ml_enabled (bool | None)

• is_bda_extraction_enabled (bool | None)

• is_data_cleanup_enabled (openapi_client.models.dataset_metadata_skip_file_header.DatasetMetadataSkipFileHeader | None)

• is_data_profiling_enabled (openapi_client.models.dataset_metadata_is_data_profiling_enabled.DatasetMetadataIsDataProfilingEnabled | None)

• is_data_validation_enabled (openapi_client.models.dataset_metadata_is_data_validation_enabled.DatasetMetadataIsDataValidationEnabled | None)

• is_delta_lake_table (bool | None)

• is_governed_table (bool | None)

• is_identity_needed (str | None)

• is_translate_ai_results_enabled (bool | None)

• keywords (List[str] | None)

• latest_record_indicator (openapi_client.models.column_name_and_type.ColumnNameAndType | None)

• life_cycle_policy_status (str | None)

• life_cycle_rules (openapi_client.models.dataset_life_cycle_rules.DatasetLifeCycleRules | None)

• local_secondary_indexes (List[Dict[str, Any]] | None)

• malware_detection_options (openapi_client.models.malware_detection_options.MalwareDetectionOptions | None)

• not_null_columns (List[str] | None)

• pagination_header_key (str | None)

• pagination_type (str | None)

• partition_key (openapi_client.models.key_schema.KeySchema | None)

• partition_keys (List[openapi_client.models.key_schema.KeySchema] | None)

• precombine_key (str | None)

• primary_key (str | None)

• primary_keys (List[str] | None)

• query_id (str | None)

• record_keys (List[str] | None)

• ser_de (str | None)

• serde_properties (Dict[str, str] | None)

• skip_file_header (openapi_client.models.dataset_metadata_skip_file_header.DatasetMetadataSkipFileHeader | None)

• skip_lz_process (openapi_client.models.dataset_metadata_skip_lz_process.DatasetMetadataSkipLZProcess | None)

• skip_row_count (Dict[str, Any] | None)

• skip_trash (openapi_client.models.dataset_metadata_skip_trash.DatasetMetadataSkipTrash | None)

• sort_key (openapi_client.models.key_schema.KeySchema | None)

• sort_keys (List[str] | None)

• sort_type (str | None)

• sql_statement (str | None)

• storage_type (str | None)

• table_name (str | None)

• table_properties (Dict[str, str] | None)

• table_update (str | None)

• target_location (str)

• target_table_prep_mode (str | None)

• unique_columns (List[str] | None)

• update_schema (str | None)

• view_type (str | None)

field dataset_name: StrictStr [Required] (alias 'DatasetName')

Unique name for the dataset. Must be unique within the domain.

Constraints:

• strict = True

field display_name: Optional[StrictStr] = None (alias 'DisplayName')

Human-readable display name shown in the UI. Defaults to DatasetName if not provided.

field dataset_description: Optional[StrictStr] = None (alias 'DatasetDescription')

Description of the dataset contents and purpose. Can be empty string but must be present.

field data_classification: Optional[List[StrictStr]] = None (alias 'DataClassification')

Data classification labels. Values must exist in the system’s DataClassification table.

field domain: StrictStr [Required] (alias 'Domain')

Domain the dataset belongs to. Must be an existing domain name.

Constraints:

• strict = True

field dataset_type: StrictStr [Required] (alias 'DatasetType')

Type of dataset being created.

Constraints:

• strict = True

field file_type: Optional[StrictStr] = None (alias 'FileType')

File format of the dataset data. Allowed values depend on TargetLocation.

field target_location: StrictStr [Required] (alias 'TargetLocation')

Storage backend where the dataset table is created.

Constraints:

• strict = True

field keywords: Optional[List[StrictStr]] = None (alias 'Keywords')

field table_update: Optional[StrictStr] = None (alias 'TableUpdate')

field file_delimiter: Optional[StrictStr] = ',' (alias 'FileDelimiter')

Column delimiter for CSV/TSV files. Defaults to comma.

field skip_file_header: Optional[DatasetMetadataSkipFileHeader] = None (alias 'SkipFileHeader')

field skip_lz_process: Optional[DatasetMetadataSkipLZProcess] = None (alias 'SkipLZProcess')

field datasource_type: Optional[StrictStr] = 'api' (alias 'DatasourceType')

Origin of data ingestion. Defaults to “api” if not provided.

field table_name: Optional[StrictStr] = None (alias 'TableName')

Table name for JDBC datasource datasets.

field directory_path: Optional[StrictStr] = None (alias 'DirectoryPath')

S3 prefix path for datasource file ingestion.

field is_data_profiling_enabled: Optional[DatasetMetadataIsDataProfilingEnabled] = None (alias 'IsDataProfilingEnabled')

field is_auto_ml_enabled: Optional[StrictBool] = None (alias 'IsAutoMLEnabled')

Enable AutoML model training on the dataset.

field is_bda_extraction_enabled: Optional[StrictBool] = None (alias 'IsBDAExtractionEnabled')

Enable BDA extraction.

field is_data_cleanup_enabled: Optional[DatasetMetadataSkipFileHeader] = None (alias 'IsDataCleanupEnabled')

field is_governed_table: Optional[StrictBool] = None (alias 'IsGovernedTable')

Whether the table is governed in Lake Formation.

field malware_detection_options: Optional[MalwareDetectionOptions] = None (alias 'MalwareDetectionOptions')

field target_table_prep_mode: Optional[StrictStr] = None (alias 'TargetTablePrepMode')

field is_data_validation_enabled: Optional[DatasetMetadataIsDataValidationEnabled] = None (alias 'IsDataValidationEnabled')

field are_ai_services_enabled: Optional[DatasetMetadataSkipFileHeader] = None (alias 'AreAIServicesEnabled')

field is_translate_ai_results_enabled: Optional[StrictBool] = None (alias 'IsTranslateAIResultsEnabled')

field life_cycle_policy_status: Optional[StrictStr] = None (alias 'LifeCyclePolicyStatus')

Lifecycle policy state for the dataset.

field life_cycle_rules: Optional[DatasetLifeCycleRules] = None (alias 'LifeCycleRules')

field data_metrics_collection_options: Optional[DataMetricsCollectionOptions] = None (alias 'DataMetricsCollectionOptions')

field advanced_config: Optional[AdvancedConfig] = None (alias 'AdvancedConfig')

field datasource_id: Optional[StrictStr] = None (alias 'DatasourceId')

Datasource identifier. Required when DatasourceType is “s3” or “jdbc”.

field dataset_s3_path: Optional[Annotated[str, Field(strict=True)]] = None (alias 'DatasetS3Path')

S3 path for external datasets. Must end with “/” and match s3://bucket/prefix/ format. Required when DatasetType is “external”.

field ser_de: Optional[StrictStr] = None (alias 'SerDe')

Glue SerDe library name. Only valid for s3athena/lf targets. Auto-assigned from FileType if not provided.

field serde_properties: Optional[Dict[str, StrictStr]] = None (alias 'SerdeProperties')

User-configurable Glue SerDe parameters for s3athena/lf tables.

field table_properties: Optional[Dict[str, StrictStr]] = None (alias 'TableProperties')

User-configurable Glue table parameters (TBLPROPERTIES) for s3athena/lf tables.

field view_type: Optional[StrictStr] = None (alias 'ViewType')

Type of DWH view to create. Required when DatasetType is “view”.

field sql_statement: Optional[StrictStr] = None (alias 'SqlStatement')

SQL CREATE VIEW statement.

field encoding: Optional[StrictStr] = None (alias 'Encoding')

Encoding of SqlStatement. When “base64”, the SQL must be base64-encoded.

field assume_role: Optional[StrictStr] = 'no' (alias 'AssumeRole')

Whether to use AssumeRole for s3athena standard views. Only valid when TargetLocation is “s3athena”.

field is_delta_lake_table: Optional[StrictBool] = None (alias 'IsDeltaLakeTable')

Whether this dataset uses Delta Lake table format.

field skip_trash: Optional[DatasetMetadataSkipTrash] = None (alias 'SkipTrash')

field auto_refresh: Optional[StrictBool] = None (alias 'AutoRefresh')

Enable auto-refresh for materialized views.

field data_cleanup_duration: Optional[StrictInt] = None (alias 'DataCleanupDuration')

Number of days after which data is automatically deleted. Only used when IsDataCleanupEnabled is true.

field update_schema: Optional[StrictStr] = None (alias 'UpdateSchema')

Schema update mode during complete-registration.

field dataset_schema: Optional[List[ColumnNameAndType]] = None (alias 'DatasetSchema')

field skip_row_count: Optional[Dict[str, Any]] = None (alias 'SkipRowCount')

field hudi_table_options: Optional[Dict[str, Any]] = None (alias 'HudiTableOptions')

field iceberg_table_options: Optional[Dict[str, Any]] = None (alias 'IcebergTableOptions')

Apache Iceberg table configuration (IsIcebergTable, IcebergCatalog, etc.).

field precombine_key: Optional[StrictStr] = None (alias 'PrecombineKey')

Hudi precombine field used to resolve duplicates during upsert.

field primary_key: Optional[StrictStr] = None (alias 'PrimaryKey')

field storage_type: Optional[StrictStr] = None (alias 'StorageType')

field input_regex: Optional[StrictStr] = None (alias 'InputRegex')

field partition_keys: Optional[List[KeySchema]] = None (alias 'PartitionKeys')

field global_secondary_indexes: Optional[List[Dict[str, Any]]] = None (alias 'GlobalSecondaryIndexes')

DynamoDB Global Secondary Index definitions.

field local_secondary_indexes: Optional[List[Dict[str, Any]]] = None (alias 'LocalSecondaryIndexes')

DynamoDB Local Secondary Index definitions.

field dist_key: Optional[StrictStr] = None (alias 'DistKey')

Redshift distribution key column name.

field dist_type: Optional[StrictStr] = None (alias 'DistType')

Redshift distribution style (KEY, EVEN, ALL, AUTO).

field is_identity_needed: Optional[StrictStr] = None (alias 'IsIdentityNeeded')

Whether to add an identity column to Redshift table.

field identity_attribute_values: Optional[List[StrictStr]] = None (alias 'IdentityAttributeValues')

Identity column seed and step values for Redshift.

field sort_keys: Optional[List[StrictStr]] = None (alias 'SortKeys')

Redshift sort key column names.

field sort_type: Optional[StrictStr] = None (alias 'SortType')

Redshift sort key type

field record_keys: Optional[List[StrictStr]] = None (alias 'RecordKeys')

field pagination_type: Optional[StrictStr] = None (alias 'PaginationType')

field pagination_header_key: Optional[StrictStr] = None (alias 'PaginationHeaderKey')

field catalog_id: Optional[StrictStr] = None (alias 'CatalogId')

field cors_dataset: Optional[StrictStr] = None (alias 'CORSDataset')

field partition_key: Optional[KeySchema] = None (alias 'PartitionKey')

field sort_key: Optional[KeySchema] = None (alias 'SortKey')

field primary_keys: Optional[List[StrictStr]] = None (alias 'PrimaryKeys')

Primary key column names for Redshift tables.

field unique_columns: Optional[List[StrictStr]] = None (alias 'UniqueColumns')

Columns with unique constraints for Redshift tables.

field not_null_columns: Optional[List[StrictStr]] = None (alias 'NotNullColumns')

Columns with NOT NULL constraints for Redshift tables.

field dq_constraints: Optional[List[Dict[str, Any]]] = None (alias 'DqConstraints')

Data quality constraint definitions

field are_auto_constraint_suggestions_enabled: Optional[StrictBool] = None (alias 'AreAutoConstraintSuggestionsEnabled')

field data_conversion_params: Optional[Dict[str, Any]] = None (alias 'DataConversionParams')

field column_table_constraints: Optional[Dict[str, Any]] = None (alias 'ColumnTableConstraints')

field dataset_key_options: Optional[Dict[str, Any]] = None (alias 'DatasetKeyOptions')

field latest_record_indicator: Optional[ColumnNameAndType] = None (alias 'LatestRecordIndicator')

field data_category: Optional[StrictStr] = None (alias 'DataCategory')

Special data category for the dataset.

field data_category_config: Optional[DatasetMetadataDataCategoryConfig] = None (alias 'DataCategoryConfig')

field query_id: Optional[StrictStr] = None (alias 'QueryId')

field file_name: Optional[StrictStr] = None (alias 'FileName')

field ingestion_type: Optional[StrictStr] = None (alias 'IngestionType')

Data ingestion type for JDBC datasource datasets.

field cost_tags: Optional[List[Dict[str, Any]]] = None (alias 'CostTags')

Cost management tags for DynamoDB target datasets.

to_str()

Returns the string representation of the model using alias

Return type:

str

to_json()

Returns the JSON representation of the model using alias

Return type:

str

classmethod from_json(json_str)

Create an instance of DatasetMetadata from a JSON string

Return type:

Optional[Self]

to_dict()

Return the dictionary representation of the model using alias.

This has the following differences from calling pydantic’s self.model_dump(by_alias=True):

• None is only added to the output dict for nullable fields that were set at model initialization. Other fields with value None are ignored.

Return type:

Dict[str, Any]

classmethod from_dict(obj)

Create an instance of DatasetMetadata from a dict

Return type:

Optional[Self]