Building with datasets

Goal: get a registered Amorphic dataset ready for downstream work — look up or create it in the right domain, put files on it (or pull them off), and clean up when you are done. The same dataset can later be attached as a knowledge-base source (see AI features).

APIs used here

  • DatasetsApi — get_dataset, list_datasets, list_domains, create_dataset, delete_dataset, get_file_upload_url_for_dataset, download_dataset_file, delete_dataset_files

Assume datasets_api = DatasetsApi(client) from Use Cases.

Look up a dataset

Confirm the dataset is active and matches the name and domain you expect before you upload files or attach it to AI resources.

dataset = datasets_api.get_dataset(id=dataset_id, role_id=role_id)

if dataset.registration_status.lower() != "completed":
    raise ValueError("Dataset is not active")
if dataset.dataset_name != expected_name:
    raise ValueError("Dataset name does not match")
if dataset.domain != expected_domain:
    raise ValueError("Domain does not match")

List domains and datasets

Discover which domains you can use and which completed datasets already exist (for example S3 Athena or DynamoDB targets) so you reuse what is there instead of creating duplicates.

domains_response = datasets_api.list_domains(role_id=role_id)
domains = [
    {
        "DomainName": domain.domain_name,
        "DisplayName": domain.display_name,
        "DomainDescription": domain.description,
    }
    for domain in (domains_response.domains or [])
]

datasets_response = datasets_api.list_datasets(
    role_id=role_id,
    filter_expression=(
        "TargetLocation:s3athena,TargetLocation:dynamodb,"
        "RegistrationStatus:completed"
    ),
    projection_expression=(
        "DatasetId,DatasetName,DisplayName,Domain,FileType,"
        "TargetLocation,DatasetSchema,RegistrationStatus"
    ),
)

Create a dataset

If nothing with that name exists yet, register a new dataset so you have a place to store files and, later, attach as an AI source. Metadata varies by target (S3 vs DynamoDB, file type, schema). This example is a simple internal S3 dataset:

existing = datasets_api.list_datasets(
    role_id=role_id,
    filter_expression=f"DatasetName:{dataset_name}",
)
if existing.datasets:
    dataset_id = existing.datasets[0].dataset_id
else:
    metadata = {
        "DatasetName": dataset_name,
        "DatasetDescription": "Example dataset",
        "Domain": domain_name,
        "DatasetType": "internal",
        "DatasourceType": "api",
        "TargetLocation": "s3",
        "FileType": "csv",
        "TableUpdate": "append",
    }
    created = datasets_api.create_dataset(
        role_id=role_id,
        dataset_metadata=metadata,
    )
    dataset_id = created.dataset_id

After create, call get_dataset when you need fields that are not always on the create response (for example dataset_s3_path or created_by).

For a DynamoDB target, set TargetLocation to dynamodb, a partition key, and an optional schema. See the Datasets API reference for the full metadata shape.

Manage files on a dataset

Move data onto the dataset so consumers (queries, knowledge bases, downloads) can use it, or pull files back out and remove ones you no longer need.

Upload — request a presigned URL, then PUT the object:

upload = datasets_api.get_file_upload_url_for_dataset(
    role_id=role_id,
    pre_signed={
        "DatasetId": dataset_id,
        "FileName": file_name,
    },
)
upload_url = upload.message  # presigned PUT URL

Download:

download = datasets_api.download_dataset_file(
    role_id=role_id,
    id=dataset_id,
    filename=amorphic_file_id,
)
download_url = download.url

Delete one or more files (permanent delete). Set TruncateDataset when the dataset should be emptied as well:

datasets_api.delete_dataset_files(
    role_id=role_id,
    id=dataset_id,
    file_add_delete={
        "Operation": "permanent_delete",
        "Files": [amorphic_file_id],
        "TruncateDataset": True,
    },
)

To remove selected files without emptying the dataset, use TruncateDataset: False.

Delete a dataset

Remove the dataset when a create should be rolled back, or when it is no longer needed as a store or AI source.

datasets_api.delete_dataset(id=dataset_id, role_id=role_id)

See also