sync_dataset_partitions

DatasetsApi.sync_dataset_partitions(id, role_id, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)

Sync partitions from source for lf and s3athena datasets

Triggers an asynchronous partition sync from the dataset S3 source via Step Functions. Supported only for TargetLocation lf or s3athena. Not supported for views, Iceberg tables, or Delta Lake tables (GE-1106: “sync partitions is/are not supported for {views| <target> datasets|deltalake/iceberg datasets}”). External datasets without partition keys return DS-1054. Internal datasets default the partition key to upload_date when none are defined. On success an email notification with more details is sent when the sync completes. No request body.

Parameters:

  • id (str) – Dataset ID to sync partitions for. Must be lf/s3athena and not a view/Iceberg/Delta Lake dataset. (required)

  • role_id (str) – Amorphic role ID the request is authorized against. (required)

  • _request_timeout (int, tuple(int, int), optional) – timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts.

  • _request_auth (dict, optional) – set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request.

  • _content_type (str, Optional) – force content-type for the request.

  • _headers (dict, optional) – set to override the headers for a single request; this effectively ignores the headers in the spec for a single request.

  • _host_index (int, optional) – set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request.

Return type:

SyncDatasetPartitionsResponse

Returns:

Returns the result object.

Request and Response Examples

HTTP: PUT /datasets/{id}/sync-partitions

Parameter examples

Name

In

Example

id

path

"b1e6c2a0-9f4d-4c7a-8f2e-1a2b3c4d5e6f"

Success (HTTP 200)

Partition sync process triggered. Email notification follows on completion.

Request / response example

{
  "Message": "Datasets partitions sync process triggered! Once completed, you will receive an email notification with more details!"
}

Errors

Documented error codes: GE-1106, DS-1054, RTE-1001.

HTTP 400

Validation failure. Body is {“Message”: “<CODE> - <text>”}. Common codes: GE-1106 (unsupported for views / non-lf-s3athena / deltalake/iceberg), DS-1054 (external table not partitioned).

Views cannot sync partitions

{
  "Message": "GE-1106 - sync partitions is/are not supported for views"
}

Iceberg/Delta Lake not supported

{
  "Message": "GE-1106 - sync partitions is/are not supported for deltalake/iceberg datasets"
}

Unsupported target location

{
  "Message": "GE-1106 - sync partitions is/are not supported for redshift datasets"
}

External dataset has no partitions

{
  "Message": "DS-1054 - Table monthly_orders is not partitioned."
}

HTTP 500

Backend/generic failure. Body is {“Message”: “<CODE> - <text>”}.

Request / response example

{
  "Message": "RTE-1001 - An error occurred."
}