trigger_iceberg_dataload_job_on_dataset

DatasetsApi.trigger_iceberg_dataload_job_on_dataset(id, type, role_id, dataload_files_input_obj, content_type=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)

Trigger Iceberg/Hudi/DeltaLake dataload for pending files

Starts an asynchronous Iceberg/Hudi/DeltaLake dataload for pending Landing Zone files on the dataset. Handler is fileUploadS3LZTrigger. Required: query type (iceberg | hudi | deltalake) and body Files (non-empty array of pending file-load FileName keys, max 750). Empty Files → IPV-1001. type missing → IPV-1052; invalid type → IPV-1071. type must match the dataset framework (GE-1034 on mismatch). Every Files entry must have LoadStatus pending (GE-1034 otherwise). Caller must be owner or editor. Rejected while DatasetStatus is {type}-data-loading (concurrent load). On success the load runs in the background; response Message is “{Framework} dataload started successfully and running in background”.

Parameters:

  • id (str) – Dataset ID (Iceberg, Hudi, or DeltaLake) to load files into. (required)

  • type (str) – Datalake framework for the load. Must match the dataset (IcebergTableOptions / HudiTableOptions / IsDeltaLakeTable). (required)

  • role_id (str) – Amorphic role ID used for authorization. Caller must be owner or editor of the dataset. (required)

  • dataload_files_input_obj (DataloadFilesInputObj) – Must include non-empty Files (pending file-load FileName keys, max 750). See DataloadFilesInputObj. (required)

  • content_type (str) – Must be application/json.

  • _request_timeout (int, tuple(int, int), optional) – timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts.

  • _request_auth (dict, optional) – set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request.

  • _content_type (str, Optional) – force content-type for the request.

  • _headers (dict, optional) – set to override the headers for a single request; this effectively ignores the headers in the spec for a single request.

  • _host_index (int, optional) – set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request.

Return type:

TriggerDataloadResponse

Returns:

Returns the result object.

Request and Response Examples

HTTP: POST /datasets/{id}/dataloads

Request

Must include non-empty Files (pending file-load FileName keys, max 750). See DataloadFilesInputObj.

Request / response example

{
  "Files": [
    "sales/monthly_orders/upload_date=1700000000/alice_49e02003-289e-41e9-a9f2-28a733349481_1700000001_orders.parquet"
  ]
}

Success (HTTP 200)

Async dataload started. Message is “{Iceberg|Hudi|Deltalake} dataload started successfully and running in background”; DatasetId echoes path id.

Request / response example

{
  "Message": "Iceberg dataload started successfully and running in background",
  "DatasetId": "49e02003-289e-41e9-a9f2-28a733349481"
}

Errors

Documented error codes: IPV-1052, IPV-1071, IPV-1001, GE-1034.

HTTP 400

Validation/auth failure. Codes/messages: IPV-1052 (missing type), IPV-1071 (type not in iceberg/hudi/deltalake), IPV-1001 (empty Files), GE-1034 (type/dataset mismatch, non-pending files, or validation failure), AUTH when caller lacks editor access, plain Message when more than 750 files or DatasetStatus is {type}-data-loading.

Missing type query param

{
  "Message": "IPV-1052 - Missing required parameter - type"
}

type not in supported frameworks

{
  "Message": "IPV-1071 - Invalid value for parameter dataload"
}

Files present but empty

{
  "Message": "IPV-1001 - Empty value - Files"
}

One or more Files are not pending

{
  "Message": "GE-1034 - Cannot start the dataload, Invalid files - [{'FileName': 'orders.parquet', 'UploadTime': '...'}] are passed for dataload only pending files are supported"
}

Dataset already loading

{
  "Message": "Dataset status is in iceberg-data-loading state, please wait for current load to finish"
}

More than 750 files selected

{
  "Message": "Only maximum of 750 files can be selected at once"
}

type does not match dataset framework

{
  "Message": "GE-1034 - Invalid querystring value given for the iceberg dataset"
}

HTTP 500

Backend/generic failure.