get_data_pipeline_file_load_manifest

DataPipelinesApi.get_data_pipeline_file_load_manifest(id, executionid, node_name, file_type, role_id, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)

Download input or output manifest for a node

Returns a GET presigned URL. file_type is required (input_manifest_file | output_manifest_file); missing/invalid is IPV-1040 (HTTP 403). input_manifest_file is file_load_validation only (uses the predecessor job temp JSON). output_manifest_file is file_load_validation or sync_to_s3 CSV. Node not reached while RUNNING/STOPPED is DP-1021; unknown node DP-1022; wrong module DP-1029; JobRunState NOT_STARTED DP-1030; S3 404 DP-1034.

Parameters:

  • id (str) – DataPipelineId. (required)

  • executionid (str) – DataPipelineExecutionId (Glue workflow run id or AI flow execution id). (required)

  • node_name (str) – NodeName of the pipeline node. (required)

  • file_type (str) – input_manifest_file (file_load_validation) or output_manifest_file (file_load_validation or sync_to_s3). (required)

  • role_id (str) – Amorphic role ID the request is authorized against. Must be a role the user belongs to. Lambda returns AUTH-1001 (HTTP 403 on v2-dataPipelines, HTTP 400 on v2-downloadLogs) if missing after gateway validation. (required)

  • _request_timeout (int, tuple(int, int), optional) – timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts.

  • _request_auth (dict, optional) – set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request.

  • _content_type (str, Optional) – force content-type for the request.

  • _headers (dict, optional) – set to override the headers for a single request; this effectively ignores the headers in the spec for a single request.

  • _host_index (int, optional) – set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request.

Return type:

GetDataPipelineNodeFileResponse

Returns:

Returns the result object.

Request and Response Examples

HTTP: GET /data-pipelines/{id}/executions/{executionid}/nodes/{node_name}/files

Success (HTTP 200)

Presigned GET URL for the manifest.

Output CSV for file_load_validation

{
  "Message": "Presigned url returned",
  "PresignedURL": "https://s3.amazonaws.com/etl-bucket/file-load-validation-reports/dp-id/exec-id/validate_load-output_manifest_file.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256"
}

Input JSON for file_load_validation

{
  "Message": "Presigned url returned",
  "PresignedURL": "https://s3.amazonaws.com/etl-bucket/job-id/temp/manifest-files/orders_enrichment_flow/exec-id-input_manifest_file.json?X-Amz-Algorithm=AWS4-HMAC-SHA256"
}

Errors

Documented error codes: IPV-1040, DP-1021, DP-1022, DP-1029, DP-1030, DP-1034, AUTH-1001, GE-1044.

HTTP 400

API Gateway request validation (missing required parameters such as role_id). Lambda InvalidInputException from this function is HTTP 403.

Gateway rejected missing required header

{
  "Message": "Invalid request"
}

HTTP 403

Invalid input or unauthorized.

file_type missing or not allowed

{
  "Message": "IPV-1040 - Invalid elements found in input fields - 'file_type'"
}

Execution has not reached the node

{
  "Message": "DP-1021 - Data pipeline execution wr_abc123 is in running state and hasn't reached the node validate_load yet."
}

Node not in the execution

{
  "Message": "DP-1022 - Node validate_load is not part of execution wr_abc123."
}

Node is not file_load_validation (input) or file_load_validation/sync_to_s3 (output)

{
  "Message": "DP-1029 - Data pipeline execution node enrich_orders is not of the expected type file_load_validation."
}

Node JobRunState is NOT_STARTED

{
  "Message": "DP-1030 - Data pipeline node validate_load execution state is NOT_STARTED. Please try again later."
}

Manifest not in S3

{
  "Message": "DP-1034 - Failed to download data pipeline execution file-load-validation-reports-output_manifest_file.csv file. File not found in s3://etl-bucket/file-load-validation-reports/dp-id/exec-id/validate_load-output_manifest_file.csv path. Please try again later!"
}

Missing role_id

{
  "Message": "AUTH-1001 - Missing role information in event header"
}

HTTP 500

Backend failure (GenericFailureException, InconsistentMetadataException, or unexpected GE-1044).

Unhandled exception

{
  "Message": "GE-1044 - Unexpected error - An unexpected error occurred."
}