create_etl_job

EtlJobsApi.create_etl_job(role_id, etl_job_post, content_type=None, quick_create=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)

Create an ETL job

Creates a Glue ETL job, IAM role, and Amorphic metadata. Use a full body for spark or pythonshell jobs, or quickCreate=yes to create from JobName and ETLJobType only. DatasetAccess is required on a full create.

Parameters:

• role_id (str) – Amorphic role ID sent for authorization. Missing this header returns AUTH-1001. (required)

• etl_job_post (ETLJobPost) – (required)

• content_type (str)

• quick_create (str) – When yes, only JobName and ETLJobType are accepted; DatasetAccess and NetworkConfiguration are defaulted. Use no or omit for a full create.

• _request_timeout (int, tuple(int, int), optional) – timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts.

• _request_auth (dict, optional) – set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request.

• _content_type (str, Optional) – force content-type for the request.

• _headers (dict, optional) – set to override the headers for a single request; this effectively ignores the headers in the spec for a single request.

• _host_index (int, optional) – set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request.

Return type:

ETLJobStatus

Returns:

Returns the result object.

Request and Response Examples

HTTP: POST /jobs

Request

Spark job with workers and dataset access

{
  "JobName": "sales_etl_daily",
  "ETLJobType": "spark",
  "Description": "Daily sales transform",
  "GlueVersion": "4.0",
  "WorkerType": "G.1X",
  "NumberOfWorkers": 2,
  "Timeout": 60,
  "MaxRetries": 0,
  "JobBookmarkOption": "disable",
  "IsAutoScalingEnabled": false,
  "IsDataLineageEnabled": "no",
  "NetworkConfiguration": "general-public-network",
  "DatasetAccess": {
    "Owner": [],
    "ReadOnly": [],
    "ReadOnlyFileLevel": []
  },
  "ParameterAccess": [],
  "SharedLibraries": [],
  "Keywords": [
    "sales",
    "etl"
  ]
}

Python shell job using MaxCapacity

{
  "JobName": "python_shell_cleanup",
  "ETLJobType": "pythonshell",
  "Description": "Lightweight Python shell cleanup",
  "PythonVersion": "3.9",
  "MaxCapacity": 1,
  "Timeout": 15,
  "JobBookmarkOption": "disable",
  "NetworkConfiguration": "general-public-network",
  "DatasetAccess": {
    "Owner": [],
    "ReadOnly": [],
    "ReadOnlyFileLevel": []
  }
}

Body when quickCreate=yes

{
  "JobName": "sales_etl_daily",
  "ETLJobType": "spark"
}

Success (HTTP 200)

Job created. Body includes Message “Successfully created a job.” and JobId (UUID of the new job).

Request / response example

{
  "Message": "Successfully created a job.",
  "JobId": "3fa85f64-5717-4562-b3fc-2c963f66afa6"
}

Errors

Documented error codes: GE-1001, IPV-1001, IPV-1040, JOB-1002, GE-1008.

HTTP 400

Validation or create failure. Body is {“Message”: “<CODE> - <text>”}. Common cases include missing required fields, unknown keys, invalid NetworkConfiguration, reserved DefaultArguments, or Glue create errors.

Empty request body

{
  "Message": "GE-1001 - Invalid input body"
}

Required field omitted

{
  "Message": "IPV-1001 - Parameter JobName is a required value."
}

Unknown field in the body

{
  "Message": "IPV-1040 - Invalid elements found in input fields - 'JobType'"
}

Reserved Glue argument key

{
  "Message": "JOB-1002 - Cannot provide ['--TempDir'] keys in JobParameters, use some other key-value pair"
}

HTTP 500

Uncaught backend failure. Typical Message is “GE-1008 - Could not complete the request. Please try again.”

Request / response example

{
  "Message": "GE-1008 - Could not complete the request. Please try again."
}