download_datasource_dataflow_logs¶

DatasourcesApi.download_datasource_dataflow_logs(datasource_id, id, role_id, servicename, action=None, logtype=None, ingestionlogtype=None, connector_type=None, filter_pattern=None, encoding=None, timestamp_range_with_next_token=None, _request_timeout=None, _request_auth=None, _content_type=None, _headers=None, _host_index=0)¶

View, generate, or download dataflow logs

Retrieves CloudWatch logs for a dataflow. The caller (identified by role_id) needs datasources.view and at least read-only access on the datasource (else AUTH-1010). Supports JDBC Bulkload (servicename=jdbc-dataflow), JDBC Advanced Load (jdbc-advanced-dataflow), Streams (streams-dataflow), and ArcGIS (arcgis-dataflow). Not supported for SaaS. JDBC uses async view/generate/download; Streams and ArcGIS return a sync download URL.

Parameters:

datasource_id (str) – UUID of the parent datasource. Rejected with AUTH-1010 if the user has no access. (required)
id (str) – UUID of the dataflow (DataflowId). Rejected with IPV-1002 if missing. (required)
role_id (str) – Amorphic role ID. Must grant datasources.view. Missing role_id returns AUTH-1001. (required)
servicename (str) – Log service selector. Must match the dataflow type: jdbc-dataflow (Bulkload), jdbc-advanced-dataflow (Advanced Load), streams-dataflow (Streams), or arcgis-dataflow (ArcGIS). (required)
action (str) – Required for JDBC Bulkload and Advanced Load: view-logs, generate-logs, or download-logs. Required for ArcGIS and must be download-logs. Not used for Streams (sync download only).
logtype (str) – ArcGIS only. Glue log stream type for the latest completed run: output or error.
ingestionlogtype (str) – ArcGIS only. metadata-logs or data-logs. data-logs allowed only when DataflowConfig.IngestionType is metadata-and-data; metadata-only allows metadata-logs only.
connector_type (str) – Required when servicename is jdbc-advanced-dataflow. Selects Kafka Connect sinkConnector or sourceConnector log streams.
filter_pattern (str) – Optional CloudWatch Logs filter pattern for JDBC Bulkload and Advanced Load.
encoding (str) – Set to base64 when filter_pattern is base64-encoded. Otherwise omit.
timestamp_range_with_next_token (TimestampRangeWithNextToken) – Required for JDBC and Streams: startTime and endTime as YYYY-MM-DDTHH:MM:SS (UTC). Optional nextToken for JDBC view-logs pagination. Not used for ArcGIS.
_request_timeout (int, tuple(int, int), optional) – timeout setting for this request. If one number provided, it will be total request timeout. It can also be a pair (tuple) of (connection, read) timeouts.
_request_auth (dict, optional) – set to override the auth_settings for an a single request; this effectively ignores the authentication in the spec for a single request.
_content_type (str, Optional) – force content-type for the request.
_headers (dict, optional) – set to override the headers for a single request; this effectively ignores the headers in the spec for a single request.
_host_index (int, optional) – set to override the host_index for a single request; this effectively ignores the host_index in the spec for a single request.

Return type:

DownloadDatasourceDataflowLogsResponse

Returns:

Returns the result object.

Request and Response Examples¶

HTTP: POST /datasources/{datasource_id}/dataflows/{id}/logs

Request¶

Required for JDBC and Streams: startTime and endTime as YYYY-MM-DDTHH:MM:SS (UTC). Optional nextToken for JDBC view-logs pagination. Not used for ArcGIS.

The request body is optional.

Time window (JDBC / Streams)

{
  "startTime": "2024-06-01T12:00:00",
  "endTime": "2024-06-01T13:00:00"
}

JDBC view-logs with nextToken

{
  "startTime": "2024-06-01T12:00:00",
  "endTime": "2024-06-01T13:00:00",
  "nextToken": "abc123"
}

Success (HTTP 200)¶

Shape depends on servicename and action. JDBC view-logs: logPreview, logTruncated, optional nextToken (last page when nextToken is absent). JDBC generate-logs: Message. JDBC download-logs: PresignedUrl or Message if not generated yet. Streams/ArcGIS: url, logPreview, logTruncated.

JDBC action=view-logs

{
  "logPreview": "2024-06-01T12:00:01 INFO Replication starting...",
  "logTruncated": false
}

JDBC action=generate-logs

{
  "Message": "Log generation started, an alert will be sent send to email after successful log retrieval"
}

JDBC action=download-logs

{
  "PresignedUrl": "https://amorphic-logs.s3.amazonaws.com/dms-task-logs/...?X-Amz-Algorithm=..."
}

Streams or ArcGIS sync download

{
  "url": "https://amorphic-logs.s3.amazonaws.com/...?X-Amz-Algorithm=...",
  "logPreview": "...",
  "logTruncated": true
}

Errors¶

Documented error codes: AUTH-1001, AUTH-1010, IPV-1041, IPV-1052, IPV-1002, IPV-1008, IPV-1054, LOG-1002, GE-1048, GE-1034.

HTTP 400

Validation or authorization failure. Body is {“Message”: “<CODE> - <text>”}. Common codes: AUTH-1001 (missing role_id), AUTH-1010 (no datasource access), IPV-1041 (invalid action/servicename/connector_type/logtype/ ingestionlogtype), IPV-1052 (missing ArcGIS params or connector_type), IPV-1002 (dataflow not found), IPV-1008 (missing startTime/endTime), IPV-1054 (bad timestamp format), LOG-1002 (1970-01-01T00:00:00), GE-1048 (Streams status blocks logs).

No datasource access

{
  "Message": "AUTH-1010 - User user1 is not authorized to perform this operation"
}

Missing startTime/endTime

{
  "Message": "IPV-1008 - Invalid request body, missing key(s) - startTime/endTime"
}

Invalid JDBC action

{
  "Message": "IPV-1041 - Invalid parameter - action, allowed values/keys are [view-logs, generate-logs, download-logs]"
}

ArcGIS missing required query params

{
  "Message": "IPV-1052 - Missing mandatory parameter(s) servicename, logtype, action in queryStringParameters/pathParameters"
}

HTTP 500

Backend failure. Body is {“Message”: “<CODE> - <text>”} or a plain Message. Example: logs unavailable for the time window, ArcGIS with no runs, or serverless Bulkload log group missing (GE-1034).

Generic fetch failure for this path

{
  "Message": "Unable to fetch logs (or) Logs not available for the selected time frame, please try again later."
}