Relay logs from AwsGlueJobOperator to Airflow log

[virendhar-aws]

Description

When GlueJobs are run directly using AwsGlueJobOperator, a feature to relay Glue Job's to Airflow log. Currently - I have to make couple of hops to access Glue Job logs.

Use case/motivation

Following is the excerpts from Ariflow logs - we see from Glue Job execution,

[2021-02-17 17:33:56,869] {{taskinstance.py:901}} INFO - Executing <Task(AwsGlueJobOperator): sample-glue-job> on 2021-02-17T17:33:51.077578+00:00
[2021-02-17 17:33:56,944] {{standard_task_runner.py:54}} INFO - Started process 7401 to run task
[2021-02-17 17:33:56,973] {{standard_task_runner.py:77}} INFO - Running: ['airflow', 'run', 'glue-job-trigger-v2', 'sample-glue-job', '2021-02-17T17:33:51.077578+00:00', '--job_id', '1761', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/dags/trigger_glue_v2/sample_dag_v2.py', '--cfg_path', '/tmp/tmp_or8jt2e']
[2021-02-17 17:33:57,071] {{standard_task_runner.py:78}} INFO - Job 1761: Subtask sample-glue-job
[2021-02-17 17:33:57,293] {{logging_mixin.py:112}} INFO - Running %s on host %s <TaskInstance: glue-job-trigger-v2.sample-glue-job 2021-02-17T17:33:51.077578+00:00 [running]> ip-10-192-21-133.ec2.internal
[2021-02-17 17:33:57,795] {{glue.py:114}} INFO - Initializing AWS Glue Job: sample-glue-job
[2021-02-17 17:33:57,906] {{logging_mixin.py:112}} INFO - [2021-02-17 17:33:57,905] {{base_aws.py:357}} INFO - Airflow Connection: aws_conn_id=dlp-aws-conn-trigger-glue-crawler-dev
[2021-02-17 17:33:58,038] {{logging_mixin.py:112}} INFO - [2021-02-17 17:33:58,038] {{base_aws.py:380}} WARNING - Unable to use Airflow Connection for credentials.
[2021-02-17 17:33:58,156] {{logging_mixin.py:112}} INFO - [2021-02-17 17:33:58,156] {{base_aws.py:381}} INFO - Fallback on boto3 credential strategy
[2021-02-17 17:33:58,223] {{logging_mixin.py:112}} INFO - [2021-02-17 17:33:58,223] {{base_aws.py:386}} INFO - Creating session using boto3 credential strategy region_name=None
[2021-02-17 17:33:58,473] {{logging_mixin.py:112}} INFO - [2021-02-17 17:33:58,473] {{glue.py:165}} INFO - Job Already exist. Returning Name of the job
[2021-02-17 17:33:58,735] {{logging_mixin.py:112}} INFO - [2021-02-17 17:33:58,735] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job current run state with status RUNNING
[2021-02-17 17:34:04,930] {{logging_mixin.py:112}} INFO - [2021-02-17 17:34:04,930] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job current run state with status RUNNING
[2021-02-17 17:34:11,148] {{logging_mixin.py:112}} INFO - [2021-02-17 17:34:11,147] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job current run state with status RUNNING
[2021-02-17 17:34:17,339] {{logging_mixin.py:112}} INFO - [2021-02-17 17:34:17,339] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job current run state with status RUNNING
[2021-02-17 17:34:23,526] {{logging_mixin.py:112}} INFO - [2021-02-17 17:34:23,526] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job current run state with status RUNNING
[2021-02-17 17:34:29,684] {{logging_mixin.py:112}} INFO - [2021-02-17 17:34:29,684] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job current run state with status RUNNING
[2021-02-17 17:34:35,833] {{logging_mixin.py:112}} INFO - [2021-02-17 17:34:35,833] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job current run state with status RUNNING
[2021-02-17 17:34:41,958] {{logging_mixin.py:112}} INFO - [2021-02-17 17:34:41,958] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job current run state with status RUNNING
[2021-02-17 17:34:48,171] {{logging_mixin.py:112}} INFO - [2021-02-17 17:34:48,171] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job current run state with status RUNNING
[2021-02-17 17:34:54,333] {{logging_mixin.py:112}} INFO - [2021-02-17 17:34:54,332] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job current run state with status RUNNING
[2021-02-17 17:35:00,488] {{logging_mixin.py:112}} INFO - [2021-02-17 17:35:00,488] {{glue.py:153}} INFO - Polling for AWS Glue Job sample-glue-job current run state with status RUNNING
[2021-02-17 17:35:06,652] {{logging_mixin.py:112}} INFO - [2021-02-17 17:35:06,651] {{glue.py:145}} INFO - Exiting Job jr_0eb8c810fa49c81fc086a3d4b27ecc20299b7a36c25de2f6d19edbc7105e234b Run State: SUCCEEDED
[2021-02-17 17:35:06,754] {{glue.py:121}} INFO - AWS Glue Job: sample-glue-job status: SUCCEEDED. Run Id: jr_0eb8c810fa49c81fc086a3d4b27ecc20299b7a36c25de2f6d19edbc7105e234b
[2021-02-17 17:35:06,886] {{taskinstance.py:1070}} INFO - Marking task as SUCCESS.dag_id=glue-job-trigger-v2, task_id=sample-glue-job, execution_date=20210217T173351, start_date=20210217T173356, end_date=20210217T173506
[2021-02-17 17:35:11,688] {{logging_mixin.py:112}} INFO - [2021-02-17 17:35:11,688] {{local_task_job.py:102}} INFO - Task exited with return code 0

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[eladkal]

Wouldnt it be better to set operator extra link?
Like we do for google: #9941

[Taragolis]

@eladkal much better than fetch all logs by Airflow, because someone could spawn Glue Job with 40 DPU and in this case Airflow should fetch logs from 80 log prefixes (error and output)

Unfortunately it is not easy integrate with AWS UI, for example Developer login in browser into Account A and Airflow execute Glue Job in Account B - that mean link wouldn't work. But even this better rather than fetch all Glue logs to Airflow

Other disgusting fact which I found when I tried to implement extra link for glue job (spoiler alert - I obsolete work on it) execution is URL pattern:

Glue Studio
https://{region_name}.console.{aws_console_endpoint}/gluestudio/home?region={region_name}#/job/{job_name}/run/{job_id}

Glue Job (Legacy)
https://{region_name}.console.{aws_console_endpoint}/glue/home?region={region_name}#jobRun:jobName={job_name};jobRunId={job_id}

Where aws_console_endpoint

• aws.amazon.com - AWS General
• amazonaws.cn - AWS China
• amazonaws-us-gov.com - AWS GovCloud (US)

Also region_name could define:

• In operator
• In connection
• As Environment Variable

All these facts make it difficult to develop Extra Link for AWS Glue (as well as other AWS services), but nothing is impossible, e.g. create link during execute() and save it in XCom, and finally fetch it by extra link

@virendhar-aws WDYT?

[shubham22]

Closed by #25142
We (AWS OSS team) ended up going with output and error logs of the Glue driver, instead of fetching and displaying logs from each and every Glue Job. This solution provides more visibility into Glue Jobs and at the same time doesn't introduce complexity of managing logs at the job-level - CloudWatch (or any other tool customers use) will remain the solution for it.