freeze AWS glue job

[andreisorokin-astrazeneca]

I use WatchTower as workaround for inability to change CloudWatch group for glue jobs written in Python.
But when I add WatchTower handler the glue jobs after successfuly finishing job freeze.

In Spark worker logs I see nothing - just normal finishing and after that final message that the worker had been killed by driver.

In Spark Driver I see repeated attempts to send stdout to CloudWatch by internal AWS mechanism - do not see how is possibly connected with WatchTower:

INFO LogPusher: uploading /tmp/spark-event-logs/ to s3://****
INFO MultipartUploadOutputStream: close closed:false s3://***.inprogress

If I set use_queues=False it finishes normally and in logs I see
INFO MultipartUploadOutputStream: close closed:false s3://***/FilteredOut/run-1705485647329-part-r-00002.gz

[andreisorokin-astrazeneca]

... and in both cases logs were successfully and fully sent to CloudWatch by WatchTower

[andreisorokin-astrazeneca]

I found workaround - if I add that in the end of glue job

logger = logging.getLogger()
logging.shutdown()
for handler in logger.handlers:
    logger.removeHandler(handler)

it finishes normally

[dovy]

Can you provide the code you used and maybe the version of cloudwatch? I can't get it to run via glue properly.

[dovy]

@andreisorokin-astrazeneca