Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Airflow] The remote logging supports the current log mechanism #172

Closed
SteNicholas opened this issue Nov 24, 2021 · 2 comments · Fixed by #174
Closed

[Airflow] The remote logging supports the current log mechanism #172

SteNicholas opened this issue Nov 24, 2021 · 2 comments · Fixed by #174
Assignees
Labels
Airflow Airflow related issue bug Something isn't working

Comments

@SteNicholas
Copy link
Contributor

FileTaskHandler, a python log handler that handles and reads task instance logs, supports the current log mechanism which uses seq_num and try_number to name the log file. The remote log handler like S3TaskHandler should support the log mechanism.

@SteNicholas SteNicholas added the bug Something isn't working label Nov 24, 2021
@SteNicholas SteNicholas added the Airflow Airflow related issue label Nov 24, 2021
@aqua7regia
Copy link
Contributor

I saw the log file path has been renamed like "{seq_num}_{try_number}.log" if seq_num in task_instance when the FileTaskHandler initial the log file relative path, and the remote log handler still use the name like "{try_number}.log" to upload or download, that may cause the bug.

file_task_handler.py line 247:

if hasattr(ti, 'seq_num') and ti.seq_num > 0:
    number = '{}_{}'.format(ti.seq_num, ti.try_number)
else:
    number = ti.try_number
relative_path = self._render_filename(ti, number)

s3_task_handler.py line 62:

self.log_relative_path = self._render_filename(ti, ti.try_number)

And GCS and Azure log handlers have the same problem.

@SteNicholas
Copy link
Contributor Author

@aqua7regia , thanks for your reporter for the remote logging handler. I have assigned to you and thanks for your contribution.

Sxnan pushed a commit that referenced this issue Feb 9, 2022
[bug fix] Force trigger a SCHEDULED dag will cause mysql duplicate key
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Airflow Airflow related issue bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants