Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Resolve OOM When Reading Large Logs in Webserver] Refactor to Use K-Way Merge for Log Streams Instead of Sorting Entire Log Records #45129

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open

[Resolve OOM When Reading Large Logs in Webserver] Refactor to Use K-Way Merge for Log Streams Instead of Sorting Entire Log Records #45129

wants to merge 23 commits into from

Conversation

jason810496
Copy link
Contributor

related: #45079

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk potiuk force-pushed the refactor/webserver-oom-for-large-log-read branch from ed334f7 to 1d0e6ed Compare December 21, 2024 08:32
@potiuk
Copy link
Member

potiuk commented Dec 21, 2024

Rebased after we fixed main issue

@jason810496 jason810496 force-pushed the refactor/webserver-oom-for-large-log-read branch from 1d0e6ed to 8617e5b Compare December 21, 2024 10:09
@jason810496 jason810496 mentioned this pull request Dec 22, 2024
Open
17 tasks
@jason810496 jason810496 force-pushed the refactor/webserver-oom-for-large-log-read branch from 0f19a8b to ef3450b Compare December 23, 2024 12:44
@jason810496
Copy link
Contributor Author

CI is failing due to: Please ask the maintainer to assign the 'legacy api' label to the PR in order to continue.

Since the get_log endpoint in both the legacy API and FastAPI uses the read_log_chunks method, it’s necessary to fix the endpoints and their corresponding tests.

@potiuk potiuk added legacy ui Whether legacy UI change should be allowed in PR legacy api Whether legacy API changes should be allowed in PR labels Dec 23, 2024
@potiuk potiuk closed this Dec 23, 2024
@potiuk potiuk reopened this Dec 23, 2024
@potiuk
Copy link
Member

potiuk commented Dec 23, 2024

Applied and closed/reopened to trigger the build

@jason810496 jason810496 force-pushed the refactor/webserver-oom-for-large-log-read branch from ef3450b to 0aaf0ab Compare December 25, 2024 04:18
@jason810496
Copy link
Contributor Author

Fix the provider tests that explicitly use the read or _read methods.

@jason810496 jason810496 force-pushed the refactor/webserver-oom-for-large-log-read branch from 3aac539 to 46e30e3 Compare December 26, 2024 04:54
@jason810496 jason810496 changed the title WIP: [Resolve OOM When Reading Large Logs in Webserver] Refactor to Use K-Way Merge for Log Streams Instead of Sorting Entire Log Records [Resolve OOM When Reading Large Logs in Webserver] Refactor to Use K-Way Merge for Log Streams Instead of Sorting Entire Log Records Dec 26, 2024
@jason810496 jason810496 force-pushed the refactor/webserver-oom-for-large-log-read branch from 46e30e3 to 1802ed1 Compare December 26, 2024 05:46
@jason810496 jason810496 marked this pull request as ready for review December 26, 2024 06:39
@jason810496
Copy link
Contributor Author

Finally fixed the tests!

This is the first (and likely the largest) PR for resolving OOM issues when reading large logs in the webserver.
Further PRs will only focus on refactoring each provider, as listed in the TODO tasks in #45079.

Even though the providers haven't yet been refactored to support stream-based log reading, the compatibility utility will transform the old read log method (which returns the entire list of logs) into a stream-based approach. Once all providers are refactored to use stream-based reading, the compatibility utility can be removed.

For the testing part:
Since the CI will run provider compatibility tests for versions 2.9.3 and 2.10.3, my approach is to copy the old test cases related to log reading into new stream-based tests. I’ve added the mark_test_for_old_read_log_method and mark_test_for_stream_based_read_log_method pytest decorators to selectively skip the corresponding test runs.
From my perspective, this approach is simpler and minimizes changes to the original test logic. Additionally, tests marked with mark_test_for_old_read_log_method can be safely removed once all providers migrate to stream-based reading.

@jason810496 jason810496 force-pushed the refactor/webserver-oom-for-large-log-read branch from 1802ed1 to 091407e Compare January 1, 2025 16:00
@potiuk potiuk added this to the Airflow 2.10.5 milestone Jan 25, 2025
@potiuk
Copy link
Member

potiuk commented Jan 25, 2025

It only really waits for your review/approval and it solves real issue.

@jason810496 jason810496 force-pushed the refactor/webserver-oom-for-large-log-read branch 2 times, most recently from 2b5c620 to dbecf19 Compare February 2, 2025 06:18
@Lee-W Lee-W self-requested a review February 3, 2025 09:15
@jason810496
Refactor _interleave_logs and _read_from_local
5e6f4ed
@jason810496
Refactor FileTaskHandler and TaskLogReader public method interface
fec6ccf 
- refactor _read and read methods in FileTaskHandler
- refactor read_log_chunks method in TaskLogReader
@jason810496
Refactor test for get from local, parse log timestamp
dd5a0c4
@jason810496
Fix line_num in parsed_log_stream, dedupe logic
29a5719
@jason810496
Fix interleave releated test
4b1c848
@jason810496
Fix format for _read and read_log_stream methods
e7eeb9f
@jason810496
Refactor _read_from_logs_server, read resp in chunk
cc8876e
@jason810496
Add compatible utility for old and new log source
6ab3de1
@jason810496
Fix test_log_handlers
b0cf055 
- add check log_stream type utils
- fix type checking for
    - test_file_task_handler_when_ti_value_is_invalid
    - test_file_task_handler
    - test_file_task_handler_running
    - test_file_task_handler_rotate_size_limit
    - test__read_when_local
    - test__read_served_logs_checked_when_done_and_no_local_or_remote_logs
- also test compatible interface for test__read_served_logs_checked_when_done_and_no_local_or_remote_logs
    - which might call _read_remote_logs
@jason810496
Refactor read_log_chunks, read_log_stream methods
f525626 
- Since read_log_chunks is public method, refactor it as same return
  type in orignial implementation to avoid breaking change
- The `host` should only show once in read_log_stream
@jason810496
Fix test_log_reader, minor fix test_log_handlers
998b09c 
- Fix mock_read to new stream-based reading in test_log_reader
- Fix test for expecting stdout of callable should be in log lines
@jason810496
Fix get_log and test_get_log in api_fastapi
f80080c
@jason810496
Fix get_log and test_get_log in api_connexion
1278333
@jason810496
Fix _interleave_logs break loop condition
5755759 
- Logs might not be add to heap in first round, should consider
  log_streams instead of heap
@jason810496
Move log_str_to_parsed_log_stream to test utils
0e44002
@jason810496
Add _get_compatible_read_for_providers function
49a7506 
- Make it compatible for providers that implemented _read method.
@jason810496
Fix providers test cases that use read or _read method
e6ba35f
@jason810496
Fix TestGCSTaskHandler log sample
97c90b2
@jason810496
Fix legacy view and corresponding test
bd4f38c
@jason810496
Fix mypy typing error
1159d82
@jason810496
Fix test_file_task_handler_rotate_size_limit
fd0abc6
@jason810496
Fix _get_compatible_parse_log_streams
289dd31 
- Should handle input list is empty.
@jason810496
Fix providers compact tests
c6b29a8 
- Copy old test case that use read or _read methods
- Add mark_test_for_stream_based_read_log_method and
  mark_test_for_old_read_log_method to skip corresponding CI tests
@jason810496 jason810496 force-pushed the refactor/webserver-oom-for-large-log-read branch from dbecf19 to c6b29a8 Compare February 8, 2025 12:01
@jason810496
Copy link
Contributor Author

Hi @potiuk, may I ask whether this PR is considered a refactor for 3.0 or 2.10? I saw the 3.0 feature freeze mentioned in the dev list, so I’m not sure which version this PR will be counted for.

Here is the related discussion on Slack: https://apache-airflow.slack.com/archives/CCZRF2U5A/p1736767159693839

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@potiuk potiuk potiuk approved these changes

@eladkal eladkal Awaiting requested review from eladkal

@o-nikolas o-nikolas Awaiting requested review from o-nikolas o-nikolas is a code owner

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton ryanahamilton is a code owner

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@bbovenzi bbovenzi Awaiting requested review from bbovenzi bbovenzi is a code owner

@pierrejeambrun pierrejeambrun Awaiting requested review from pierrejeambrun pierrejeambrun is a code owner

@jscheffl jscheffl Awaiting requested review from jscheffl jscheffl is a code owner

@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy ephraimbuddy is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

@hussein-awala hussein-awala Awaiting requested review from hussein-awala hussein-awala is a code owner

@Lee-W Lee-W Awaiting requested review from Lee-W

Assignees
No one assigned
Labels
area:logging legacy api Whether legacy API changes should be allowed in PR legacy ui Whether legacy UI change should be allowed in PR
Projects
None yet
Milestone
Airflow 2.10.5
Development

Successfully merging this pull request may close these issues.
2 participants
@jason810496 @potiuk