Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core][logging][ipython] Fix log buffering when consecutive runs within ray log dedup window #37134

Merged
merged 3 commits into from
Jul 6, 2023

Conversation

rickyyx
Copy link
Contributor

@rickyyx rickyyx commented Jul 6, 2023

Why are these changes needed?

Related issue number

Fixes #34831

The issue is when tasks runs within the same job are executed and producing logs, log lines will be dedup for better observability.

So ray might be buffering some lines, in case there are logs that could be dedup within the same RAY_DEDUP_LOGS_AGG_WINDOW_S. This will show up as logs line not available until future ray tasks runs since buffer is not flushed when there's no activity (no job done, no new task)

This PR adds callbacks to the post_execute to flush the log lines in log monitor when execution in shell took place.

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

Signed-off-by: Ricky Xu <xuchen727@hotmail.com>
rickyyx and others added 2 commits July 5, 2023 20:35
Signed-off-by: Ricky Xu <xuchen727@hotmail.com>
@ericl ericl merged commit 10e11f9 into ray-project:master Jul 6, 2023
rickyyx added a commit to rickyyx/ray that referenced this pull request Jul 6, 2023
bveeramani pushed a commit that referenced this pull request Jul 10, 2023
arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023
…in ray log dedup window (ray-project#37134)

Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Core] Logs look weird in Jupyter environment
2 participants