Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[autoscaler][logs] Improve status logging #26670

Open
DmitriGekhtman opened this issue Jul 18, 2022 · 2 comments
Open

[autoscaler][logs] Improve status logging #26670

DmitriGekhtman opened this issue Jul 18, 2022 · 2 comments
Labels
enhancement Request for new feature and/or capability good-first-issue Great starter issue for someone just starting to contribute to Ray infra autoscaler, ray client, kuberay, related issues observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling P1 Issue that should be fixed within a few weeks Ray-2.4 usability
Milestone

Comments

@DmitriGekhtman
Copy link
Contributor

Description

The "ray status" logs periodically emitted by the autoscaler have the following issues:

  • (biggest issue) They are emitted once per several seconds, even when the cluster status hasn't changed. This drowns out information about actual events and issues. Only log status when there is a change. The definition of "change" is TBD, but for a first cut, a literal change in the status string (ignoring timestamp) could probably work. This change should be accompanied by an info message and a feature flag to disable it.
  • (smaller issue) The logs are multi-line, i.e. they contain newline characters. That's bad for structured logging. We've had some complaints about this. Consider logging the status line-by-line.

Use case

Improved log-viewing experience.

@DmitriGekhtman DmitriGekhtman added enhancement Request for new feature and/or capability P1 Issue that should be fixed within a few weeks labels Jul 18, 2022
@DmitriGekhtman DmitriGekhtman added this to the Infra Backlog milestone Jul 18, 2022
@DmitriGekhtman
Copy link
Contributor Author

Going to tentatively assign @wuisawesome, who is free to reject the assignment :)

@wuisawesome wuisawesome added usability core-autoscaler autoscaler related issues labels Aug 16, 2022
@wuisawesome wuisawesome removed their assignment Aug 16, 2022
@wuisawesome
Copy link
Contributor

Unassigning until I'm sure that I will take this on.

@wuisawesome wuisawesome added the good-first-issue Great starter issue for someone just starting to contribute to Ray label Aug 16, 2022
@scottsun94 scottsun94 added the observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling label Oct 13, 2022
@richardliaw richardliaw added infra autoscaler, ray client, kuberay, related issues and removed core-autoscaler autoscaler related issues labels Nov 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Request for new feature and/or capability good-first-issue Great starter issue for someone just starting to contribute to Ray infra autoscaler, ray client, kuberay, related issues observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling P1 Issue that should be fixed within a few weeks Ray-2.4 usability
Projects
None yet
Development

No branches or pull requests

5 participants