Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: set job's backoffLimit to 0 and report job's active/ready status in workspace #575

Merged
merged 2 commits into from
Aug 22, 2024

Conversation

Fei-Guo
Copy link
Collaborator

@Fei-Guo Fei-Guo commented Aug 21, 2024

For Job workload, the backoffLimit specifies the number of retries before marking this job failed. Defaults to 6. In Kaito finetuning, there are not many benefits to recreate pods 6 times if the tuning fails, because the failures usually require user interventions.

This change also reports the job's ready pod count in workspace CR so that users do not need to frequently check the job object for status check. When tuning job is working, the ready count is 1. If the tuning container completes but the docker sidecar containers fails with infinite retry, the ready count becomes 0.

Add another tip in the troubleshoot guide.

@Fei-Guo Fei-Guo merged commit d4db63d into main Aug 22, 2024
4 of 6 checks passed
@Fei-Guo Fei-Guo deleted the fguo-dev1 branch August 22, 2024 01:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants