Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add readiness probe to execution pods #3720

Closed
sotojn opened this issue Aug 14, 2024 · 0 comments
Closed

Add readiness probe to execution pods #3720

sotojn opened this issue Aug 14, 2024 · 0 comments

Comments

@sotojn
Copy link
Contributor

sotojn commented Aug 14, 2024

When creating a new execution controller job resource in kubernetes, we check to see if the execution controller pod is operational by looking at the pod status. If the pod status is in a running state, we proceed to create the worker deployment.

The issue is that checking to see if the pod is in a running state is not a good indication that the execution service is ready to have workers connect and start distributing slices.

Adding a readiness probe to the container and checking to see if the readiness probe currently is returning true would solve this.

I'm looking to see if there is already a built in health check on the execution controller to see what we can use to verify it's ready.

@sotojn sotojn self-assigned this Aug 14, 2024
godber added a commit that referenced this issue Aug 22, 2024
This PR makes the following changes:

- Adds readiness probe to `execution controller` job (_**Only applies to
k8s V2 backend**_)
- Worker deployments will now wait for the ex readiness probe to come
back as `ready` before creation.
- Adds `/ready` endpoint to execution controller server 
  - Will return with status code `200` when initializing is complete 
- Will return with status code `503` when initializing either fails or
hasn't yet completed
- Bumps **@terascope/teraslice-messaging** from `v1.2.0` to `v1.3.0`

Ref to issue #3720

---------

Co-authored-by: Austin Godber <godber@terascope.io>
@godber godber closed this as completed Aug 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants