set default resources for all dagster repos #324
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Minimum allowed line rate is |
kgrunert
reviewed
Dec 7, 2023
There was a problem hiding this comment.
This PR should create a test version for MPyL.
I recommend on building and deploying a dagster user deployment with this test version and then checking with
helm get values or helm get manifest that the values are being set in the release that you expect!
And then also run the MPyL Test Pipeline on jenkins to verify the build of this mpyl version :-)
|
New release |
kgrunert
reviewed
Dec 7, 2023
kgrunert
approved these changes
Dec 13, 2023
ba5a4ea to
a123fff
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See here how resources of user code deployments should be set: https://github.com/dagster-io/dagster/blob/master/helm/dagster/charts/dagster-user-deployments/values.yaml#L114
This PR attempts to implement this.
Check any pod starting with
udcin grafana (e.g. this one) and conclude with me that the resources I set are sensible.📕 DATA-1159 As a developer I don't want to waste money by overcommitting my app its resource requests in Kubernetes
– This ticket is a reminder for Pieter to follow-up with Pim on resources –
Story
Every app we make in-house will be deployed to Kubernetes. Kubernetes uses so called resources requests (and limits) to make sure you app gets the resources it needs, but doesn't use too much and breaks other apps or the underlaying nodes.These are manual configurations, which can be found in the
project.yamland must be refined before deployment to production and each time you modify the app.To use them correctly, please read/watch on of the articles below. In short:** Kubernetes defines Limits* as the maximum amount of a resource to be used by a container. This means that the container can never consume more than the memory amount or CPU amount indicated. It will either be throttled (CPU) or killed OOM (RAM) after hitting the limits.
** Requests*, on the other hand, are the minimum guaranteed amount of a resource that is reserved for a container. CPU or RAM requested by one pod, can't be requested/used by another.
For this item we will focus on the CPU requests, not the memory requests and cpu/memory limits. Why?
Because we reserve 75% CPU while we're mostly only using 10%. See image below. This is expensive, but also makes it impossible to deploy sometimes, because the cluster seems "full".
How?
For each of the apps/deployments (not jobs, for now) your team is responsible for, verify it's actual CPU use here: GRAFANA PROD.
You can filter on namespace and app name. In the
CPU Usagegraph you can see the "normal" CPU usage of the app. Don't look at the spikes, just try to find the trend of normal usage.In the URL I added as an example to GRAFANA PROD above, you can see that the app:
The preferred modification here should be reducing the CPU requests rom 0.4 to +/- 0.05. We'll keep the limits untouched for now. Redeploy.
Please have a quick look if the CPU requests could be lower in
acce}} and {{test}} in comparison to {{prod. Those clusters are smaller and the apps are being used less, usually!Resourcesproject.yamlexample: https://github.com/Vandebron/Vandebron/blob/master/apps/customer-dashboard/deployment/project.yml#L33🏗️ Build 3 ✅ Successful, started by Pieter Custers
🚀 cloudfront-service, example-dagster-user-code, job, kong-sync, nodeservice, ocpp-first, ocpp-second, sbtservice, sparkJob-first, sparkJob-second