From dea9d0fb41bdb2dedb23724ab9a3b8a243112447 Mon Sep 17 00:00:00 2001 From: Tim Collins <45351296+tico24@users.noreply.github.com> Date: Mon, 3 Feb 2025 16:30:51 +0000 Subject: [PATCH] docs: Improvements to ContainerSet Templates docs (#14143) Signed-off-by: Tim Collins --- .spelling | 4 ++ docs/container-set-template.md | 100 +++++++++++++++++---------------- 2 files changed, 56 insertions(+), 48 deletions(-) diff --git a/.spelling b/.spelling index dba0646e3886..661c7e52d415 100644 --- a/.spelling +++ b/.spelling @@ -1,6 +1,7 @@ # markdown-spellcheck dictionary 000s 0s +100Gi 100Mi 100s 10h @@ -11,6 +12,7 @@ 1Mi 1h 1m +200Ki 2Gi 2h 30s @@ -41,6 +43,7 @@ ClusterWorkflowTemplate Codespaces ConfigMap ConfigMaps +ContainerSet Couler CronWorkflow CronWorkflows @@ -100,6 +103,7 @@ PDBs PProf PVCs Peixuan +PersistentVolumeClaims Ploomber Postgres PriorityClass diff --git a/docs/container-set-template.md b/docs/container-set-template.md index 2108fa3d08b9..e6c22073036f 100644 --- a/docs/container-set-template.md +++ b/docs/container-set-template.md @@ -1,12 +1,17 @@ -# Container Set Template +# ContainerSet Template > v3.1 and after -A container set template is similar to a normal container or script template, but allows you to specify multiple -containers to run within a single pod. +A ContainerSet template is similar to a normal container template, but it allows you to run multiple containers within a single Pod. -Because you have multiple containers within a pod, they will be scheduled on the same host. You can use cheap and fast -empty-dir volumes instead of persistent volume claims to share data between steps. +Since multiple containers run within a single Pod, they schedule on the same host. +You can use cheap and fast `emptyDir` volumes instead of PersistentVolumeClaims (PVCs) to share data between steps. + +However, running all containers on the same host limits you to the host's resources. +Running all containers simultaneously may use more resources than running them sequentially. + +Use ContainerSet templates strategically to avoid waiting for Pods to start and reduce the overhead of creating multiple init and wait containers. +This can be more efficient than using a DAG template. ```yaml apiVersion: argoproj.io/v1alpha1 @@ -47,83 +52,82 @@ spec: path: /workspace/message ``` -There are a couple of caveats: +There are a few caveats: -1. You must use the [Emissary Executor](workflow-executors.md#emissary-emissary). -2. Or all containers must run in parallel - i.e. it is a graph with no dependencies. -3. You cannot use [enhanced depends logic](enhanced-depends-logic.md). -4. It will use the sum total of all resource requests, maybe costing more than the same DAG template. This will be a problem if your requests already cost a lot. See below. +1. You cannot use [enhanced depends logic](enhanced-depends-logic.md). +1. The ContainerSet uses the sum total of all resource requests, which may cost more than using the same DAG template. This can be problematic if your [resource requests](#️-resource-requests) are already high. +1. ContainerSet templates can only run container templates. -The containers can be arranged as a graph by specifying dependencies. This is suitable for running 10s rather than 100s -of containers. +You can arrange the containers as a graph by specifying dependencies. -## Inputs and Outputs +ContainerSet templates are suitable for running tens rather than hundreds of containers. -As with the container and script templates, inputs and outputs can only be loaded and saved from a container -named `main`. +## Inputs and Outputs -All container set templates that have artifacts must/should have a container named `main`. +As with container and script templates, you can only load and save inputs and outputs from a container named `main`. -If you want to use base-layer artifacts, `main` must be last to finish, so it must be the root node in the graph. +Include a container named `main` in all ContainerSet templates that have artifacts. -That may not be practical. +If you want to use base-layer artifacts, ensure `main` is the last to finish, making it the root node in the graph. +This may not always be practical. -Instead, have a workspace volume and make sure all artifacts paths are on that volume. +Instead, use a workspace volume and ensure all artifact paths are on that volume. ## ⚠️ Resource Requests -A container set actually starts all containers, and the Emissary only starts the main container process when the containers it depends on have completed. This mean that even though the container is doing no useful work, it is still consuming resources and you're still getting billed for them. +A ContainerSet starts all containers, and the [workflow executor](workflow-executors.md#emissary-emissary) only starts the main container process when the containers it depends on have completed. + +This means that even though the container is doing no useful work, it still consumes resources and you are still billed for them. If your requests are small, this won't be a problem. If your requests are large, set the resource requests so the sum total is the most you'll need at once. -Example A: a simple sequence e.g. `a -> b -> c` +### Example A: Simple Sequence (a -> b -> c) * `a` needs 1Gi memory * `b` needs 2Gi memory * `c` needs 1Gi memory -Then you know you need only a maximum of 2Gi. You could set as follows: +You need a maximum of 2Gi. You could set the requests as follows: * `a` requests 512Mi memory * `b` requests 1Gi memory * `c` requests 512Mi memory -The total is 2Gi, which is enough for `b`. We're all good. - -Example B: Diamond DAG e.g. a diamond `a -> b -> d and a -> c -> d`, i.e. `b` and `c` run at the same time. +The total is 2Gi, which is enough for `b`. -* `a` needs 1000 cpu -* `b` needs 2000 cpu -* `c` needs 1000 cpu -* `d` needs 1000 cpu +### Example B: Diamond DAG (a -> b -> d and a -> c -> d) -I know that `b` and `c` will run at the same time. So I need to make sure the total is 3000. +* `a` needs 1000 CPU +* `b` needs 2000 CPU +* `c` needs 1000 CPU +* `d` needs 1000 CPU -* `a` requests 500 cpu -* `b` requests 1000 cpu -* `c` requests 1000 cpu -* `d` requests 500 cpu +You know that `b` and `c` will run at the same time. So you need to make sure the total is 3000 CPU. -The total is 3000, which is enough for `b + c`. We're all good. +* `a` requests 500 CPU +* `b` requests 1000 CPU +* `c` requests 1000 CPU +* `d` requests 500 CPU -Example B: Lopsided requests, e.g. `a -> b` where `a` is cheap and `b` is expensive +The total is 3000 CPU, which is enough for `b + c`. -* `a` needs 100 cpu, 1Mi memory, runs for 10h -* `b` needs 8Ki GPU, 100 Gi memory, 200 Ki GPU, runs for 5m +### Example C: Lopsided Requests (a -> b) -Can you see the problem here? `a` only has small requests, but the container set will use the total of all requests. So it's as if you're using all that GPU for 10h. This will be expensive. +* `a` needs 100 CPU, 1Mi memory, runs for 10h +* `b` needs 8Ki GPU, 100Gi memory, 200Ki GPU, runs for 5m -Solution: do not use container set when you have lopsided requests. +In this case, `a` only has small requests, but the ContainerSet uses the total of all requests. So it's as if you're using all that GPU for 10h. This will be expensive. +This is a good example of when using a ContainerSet template would not be efficient. ## Inner `retryStrategy` usage > v3.3 and after -You can set an inner `retryStrategy` to apply to all containers of a container set, including the `duration` between each retry and the total number of `retries`. +Set an inner `retryStrategy` to apply to all containers of a container set, including the `duration` between each retry and the total number of `retries`. -See an example below: +See the example below: ```yaml apiVersion: argoproj.io/v1alpha1 @@ -132,7 +136,7 @@ metadata: name: containerset-with-retrystrategy annotations: workflows.argoproj.io/description: | - This workflow creates a container set with a retryStrategy. + This workflow creates a ContainerSet with a retryStrategy. spec: entrypoint: containerset-retrystrategy-example templates: @@ -161,12 +165,12 @@ spec: -!!! Note "Template-level `retryStrategy` vs Container Set `retryStrategy`" +!!! Note "Template-level `retryStrategy` vs ContainerSet `retryStrategy`" `containerSet.retryStrategy` works differently from [template-level retries](retries.md): - 1. Your `command` will be re-ran by the Executor inside the same container if it fails. + 1. The Executor re-runs your `command` inside the same container if it fails. - - As no new containers are created, the nodes in the UI remain the same, and the retried logs are appended to original container's logs. For example, your container logs may look like: + - Since no new containers are created, the nodes in the UI remain the same, and the retried logs are appended to the original container's logs. For example, your container logs may look like: ```text time="2024-03-29T06:40:25 UTC" level=info msg="capturing logs" argo=true intentional failure @@ -181,8 +185,8 @@ spec: Error: exit status 1 ``` - 1. If a container's `command` cannot be located, it will not be retried. + 2. If the Executor cannot locate a container's `command`, it will not retry. - - As it will fail each time, the retry logic is short-circuited. + - Since it will fail each time, the retry logic is short-circuited.