Skip to content

Commit

Permalink
docs: Improvements to ContainerSet Templates docs (argoproj#14143)
Browse files Browse the repository at this point in the history
Signed-off-by: Tim Collins <tim@thecollins.team>
  • Loading branch information
tico24 authored Feb 3, 2025
1 parent e00a004 commit dea9d0f
Show file tree
Hide file tree
Showing 2 changed files with 56 additions and 48 deletions.
4 changes: 4 additions & 0 deletions .spelling
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# markdown-spellcheck dictionary
000s
0s
100Gi
100Mi
100s
10h
Expand All @@ -11,6 +12,7 @@
1Mi
1h
1m
200Ki
2Gi
2h
30s
Expand Down Expand Up @@ -41,6 +43,7 @@ ClusterWorkflowTemplate
Codespaces
ConfigMap
ConfigMaps
ContainerSet
Couler
CronWorkflow
CronWorkflows
Expand Down Expand Up @@ -100,6 +103,7 @@ PDBs
PProf
PVCs
Peixuan
PersistentVolumeClaims
Ploomber
Postgres
PriorityClass
Expand Down
100 changes: 52 additions & 48 deletions docs/container-set-template.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,17 @@
# Container Set Template
# ContainerSet Template

> v3.1 and after
A container set template is similar to a normal container or script template, but allows you to specify multiple
containers to run within a single pod.
A ContainerSet template is similar to a normal container template, but it allows you to run multiple containers within a single Pod.

Because you have multiple containers within a pod, they will be scheduled on the same host. You can use cheap and fast
empty-dir volumes instead of persistent volume claims to share data between steps.
Since multiple containers run within a single Pod, they schedule on the same host.
You can use cheap and fast `emptyDir` volumes instead of PersistentVolumeClaims (PVCs) to share data between steps.

However, running all containers on the same host limits you to the host's resources.
Running all containers simultaneously may use more resources than running them sequentially.

Use ContainerSet templates strategically to avoid waiting for Pods to start and reduce the overhead of creating multiple init and wait containers.
This can be more efficient than using a DAG template.

```yaml
apiVersion: argoproj.io/v1alpha1
Expand Down Expand Up @@ -47,83 +52,82 @@ spec:
path: /workspace/message
```
There are a couple of caveats:
There are a few caveats:
1. You must use the [Emissary Executor](workflow-executors.md#emissary-emissary).
2. Or all containers must run in parallel - i.e. it is a graph with no dependencies.
3. You cannot use [enhanced depends logic](enhanced-depends-logic.md).
4. It will use the sum total of all resource requests, maybe costing more than the same DAG template. This will be a problem if your requests already cost a lot. See below.
1. You cannot use [enhanced depends logic](enhanced-depends-logic.md).
1. The ContainerSet uses the sum total of all resource requests, which may cost more than using the same DAG template. This can be problematic if your [resource requests](#️-resource-requests) are already high.
1. ContainerSet templates can only run container templates.
The containers can be arranged as a graph by specifying dependencies. This is suitable for running 10s rather than 100s
of containers.
You can arrange the containers as a graph by specifying dependencies.
## Inputs and Outputs
ContainerSet templates are suitable for running tens rather than hundreds of containers.
As with the container and script templates, inputs and outputs can only be loaded and saved from a container
named `main`.
## Inputs and Outputs
All container set templates that have artifacts must/should have a container named `main`.
As with container and script templates, you can only load and save inputs and outputs from a container named `main`.

If you want to use base-layer artifacts, `main` must be last to finish, so it must be the root node in the graph.
Include a container named `main` in all ContainerSet templates that have artifacts.

That may not be practical.
If you want to use base-layer artifacts, ensure `main` is the last to finish, making it the root node in the graph.
This may not always be practical.

Instead, have a workspace volume and make sure all artifacts paths are on that volume.
Instead, use a workspace volume and ensure all artifact paths are on that volume.

## ⚠️ Resource Requests

A container set actually starts all containers, and the Emissary only starts the main container process when the containers it depends on have completed. This mean that even though the container is doing no useful work, it is still consuming resources and you're still getting billed for them.
A ContainerSet starts all containers, and the [workflow executor](workflow-executors.md#emissary-emissary) only starts the main container process when the containers it depends on have completed.

This means that even though the container is doing no useful work, it still consumes resources and you are still billed for them.

If your requests are small, this won't be a problem.

If your requests are large, set the resource requests so the sum total is the most you'll need at once.

Example A: a simple sequence e.g. `a -> b -> c`
### Example A: Simple Sequence (a -> b -> c)

* `a` needs 1Gi memory
* `b` needs 2Gi memory
* `c` needs 1Gi memory

Then you know you need only a maximum of 2Gi. You could set as follows:
You need a maximum of 2Gi. You could set the requests as follows:

* `a` requests 512Mi memory
* `b` requests 1Gi memory
* `c` requests 512Mi memory

The total is 2Gi, which is enough for `b`. We're all good.

Example B: Diamond DAG e.g. a diamond `a -> b -> d and a -> c -> d`, i.e. `b` and `c` run at the same time.
The total is 2Gi, which is enough for `b`.

* `a` needs 1000 cpu
* `b` needs 2000 cpu
* `c` needs 1000 cpu
* `d` needs 1000 cpu
### Example B: Diamond DAG (a -> b -> d and a -> c -> d)

I know that `b` and `c` will run at the same time. So I need to make sure the total is 3000.
* `a` needs 1000 CPU
* `b` needs 2000 CPU
* `c` needs 1000 CPU
* `d` needs 1000 CPU

* `a` requests 500 cpu
* `b` requests 1000 cpu
* `c` requests 1000 cpu
* `d` requests 500 cpu
You know that `b` and `c` will run at the same time. So you need to make sure the total is 3000 CPU.

The total is 3000, which is enough for `b + c`. We're all good.
* `a` requests 500 CPU
* `b` requests 1000 CPU
* `c` requests 1000 CPU
* `d` requests 500 CPU

Example B: Lopsided requests, e.g. `a -> b` where `a` is cheap and `b` is expensive
The total is 3000 CPU, which is enough for `b + c`.

* `a` needs 100 cpu, 1Mi memory, runs for 10h
* `b` needs 8Ki GPU, 100 Gi memory, 200 Ki GPU, runs for 5m
### Example C: Lopsided Requests (a -> b)

Can you see the problem here? `a` only has small requests, but the container set will use the total of all requests. So it's as if you're using all that GPU for 10h. This will be expensive.
* `a` needs 100 CPU, 1Mi memory, runs for 10h
* `b` needs 8Ki GPU, 100Gi memory, 200Ki GPU, runs for 5m

Solution: do not use container set when you have lopsided requests.
In this case, `a` only has small requests, but the ContainerSet uses the total of all requests. So it's as if you're using all that GPU for 10h. This will be expensive.
This is a good example of when using a ContainerSet template would not be efficient.

## Inner `retryStrategy` usage

> v3.3 and after

You can set an inner `retryStrategy` to apply to all containers of a container set, including the `duration` between each retry and the total number of `retries`.
Set an inner `retryStrategy` to apply to all containers of a container set, including the `duration` between each retry and the total number of `retries`.

See an example below:
See the example below:

```yaml
apiVersion: argoproj.io/v1alpha1
Expand All @@ -132,7 +136,7 @@ metadata:
name: containerset-with-retrystrategy
annotations:
workflows.argoproj.io/description: |
This workflow creates a container set with a retryStrategy.
This workflow creates a ContainerSet with a retryStrategy.
spec:
entrypoint: containerset-retrystrategy-example
templates:
Expand Down Expand Up @@ -161,12 +165,12 @@ spec:

<!-- markdownlint-disable MD046 -- allow indentation within the admonition -->

!!! Note "Template-level `retryStrategy` vs Container Set `retryStrategy`"
!!! Note "Template-level `retryStrategy` vs ContainerSet `retryStrategy`"
`containerSet.retryStrategy` works differently from [template-level retries](retries.md):

1. Your `command` will be re-ran by the Executor inside the same container if it fails.
1. The Executor re-runs your `command` inside the same container if it fails.

- As no new containers are created, the nodes in the UI remain the same, and the retried logs are appended to original container's logs. For example, your container logs may look like:
- Since no new containers are created, the nodes in the UI remain the same, and the retried logs are appended to the original container's logs. For example, your container logs may look like:
```text
time="2024-03-29T06:40:25 UTC" level=info msg="capturing logs" argo=true
intentional failure
Expand All @@ -181,8 +185,8 @@ spec:
Error: exit status 1
```

1. If a container's `command` cannot be located, it will not be retried.
2. If the Executor cannot locate a container's `command`, it will not retry.

- As it will fail each time, the retry logic is short-circuited.
- Since it will fail each time, the retry logic is short-circuited.

<!-- markdownlint-enable MD046 -->

0 comments on commit dea9d0f

Please sign in to comment.