Skip to content

Latest commit

 

History

History
123 lines (82 loc) · 4.85 KB

2023-06-02.md

File metadata and controls

123 lines (82 loc) · 4.85 KB

2 June 2023

or, Nextstrain runtime env

previously (important context)

New global runner options (akin to --exec) registered nextstrain/cli/runner/__init__.py:register_arguments:

--env NAME
--env NAME=VALUE
--envdir PATH

Centrally handled in nextstrain/cli/runner/__init__.py:run and presented as a single set of env to pass thru to the runtime (e.g. the same way cpus or working_volume are).

Each runner would then be responsible doing the most reasonable thing to get the values into the runtime environment.

Implementation 1: Pass via disk (write an envdir)

A temporary envdir in a private location is created. Files in any --envdir options are copied to it. Values from any --env options are written to it.

  • Docker would mount the envdir on the host as /nextstrain/env.d in the container and arrange to exec thru envdir /nextstrain/env.d (possibly via the entrypoint).

  • Singularity would do the same as Docker.

  • Conda would arrange to exec thru envdir LOCAL-DIR.

  • Ambient would exec within with envdir.Env(…) or possibly thru envdir LOCAL-DIR. Consideration is mostly one of backcompat for folks without envdir.

  • AWS Batch would upload the envdir to S3 separately from the workdir and arrange to exec thru envdir /nextstrain/env.d (possibly via the entrypoint).

    Additionally, this would change how the existing env is passed so it too is not incorporated into the job submission itself.

    Possibly, instead of a separate workdir archive, the archive could become a a combined archive rooted at /nextstrain instead of /nextstrain/build? But then envdir values are potentially downloaded by attaching clients later.

While there are non-envdir methods for all runtimes, the above all go thru a written-to-disk envdir because because that avoids revealing the env values in side-channels like docker inspect, AWS Batch job submissions, process listings, etc.

On the other hand, it does require writing values to disk, at least transiently, when some runtimes (Conda, Singularity, ambient) could keep values in memory across an exec. This can be mitigated by using temporary directories under the user's home (e.g. ~/.nextstrain/tmp/…) with restricted permissions and removing them immediately after use by a specialized envdir (or an rm -rf that exec-chains?). (Although this means that any given envdir is no longer acceptable, so maybe we do something else after all…? Or provide our own envdir to all runtimes.)

Implemention 2: Pass via memory (mostly)

A temporary dict is created (or an existing one, e.g. extra_env, is used). Any --envdir options are read into it (i.e. not into the current environment). Values from any --env options are added to it.

  • Docker would pass values thru the existing extra_env parameters.

    This passes via memory to docker run (the Docker client) and then passes to dockerd (the Docker server) via socket. It's persisted to disk by dockerd, but is accessible only to root.

    These values (as now) would be visible to docker inspect, but like the ability to docker run, that command requires root or delegated access to dockerd. So it doesn't change trust levels.

  • Singularity would pass values thru the existing extra_env parameters.

  • Conda would pass values thru the existing extra_env parameters.

  • Ambient would pass values thru the existing extra_env parameters.

  • AWS Batch would pass values thru the existing extra_env parameters.

    The values (as now) would be visible in the job submission details.

Implementation 3: Minimize persistence outside of process memory

Implementation 1 for Docker and AWS Batch, to avoid persistence in side-channels (docker inspect, AWS Batch job submissions) and minimize persistence on storage (disk, S3).

Implementation 2 for everything else, to avoid persistence entirely (i.e. outside of process memory).

Bits to do

In no particular order:

  • Create /nextstrain/env.d in the container/image (PR)
  • Exec thru envdir /nextstrain/env.d in container entrypoint (PR)
    • Consider how to remove /nextstrain/env.d contents after exec
  • Download env.d from S3 in container entrypoint for AWS Batch (PR)
  • Add --env and --envdir options (PR)
  • Implement their behaviour in each runner (PR)

Bonus tidy up

The hostenv concept in the runner code predates the extra_env concept. Currently each runner handles hostenv independently, while extra_env is centralized thru the runner dispatcher. In the future, hostenv handling should be centralized the same way and end up merely as another (automatic) part of extra_env.