[WIP] RFC: symbolic import mapping

[claymcleod]

Note

This RFC is a work in progress and is currently not complete in its reasoning.

Concerns left to address.

• Cascaded importing of WDL through the dependency tree while also decoupling it from the source code.

Overview

Symbolic import mapping is proposed as a mechanism for dynamically mapping import items using a symbolic name at execution time in WDL. Here, we lay out the case that importing within WDL today is not flexible enough to support a mature package ecosystem in WDL—a desired expressed by many groups to encourage code reuse across the community. To get an idea, here's an example of what symbolic importing would look like in practice.

import fastp from biowdl

workflow hello {
  # ...

  call fastp.Fastp {
    # ...
  }

  # ...
}

Motivation

Importing in WDL today is a relatively constrained operation—only local file references or references to URLs are supported during import. This leaves quite a few things to be desired.

• Importing files require you to import things either (a) relative to your current working directory [not ergonomic for anything besides files within your existing project] or (b) absolutely [not portable].
• Importing URLs solves the two above issues but require a statically hosted URL at which the WDL document is made available.

Importing via the URL mechanism is fine for simple imports that rarely change, but that approach is far from enabling a package management ecosystem. For example,

• This approach is not conducive to proper versioning. Because the concept of a version is not really a first-class concept in URLs, tasks and workflows are rarely versioned at all. In practice, imports typically just point to the most up-to-date module on a main/master branch.
  • You might, for example, try to version your tools by importing the content at a particular git tag, but, when you go from v2.0.1 to v2.0.2, all of those URLs in your source code need to be updated. This is unnecessary work and noise for the git commit history.
• Similarly, many common workflows are impeded in this model. For example, consider the common paradigm of developing new tasks/workflows on branch in your GitHub repository—a task that is exceedingly cumbersome today.
  • When you create a branch, you're required to go through all of your files, update the imports to point to your branch, ask the reviewer to ignore all of the noise of these updates, and then, finally, to not forget to change the imports back to the main branch right before merging. More than once this has tripped us up on the St. Jude Cloud team, and I'm sure it's affected others.
• Last, because these imports are individually scattered throughout the code base, updates to dependencies are neither concise nor straightforward. This introduces a load of situations where human error can occur.

The above, plus the numerous issues regarding how package management can be improved in WDL, should convince you that the current state of importing in WDL is not a solid foundation upon which to build a package management ecosystem. This begs the question, "what is the criteria for a modern package management ecosystem"?

A more ideal state

We consider the following requirements of a solid foundation upon which to build.

• Symbolic resolution of package names should be supported—the resolution of these symbolic package names should be execution engine specific (e.g., where the execution engine looks like map a symbolic name to it's source path may be different across the engines), but the behavior should be dictated by the WDL specification such that this remains portable (e.g., the specification dictates that all executions make readily available documentation regarding how to link in these imports, and the behavior once linked in is the same).
• Imports should be supported across a variety of context-specific modalities, and those should be enabled by requirements on execution engines. For example, I'd start with the following requirements:
  • Symbolic linking to local directories.
  • Symbolic linking to publicly accessible git projects.
    • Within this, support for pulling a specific tag or branch in one location.
• The ability to cascade packages throughout the dependency tree.
• Versioning of packages should be so straightforward that it is the norm: a community of unversioned packages in a recipe for disaster. Those versions should be required to conform to some community standard, such as semantic versioning, such that tooling around package resolution and management could take advantage of the standards.
• The user should have flexibility to pull in package scopes and rename them as necessary to avoid naming conflicts in their code base.

Specific suggestions

Note

This section specifically uses the sprocket engine that my team is writing as an example. This is because it's hard to make a concrete case for what a solution might look like outside of both (a) the WDL specification and (b) a concrete execution engine. That being said, none of this is binding to what sprocket might eventually do—just consider it as an example situation.

Overview

I suggest we do the following things: all of which get their own sections below.

• Introduce syntax for symbolic imports.
• Introduce requirements on execution engines for these imports.
• Deprecate the importing of individual files.

Introduce syntax for symbolic imports

First, let's address the relatively straightforward question of what for the imports would take. Because imports today must be enclosed in quotes, a backwards compatible change that seems sufficient would be that any non-quoted import is assumed to be a symbolic import. For example,

# This is assumed to be a local file import.
import "biowdl"

# This is assumed to be a symbolic import.
import biowdl

Next, there are a number of different ways you can structure the identifier that symbolic imports link into—particularly as it relates to conflicts in the identifiers. To start, we suggest to start with the simplest approach, which is an identifier conforming to a word pattern ([a-zA-Z0-9_]+). The lack of namespacing means that everything must be explicitly handled by the user.

Consider a Sprocket.toml file that contains a dependency section like so:

# ...

[dependencies]
# Importing a package with the same name as the scope.
tasks = { git = "https://github.com/stjudecloud/tasks", tag = "v1.0.0" }

# An import that is explicitly renamed by the user. Notably, under other paradigms, 
# this would cause a name conflict with the import above.
biowdl = { git = "https://github.com/biowdl/tasks", branch = "develop" }

Effectively, we're empowering (and requiring) users to solve name conflicts themselves. The authors feel this a reasonable first step in this direction rather than, say, a complicated set of name conflict requirements.

Concretely, the following import modalities would be required (as show through examples in the fictional Sprocket.toml file).

[dependencies]
# Importing packages on the local filesystem is required. 
# No further options are specified—the files are just imported as is.
package = { path = "../../path/to/my/package" }

# Importing packages from a public git repository is required.
# Packages in this form _must_ specify the branch, tag, or commit that 
# needs to be checked out.
package = { git = "https://github.com/package/name", branch = "main" }
package = { git = "https://github.com/package/name", tag = "v1.0.0" }
package = { git = "https://github.com/package/name", commit = "bcd123c" }

Notably, the specifics of how these imports would be linked in across the various execution engines is left up to the execution engine—there is no requirement from the specification other than (a) these modalities must be supported and (b) they must be clearly documented for users to be able to take advantage of.