Nut: Under The Hood

This wiki targets contributors, advanced users, and curious people. The goal is to give the reader a better understanding about how Nut works.

CLI

Nut parses command line arguments using codegangsta/cli. Everything happens in main.go:

• parse nut.yml to retrieve the configuration of the project (more about this later)
• saves the context in which nut has been called: folder where the user called nut, and path where nut.yml was found
• for each macro defined in the configuration, create a CLI command (e.g. run and build)
• add flags for operations that are not defined by the configuration (e.g. --init and --exec)
• parse CLI arguments, and execute the corresponding macro, given the context in which nut was called.

Execution of a macro

core.go defines how to process a macro, given the context in which nut was called:

• connect to Docker host, either with socket unix:///var/run/docker.sock, or by using Docker Machine environment variables (e.g. DOCKER_HOST), using fsouza/go-dockerclient
• read the name of the docker image from the configuration (i.e, from the macro), and pull the image from registry, if need be
• prepare arguments to create and start the container, based on the configuration (which directory should be mounted in the container? Which port should be bound?)
• create the container
• start the container with the list of commands defined by the macro, and attach to it, using fgrehm/go-dockerpty (right now it's a fork that supports Windows: matthieudelaro/go-dockerpty)
• stop and destroy the container upon task completion

Note: By default, Nut does not rely on docker cli. It connect directly to docker host. This means that you can connect to a distant docker host, without ever installing docker on your machine. It also means that each parameter of the configuration has to be translated into a set of options for Docker API. However, on Windows, Nut relies on Docker CLI by default (this behavior should be changed). Use flags --dockerapi and --dockercli to specify which way you wish to target docker.

Configuration files

Nut parses configuration from nut.yml in current folder. If none is found, then it looks in parent folders until it reaches the root of the file system. YAML is parsed using gopkg.in/yaml.v2. Configuration file syntaxes are defined in sub-package nut/config:

• basic syntax elements are defined in nut/config/config_base.go. Main elements are projects and macros, which both inherit elements of config.
• those main elements comply to interfaces Project, Macro, and Config. Note that both Project and Macro define Config.
• each version XXX of the syntax is defined in nut/config/config_vXXX.go. Former versions of the syntax should not be modified, in order to ensure backward compatibility.
• functions handling Project, Macro and Config are defined in nut/config/functions_over_interfaces.go. For example, GetEnvironmentVariables(config Config) will process the given Config (e.g a Macro), and return its environment variables, merged with those of its parents (e.g the Project to which the Macro belongs). This is where configuration overload occurs.

GUI

Most linux graphical applications connect to server X, which renders windows on the screen. This designs allows to open windows on a remote machine. Since there are implementations of server X on OSX and Windows, Nut connects the proper sockets together to let the magic happen when you declare enable_gui: true in nut.yml. Further readings:

• https://blog.jessfraz.com/post/docker-containers-on-the-desktop/
• https://blog.bennycornelissen.nl/bwc-gui-apps-in-docker-on-osx/
• https://github.com/docker/docker/issues/8710

Nvidia devices

For data scientists, having access to graphic cards is a requirement. Nut consumes Nvidia Docker Plugin REST API to retrieve the list of Nvidia devices. This means that Nvidia Docker daemon must be running when Nut is called.

Synchronize timezones

Docker takes care of synchronizing the clock of the host and of the container. However, the timezone is not taken care of. It's annoying when the container creates files on the host file system, or displays time in logs. So Nut fixes this issue by setting the environment variable $TZ to "UTC-09:00" (for example). See http://www.cyberciti.biz/faq/linux-unix-set-tz-environment-variable/ for details about $TZ.

.nut/

Nut stores some information in .nut folder, next to nut.yml. Right now, the only thing that are stored are the environments that nut.yml overrides (using based_on parameter). If the configuration is based from a Git repository, that the configuration from the repo will be downloaded and saved to .nut/environments/name/of/the/repot/nut.yml.

Later .nut/ will store key/value pairs to reuse the same container multiple time, etc.