[openSUSE] configure docker daemon using json configuration file

[FroggyFlox]

Fixes #2032.
Fixes #2088.
@phillxnet, ready for review.

This pull request proposes to use the docker daemon configuration file (located at /etc/docker/daemon.json) to customize and configure the docker daemon with our required options, in replacement of our current docker-wrapper. Notably, these changes would be done in Rockstor-NG only, so that compatibility with CentOS Rockstor is maintained (and fully functional).
This PR would thus fix two issues: #2032 and #2088.

Aims & Logic

In openSUSE flavors only (detected using distro.id()), this PR creates a dedicated docker-daemon.json file using the daemon.json file provided by the package as a template. It then updates or adds new keys and values as needed by Rockstor. These options are the same that were previously defined through our docker-wrapper:

• data-root
• storage-driver
• storage-opts
• log-driver

In addition, a new option related to logs has been added:

• log-opts: "tag": "{{.ImageName}}/{{.Name}}". This option is purely "cosmetic" and simply prepends all log lines coming from a container with the names of the docker image and container (see demonstration below). Although it adds more characters to each line of corresponding logs, it does help identify what container is producing those lines and allows subsequent filtering of logs by container name or image name. @phillxnet, we can easily do without this option if preferred.

Once all options are defined, we write the corresponding json file in settings.CONFROOT/docker-daemon.json).
Accordingly, we then edit the docker.service file to:

• not replace /usr/bin/dockerd by our docker-wrapper
• not define the data-root as argument to dockerd
• instead point to the config file written above using the --config-file flag.

This results in the docker.service file triggering /usr/bin/dockerd directly (rather than through our docker-wrapper), which maintains full compatiblity of systemd control over its processes when the docker service (Rock-on service) is turned off (thereby fixing #2088).
Furthermore, because the dockerd configurations are not defined as arguments in the dockerd call but entirely in our new settings.CONFROOT/docker-daemon.json file, there is no conflict of options being defined in two different places (thereby fixing #2032).

Demonstration

In openSUSE Leap15.1, configuring the Rock-on service to use the share named rockons_root and turning it on completes successfully (fixes #2032). The resulting systemd file is:

rockdev:~ # systemctl status -l docker
â—� docker.service - Docker Application Container Engine
   Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2019-11-22 14:52:28 EST; 8s ago
     Docs: http://docs.docker.com
 Main PID: 3235 (dockerd)
    Tasks: 23
   Memory: 213.0M
      CPU: 724ms
   CGroup: /system.slice/docker.service
           ├─3235 /usr/bin/dockerd --add-runtime oci=/usr/sbin/docker-runc --config-file /opt/build/conf/docker-daemon.json
           └─3276 docker-containerd --config /var/run/docker/containerd/containerd.toml --log-level info

Nov 22 14:52:26 rockdev dockerd[3235]: time="2019-11-22T14:52:26.177148720-05:00" level=warning msg="Your kernel does not support swap memory limit"
Nov 22 14:52:26 rockdev dockerd[3235]: time="2019-11-22T14:52:26.177176422-05:00" level=warning msg="Your kernel does not support cgroup rt period"
Nov 22 14:52:26 rockdev dockerd[3235]: time="2019-11-22T14:52:26.177185526-05:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Nov 22 14:52:26 rockdev dockerd[3235]: time="2019-11-22T14:52:26.177375398-05:00" level=info msg="Loading containers: start."
Nov 22 14:52:27 rockdev dockerd[3235]: time="2019-11-22T14:52:27.550523259-05:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16.>
Nov 22 14:52:28 rockdev dockerd[3235]: time="2019-11-22T14:52:28.044087899-05:00" level=info msg="Loading containers: done."
Nov 22 14:52:28 rockdev dockerd[3235]: time="2019-11-22T14:52:28.414342975-05:00" level=info msg="Docker daemon" commit=74b1e89e8ac6 graphdriver(s)=btrfs version=19.03.1
Nov 22 14:52:28 rockdev dockerd[3235]: time="2019-11-22T14:52:28.414498991-05:00" level=info msg="Daemon has completed initialization"
Nov 22 14:52:28 rockdev dockerd[3235]: time="2019-11-22T14:52:28.525473149-05:00" level=info msg="API listen on /var/run/docker.sock"
Nov 22 14:52:28 rockdev systemd[1]: Started Docker Application Container Engine.

Note how the MAINPID now points to dockerd: Main PID: 3235 (dockerd).

The docker.service file now reads:

rockdev:~ # cat /etc/systemd/system/docker.service 
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target lvm2-monitor.service SuSEfirewall2.service rockstor-bootstrap.service

[Service]
EnvironmentFile=/etc/sysconfig/docker

# While Docker has support for socket activation (-H fd://), this is not
# enabled by default because enabling socket activation means that on boot your
# containers won't start until someone tries to administer the Docker daemon.
Type=notify
NotifyAccess=all
ExecStart=/usr/bin/dockerd --add-runtime oci=/usr/sbin/docker-runc $DOCKER_NETWORK_OPTIONS $DOCKER_OPTS --config-file /opt/build/conf/docker-daemon.json
ExecReload=/bin/kill -s HUP $MAINPID

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity

# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this property.
TasksMax=infinity

# Set delegate yes so that systemd does not reset the cgroups of docker containers
# Only systemd 218 and above support this property.
Delegate=yes

# Kill only the docker process, not all processes in the cgroup.
KillMode=process

# Restart the docker process if it exits prematurely.
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

The resulting settings.CONFROOT/docker-daemon.json is:

rockdev:~ # cat /opt/build/conf/docker-daemon.json 
{
  "data-root": "/mnt2/rockons_root", 
  "log-driver": "journald", 
  "log-level": "warn", 
  "log-opts": {
    "tag": "{{.ImageName}}/{{.Name}}"
  }, 
  "storage-driver": "btrfs", 
  "storage-opts": [
    "btrfs.min_space=1G"
  ]
}

The rock-on service can be toggled OFF and ON repeatedly without errors (fixes #2088).
Moreover, the journalctl logs now shows the following details when starting the Emby server rock-on, for instance:

Nov 22 14:58:00 rockdev emby/embyserver:latest/embyserver[5949]: [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
Nov 22 14:58:00 rockdev emby/embyserver:latest/embyserver[5949]: [s6-init] ensuring user provided files have correct perms...exited 0.
Nov 22 14:58:00 rockdev emby/embyserver:latest/embyserver[5949]: [fix-attrs.d] applying ownership & permissions fixes...
Nov 22 14:58:00 rockdev emby/embyserver:latest/embyserver[5949]: [fix-attrs.d] done.
Nov 22 14:58:00 rockdev emby/embyserver:latest/embyserver[5949]: [cont-init.d] executing container initialization scripts...
Nov 22 14:58:00 rockdev emby/embyserver:latest/embyserver[5949]: [cont-init.d] done.
Nov 22 14:58:00 rockdev emby/embyserver:latest/embyserver[5949]: [services.d] starting services
Nov 22 14:58:00 rockdev emby/embyserver:latest/embyserver[5949]: [services.d] done.
Nov 22 14:58:04 rockdev emby/embyserver:latest/embyserver[5949]: Info Main: Application path: /system/EmbyServer.dll
Nov 22 14:58:05 rockdev emby/embyserver:latest/embyserver[5949]: Info Main: Emby

Note how both the docker image name (emby/embyserver:latest) and container name (embyserver) are detailed.

Of note, the same procedure and its results have been confirmed in Tumbleweed.
In CentOS, however, everything remains the same as before and the docker-wrapper is still used.

In addition to the main logic, a small check is done on whether or not the docker-daemon.json file can be written, and returns a legible error in case of error:

Testing

As detailed in #2088 (#2088 (comment)), the default /etc/docker/daemon.json is not affected by a docker package update as long as it is still present. As a result, it seems that using it as template for our docker-daemon.json should be OK and won't be affected by package updates.
/etc/docker/daemon.json will be replaced (as reported by @Hooverdan96) only if it does not exist at the time of the docker package update.

As we wouldn't be using the docker-wrapper in openSUSE anymore, it seems important to review what it was doing and whether or not we are loosing some sanity checks (spoiler: we are not). Indeed, we had a check for:

1. the share used as rockons_root does exist.
2. this share is currently mounted.

Notably, we already have both of these checks in docker_service.py:

            share = self._validate_root(request, config["root_share"])
            mnt_pt = "{}{}".format(settings.MNT_PT, share.name)
            if not share.is_mounted:
                mount_share(share, mnt_pt)

    def _validate_root(self, request, root):
        try:
            return Share.objects.get(name=root)
        except Exception as e:
            logger.exception(e)
            e_msg = "Share name ({}) does not exist.".format(root)
            handle_exception(Exception(e_msg), request)

Caveats & shortcomings

@phillxnet, not a real shortcoming, but let me know if you think of a better location for our docker-daemon.jon than its current home: settings.CONFROOT.

[phillxnet]

@FroggyFlox Very nice solution and thanks for the clear and full explanation. Much appreciated.

This was a tricky one that caught us by surprise so thank for stepping up and sorting it. Super timely given our imminent openSUSE based testing releases.

Tested and worked exactly as stated re fixes and function.

Thanks again for modernising this area of Rockstor's docker management; and nice touch on the file write check and logging enhancements.

[FroggyFlox]

Thanks, @phillxnet!
Let's hope we'll be good with this setup for a little while, at least.

Cheers,