[openSUSE] Rock-on service does not stop correctly

[FroggyFlox]

On openSUSE Leap15.1 (expected to be the case in Tumbleweed as well, but untested), while the Rock-on service turns ON without problem the first time, it errors out on subsequent starts (after being turned OFF, of course). The error is:

CommandException: Error running a command. cmd = /usr/bin/systemctl start docker. rc = 1. stdout = ['']. stderr = ['Job for docker.service failed because the control process exited with error code.', 'See "systemctl  status docker.service" and "journalctl  -xe" for details.', '']

Full error

[16/Nov/2019 10:26:32] ERROR [storageadmin.middleware:32] Exception occurred while processing a request. Path: /api/sm/services/docker/start method: POST
[16/Nov/2019 10:26:32] ERROR [storageadmin.middleware:33] Error running a command. cmd = /usr/bin/systemctl start docker. rc = 1. stdout = ['']. stderr = ['Job for docker.service failed because the control process exited with error code.', 'See "systemctl  status docker.service" and "journalctl  -xe" for details.', '']
Traceback (most recent call last):
  File "/opt/build/eggs/Django-1.8.16-py2.7.egg/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/build/eggs/Django-1.8.16-py2.7.egg/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/build/eggs/Django-1.8.16-py2.7.egg/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/build/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 452, in dispatch
    response = self.handle_exception(exc)
  File "/opt/build/eggs/djangorestframework-3.1.1-py2.7.egg/rest_framework/views.py", line 449, in dispatch
    response = handler(request, *args, **kwargs)
  File "/opt/build/eggs/Django-1.8.16-py2.7.egg/django/utils/decorators.py", line 145, in inner
    return func(*args, **kwargs)
  File "/opt/build/src/rockstor/smart_manager/views/docker_service.py", line 106, in post
    systemctl(self.name, 'start')
  File "/opt/build/src/rockstor/system/services.py", line 63, in systemctl
    return run_command([SYSTEMCTL_BIN, switch, service_name])
  File "/opt/build/src/rockstor/system/osi.py", line 120, in run_command
    raise CommandException(cmd, out, err, rc)
CommandException: Error running a command. cmd = /usr/bin/systemctl start docker. rc = 1. stdout = ['']. stderr = ['Job for docker.service failed because the control process exited with error code.', 'See "systemctl  status docker.service" and "journalctl  -xe" for details.', '']

The journalctl log shows more details:

Error running a command. cmd = /usr/bin/dockerd --log-driver=journald --storage-driver btrfs --storage-opt btrfs.min_space=1G --add-runtime oci=/usr/sbin/docker-runc --data-root /mnt2/rockons_root. rc = 1. stdout = ['']. stderr = ['time="2019-11-16T10:26:37.433837429-05:00" level=info msg="Starting up"', 'failed to start daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid', '']

Full error

Nov 16 10:26:35 rockdev systemd[1]: Starting Docker Application Container Engine...
Nov 16 10:26:37 rockdev docker-wrapper[28856]: Traceback (most recent call last):
Nov 16 10:26:37 rockdev docker-wrapper[28856]:   File "/opt/build/bin/docker-wrapper", line 39, in <module>
Nov 16 10:26:37 rockdev docker-wrapper[28856]:     sys.exit(scripts.docker_wrapper.main())
Nov 16 10:26:37 rockdev docker-wrapper[28856]:   File "/opt/build/src/rockstor/scripts/docker_wrapper.py", line 52, in main
Nov 16 10:26:37 rockdev docker-wrapper[28856]:     run_command(cmd)
Nov 16 10:26:37 rockdev docker-wrapper[28856]:   File "/opt/build/src/rockstor/system/osi.py", line 120, in run_command
Nov 16 10:26:37 rockdev docker-wrapper[28856]:     raise CommandException(cmd, out, err, rc)
Nov 16 10:26:37 rockdev docker-wrapper[28856]: system.exceptions.CommandException: Error running a command. cmd = /usr/bin/dockerd --log-driver=journald --storage-driver btrfs --storage-opt btrfs.min_space=1G --add-runtime oci=/usr/sbin/docker-runc --data-root /mnt2/rockons_root. rc = 1. stdout = ['']. stderr = ['time="2019-11-16T10:26:37.433837429-05:00" level=info msg="Starting up"', 'failed to start daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid', '']

As we can see in the error above, Docker fails to start because it is found to still be running. Indeed, we have:

rockdev:~ # ps aux | grep -v grep | grep docker
root     19321  0.1  4.3 972104 89280 ?        Sl   09:55   0:02 /usr/bin/dockerd --log-driver=journald --storage-driver btrfs --storage-opt btrfs.min_space=1G --add-runtime oci=/usr/sbin/docker-runc --data-root /mnt2/rockons_root
root     19330  0.0  2.2 970240 45756 ?        Ssl  09:55   0:02 docker-containerd --config /var/run/docker/containerd/containerd.toml --log-level info

After a little investigation, I believe the problem is linked to how systemd is stopping the docker service. Indeed, in the docker.service files we source directly from the package (since #2046), the docker service is meant to be stopped by terminating the dockerd daemon process--which is thus the way it's meant to be stopped. This is illustrated in its service file by the following line:

# Kill only the docker process, not all processes in the cgroup.
KillMode=process

Full file

rockdev:~ # cat /etc/systemd/system/docker.service 
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target lvm2-monitor.service SuSEfirewall2.service rockstor-bootstrap.service

[Service]
EnvironmentFile=/etc/sysconfig/docker

# While Docker has support for socket activation (-H fd://), this is not
# enabled by default because enabling socket activation means that on boot your
# containers won't start until someone tries to administer the Docker daemon.
Type=notify
NotifyAccess=all
ExecStart=/opt/build/bin/docker-wrapper --add-runtime oci=/usr/sbin/docker-runc $DOCKER_NETWORK_OPTIONS $DOCKER_OPTS /mnt2/rockons_root
ExecReload=/bin/kill -s HUP $MAINPID

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity

# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this property.
TasksMax=infinity

# Set delegate yes so that systemd does not reset the cgroups of docker containers
# Only systemd 218 and above support this property.
Delegate=yes

# Kill only the docker process, not all processes in the cgroup.
KillMode=process

# Restart the docker process if it exits prematurely.
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

Using the standard docker "setup", this works well as the main process being killed is the dockerd daemon. Indeed:

rockdev:~ # systemctl status -l docker
- docker.service - Docker Application Container Engine
   Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-11-16 11:17:33 EST; 1min 18s ago
     Docs: http://docs.docker.com
 Main PID: 11475 (dockerd)
    Tasks: 21
   Memory: 70.2M
      CPU: 718ms
   CGroup: /system.slice/docker.service
           - 11475 /usr/bin/dockerd --add-runtime oci=/usr/sbin/docker-runc
           - 11483 docker-containerd --config /var/run/docker/containerd/containerd.toml --log-level info

Nov 16 11:17:33 rockdev dockerd[11475]: time="2019-11-16T11:17:33.576608523-05:00" level=warning msg="Your kernel does not support swap memory limit"
(...)

The main PID actually is the dockerd daemon, and because the KillMode is set to process in the service file, the docker daemon is actually killed when stopping the docker service.

Using the Rockstor's customized docker service file, however, we see the difference:

rockdev:~ # systemctl status -l docker
- docker.service - Docker Application Container Engine
   Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-11-16 11:31:37 EST; 8s ago
     Docs: http://docs.docker.com
 Main PID: 16307 (docker-wrapper)
    Tasks: 24
   Memory: 104.3M
      CPU: 2.590s
   CGroup: /system.slice/docker.service
           - 16307 /usr/bin/python /opt/build/bin/docker-wrapper --add-runtime oci=/usr/sbin/docker-runc /mnt2/rockons_root
           - 16373 /usr/bin/dockerd --log-driver=journald --storage-driver btrfs --storage-opt btrfs.min_space=1G --add-runtime oci=/usr/sbin/docker-runc --data-root /mnt2/rockons_root
           - 16380 docker-containerd --config /var/run/docker/containerd/containerd.toml --log-level info

Nov 16 11:31:34 rockdev systemd[1]: Starting Docker Application Container Engine...
Nov 16 11:31:37 rockdev systemd[1]: Started Docker Application Container Engine.

As we can see above, the main PID refers here to our docker-wrapper and not directly the dockerd daemon. In this context, the KillMode set to process understandably kills only the docker-wrapper and not the docker daemon.

As a result, setting the KillMode to control-group (default in systemd) seems to resolve the problem. This was tested after starting the Rock-on service (thus re-setting the docker.service file to Rockstor's customized version), and then:

rockdev:~ # sed -i 's:KillMode=process:#KillMode=process:g' /etc/systemd/system/docker.service
rockdev:~ # systemctl stop docker
rockdev:~ # ps aux | grep -v grep | grep docker
rockdev:~ # 

We could thus change the KillMode setting in the service file while we're customizing it:

rockstor-core/src/rockstor/smart_manager/views/docker_service.py

Lines 85 to 101 in ec3903d

	with open(inf) as ino, open(outf, 'w') as outo:
	for l in ino.readlines():
	if re.match('ExecStart=', l) is not None:
	outo.write('{} {}\n'.format(
	l.strip().replace(DOCKERD, docker_wrapper, 1),
	mnt_pt))
	elif re.match('Type=notify', l) is not None:
	# Our docker wrapper use need NotifyAccess=all: avoids
	# "Got notification message from PID ####1, but
	# reception only permitted for main PID ####2"
	outo.write(l)
	outo.write('NotifyAccess=all\n')
	elif re.match('After=', l) is not None:
	outo.write('{} {}\n'.format(
	l.strip(), 'rockstor-bootstrap.service'))
	else:
	outo.write(l)

@phillxnet, what do you think of this approach? It seems it's a regression I introduced in #2046 and deeply apologize about it... I'm just surprised I didn't notice that when testing that PR and that it hasn't emerged since then...

[phillxnet]

@FroggyFlox

@phillxnet, what do you think of this approach?

I thinks that's just fine. We are customising a tad anyway and if your sure of the KillMode modification I say we go for it. As for regressions we are dealing with 3 moving targets, but at least we have the pending testing channel to help with just this kind of think: just need to get one or two more things lined up ready.

[phillxnet]

@FroggyFlox Nice find and proposed fix by the way. And again, quite timely.

[FroggyFlox]

if your sure of the KillMode modification

I've never messed with that option before and only found out about it while trying to debug that issue. That made me learn a lot about a service file configuration, actually... I'll need to test more of course and see if it breaks anything else down the line but it seems to be OK so far; this is the KillMode we use currently in CentOS, after all. I just need to make sure there's not a very good reason why the package comes with the process KillMode.

The other option would be to do without our docker-wrapper but that seems like a more risky route, especially given I'm not particularly familiar with its reasons and history.

I'll try to go on with changing the KillMode and test that out, then.

[FroggyFlox]

@phillxnet , a quick update after a brief investigation:

As I was trying to see why the default systemd docker.service file specifically does NOT use the default KillMode, it seems that not using the process KillMode isn't a good idea and can break a few features of the docker daemon.
The commit in question in which this was specified is moby/moby@db435f5 and not following this should lead to issues with containers down the line. See these few examples:

• https://success.docker.com/article/error-starting-containers-after-docker-engine-restart
• Missing KillMode=process in docker.service cause live-restore fail. docker/machine#3963

If we keep KillMode=process, however (which would be the recommended way as far I can see), we need to ensure the MAINPID refers to /usr/bin/dockerd and not our docker_wrapper. In this context, I'm actually wondering whether we shouldn't "simply" do without this wrapper altogether.

It seems to me that its only purpose is to pass along our Rockstor-specific customizations (--data-root, --log-driver, --storage-driver, and --storage-opt); but please correct me if I'm wrong. The Docker documentation does recommend to use the /etc/docker/daemon.json file to set these options instead of passing them as arguments to dockerd as it is platform independent and a lot more portable:

There are a number of ways to configure the daemon flags and environment variables for your Docker daemon. The recommended way is to use the platform-independent daemon.json file, which is located in /etc/docker/ on Linux by default. See Daemon configuration file.
https://docs.docker.com/config/daemon/systemd/

Interestingly, if we simply use this file, we may fix our current issue we have in both openSUSE flavors re-docker settings being defined in both the daemon.json and dockerd call (#2032). One option would be to simply specify these options by customizing this file (or a separate one that is then referenced to the dockerd call with the --config-file flag) rather than by customizing docker.service in docker_service.py :

The default location of the configuration file on Linux is /etc/docker/daemon.json. The --config-file flag can be used to specify a non-default location.
https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file

We still have customizations to make in the docker.service file, but it seems we could do without our wrapper and thus simplify the situation by using a more "default" way to customize the docker daemon. What do you think? Am I missing an obvious need for the docker_wrapper here?

[phillxnet]

@FroggyFlox Re:

I'm actually wondering whether we shouldn't "simply" do without this wrapper altogether.

Yes, I was considering this, if the quick fix you suggested earlier is a no go. Which it looks to be here.

The Docker documentation does recommend to use the /etc/docker/daemon.json

Exactly, that way we go with the flow, and edit that file instead for our logging, driver, etc.

Docker has moved on some since Rockstor first adopted it and maybe, as you suggest it's time we adapted to that via embracing the daemon.json. But we may still need to set dependencies up as we are responsible for mounting the Rock-ons root and then starting docker. So we just have to ensure we add our appropriate service as a dependency to the docker service file so that it waits for us to mount it's required subvols.

Interestingly, if we simply use this file, we may fix our current issue we have in both openSUSE flavors re-docker settings being defined in both the daemon.json and dockerd call (#2032).

Yes, again going with the flow of change. But I'd rather do this only with our openSUSE offerings.

We still have customizations to make in the docker.service file, but it seems we could do without our wrapper and thus simplify the situation by using a more "default" way to customize the docker daemon. What do you think?

I think you have it. I've only dabbled with that element of the code of late so the wrapper will need to be fully understood but I definitely think you are on the right track. If your game to steam in we can check the results. Again if done in openSUSE only then we only affect the testing channel anyway, which of course is what it's for. Let me know if you are game to take this one on as it looks like you are far further along than I anyway. I definitely think we can pass more work back to systemd here and go the default config route for docker. Nice call. I think our current wrapper was initially, as you say, just to pass our customisation as docker was a lot younger at the time (in computer years).

So looks like we are all turned around on this one, but it also looks to be a good thing. We may end up with a code simplification which would be nice, at least on the openSUSE side. Note here if you are game to continue tackling this one as it's a key one for our soon to be release installer / testing channel.

[FroggyFlox]

Thanks a lot for a prompt, yet exhaustive feedback, @phillxnet!

I will of course try to tackle that one, especially considering it is fresh in my head. I'll still need to make sure I'm not missing one of the functions of the docker_wrapper, but what you described related to its use actually fits what I thought so it makes a lot of sense.

I'll thus make sure to keep the systemd customizations to their minimum and switch as much as I can to the daemon.json config file. Interestingly, as pretty much everything related to the docker daemon can be configured there, we could even offer further advanced config of the Rock-on service for those who want it (something with a text box to pass all arguments to the config file similar to what we do for the SMART or Samba services, for instance). I can already see a few things that can break the rock-on service if we do that, but at least it's an option that would open to us in the future, were we interested in offering it.

Going back to this issue, I will, of course, only make change to openSUSE flavors as I'm not even sure our docker version in CentOS would support this way of config anyway.

Thanks again, and hopefully I'll have a PR ready very soon.

[FroggyFlox]

we could even offer further advanced config of the Rock-on service for those who want it (something with a text box to pass all arguments to the config file similar to what we do for the SMART or Samba services, for instance).

Just a quick mention about this: I was thinking of a use of such feature for users who need to use a different runtime than the default one, such as the nvidia one, which has come up a couple of time in the forum. That's a rather advanced feature, but it could be of use for some and thus worth considering in the future.

[FroggyFlox]

I now have an implementation seemingly working in Leap15.1 (see https://github.com/FroggyFlox/rockstor-core/tree/WIP_Docker_KillMode):

• can turn on the rock-on service ON for the first time without deleting /etc/docker/daemon.json
• can toggle the Rock-on service ON/OFF repeatedly without error
• a running rock-on successfully shuts downs and restarts upon toggling OFF/ON the Rock-on service while said rock-on was running.

Both this issue and #2032 thus seem fixed with this branch.

Now that we have proof of concept, I will proceed with refining, writing tests, and further testing the new logic.

[FroggyFlox]

@phillxnet, may I have your feedback on the following?

In this new branch, I currently load the default daemon.json file (located at /etc/docker/daemon.json) that is shipped with the docker package. I thus update it with our needed Rockstor customizations and then write it to a new config file that is called by dockerd. I originally started this way in the spirit of thus taking automatically any change to this default conf file that may come in a future docker package update.

The risk, however, is to also carry a change in this config that may be incompatible with the options we need to set. An example of that actually already happened with the default config file having log-opts that are incompatible with the journald driver we use for logging. With these options, the Rock-on service fails to start, for instance, and I thus have to replace them.

In this context, I see two options:

1. keep the current method and take the default config file as template and update it with our own options. This has the advantages of keeping up-to-date with this default config file automatically every time the Rock-on service is started. This has the inconvenient, however, of potentially introducing a breaking change automatically.
2. Simply start from scratch and only write the options we need. This has the advantage of ensuring reliability, but the inconvenient of having to manually update this file if we want to keep up-to-date with the package's default conf file.

The more I think about it, the more I prefer option 2, as I see reliability has having priority over being up-to-date. Also, I would be surprised if this default config file changes that often (if at all). Moreover, as mention above, we can even implement the possibility for users to add their own custom settings in the future if we'd like.

Thanks in advance for your feedback!

[phillxnet]

@FroggyFlox I favour number 1. We update the current config file. Post initial install any package updates should leave it be anyway is my understanding. Although a package update may end up dumping another ineffectual file their for reference but I don't think it will change our config, once established.

This has the inconvenient, however, of potentially introducing a breaking change automatically.

Given the above assumption of mine that a package update won't change this file if it has already been changed, do we not avoid a potential package breakage by this means?

2. Simply start from scratch and only write the options we need. This has the advantage of ensuring reliability, but the inconvenient of having to manually update this file if we want to keep up-to-date with the package's default conf file.

I'd shy away from this one myself as once we've changed this file I'm assuming the package manager will not change it in an update.

The more I think about it, the more I prefer option 2

I'm currently favouring 1, but with the caveat that once we have made our changed the file is ours anyway. And if we want to support custom changes by users then with 2 will overwrite those changes: where changes are made directly to the file that our not ones we inline edit.

So rather unusually we have different selections. Could it be that once we establish that a package update will leave our edited json unchanged we may be in agreement. As you say this file may not change a howl lot anyway and is unlikely to receive critical functional updates.

A point on 1:

1. This has the advantages of keeping up-to-date with this default config file automatically every time the Rock-on service is started.
   I don't see that this would be the case if the file, once altered, will not be overwritten by the package manger.

Your call on this as you are doing it and most likely more up on this but I think we need to establish if this file, once updated by us, remains as we left it. I.e. akin to package files installed in /etc that will not be overwritten by a package manager (normally) but will be written if the file don't exist. Such as @Hooverdan96 reported in #2032. This may help to clarify our choices.

I see reliability has having priority over being up-to-date.

Yes, unless docker arbitrarily decides that a missing config element (or repeat as we've just had) is a breaking change (given precedence was apparently impossible). If we edit what we find a fix is to copy over the backup created by the package (hopefully) and cycle our Rock-ons service. It will then edit what we need to an leave all other options as is, including custom ones edited by other programs / users.

Sorry that's not all that clear but I think you are assuming this file will be updated by the package manger post our changes and I am assuming otherwise. So if we first clear that up then we know for sure and can take it form their. Plus the nuke and pave approach on config rather than edit has left us previously with really old config arrangements and an inline edit such as in 1. would not do that as at least each time we roll an iso we edit what we find.

Let double check that files treatment over an upgrade post modification and take it from their. Personally I think it 'should' be left unaltered and the update should dump a copy of what it would have put their but not in play. This then means docker updates don't break existing config such as all other updates generally. However I also think that precedence should existing in options and warning not breakage should result if their are mulltiple competing options. This was not what we found so lets get more info on how this file is treated and take it form their.

Hope thats of use and when we have more info we can take it from their. But again, your call and this will be in testing channel first (assuming we are still openSUSE only on this one). But nice to do right first time if we can. Plus we may loose either way :)

[FroggyFlox]

Thanks a lot for your feedback, @phillxnet!

Your call on this as you are doing it and most likely more up on this but I think we need to establish if this file, once updated by us, remains as we left it. I.e. akin to package files installed in /etc that will not be overwritten by a package manager (normally) but will be written if the file don't exist. Such as @Hooverdan96 reported in #2032. This may help to clarify our choices.

Very good idea, indeed... I'll try to figure that out by testing after a docker-ce update.

The "nice" thing is that dockerd doesn't seem to care about the existence of the file as it's only there to optionally customize dockerd configuration. If no value for a given option is specified in daemon.json, the dockerd default will then apply.

Given the above assumption of mine that a package update won't change this file if it has already been changed, do we not avoid a potential package breakage by this means?

Correct.

2. Simply start from scratch and only write the options we need. This has the advantage of ensuring reliability, but the inconvenient of having to manually update this file if we want to keep up-to-date with the package's default conf file.

I'd shy away from this one myself as once we've changed this file I'm assuming the package manager will not change it in an update.

I should have mentioned that in my current PR preparation, I'm using a distinct json file for configuration and propose to write it in <settings.CONFROOT>/docker-daemon.json and telling dockerd to use this file for configuration instead of the default /etc/docker/daemon.json. As a result, we keep the default in its default location and we only have to decide whether or not we want to take default customizations from it (it's not even the dockerd defaults). /etc/docker/daemon.json thus remain unchanged.
https://github.com/FroggyFlox/rockstor-core/blob/e3721566559202a839093894e4ae75ac0d652952/src/rockstor/smart_manager/views/docker_service.py#L152-L154

So rather unusually we have different selections. Could it be that once we establish that a package update will leave our edited json unchanged we may be in agreement. As you say this file may not change a howl lot anyway and is unlikely to receive critical functional updates.

It may also be that I failed to give you all the information, so we're not making our reasoning from the same resources. This is also why I agree with you on the fact that I need to figure out what a package update will or will not do to this default json file. It will for sure leave our edited json alone as it has no way to even be aware of its existence (or so I believe, at least). I personally highly doubt that this file will receive critical updates as it isn't its purpose (well... it shouldn't), Its purpose is only to customize and not define dockerd, As mentioned above, if a setting is not defined in this json file, the "hardcoded" dockerd default will apply.

If we edit what we find a fix is to copy over the backup created by the package (hopefully) and cycle our Rock-ons service. It will then edit what we need to an leave all other options as is, including custom ones edited by other programs / users.

Agreed... and that's actually pretty much what I coded so far:
1a. if /etc/docker/daemon.json exists, import it, stream edit it with our custom options, and write a new file in <settings.CONFROOT>/docker-daemon.json.
1b. if /etc/docker/daemon.json does not exist, start from an empty dict, edit it with our custom otpions, and write a new file in <settings.CONFROOT>/docker-daemon.json.
2. Change docker.service to point to this file with --config-file <settings.CONFROOT>/docker-daemon.json.

This way, /etc/docker/daemon.json always remains intact and available to be used as template when we start the Rock-on service.

I think you are assuming this file will be updated by the package manger post our changes

Yes, I am... in an effort to think of a worst-case scenario but that probably leads me to give too much importance to something that may very well never happen. Like you said, then, I'd better figure out exactly what a docker-ce update will do to this file (if any).

Thanks again for spending some of your time on this, that is very helpful!

[FroggyFlox]

I had some time to play around with docker update via zypper and observe the fate of /etc/docker/daemon.json and things look fine as it doesn't touch the file. It only copies the default /etc/docker/daemon.json there if the file is absent.

In Leap 15.1, using zypper install --oldpackage docker-18.09.6_ce-lp151.2.9.1 and then updating with zypper update docker, I could test several modifications to /etc/docker/daemon.json and see whether or not they would be affected by the package update. I could thus tried to modify /etc/docker/daemon.json in the following ways:

1. change the value of one key
2. remove a key entirely
3. remove the file altogether

In 1 and 2, the changes were kept after the docker package install. In 3, however, I observed the same as @Hooverdan96 and /etc/docker/daemon.json was restored.

It thus appears that the file won't be updated after package install no matter what we do to it, as long as we don't rename/remove it.