docker breaks on latest zfsonlinux

[wphilips]

I run fedora22 on top of a zfsonlinux root file system. After updating zfsonlinux
to version 0.6.5 docker-1.8.2-1 (and earlier versions) fail. The details of the failure
are listed below. if I move back to older zfs/spl modules, while keeping everything else
exactly the same (including the /usr/sbin/zfs command and the zfs libs), docker works as expected.

My initial hunch was that the cause might some changes in the output of some zfs commands.
I therefore checked the output of zfs list and zfs get for the two cases (good and bad) and
I do see some differences in zfs list, but these are only in the reported size of filesystems.
So I assume mny hunch was false.

///////////////////////////
Last known good versions (test below succeeds)
SPL: Loaded module v0.6.4.1-1
ZFS: Loaded module v0.6.4.2-1,

Bad versions (test below fails with the listed error message):
SPL: 0.6.5
ZFS: 0.6.5
docker-engine-1.8.2-1.fc22.x86_64
Linux 4.1.2-200.fc22.x86_64 #1 SMP Wed Jul 15 20:12:12 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
//////////////////////////////////////////////////////////////
docker daemon is started as follows
sudo docker daemon -D --bip=10.66.33.10/24

dockter test command line is:
sudo docker run -ti fedora /bin/bash
Error response from daemon: error creating zfs mount of zfs/docker.v1/6259375370e56d5d450cef0b4b1cdbb6d8859c3f95e02db93eeee66ad315a911-init to /var/lib/docker/zfs/graph/6259375370e56d5d450cef0b4b1cdbb6d8859c3f95e02db93eeee66ad315a911-init: invalid argument
$

[wphilips]

docker version
Client:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Thu Sep 10 19:20:17 UTC 2015
OS/Arch: linux/amd64
Get http:///var/run/docker.sock/v1.20/version: dial unix /var/run/docker.sock: no such file or directory.

• Are you trying to connect to a TLS-enabled daemon without TLS?
• Is your docker daemon up and running?

docker info
Containers: 10
Images: 9
Storage Driver: zfs
Zpool: zfs
Zpool Health: ONLINE
Parent Dataset: zfs/docker.v1
Space Used By Parent: 1146806870
Space Available: 918566527362
Parent Quota: no
Compression: gzip
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.6-201.fc22.x86_64
Operating System: Fedora 22 (Twenty Two)
CPUs: 4
Total Memory: 31.31 GiB
Name: eltodo.telin
ID: NIUR:HKEV:CO6Q:QUDV:7GWD:AASJ:PL62:4G3A:S3D5:LZOA:J2BC:YQ67

#uname -a
Linux eltodo.telin 4.1.6-201.fc22.x86_64 #1 SMP Fri Sep 4 17:49:24 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

List the steps to reproduce the issue:
See above

[bjin]

I have exactly the same issue, zfs 0.6.5, docker 1.8.2/1.8.1 and kernel 4.1.6

[bjin]

cc the zfs graphdriver maintainers: @Mic92 @baloo

[bjin]

Most likely a bug on zfsonlinux side, or at least a bug in syscall Go package.

[Mic92]

I would say, because it is hard to change the syscall package and we have to deal with the current zfs release, the easiest way would be to do this mount in plain C.
UPDATE using zfs mount might be easier but slower. For the C version at least 2 variants for freebsd and linux is needed.

UPDATE zfs mount is not an option, because selinux labels have to passed with the mount.

[bjin]

I think we can close this issue now. zfs 0.6.5.1 is released with a fix to this bug. And I don't think anyone will stick with zfs 0.6.5 since the 0.6.5.1 release also fixes a major bug which might lead to data loss on zvol.

[Mic92]

Upstream was faster then me, I suppose. Most downstream sources should have this package by now anyway.

[calavera]

Thanks @bjin. I'm closing this issue 🎉

Please, feel free to reopen or comment here if the problem persists.