Docker disables btrfs quotas creating an unrecoverable broken btrfs volume

[hutzl]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

When using docker on a btrfs volume where the quotas are enabled and certain limits on certain qgroups are set those should be retained and still be there after a reboot of the system.

Actual behavior

When using docker on a btrfs volume where quotas are enabled and certain limits on certain qgroups are set the btrfs volume gets corrupted in a way that seems unrecoverable as soon as you start the first container. This behavior does not show immediately but after a reboot the quotas are disabled and the qgroup limits are gone. I found no other way to recover from that state except to reformat the btrfs volume.

Steps to reproduce the behavior

Output of docker version:

Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   3dfb8343
 Built:        Wed Jul 26 18:03:33 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87617
 Built:        Wed Jul 26 20:03:39 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 17.06.0-ce
Storage Driver: btrfs
 Build Version: Btrfs v4.11.1
 Library Version: 102
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.12.4-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.686GiB
Name: peregrin
ID: WGK4:K2IQ:GTX5:JMQQ:45EB:T3MU:OA62:TIUD:VIMX:VV3P:OISY:IRC6
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Steps to reproduce (run all commands as root for simplicity):

1. Install Linux using btfs root (I use arch linux)
2. enable btrfs quotas

btrfs quota enable /

3. Create a btrfs subvolume

btrfs subvolume create /sub1

4. check quota

btrfs qgroup show -f -re /sub1 
qgroupid         rfer         excl     max_rfer     max_excl 
--------         ----         ----     --------     -------- 
0/265        16.00KiB     16.00KiB         none         none

5. set a limit for the new subvolume

btrfs qgroup limit 10M 0/265 /

6. check quota

btrfs qgroup show -f -re /sub1 
qgroupid         rfer         excl     max_rfer     max_excl 
--------         ----         ----     --------     -------- 
0/265        16.00KiB     16.00KiB     10.00MiB         none 

=> You see the limit is set!

7. install docker
8. run any docker container (e.g busybox)

docker run -ti busybox

9. check quota

btrfs qgroup show -f -re /sub1 
qgroupid         rfer         excl     max_rfer     max_excl 
--------         ----         ----     --------     -------- 
0/265        16.00KiB     16.00KiB     10.00MiB         none 

=> All seems still fine up to here.
10. reboot the system
11. check quota

btrfs qgroup show -f -re /sub1
ERROR: can't list qgroups: quotas not enabled

=> Quota support is gone.
12. re-enable quota to see what it does

btrfs quota enable /

13. check quotas

btrfs qgroup show -f -re /sub1
qgroupid         rfer         excl     max_rfer     max_excl 
--------         ----         ----     --------     -------- 
0/265        16.00KiB     16.00KiB         none         none

=> The previously set quota is gone!

[hutzl]

I have already tracked this issue down to some code which I believe is at least contributing to the problem:

file: docker-ce/components/engine/daemon/graphdriver/btrfs/btrfs.go

This procedure:

func (d *Driver) subvolDisableQuota() error {
	d.updateQuotaStatus()

	if !d.quotaEnabled {
		return nil
	}

	dir, err := openDir(d.home)
	if err != nil {
		return err
	}
	defer closeDir(dir)

	var args C.struct_btrfs_ioctl_quota_ctl_args
	args.cmd = C.BTRFS_QUOTA_CTL_DISABLE
        // this seems to be the problematic line (comment by hutzl):
	_, _, errno := unix.Syscall(unix.SYS_IOCTL, getDirFd(dir), C.BTRFS_IOC_QUOTA_CTL,
		uintptr(unsafe.Pointer(&args)))
	if errno != 0 {
		return fmt.Errorf("Failed to disable btrfs quota for %s: %v", dir, errno.Error())
	}

	d.quotaEnabled = false

	return nil
}

It looks like the btrfs storage backend tries to enable/disable the quotas on a subvolume base. But with all I have read about btrfs I don't think that is at all possible. Sure you can set/delete the quotas on a subvolume base, but the quota support itself (enabling/disabling qgroups) seems to be a setting that applies to the whole btrfs volume. I even tried to uncomment the line I marked above and the resulting dockerd is not creating the problem any more. I don't say that this is a valid fix for the problem but at least it proves that the problem might be located there. By the way, the procedure subvolEnableQuota() has the same issue.

The root cause is probably a combination of docker trying to do something it should not do and btrfs causing some havoc as a result of that.

[aki-k]

This just happened to me and my btrfs quotas are gone:

# btrfs qgroup show -pcre /btrfs_root/vdb1
ERROR: can't list qgroups: quotas not enabled

My data is still there, which is nice. This happened after removing two docker volumes resulting in 0 docker volumes left and a VM reboot. I was just saying today to my friend that I've had 0 problems with btrfs and these VMs.

Operating system: CentOS 7.7.1908 x86_64
Kernel: 5.4.15-1.el7.elrepo.x86_64
Docker: 19.03.6

Storage Driver: btrfs
Build Version: Btrfs v4.9.1
Library Version: 102

[plevart]

I think I have been beaten by the same problem. I use the same BTRFS filesystem for docker graph driver and for other things (data files of a DB). I have been playing with brtfs quota enable / disable and quota groups for this other thing not realizing that docker also does something with quotas. Suddenly (perhaps accidentaly at the same time as stoping and removing a docker container) I experienced some BTRFS errors and the filesystem entered read-only mode (the docker stop command reported error not being able to remove a subvolume due to read-only filesystem). I unmounted the silesystem did a btrfs check which found inconsistencies in free block cache and in quota groups. I repeated the check with --repair option to try to repair the inconsistencies. All but quota groups managed to get repaired but btrfs check dumped core (an assert failed) when checking the quota groups. I briefly mounted the filesystem just to disable quotas and unmounted it again. After that, btrfs check skipped checking quota groups and the filesystem was fully checked without errors (including CRC check on data blocks). So I'm now using it with quotas disabled.
Perhaps I need two independent filesystems: one for docker and the other for custom data.

[smlx]

Perhaps I need two independent filesystems: one for docker and the other for custom data.

yep, just mount it on /var/lib/docker. I've been using this for a while now with no issues.