Receiving zvol used as VM disk results in permanently busy dataset

[chrsigg]

The following steps create a dataset that is permanently busy (i.e. across reboots), even though nothing (that I know of) is accessing it.

Create two pools:

dd if=/dev/zero of=/root/source.pool bs=1G count=5
dd if=/dev/zero of=/root/target.pool bs=1G count=5
zpool create source /root/source.pool
zpool create target /root/target.pool

Create a sparse ZVOL and use it as the disk for a freshly minted VM:

zfs create -s -o compression=lz4 -V 4G source/vmvol

virt-install --name vmvol --memory 2048 --location=/var/lib/libvirt/images/CentOS-7-x86_64-Minimal-1503-01.iso --os-variant centos7.0 --disk /dev/zvol/source/vmvol --network network:vmnet --initrd-inject=/var/lib/libvirt/images/ks_testvm.cfg --extra-args='console=tty0 console=ttyS0,115200n8 ks=file:/ks_testvm.cfg'

Create a recursive snapshot of source and send it to target:

zfs snapshot -r source@snap1
zfs send -Rv source@snap1 | zfs receive -Fv target

At this point, I can still successfully export target. However, after I re-import this pool (or don't export it in the first place) and reboot the system, this is no longer possible:

[root@vmhost ~]# zpool export target
cannot export 'target': pool is busy

[root@vmhost ~]# zfs destroy -r target/vmvol
cannot destroy 'target/vmvol': dataset is busy

fuser -am /dev/zvol/target/vmvol* reports no processes. Rebooting again has no effect, the target pool is still busy.

However, if I export the source pool and reboot the system, then I can successfully export the target pool. If I re-import both pools and do a reboot, I am back to a busy target pool.

This is on CentOS 7 with kernel 3.10.0-229.4.2.el7.x86_64 and SPL/ZFS version 0.6.4.1.

I don't know any ZFS internals. Let me know what diagnostic information I can provide, or if I can further simplify the test case.

[chrsigg]

I retested with kernel 3.10.0-229.11.1.el7.x86_64 and ZFS version 0.6.4.2 and the problem persists.

It's also not necessary to do a reboot. Shutting down the VM after receiving the snapshot is enough to produce a busy target pool:

[root@vmhost ~]# zfs send -Rv source@snap1 | zfs receive -Fv target
[root@vmhost ~]# virsh shutdown vmvol

[root@vmhost ~]# zpool export target
cannot export 'target': pool is busy

Likely the same issue was already posted to zfs-discuss back in March.

[chrsigg]

Thanks for looking into it.

This issue breaks my incremental backups. If anyone can suggest a work-around, I would be grateful.

[kernelOfTruth]

referencing:

https://forums.freenas.org/index.php?threads/zvol-replication-dataset-is-busy.20074/
#1761 Zfs incremental recv fails with dataset is busy error

(both slightly related)

[mrqwer88]

Hello!

@chrsigg - please check lvm.
I have problem, that looks same on centos 7 in libvirt with zfs storage.
I cannot delete zvol with error dataset is busy.
On my node it was busy becouse lvm start use it from node -

 pvdisplay 
  "/dev/zd0p2" is a new physical volume of "19,51 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/zd0p2
  VG Name               
  PV Size               19,51 GiB
  Allocatable           NO
  PE Size               0   
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               6kMmYk-jHrG-Y2ME-RvOL-khHe-J6DM-eI144v

If delete it via pvremove zvol destroy succsessful.

I try to fix it like this https://pve.proxmox.com/wiki/Storage:_ZFS#How_to_prevent_lvm_of_scanning_zvols
but it not help

[root@kvm0 ~]# pvscan 
  WARNING: Device for PV k2Gr2X-bqrH-Xm8a-pBMX-laJo-b6KP-lVHBJ3 not found or rejected by a filter.
  WARNING: Device for PV k2Gr2X-bqrH-Xm8a-pBMX-laJo-b6KP-lVHBJ3 not found or rejected by a filter.
  No matching physical volumes found
[root@kvm0 ~]# vgscan 
  Reading all physical volumes.  This may take a while...
  WARNING: Device for PV k2Gr2X-bqrH-Xm8a-pBMX-laJo-b6KP-lVHBJ3 not found or rejected by a filter.
  Found volume group "VolGroup" using metadata type lvm2
[root@kvm0 ~]# pvdisplay 
  WARNING: Device for PV k2Gr2X-bqrH-Xm8a-pBMX-laJo-b6KP-lVHBJ3 not found or rejected by a filter.
  WARNING: Device for PV k2Gr2X-bqrH-Xm8a-pBMX-laJo-b6KP-lVHBJ3 not found or rejected by a filter.
  --- Physical volume ---
  PV Name               unknown device
  VG Name               VolGroup
  PV Size               19,51 GiB / not usable 3,00 MiB
  Allocatable           yes (but full)
  PE Size               4,00 MiB
  Total PE              4994
  Free PE               0
  Allocated PE          4994
  PV UUID               k2Gr2X-bqrH-Xm8a-pBMX-laJo-b6KP-lVHBJ3

But it not looks like zfs bug, imho.

UPD:
How temporary fix I in /etc/lvm/lvm.conf set incorrect scan param -

- scan = [ "/dev" ]
+ scan = [ "/etc/lvm/" ]

• I not need lvm in fact on node.
  Maybe filter not full, as variant in filter can add /dev/disk/*.