Mounting a dataset as read-only "alters" it and incremental snapshot receiving made impossible

[sjau]

System information

Type	Version/Name
Distribution Name	Nixos
Distribution Version	Unstable
Linux Kernel	4.14.14
Architecture	Linux
ZFS Version	1d53657
SPL Version	23602fd

Describe the problem you're observing

I have the following scenario:

In a small office with only a handful of people there's a need (quite regularly) to get old versions of some database (ms access) files. So this far I've been using rsync + hardlinking to create daily versions (if altered). Those rsynced data is stored on a seperate drive on the server and there's also 2 offsite backups as well. However, I'm still fairly new to ZFS (but use it a lot now), I want to change that to snapshots and incremental sending.

So I've been testing how to best set this up. My thoughts was to just mount this "backup" dataset as read-only. No write access needed. Since I mount it as read-only my thought of train was, that it can still receive incremental snapshots. However that seems not to be the case as it keeps telling me it has been altered.

So what I did then is mount that dataset as read-only, roll it back and try again an incremental send. This seems to work and new snapshots actually appear in the .zfs folder.

For me, mounting as read-only shouldn't alter the dataset so I wonder why there's such a behaviour? Also having it mounted, rolling back the snapshot wil being mounted as ro and then being able to receive new snapshots is somehwat a mystery to me as why it works (I'm glad it does work).

So is there bug that the dataset "advances" even when mounted as read-only?

Describe how to reproduce the problem

root@subi:~# ssh root@10.200.0.2 "zfs send tankJL/Nixos@zfs-auto-snap_hourly-2018-02-05-12h00" | zfs receive tankSubi/encZFS/TEST
root@subi:~# mkdir /tmp/TEST
root@subi:~# mount -o ro -t zfs tankSubi/encZFS/TEST /tmp/TEST/
root@subi:~# cd /tmp/TEST
root@subi:/tmp/TEST# cd .zfs
root@subi:/tmp/TEST/.zfs# cd snapshot
root@subi:/tmp/TEST/.zfs/snapshot# ls
zfs-auto-snap_hourly-2018-02-05-12h00
root@subi:/tmp/TEST/.zfs/snapshot# ssh root@10.200.0.2 "zfs send -i tankJL/Nixos@zfs-auto-snap_hourly-2018-02-05-12h00 tankJL/Nixos@zfs-auto-snap_hourly-2018-02-05-13h00" | zfs receive tankSubi/encZFS/TEST
cannot receive incremental stream: destination tankSubi/encZFS/TEST has been modified since most recent snapshot
root@subi:/tmp/TEST/.zfs/snapshot# zfs rollback tankSubi/encZFS/TEST@zfs-auto-snap_hourly-2018-02-05-12h00
root@subi:/tmp/TEST/.zfs/snapshot# ssh root@10.200.0.2 "zfs send -i tankJL/Nixos@zfs-auto-snap_hourly-2018-02-05-12h00 tankJL/Nixos@zfs-auto-snap_hourly-2018-02-05-13h00" | zfs receive tankSubi/encZFS/TEST
root@subi:/tmp/TEST/.zfs/snapshot# ls
zfs-auto-snap_hourly-2018-02-05-12h00  zfs-auto-snap_hourly-2018-02-05-13h00
root@subi:/tmp/TEST/.zfs/snapshot# ssh root@10.200.0.2 "zfs send -i tankJL/Nixos@zfs-auto-snap_hourly-2018-02-05-13h00 tankJL/Nixos@zfs-auto-snap_hourly-2018-02-05-14h00" | zfs receive tankSubi/encZFS/TEST
root@subi:/tmp/TEST/.zfs/snapshot# ls
zfs-auto-snap_hourly-2018-02-05-12h00  zfs-auto-snap_hourly-2018-02-05-13h00  zfs-auto-snap_hourly-2018-02-05-14h00
root@subi:/tmp/TEST/.zfs/snapshot# 

So is this intended behaviour that mounting a dataset as read-only will alter/advance it so that a rollback is required before being able to receive an incremental snapshot?
Is it also intended behaviour that once it's mounted as read-only and rolled backup, that new snapshots will just show up in the .zfs folder without anew rollback?

[richardelling]

best practice is to keep the snapshots unmounted and use clones to access the data.

[loli10K]

However that seems not to be the case as it keeps telling me it has been altered.

@sjau i may be missing something here, but i'm not finding this information in the output you pasted in the issue description.

I'm only seeing

WARNING: could not send tankJL/Nixos@zfs-auto-snap_hourly-2018-02-05-12h00:
incremental source (tankJL/Nixos@zfs-auto-snap_hourly-2018-02-05-12h00) is not earlier than it

which is a zfs send error, so not related to receiving "altered" datasets, and

cannot receive: failed to read from stream

which is only because the input stream being received is empty (due the first error).

[sjau]

Sorry, deleted the wrong lines... here again fresh:

root@subi:~# zfs list
NAME                    USED  AVAIL  REFER  MOUNTPOINT
tankSubi                790G   109G   120K  none
tankSubi/encZFS         790G   109G   184K  legacy
tankSubi/encZFS/Media   574G   109G   566G  legacy
tankSubi/encZFS/Nixos   114G   109G  95.9G  legacy
tankSubi/encZFS/VMs     102G   109G  99.2G  legacy
root@subi:~# rmdir /tmp/TEST
rmdir: failed to remove '/tmp/TEST': No such file or directory
root@subi:~# ssh root@10.200.0.2 "zfs send tankJL/Nixos@zfs-auto-snap_hourly-2018-02-08-12h00" | zfs receive tankSubi/encZFS/TEST
root@subi:~# mkdir /tmp/TEST
root@subi:~# mount -o ro -t zfs tankSubi/encZFS/TEST /tmp/TEST/
root@subi:~# cd /tmp/TEST
root@subi:/tmp/TEST# cd .zfs
root@subi:/tmp/TEST/.zfs# cd snapshot
root@subi:/tmp/TEST/.zfs/snapshot# ls
zfs-auto-snap_hourly-2018-02-08-12h00
root@subi:/tmp/TEST/.zfs/snapshot# ssh root@10.200.0.2 "zfs send -i tankJL/Nixos@zfs-auto-snap_hourly-2018-02-08-12h00 tankJL/Nixos@zfs-auto-snap_hourly-2018-02-08-13h00" | zfs receive tankSubi/encZFS/TEST
cannot receive incremental stream: destination tankSubi/encZFS/TEST has been modified
since most recent snapshot
root@subi:/tmp/TEST/.zfs/snapshot# zfs rollback tankSubi/encZFS/TEST@zfs-auto-snap_hourly-2018-02-08-12h00
root@subi:/tmp/TEST/.zfs/snapshot# ssh root@10.200.0.2 "zfs send -i tankJL/Nixos@zfs-auto-snap_hourly-2018-02-08-12h00 tankJL/Nixos@zfs-auto-snap_hourly-2018-02-08-13h00" | zfs receive tankSubi/encZFS/TEST
root@subi:/tmp/TEST/.zfs/snapshot# ls
zfs-auto-snap_hourly-2018-02-08-12h00  zfs-auto-snap_hourly-2018-02-08-13h00
root@subi:/tmp/TEST/.zfs/snapshot# ssh root@10.200.0.2 "zfs send -i tankJL/Nixos@zfs-auto-snap_hourly-2018-02-08-13h00 tankJL/Nixos@zfs-auto-snap_hourly-2018-02-08-14h00" | zfs receive tankSubi/encZFS/TEST
root@subi:/tmp/TEST/.zfs/snapshot# ls
zfs-auto-snap_hourly-2018-02-08-12h00  zfs-auto-snap_hourly-2018-02-08-13h00  zfs-auto-snap_hourly-2018-02-08-14h00

Still the problem exists. Feel free to try it yourself.

[loli10K]

Still the problem exists. Feel free to try it yourself.

I'm not questioning the fact this can be reproduced, i was actually trying to, but the issue description was very misleading.

[pcd1193182]

I think this bug used to exist in our operating system, but I think we fixed it at some point. Let me look around and see if I can find a record of it.

[pcd1193182]

Ok, I tried reproducing this on my ZoL box, and I haven't been able to do so. I also can't find any history of issues related to this in our OS; I may have been imagining it, or I may have not been looking hard enough. I would suggest trying to gather more debugging information; if you can get it into the state where you can't receive an incremental, try zfs snapshot tankSubi/encZFS/TEST@test and then use zfs diff to compare that snapshot with the previous one. If it shows files that have been modified, that might help us track down what has been going on.

EDIT: Using zfs diff itself may run afoul of #6335. If that doesn't produce useful results, you can also try doing an incremental send instead, and piping the output to zstreamdump -v.

[loli10K]

I think this bug used to exist in our operating system, but I think we fixed it at some point.

I tried reproducing this issue with commits prior to 6a21856 (OpenZFS 8930 - zfs_zinactive: do not remove the node if the filesystem is readonly) without success.

[loli10K]

Duplicate of #6741

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.