zfs send -p | zfs receive -F destroys all other snapshots

[tobia]

From the man page of zfs receive:

When a snapshot replication package stream that is generated by using the zfs send -R command is received, any snapshots that do not exist on the sending location are destroyed by using the zfs destroy -d command.

I never used -R, but I did add -p to the send command (zfs send -p -I ...) because I wanted to transfer the properties along with the snapshots. But then the receive side (with -F) behaved as if I was sending a replication stream: it deleted all snapshots that didn't exist on the sending side.

This was not expected at all. Is this a bug in send, receive, or in the documentation?

Edit: if this is the expected behavior, I'll submit a PR to make it clearer in the man pages.

[tobia]

Sure, but at this point I don't know yet if this is the expected behavior (then I can submit a PR for the man pages) or a bug, in which case there's probably not much I could do.

[loli10K]

@kpande -R implies -p: this doesn't mean it is necessarily the other way around.

[loli10K]

@kpande i was referring to the zfs send part of the man page you were quoting:

This flag is implicit when -R is specified

It's not the other way around, -p doesn't imply -R, so when you zfs receive -F a stream sent with -p it may not qualify as an "incremental replication stream".

Anyway, other ZFS implementation seems to behave in the same way.

[tobia]

@kpande for me -p should not have implied the -R behavior either.

it would be nice to break down -F into various options to roll back first snapshot only, or remove nonexistant snapshots, etc

Yes. On the receiving side I use -F to roll back local changes, otherwise any local change (including atime) will prevent it from receiving the next incremental stream (-I)

I certainly didn't expect that adding -p on the sending side would make -F destroy all past snapshots!

[tobia]

Silly me. I just set readonly=on (and atime=off for good measure) on the entire destination pool, since it's a backup server. This allowed me to remove -F from zfs recv, which should prevent further disappearance of snapshots.

Still, if you are of the opinion that what I reported above is the expected behavior (that is, -p generates a replication stream even without -R) then I'll go ahead and prepare a PR on the man pages.

[GregorKopka]

According to the name, and the documentation, it is ment to attach the properties of the dataset to the snapshot data, so after receiving the target will have the properties set identical to the source. Apart from that the behaviour shouldn't be different than the same operation without the -p on the sending side.

Having zfs send -p destroying unrelated stuff on the receiving side when received with -F (for the rollback to the last snapshot on the target) isn't expected, at least by me.

What is expected is that -F performs the equivalent of zfs rollback dataset@(incremental start snaphot) on the target and then to receive the stream. So should the issue be that snapshots after the common incremental start point were discarded on the target then it works as expected (it needs to rollback to the common point to be able to accept the data, since it can't rewrite existing stuff). This wouldn't be related to -p or not to -p.

But there is IMHO no sane reason why it should modify the snapshot chain on the target prior to the common snapshot used as the start of the incremental (the one specified with -i/-I). I would see this as a bug. Same should it destroy child datasets on the target that do not exist at the source, since -p dosn't (and shouldn't) imply that the operation is performed on children. Unless -R is specified the send/recv should be limited to the one dataset specified, nothing else.

@tobia could you please post the exact commands executed, the list of snapshots of the dataset in question (source side) and more details about what was destroyed on the target? You should be able to get the list of the destroyed snapshots in zpool history of the target pool.

[tobia]

@GregorKopka My source systems take snapshots of all their zfs filesystems every 15 minutes, with zfs snapshot $fs@$now, where $now is a string with the current date and time, such as 2016-11-04_21:30.

After that, they send them over to the backup server, with zfs send -I $last $fs@$now | ssh $server "zfs receive -F $fs", where $last is the name of the last snapshot that was successfully sent to the server. My purpose for using -F was to rollback any changes made on the destination after $last was received. After every successful send, my script would remove from the source all snapshots up to $now (excluded) and save the value of $now to be used as $last in the next execution.

This worked fine, but then I thought about transferring the properties along with the snapshots, so I added -p to the send command and tested it. The result was that every old snapshot was deleted from the target system, where only the snapshots between $last and $now would remain.

For this reason I quickly decided against using -p. After @kpande's advice I set readonly=on and atime=off on the target system, which allowed me to remove -F from the receive command. (Because at this point I think not having -F is safer in my use case, which is incremental backups.)

I looked at zpool history, but it only shows the zfs receive -F commands with no additional info.

[GregorKopka]

This looks like a bug to me, there should be no reason for recv -F (that is getting an incremental) to destroy snapshots prior to the common one (specified by -I on the source).

Reproducer:

DATASET="tank/ZOL-5341"
zfs create $DATASET/src -p
zfs snapshot $DATASET/src@a
zfs snapshot $DATASET/src@b
zfs snapshot $DATASET/src@c
zfs snapshot $DATASET/src@d
zfs list $DATASET -r -tall
zfs send $DATASET/src@a | zfs recv -F $DATASET/dst
zfs list $DATASET -r -tall
zfs send -I a $DATASET/src@b | zfs recv -F $DATASET/dst
zfs list $DATASET -r -tall
zfs destroy $DATASET/src@a
zfs send -I b $DATASET/src@c | zfs recv -F $DATASET/dst
zfs list $DATASET -r -tall
# $DATASET/dst@a is still there
zfs destroy $DATASET/src@b
zfs send -p -I c $DATASET/src@d | zfs recv -F $DATASET/dst
zfs list $DATASET -r -tall
# $DATASET/dst@a and @b are gone

# zfs destroy $DATASET -r # uncomment line for cleanup

Output on my system:

~ $ DATASET="tank/ZOL-5341"
~ $ zfs destroy tank/send-p -r
~ $ zfs create $DATASET
~ $ zfs create $DATASET/src
~ $ zfs snapshot $DATASET/src@a
~ $ zfs snapshot $DATASET/src@b
~ $ zfs snapshot $DATASET/src@c
~ $ zfs snapshot $DATASET/src@d
~ $ zfs list $DATASET -r -tall
NAME                  USED  AVAIL  REFER  MOUNTPOINT
tank/ZOL-5341         447K   395G   230K  /tank/ZOL-5341
tank/ZOL-5341/src     217K   395G   217K  /tank/ZOL-5341/src
tank/ZOL-5341/src@a      0      -   217K  -
tank/ZOL-5341/src@b      0      -   217K  -
tank/ZOL-5341/src@c      0      -   217K  -
tank/ZOL-5341/src@d      0      -   217K  -
~ $ zfs send $DATASET/src@a | zfs recv -F $DATASET/dst
~ $ zfs list $DATASET -r -tall
NAME                  USED  AVAIL  REFER  MOUNTPOINT
tank/ZOL-5341         665K   395G   230K  /tank/ZOL-5341
tank/ZOL-5341/dst     217K   395G   217K  /tank/ZOL-5341/dst
tank/ZOL-5341/dst@a      0      -   217K  -
tank/ZOL-5341/src     217K   395G   217K  /tank/ZOL-5341/src
tank/ZOL-5341/src@a      0      -   217K  -
tank/ZOL-5341/src@b      0      -   217K  -
tank/ZOL-5341/src@c      0      -   217K  -
tank/ZOL-5341/src@d      0      -   217K  -
~ $ zfs send -I a $DATASET/src@b | zfs recv -F $DATASET/dst
~ $ zfs list $DATASET -r -tall
NAME                  USED  AVAIL  REFER  MOUNTPOINT
tank/ZOL-5341         690K   395G   243K  /tank/ZOL-5341
tank/ZOL-5341/dst     230K   395G   217K  /tank/ZOL-5341/dst
tank/ZOL-5341/dst@a  12,8K      -   217K  -
tank/ZOL-5341/dst@b      0      -   217K  -
tank/ZOL-5341/src     217K   395G   217K  /tank/ZOL-5341/src
tank/ZOL-5341/src@a      0      -   217K  -
tank/ZOL-5341/src@b      0      -   217K  -
tank/ZOL-5341/src@c      0      -   217K  -
tank/ZOL-5341/src@d      0      -   217K  -
~ $ zfs destroy $DATASET/src@a
~ $ zfs send -I b $DATASET/src@c | zfs recv -F $DATASET/dst
~ $ zfs list $DATASET -r -tall
NAME                  USED  AVAIL  REFER  MOUNTPOINT
tank/ZOL-5341         703K   395G   243K  /tank/ZOL-5341
tank/ZOL-5341/dst     243K   395G   217K  /tank/ZOL-5341/dst
tank/ZOL-5341/dst@a  12,8K      -   217K  -
tank/ZOL-5341/dst@b  12,8K      -   217K  -
tank/ZOL-5341/dst@c      0      -   217K  -
tank/ZOL-5341/src     217K   395G   217K  /tank/ZOL-5341/src
tank/ZOL-5341/src@b      0      -   217K  -
tank/ZOL-5341/src@c      0      -   217K  -
tank/ZOL-5341/src@d      0      -   217K  -
~ $ # $DATASET/dst@a is still there
~ $ zfs destroy $DATASET/src@b
~ $ zfs send -p -I c $DATASET/src@d | zfs recv -F $DATASET/dst
~ $ zfs list $DATASET -r -tall
NAME                  USED  AVAIL  REFER  MOUNTPOINT
tank/ZOL-5341         690K   395G   243K  /tank/ZOL-5341
tank/ZOL-5341/dst     230K   395G   217K  /tank/ZOL-5341/dst
tank/ZOL-5341/dst@c  12,8K      -   217K  -
tank/ZOL-5341/dst@d      0      -   217K  -
tank/ZOL-5341/src     217K   395G   217K  /tank/ZOL-5341/src
tank/ZOL-5341/src@c      0      -   217K  -
tank/ZOL-5341/src@d      0      -   217K  -
~ $ # $DATASET/dst@a and @b are gone
~ $ zfs destroy $DATASET -r
~ $

I would maybe expect that from send -R, but not from send -p. This should be fixed in the code.

[GregorKopka]

Better IMHO would be if the directive to destroy snapshots (and if so: which) would be completely anchored at recv (instead depending on send), with a fine granularity for the user to decide what he actually wants for incrementals (using send -i/-I a src@b | zfs recv dst for example):

• fail if dst not clean @A, else recv (as-is without -F now)
• fail if last dst snapshot isn't @A, else rollback to it, if needed, and recv
• fail if dst dosn't have @A, else rollback to it, if needed, and recv

Additionally flags to explicitely allow recv to:

• work on child datasets at the destination
• apply properties found in the received stream
• destroy destination snapshots not contained in a replication stream
• allow an existing destination to be destroyed for a full stream receive

would be helpful, to give better control of the outcome.

Especially in light of backup scenarios where the backup host calls into source systems with a zfs send, should the source be compromised it could modify the send invocation to deliver a specially crafted dataset that overloads parts of the target system (through setting overlay, mountpoint, canmount) which compromises the destination system (/root/.ssh filesystem with attacker controlled keys or overloading /etc/cron.* directories).

Back to topic:
@kpande Then we have an inconsistency with zfs send -I b $DATASET/src@c | zfs recv -F $DATASET/dst (in my reproducer) as that dosn't destroy @A.

IMHO -p generating a replication stream (or something that recv treats as one) is a bug.

[tobia]

IMHO -p generating a replication stream (or something that recv treats as one) is a bug.

Yes, that was exactly my point.

[pklapperich]

I currently rely on zfs send -p | zfs receive -F to delete old snapshots, but I agree with @GregorKopka that this is a bug. IMHO, my usage is merely a work around (taking advantage of a side effect from a bug) until zfs receive has more options. I happened on this behavior accidentally and kept using it. An option like zfs receive -D - delete old snaphots. Works only on streams sent with zfs send -R, -p, or... [insert exhaustive list here] would be ideal.

[GregorKopka]

@behlendorf this is closed (and without by whom and when) and marked as documentation.

But this issue describes an obvious bug in the code, it shouldn't be closed unless there is a merged pull request that fixes the issue. Please reopen and tag accordingly.

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

[tobia]

Why is there a bot closing issues that are not solved

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

[decayingabstractions]

The following commit may have fixed this issue a while ago:
OpenZFS 5380 - receive of a send -p stream doesn't need to try renaming snapshots

FWIW, in OpenZFS 2.1.5 on Ubuntu 22.04 this no problem no longer seems to occur.

Running @GregorKopka's reproducer, but getting different results:

$ DATASET="sandpool/sandbox5341"
$ sudo zfs create $DATASET/src -p
$ sudo zfs allow -u sandperson create,snapshot,send,destroy,mount,receive,hold $DATASET
$ zfs snapshot $DATASET/src@a
$ zfs snapshot $DATASET/src@b
$ zfs snapshot $DATASET/src@c
$ zfs snapshot $DATASET/src@d
$
$ zfs list $DATASET -r -tall
NAME                             USED  AVAIL     REFER  MOUNTPOINT
sandpool/sandbox5341          48K  23.0M       24K  /sandpool/sandbox5341
sandpool/sandbox5341/src      24K  23.0M       24K  /sandpool/sandbox5341/src
sandpool/sandbox5341/src@a     0B      -       24K  -
sandpool/sandbox5341/src@b     0B      -       24K  -
sandpool/sandbox5341/src@c     0B      -       24K  -
sandpool/sandbox5341/src@d     0B      -       24K  -
$ zfs send $DATASET/src@a | zfs recv -F $DATASET/dst
$ zfs list $DATASET -r -tall
NAME                             USED  AVAIL     REFER  MOUNTPOINT
sandpool/sandbox5341          72K  23.0M       24K  /sandpool/sandbox5341
sandpool/sandbox5341/dst      24K  23.0M       24K  /sandpool/sandbox5341/dst
sandpool/sandbox5341/dst@a     0B      -       24K  -
sandpool/sandbox5341/src      24K  23.0M       24K  /sandpool/sandbox5341/src
sandpool/sandbox5341/src@a     0B      -       24K  -
sandpool/sandbox5341/src@b     0B      -       24K  -
sandpool/sandbox5341/src@c     0B      -       24K  -
sandpool/sandbox5341/src@d     0B      -       24K  -
$ zfs send -I a $DATASET/src@b | zfs recv -F $DATASET/dst
$ zfs list $DATASET -r -tall
NAME                             USED  AVAIL     REFER  MOUNTPOINT
sandpool/sandbox5341          72K  23.0M       24K  /sandpool/sandbox5341
sandpool/sandbox5341/dst      24K  23.0M       24K  /sandpool/sandbox5341/dst
sandpool/sandbox5341/dst@a     0B      -       24K  -
sandpool/sandbox5341/dst@b     0B      -       24K  -
sandpool/sandbox5341/src      24K  23.0M       24K  /sandpool/sandbox5341/src
sandpool/sandbox5341/src@a     0B      -       24K  -
sandpool/sandbox5341/src@b     0B      -       24K  -
sandpool/sandbox5341/src@c     0B      -       24K  -
sandpool/sandbox5341/src@d     0B      -       24K  -
$ zfs destroy $DATASET/src@a
$ zfs send -I b $DATASET/src@c | zfs recv -F $DATASET/dst
$ zfs list $DATASET -r -tall
NAME                             USED  AVAIL     REFER  MOUNTPOINT
sandpool/sandbox5341          72K  23.0M       24K  /sandpool/sandbox5341
sandpool/sandbox5341/dst      24K  23.0M       24K  /sandpool/sandbox5341/dst
sandpool/sandbox5341/dst@a     0B      -       24K  -
sandpool/sandbox5341/dst@b     0B      -       24K  -
sandpool/sandbox5341/dst@c     0B      -       24K  -
sandpool/sandbox5341/src      24K  23.0M       24K  /sandpool/sandbox5341/src
sandpool/sandbox5341/src@b     0B      -       24K  -
sandpool/sandbox5341/src@c     0B      -       24K  -
sandpool/sandbox5341/src@d     0B      -       24K  -
$ # $DATASET/dst@a is still there
$ zfs destroy $DATASET/src@b
$ zfs send -p -I c $DATASET/src@d | zfs recv -F $DATASET/dst
$ zfs list $DATASET -r -tall
NAME                             USED  AVAIL     REFER  MOUNTPOINT
sandpool/sandbox5341          72K  23.0M       24K  /sandpool/sandbox5341
sandpool/sandbox5341/dst      24K  23.0M       24K  /sandpool/sandbox5341/dst
sandpool/sandbox5341/dst@a     0B      -       24K  -
sandpool/sandbox5341/dst@b     0B      -       24K  -
sandpool/sandbox5341/dst@c     0B      -       24K  -
sandpool/sandbox5341/dst@d     0B      -       24K  -
sandpool/sandbox5341/src      24K  23.0M       24K  /sandpool/sandbox5341/src
sandpool/sandbox5341/src@c     0B      -       24K  -
sandpool/sandbox5341/src@d     0B      -       24K  -
$ 
$ # $DATASET/dst@a and @b still exist
$ 

I believe the behavior now matches the documentation.