"invalid backup stream" error on second incremental stream sent in with -RI

[ryanolf]

I'm using the 4~natty version of ubuntu-zfs from the ppa. I have a filesystem with a number of snapshots that I'm trying to send as an incremental replication stream to a new pool. When I do this, one snapshot (the oldest) is sent successfully, but then the next snapshot fails. I restart the replication stream from the failed snapshot, and once again one snapshot is sent successfully while the next one fails. I can transfer all of my snapshots like this, one snapshot at a time.

ryan@oxford:~$ sudo zfs send -vRDI @AutoM-2010-02 tank/oxford@AutoD-2011-12-09 | \
sudo zfs recv -vFd btank/recv

sending from @AutoM-2010-02 to tank/oxford@AutoM-2010-03
receiving incremental stream of tank/oxford@AutoM-2010-03 into btank/recv/oxford@AutoM-2010-03
received 7.05GB stream in 3965 seconds (1.82MB/sec)
sending from @AutoM-2010-03 to tank/oxford@AutoM-2010-04
receiving incremental stream of tank/oxford@AutoM-2010-04 into btank/recv/oxford@AutoM-2010-04
cannot receive incremental stream: invalid backup stream

ryan@oxford:~$ sudo zfs send -vRDI @AutoM-2010-03 tank/oxford@AutoD-2011-12-09 |  \
sudo zfs recv -vFd btank/recv

sending from @AutoM-2010-03 to tank/oxford@AutoM-2010-04
receiving incremental stream of tank/oxford@AutoM-2010-04 into btank/recv/oxford@AutoM-2010-04
received 1.41GB stream in 1201 seconds (1.17MB/sec)
sending from @AutoM-2010-04 to tank/oxford@AutoM-2010-05
receiving incremental stream of tank/oxford@AutoM-2010-05 into btank/recv/oxford@AutoM-2010-05
cannot receive incremental stream: invalid backup stream

Please let me know if I can provide more useful information.

[behlendorf]

Interesting, that sure looks like a bug. Thanks for filing it, we'll take a look time permitting.

[gunnarbeutner]

Hi ryanolf,

can you please post the exact version of the ZFS Ubuntu packages you are using? "4~natty" is just the version of the ubuntu-zfs meta package. You can get the relevant information using "dpkg -l | grep -E spl-|zfs-" and "modinfo zfs".

The reason why I'm asking is that this bug looks like a duplicate of #372 which has been fixed a while ago. Can you try and check if zfs send without -D works?

Regards,
Gunnar

[ryanolf]

modinfo zfs gives:

filename:       /lib/modules/2.6.38-11-server/updates/dkms/zfs.ko
license:        CDDL
author:         Sun Microsystems/Oracle, Lawrence Livermore National Laboratory
description:    ZFS
srcversion:     94FACEC00CC5490AB09E6BE
depends:        spl,znvpair,zcommon,zunicode,zavl
vermagic:       2.6.38-11-server SMP mod_unload modversions 

Here's more package info:

ryan@oxford:~$ sudo dpkg -l | grep -E "spl-|zfs-"
ii  spl-dkms                              0.6.0.34-0ubuntu1~natty2                   Solaris Porting Layer kernel modules for Linux
ii  zfs-dkms                              0.6.0.34-0ubuntu1~natty1                   Native ZFS filesystem kernel modules for Linux

I'll try without the -D soon. Right now I have a send/receive in progress. I don't think this is the same as #372, because here the first incremental stream is sent and received successfully. The second in the series produces an error along the lines of #372, so they may be related. I don't think this is a duplicate, though.

[ryanolf]

I can note that dropping the "-R" does not change the issue. I already tried that one. I have not tried without the "-D" yet.

[ryanolf]

I'm trying to replicate the bug on a test filesystem (with less data so that the tests are faster), and the problem does not appear. This is on the same machine, in the same pools.

root@oxford:~# zfs send -vRD tank/test@1 | zfs recv -vFd btank/recv
sending from @ to tank/test@1
receiving full stream of tank/test@1 into btank/recv/test@1
received 45.3KB stream in 60 seconds (772B/sec)
root@oxford:~# zfs send -vRDI @1 tank/test@4 | zfs recv -vFd btank/recv
sending from @1 to tank/test@2
receiving incremental stream of tank/test@2 into btank/recv/test@2
sending from @2 to tank/test@3
sending from @3 to tank/test@4
received 12.2KB stream in 102 seconds (122B/sec)
receiving incremental stream of tank/test@3 into btank/recv/test@3
received 11.1KB stream in 106 seconds (106B/sec)
receiving incremental stream of tank/test@4 into btank/recv/test@4
received 10.8KB stream in 104 seconds (106B/sec)

I have other (used) filesystems I can try to replicate the bug on, and try without the -D option after my current large send/recv is completed.

[ryanolf]

Ok, I think I have a good handle on how to reproduce this bug on my system. The issue only occurs with the -D option on zfs send (my pool also has dedup enabled globally, not sure if that matters), and the invalid backup stream is produced only if there has been data REMOVED between snapshots. For example, I can run this test in a fresh filesystem called "test."

ryan@oxford:/tank/test$ cat /dev/urandom > file1
ryan@oxford:/tank/test$ sudo zfs snapshot tank/test@1
ryan@oxford:/tank/test$ cat /dev/urandom > file2
ryan@oxford:/tank/test$ sudo zfs snapshot tank/test@2
ryan@oxford:/tank/test$ cat /dev/urandom > file3
ryan@oxford:/tank/test$ sudo zfs snapshot tank/test@3
ryan@oxford:/tank/test$ du -hs *
1.6M file1
3.8M file2
20.1M file3

There are three files, and three snapshots. Nothing has been deleted between snapshots. Now I send it to another pool using the -D option (without -D there is no problem):

ryan@oxford:~$ sudo zfs send -vRD tank/test@3 | sudo zfs recv -vFd btank/recv
sending from @ to tank/test@1
receiving full stream of tank/test@1 into btank/recv/test@1
received 1.55MB stream in 2 seconds (792KB/sec)
sending from @1 to tank/test@2
receiving incremental stream of tank/test@2 into btank/recv/test@2
sending from @2 to tank/test@3
received 3.77MB stream in 5 seconds (772KB/sec)
receiving incremental stream of tank/test@3 into btank/recv/test@3
received 20.1MB stream in 23 seconds (893KB/sec)

No problem, it works. Now instead of creating a third file in the third snapshot, let's instead (or additionally, it doesn't matter) delete a file:

ryan@oxford:/tank/test$ sudo zfs rollback -r tank/test@2
ryan@oxford:/tank/test$ ls
file1 file2
ryan@oxford:/tank/test$ rm file1
ryan@oxford:/tank/test$ sudo zfs snapshot tank/test@3

This time when I send it, there is the error (only if the the -D option is present):

ryan@oxford:~$ sudo zfs destroy -r btank/recv/test
ryan@oxford:~$ sudo zfs send -vRD tank/test@3 | sudo zfs recv -vFd btank/recv
sending from @ to tank/test@1
receiving full stream of tank/test@1 into btank/recv/test@1
received 1.55MB stream in 4 seconds (396KB/sec)
sending from @1 to tank/test@2
receiving incremental stream of tank/test@2 into btank/recv/test@2
sending from @2 to tank/test@3
received 3.77MB stream in 6 seconds (643KB/sec)
receiving incremental stream of tank/test@3 into btank/recv/test@3
cannot receive incremental stream: invalid backup stream

The issue, then, is sending multiple snapshots, either with a replication stream (-R) and/or incremental (-I), along with the -D option when files have been deleted between snapshots. I'm using the latest files from the ubuntu PPA. If you need some more info on my setup, please let me know what commands to run so I can give you the appropriate output.

I hope this helps track down the bug. I'm happy to try other tests to further troubleshoot what is going on. In the mean time, a workaround is to use a script to send each snapshot one at a time using the "-i" option, so this isn't a showstopper.

[ryanolf]

FYI, I've updated to Oneiric, and my package info now reads:

ryan@oxford:~$ sudo dpkg -l | grep -E "spl-|zfs-"
ii  spl-dkms                               0.6.0.34-0ubuntu1~oneiric2                 Solaris Porting Layer kernel modules for Linux
ii  zfs-dkms                               0.6.0.34-0ubuntu1~oneiric1                 Native ZFS filesystem kernel modules for Linux
rc  zfs-fuse                               0.6.9-1                                    ZFS on FUSE
ryan@oxford:~$ modinfo zfs
filename:       /lib/modules/3.0.0-15-server/updates/dkms/zfs.ko
license:        CDDL
author:         Sun Microsystems/Oracle, Lawrence Livermore National Laboratory
description:    ZFS
srcversion:     94FACEC00CC5490AB09E6BE
depends:        spl,znvpair,zcommon,zunicode,zavl
vermagic:       3.0.0-15-server SMP mod_unload modversions

And then a bunch of parm: details. Let me know if you need those.

[ryanolf]

Oops. I didn't mean to hit the "close" button. Sorry! Issue is still open.

I thought I'd just add that the "tank" pool was created using zfs-fuse but is running the latest pool version via zpool upgrade. I can post detailed properties for the zpool and the filesystem if you want.

[behlendorf]

Closing as a duplicate of #2210. Let's track this defect under a single issue.