Unable to mount encrypted datasets after raw send/recv to new pool

[cytrinox]

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	Debian GNU/Linux 9.1 (stretch)
Linux Kernel	Linux srv-demo-1 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u3 (2017-08-06) x86_64 GNU/Linux
Architecture	x86_64
ZFS Version	v0.7.0-167_g23ea00a1f
SPL Version	v0.7.0-20_g35a44fc

Describe the problem you're observing

When transferring an encrypted dataset with sub-datasets with raw sends from one pool to another pool, only the direct descendants of the encrypted dataset are mountable. zfs refused to mount sub-datasets of the direct descendants.

Describe how to reproduce the problem

The origin pool dataset configuration is:

root@srv-demo-1:~# zfs list
NAME                  USED  AVAIL  REFER  MOUNTPOINT
tank1                12.3M  48.1G  30.6K  /tank1
tank1/main           12.1M  48.1G  1.40M  /tank1/main
tank1/main/art       3.98M  48.1G  1.33M  /tank1/main/art
tank1/main/art/raws  1.33M  48.1G  1.33M  /tank1/main/art/raws
tank1/main/music     2.65M  48.1G  1.33M  /tank1/main/music
tank1/main/videos    2.65M  48.1G  1.33M  /tank1/main/videos

When transferring to tank2, the tank2/main/art/raws dataset is not mountable.

I use this script to reproduce the problem:

#!/bin/bash

# Steps to reproduce

zpool create -f tank1 raidz sda{5,6,7}
zfs create -o encryption=on -o keyformat=passphrase -o compression=lz4 tank1/main
zfs create -o atime=off tank1/main/videos
zfs create -o atime=off tank1/main/music
zfs create -o atime=off tank1/main/art
zfs create -o atime=off tank1/main/art/raws
zpool create -f tank2 raidz sdb{5,6,7}
zfs snapshot -r tank1/main@snap1
zfs send -cwvR tank1/main@snap1 | zfs recv -d tank2
zpool export tank1
zfs load-key tank2/main
zfs mount -a

root@srv-demo-1:~# bash repro.sh
Enter passphrase:
Re-enter passphrase:
full send of tank1/main@snap1 estimated size is 717K
full send of tank1/main/videos@snap1 estimated size is 683K
full send of tank1/main/music@snap1 estimated size is 683K
full send of tank1/main/art@snap1 estimated size is 684K
full send of tank1/main/art/raws@snap1 estimated size is 683K
total estimated size is 3.37M
TIME        SENT   SNAPSHOT
TIME        SENT   SNAPSHOT
TIME        SENT   SNAPSHOT
TIME        SENT   SNAPSHOT
TIME        SENT   SNAPSHOT
Enter passphrase for 'tank2/main':
cannot mount '/tank2/main/art/raws': encryption key not loaded
root@srv-demo-1:~#

Try to load a key specific for this dataset (which is nonsense but tried it anyway):

root@srv-demo-1:~# zfs load-key tank2/main/art/raws
Key load error: Keys must be loaded for encryption root of 'tank2/main/art/raws' (tank2/main/art).
root@srv-demo-1:~# zfs load-key tank2/main/art
Key load error: Keys must be loaded for encryption root of 'tank2/main/art' (tank2/main).
root@srv-demo-1:~# zfs load-key tank2/main
Key load error: Key already loaded for 'tank2/main'.

[cytrinox]

@tcaputi Can you take a look on this issue? Thank you.

[tcaputi]

thanks for the easy script to reproduce the problem. I'm working on a few other things but I should be able to get to this soon.

[tcaputi]

This diff should fix it for you. I'll get it up as an official PR soon.

diff --git a/module/zfs/dsl_crypt.c b/module/zfs/dsl_crypt.c
index a711426..59562d1 100644
--- a/module/zfs/dsl_crypt.c
+++ b/module/zfs/dsl_crypt.c
@@ -1454,7 +1454,8 @@ spa_keystore_change_key_sync(void *arg, dmu_tx_t *tx)
                    0, 0, NULL, tx);
 
                rddobj = ds->ds_dir->dd_object;
-               new_rddobj = ds->ds_dir->dd_parent->dd_object;
+               VERIFY0(dsl_dir_get_encryption_root_ddobj(ds->ds_dir->dd_parent,
+                   &new_rddobj));
        }
 
        if (wkey == NULL) {

[cytrinox]

I confirm that this fixes the bug. Thank you!

While testing zfs raw send/recv, I stumbled over a question: when using zfs send -R for an encrypted dataset, all descendant datasets of main inherits the crypt key from main. When transferring each dataset step by step (first tank1/main -> tank2/main followed by tank1/main/art -> tank2/main/art...) each dataset has an individual key. All keys are unlocked with the same passphrase, but I can't just load the key for tank2/main and all descendant datasets reuse this key, I've to load each key individually.
Is there a procedure to transfer each dataset with raw send but keep the inheritance of the main key?

[tcaputi]

Once the raw receive has completed, you can fixup the heirarchy with zfs change-key -i but that will force you to load the key. We currently don't have any plans to expose the internal ioctl which allows recursive sends to adjust the encryption root heirarchy without the key loaded because it is extremely easy to make a mistake when using it and zfs has no way to confirm that you aren't making your dataset unreadable.

[cytrinox]

Okay, all I need is change-key -i. Thanks for the hint.