Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

zfs mount fails silently on encrypted dataset #12418

Open
mtippmann opened this issue Jul 22, 2021 · 21 comments
Open

zfs mount fails silently on encrypted dataset #12418

mtippmann opened this issue Jul 22, 2021 · 21 comments
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@mtippmann
Copy link

mtippmann commented Jul 22, 2021 •
edited
Loading

System information

Type Version/Name
Distribution Name Arch Linux
Distribution Version rolling
Kernel Version 5.13.4-zen1-1-zen + 5.10.52-1
Architecture amd64
OpenZFS Version zfs-2.1.99-372_g1b50749ce9 / zfs-kmod-2.1.99-1

Describe the problem you're observing

Trying to mount an encrypted dataset with a different passphrase below other encryped datasets. This used to work fine in the past but since some unknown date zfs mount fails silently - zfs mount -vvv is also empty. It works after changing the mountpoint with zfs set mountpoint=/mnt/foo tank/enc/p but after reboot mount fails.

tank/enc encryption root
tank/enc/p dataset that I'm not able to mount with canmount=noauto with a different passphrase

Describe how to reproduce the problem

I could only test with zfs git and kernel 5.13 / 5.10

create an encrypted dataset and a child dataset below that, change the passphrase for a child dataset and try to mount it after rebooting - doing a zfs set mountpoint=somethingnew tank/enc/p allows mounting the dataset - zfs unmount / zfs mount work fine after that until rebooting, after reboot without changing the mountpoint mounting fails silent again.

Include any warning/errors/backtraces from the system logs

zfs list

~ » zfs list
NAME                                                                                        USED  AVAIL     REFER  MOUNTPOINT
tank                                                                                        577G   322G      208K  /tank
tank/enc                                                                                    577G   322G      416K  /tank/enc
tank/enc/ROOT                                                                               124G   322G      336K  /tank/enc/ROOT
tank/enc/ROOT/arch                                                                          124G   322G      124G  /tank/enc/ROOT/arch
tank/enc/data                                                                               426G   322G      336K  /tank/enc/data
tank/enc/data/home                                                                          426G   322G      426G  /home
tank/enc/data/tmp                                                                           340M   322G      340M  /tmp
tank/enc/docker                                                                            9.35G   322G     9.35G  -
tank/enc/dockerjournal                                                                     2.03G   322G     2.03G  -
tank/enc/p                                                                                 15.1G   322G     15.1G  /mnt/p

zpool get all

~ » zpool get all tank
NAME  PROPERTY                       VALUE                          SOURCE
tank  size                           928G                           -
tank  capacity                       62%                            -
tank  altroot                        -                              default
tank  health                         ONLINE                         -
tank  guid                           15984558290016131493           -
tank  version                        -                              default
tank  bootfs                         tank/enc/ROOT/arch             local
tank  delegation                     on                             default
tank  autoreplace                    off                            default
tank  cachefile                      -                              default
tank  failmode                       wait                           default
tank  listsnapshots                  off                            default
tank  autoexpand                     off                            default
tank  dedupratio                     1.00x                          -
tank  free                           351G                           -
tank  allocated                      577G                           -
tank  readonly                       off                            -
tank  ashift                         13                             local
tank  comment                        -                              default
tank  expandsize                     -                              -
tank  freeing                        0                              -
tank  fragmentation                  29%                            -
tank  leaked                         0                              -
tank  multihost                      off                            default
tank  checkpoint                     -                              -
tank  load_guid                      8427598664856948575            -
tank  autotrim                       off                            default
tank  compatibility                  off                            default
tank  feature@async_destroy          enabled                        local
tank  feature@empty_bpobj            active                         local
tank  feature@lz4_compress           active                         local
tank  feature@multi_vdev_crash_dump  enabled                        local
tank  feature@spacemap_histogram     active                         local
tank  feature@enabled_txg            active                         local
tank  feature@hole_birth             active                         local
tank  feature@extensible_dataset     active                         local
tank  feature@embedded_data          active                         local
tank  feature@bookmarks              enabled                        local
tank  feature@filesystem_limits      enabled                        local
tank  feature@large_blocks           enabled                        local
tank  feature@large_dnode            active                         local
tank  feature@sha512                 enabled                        local
tank  feature@skein                  enabled                        local
tank  feature@edonr                  enabled                        local
tank  feature@userobj_accounting     active                         local
tank  feature@encryption             active                         local
tank  feature@project_quota          active                         local
tank  feature@device_removal         enabled                        local
tank  feature@obsolete_counts        enabled                        local
tank  feature@zpool_checkpoint       enabled                        local
tank  feature@spacemap_v2            active                         local
tank  feature@allocation_classes     enabled                        local
tank  feature@resilver_defer         enabled                        local
tank  feature@bookmark_v2            enabled                        local
tank  feature@redaction_bookmarks    enabled                        local
tank  feature@redacted_datasets      enabled                        local
tank  feature@bookmark_written       enabled                        local
tank  feature@log_spacemap           active                         local
tank  feature@livelist               enabled                        local
tank  feature@device_rebuild         enabled                        local
tank  feature@zstd_compress          enabled                        local
tank  feature@draid                  disabled                       local

zfs get all for the dataset that fails to mount (zfs load-key works fine):

~ » zfs get all tank/enc/p
NAME        PROPERTY              VALUE                 SOURCE
tank/enc/p  type                  filesystem            -
tank/enc/p  creation              Mi Aug 26  0:45 2020  -
tank/enc/p  used                  15.1G                 -
tank/enc/p  available             322G                  -
tank/enc/p  referenced            15.1G                 -
tank/enc/p  compressratio         1.03x                 -
tank/enc/p  mounted               no                    -
tank/enc/p  quota                 none                  default
tank/enc/p  reservation           none                  default
tank/enc/p  recordsize            128K                  default
tank/enc/p  mountpoint            /mnt/p                local
tank/enc/p  sharenfs              off                   default
tank/enc/p  checksum              on                    default
tank/enc/p  compression           lz4                   inherited from tank/enc
tank/enc/p  atime                 off                   local
tank/enc/p  devices               on                    default
tank/enc/p  exec                  on                    default
tank/enc/p  setuid                on                    default
tank/enc/p  readonly              off                   default
tank/enc/p  zoned                 off                   default
tank/enc/p  snapdir               hidden                default
tank/enc/p  aclmode               discard               default
tank/enc/p  aclinherit            restricted            default
tank/enc/p  createtxg             5085819               -
tank/enc/p  canmount              noauto                local
tank/enc/p  xattr                 sa                    inherited from tank/enc
tank/enc/p  copies                1                     default
tank/enc/p  version               5                     -
tank/enc/p  utf8only              off                   -
tank/enc/p  normalization         none                  -
tank/enc/p  casesensitivity       sensitive             -
tank/enc/p  vscan                 off                   default
tank/enc/p  nbmand                off                   default
tank/enc/p  sharesmb              off                   default
tank/enc/p  refquota              none                  default
tank/enc/p  refreservation        none                  default
tank/enc/p  guid                  13385462292017961694  -
tank/enc/p  primarycache          all                   inherited from tank
tank/enc/p  secondarycache        all                   default
tank/enc/p  usedbysnapshots       0B                    -
tank/enc/p  usedbydataset         15.1G                 -
tank/enc/p  usedbychildren        0B                    -
tank/enc/p  usedbyrefreservation  0B                    -
tank/enc/p  logbias               latency               default
tank/enc/p  objsetid              29578                 -
tank/enc/p  dedup                 off                   default
tank/enc/p  mlslabel              none                  default
tank/enc/p  sync                  standard              default
tank/enc/p  dnodesize             auto                  inherited from tank
tank/enc/p  refcompressratio      1.03x                 -
tank/enc/p  written               15.1G                 -
tank/enc/p  logicalused           15.5G                 -
tank/enc/p  logicalreferenced     15.5G                 -
tank/enc/p  volmode               default               default
tank/enc/p  filesystem_limit      none                  default
tank/enc/p  snapshot_limit        none                  default
tank/enc/p  filesystem_count      none                  default
tank/enc/p  snapshot_count        none                  default
tank/enc/p  snapdev               hidden                default
tank/enc/p  acltype               off                   default
tank/enc/p  context               none                  default
tank/enc/p  fscontext             none                  default
tank/enc/p  defcontext            none                  default
tank/enc/p  rootcontext           none                  default
tank/enc/p  relatime              on                    inherited from tank
tank/enc/p  redundant_metadata    most                  inherited from tank
tank/enc/p  overlay               off                   local
tank/enc/p  encryption            aes-128-gcm           -
tank/enc/p  keylocation           prompt                local
tank/enc/p  keyformat             passphrase            -
tank/enc/p  pbkdf2iters           350000                -
tank/enc/p  encryptionroot        tank/enc/p            -
tank/enc/p  keystatus             available             -
tank/enc/p  special_small_blocks  0                     default

zdb:

~ » zdb
tank:
    version: 5000
    name: 'tank'
    state: 0
    txg: 3500
    pool_guid: 15984558290016131493
    errata: 0
    hostname: 'archlinux'
    com.delphix:has_per_vdev_zaps
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 15984558290016131493
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 8329439666482003506
            path: '/dev/disk/by-id/ata-Samsung_SSD_860_EVO_1TB_S3Z9NX0N301403X-part1'
            devid: 'ata-Samsung_SSD_860_EVO_1TB_S3Z9NX0N301403X-part1'
            phys_path: 'pci-0000:00:1f.2-ata-2'
            whole_disk: 1
            metaslab_array: 65
            metaslab_shift: 33
            ashift: 13
            asize: 1000189984768
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_leaf: 129
            com.delphix:vdev_zap_top: 130
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data

strace (mount() syscall returns 0 but mount does not appear in /proc/mounts)

~ » LANG=C sudo strace -f zfs mount tank/enc/p
execve("/bin/zfs", ["zfs", "mount", "tank/enc/p"], 0x7fff96f5fd48 /* 22 vars */) = 0
brk(NULL)                               = 0x562fa004b000
arch_prctl(0x3001 /* ARCH_??? */, 0x7fff4694b410) = -1 EINVAL (Invalid argument)
access("/etc/ld.so.preload", R_OK)      = 0
openat(AT_FDCWD, "/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0
close(3)                                = 0
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=253704, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 253704, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f50c4ae8000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libzfs.so.4", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \340\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=424272, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f50c4ae6000
mmap(NULL, 440072, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c4a7a000
mmap(0x7f50c4a88000, 274432, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe000) = 0x7f50c4a88000
mmap(0x7f50c4acb000, 81920, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x51000) = 0x7f50c4acb000
mmap(0x7f50c4adf000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x64000) = 0x7f50c4adf000
mmap(0x7f50c4ae3000, 9992, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f50c4ae3000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libzfs_core.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 p\0\0\0\0\0\0"..., 832) = 832
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\3660\303\240\276\343\17\t\271\311}\371\256\246\337\246"..., 36, 800) = 36
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=109896, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 116480, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c4a5d000
mmap(0x7f50c4a64000, 57344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f50c4a64000
mmap(0x7f50c4a72000, 20480, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7f50c4a72000
mmap(0x7f50c4a77000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19000) = 0x7f50c4a77000
mmap(0x7f50c4a79000, 1792, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f50c4a79000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libnvpair.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 P\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=100176, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 102496, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c4a43000
mmap(0x7f50c4a48000, 53248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f50c4a48000
mmap(0x7f50c4a55000, 24576, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12000) = 0x7f50c4a55000
mmap(0x7f50c4a5b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f50c4a5b000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libuutil.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0PP\0\0\0\0\0\0"..., 832) = 832
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0>\30\326.J\26\36\266\311\232?\0\220\346r\327"..., 36, 800) = 36
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=64024, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 70688, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c4a31000
mmap(0x7f50c4a36000, 24576, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f50c4a36000
mmap(0x7f50c4a3c000, 16384, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xb000) = 0x7f50c4a3c000
mmap(0x7f50c4a40000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe000) = 0x7f50c4a40000
mmap(0x7f50c4a42000, 1056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f50c4a42000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\200\0\0\0\0\0\0"..., 832) = 832
pread64(3, "\4\0\0\0@\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0"..., 80, 792) = 80
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\7\310\371[O2Q\320\205P!z\330\241\363\20"..., 68, 872) = 68
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=154040, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 131472, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c4a10000
mprotect(0x7f50c4a17000, 81920, PROT_NONE) = 0
mmap(0x7f50c4a17000, 61440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f50c4a17000
mmap(0x7f50c4a26000, 16384, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f50c4a26000
mmap(0x7f50c4a2b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a000) = 0x7f50c4a2b000
mmap(0x7f50c4a2d000, 12688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f50c4a2d000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`|\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
pread64(3, "\4\0\0\0@\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0"..., 80, 848) = 80
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0K@g7\5w\10\300\344\306B4Zp<G"..., 68, 928) = 68
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=2150424, ...}, AT_EMPTY_PATH) = 0
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 1880536, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c4844000
mmap(0x7f50c486a000, 1355776, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f50c486a000
mmap(0x7f50c49b5000, 311296, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x171000) = 0x7f50c49b5000
mmap(0x7f50c4a01000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bc000) = 0x7f50c4a01000
mmap(0x7f50c4a07000, 33240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f50c4a07000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0  \0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=30616, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f50c4842000
mmap(NULL, 32792, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c4839000
mmap(0x7f50c483b000, 16384, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f50c483b000
mmap(0x7f50c483f000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f50c483f000
mmap(0x7f50c4840000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f50c4840000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\2207\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=39408, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 43520, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c482e000
mmap(0x7f50c4831000, 16384, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f50c4831000
mmap(0x7f50c4835000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f50c4835000
mmap(0x7f50c4837000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f50c4837000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libblkid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\240\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=347112, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 350008, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c47d8000
mprotect(0x7f50c47e2000, 286720, PROT_NONE) = 0
mmap(0x7f50c47e2000, 217088, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0x7f50c47e2000
mmap(0x7f50c4817000, 65536, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3f000) = 0x7f50c4817000
mmap(0x7f50c4828000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4f000) = 0x7f50c4828000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libtirpc.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \200\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=186672, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 192112, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c47a9000
mprotect(0x7f50c47b1000, 151552, PROT_NONE) = 0
mmap(0x7f50c47b1000, 110592, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f50c47b1000
mmap(0x7f50c47cc000, 36864, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f50c47cc000
mmap(0x7f50c47d6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2c000) = 0x7f50c47d6000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\363\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1323472, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 1323032, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c4665000
mprotect(0x7f50c4674000, 1257472, PROT_NONE) = 0
mmap(0x7f50c4674000, 630784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf000) = 0x7f50c4674000
mmap(0x7f50c470e000, 622592, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa9000) = 0x7f50c470e000
mmap(0x7f50c47a7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x141000) = 0x7f50c47a7000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libcrypto.so.1.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0p\7\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=2986824, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 3006384, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c4387000
mmap(0x7f50c43fd000, 1724416, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x76000) = 0x7f50c43fd000
mmap(0x7f50c45a2000, 593920, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21b000) = 0x7f50c45a2000
mmap(0x7f50c4633000, 188416, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2ab000) = 0x7f50c4633000
mmap(0x7f50c4661000, 16304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f50c4661000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=100096, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f50c4385000
mmap(NULL, 102416, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c436b000
mprotect(0x7f50c436e000, 86016, PROT_NONE) = 0
mmap(0x7f50c436e000, 57344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f50c436e000
mmap(0x7f50c437c000, 24576, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x11000) = 0x7f50c437c000
mmap(0x7f50c4383000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f50c4383000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\"\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=22704, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 24720, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c4364000
mmap(0x7f50c4366000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f50c4366000
mmap(0x7f50c4368000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f50c4368000
mmap(0x7f50c4369000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f50c4369000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=475944, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 107240, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c4349000
mprotect(0x7f50c434c000, 90112, PROT_NONE) = 0
mmap(0x7f50c434c000, 73728, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f50c434c000
mmap(0x7f50c435e000, 12288, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7f50c435e000
mmap(0x7f50c4362000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7f50c4362000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libgssapi_krb5.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \300\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=344176, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 346976, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c42f4000
mprotect(0x7f50c4300000, 286720, PROT_NONE) = 0
mmap(0x7f50c4300000, 229376, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc000) = 0x7f50c4300000
mmap(0x7f50c4338000, 53248, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x44000) = 0x7f50c4338000
mmap(0x7f50c4346000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x51000) = 0x7f50c4346000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libkrb5.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 0\2\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=940440, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 943392, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c420d000
mmap(0x7f50c4230000, 434176, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f50c4230000
mmap(0x7f50c429a000, 299008, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8d000) = 0x7f50c429a000
mmap(0x7f50c42e3000, 69632, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd5000) = 0x7f50c42e3000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libk5crypto.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 P\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=194544, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f50c420b000
mmap(NULL, 200560, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c41da000
mmap(0x7f50c41df000, 114688, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f50c41df000
mmap(0x7f50c41fb000, 53248, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21000) = 0x7f50c41fb000
mmap(0x7f50c4208000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2d000) = 0x7f50c4208000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libcom_err.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0  \0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=18184, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 20648, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c41d4000
mmap(0x7f50c41d6000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f50c41d6000
mmap(0x7f50c41d7000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f50c41d7000
mmap(0x7f50c41d8000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f50c41d8000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libkrb5support.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=55352, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 57992, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c41c5000
mmap(0x7f50c41c8000, 28672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f50c41c8000
mmap(0x7f50c41cf000, 12288, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0x7f50c41cf000
mmap(0x7f50c41d2000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc000) = 0x7f50c41d2000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libkeyutils.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0  \0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=22200, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 24592, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c41be000
mmap(0x7f50c41c0000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f50c41c0000
mmap(0x7f50c41c2000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f50c41c2000
mmap(0x7f50c41c3000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f50c41c3000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20G\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=92496, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 104608, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f50c41a4000
mprotect(0x7f50c41a8000, 73728, PROT_NONE) = 0
mmap(0x7f50c41a8000, 57344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f50c41a8000
mmap(0x7f50c41b6000, 12288, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12000) = 0x7f50c41b6000
mmap(0x7f50c41ba000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7f50c41ba000
mmap(0x7f50c41bc000, 6304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f50c41bc000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f50c41a2000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f50c41a0000
mmap(NULL, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f50c419b000
arch_prctl(ARCH_SET_FS, 0x7f50c419d7c0) = 0
mprotect(0x7f50c4a01000, 12288, PROT_READ) = 0
mprotect(0x7f50c41ba000, 4096, PROT_READ) = 0
mprotect(0x7f50c41c3000, 4096, PROT_READ) = 0
mprotect(0x7f50c4369000, 4096, PROT_READ) = 0
mprotect(0x7f50c41d2000, 4096, PROT_READ) = 0
mprotect(0x7f50c4a2b000, 4096, PROT_READ) = 0
mprotect(0x7f50c41d8000, 4096, PROT_READ) = 0
mprotect(0x7f50c4208000, 8192, PROT_READ) = 0
mprotect(0x7f50c42e3000, 61440, PROT_READ) = 0
mprotect(0x7f50c4346000, 8192, PROT_READ) = 0
mprotect(0x7f50c4362000, 4096, PROT_READ) = 0
mprotect(0x7f50c4383000, 4096, PROT_READ) = 0
mprotect(0x7f50c4633000, 180224, PROT_READ) = 0
mprotect(0x7f50c47a7000, 4096, PROT_READ) = 0
mprotect(0x7f50c47d6000, 4096, PROT_READ) = 0
mprotect(0x7f50c4828000, 20480, PROT_READ) = 0
mprotect(0x7f50c4837000, 4096, PROT_READ) = 0
mprotect(0x7f50c4840000, 4096, PROT_READ) = 0
mprotect(0x7f50c4a40000, 4096, PROT_READ) = 0
mprotect(0x7f50c4a5b000, 4096, PROT_READ) = 0
mprotect(0x7f50c4a77000, 4096, PROT_READ) = 0
mprotect(0x7f50c4adf000, 8192, PROT_READ) = 0
mprotect(0x562f9e174000, 4096, PROT_READ) = 0
mprotect(0x7f50c4b54000, 8192, PROT_READ) = 0
munmap(0x7f50c4ae8000, 253704)          = 0
set_tid_address(0x7f50c419da90)         = 107444
set_robust_list(0x7f50c419daa0, 24)     = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x7f50c4a17b70, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f50c4a23870}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x7f50c4a17c10, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f50c4a23870}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
brk(NULL)                               = 0x562fa004b000
brk(0x562fa006c000)                     = 0x562fa006c000
access("/sys/module/zfs", F_OK)         = 0
openat(AT_FDCWD, "/dev/zfs", O_RDWR|O_CLOEXEC) = 3
close(3)                                = 0
openat(AT_FDCWD, "/dev/zfs", O_RDWR|O_EXCL|O_CLOEXEC) = 3
openat(AT_FDCWD, "/dev/zfs", O_RDWR|O_CLOEXEC) = 4
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/redundant_metadata", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/sync", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/checksum", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/dedup", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/compression", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/snapdir", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/snapdev", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/aclmode", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/acltype", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/aclinherit", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/copies", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/primarycache", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/secondarycache", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/logbias", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/xattr", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/dnodesize", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/volmode", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/atime", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/relatime", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/devices", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/exec", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/setuid", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/readonly", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/zoned", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/vscan", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/nbmand", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/overlay", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/version", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/canmount", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/mounted", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/defer_destroy", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/keystatus", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/normalization", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/casesensitivity", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/keyformat", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/encryption", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/utf8only", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/origin", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/clones", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/mountpoint", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/sharenfs", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/type", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/sharesmb", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/mlslabel", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/context", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/fscontext", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/defcontext", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/rootcontext", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/receive_resume_token", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/encryptionroot", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/keylocation", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/redact_snaps", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/used", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/available", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/referenced", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/compressratio", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/refcompressratio", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/volblocksize", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/usedbysnapshots", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/usedbydataset", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/usedbychildren", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/usedbyrefreservation", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/userrefs", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/written", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/logicalused", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/logicalreferenced", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/filesystem_count", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/snapshot_count", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/guid", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/createtxg", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/pbkdf2iters", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/objsetid", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/quota", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/reservation", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/volsize", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/refquota", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/refreservation", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/filesystem_limit", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/snapshot_limit", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/recordsize", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/special_small_blocks", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/numclones", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/name", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/iscsioptions", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/stmf_sbd_lu", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/useraccounting", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/unique", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/inconsistent", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/ivsetguid", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/prevsnap", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/pbkdf2salt", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/keyguid", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/redacted", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/remaptxg", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.dataset/creation", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/altroot", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/bootfs", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/cachefile", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/comment", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/compatibility", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/size", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/free", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/freeing", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/checkpoint", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/leaked", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/allocated", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/expandsize", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/fragmentation", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/capacity", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/guid", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/load_guid", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/health", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/dedupratio", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/version", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/ashift", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/delegation", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/autoreplace", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/listsnapshots", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/autoexpand", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/readonly", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/multihost", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/failmode", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/autotrim", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/name", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/maxblocksize", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/tname", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/maxdnodesize", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/properties.pool/dedupditto", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:async_destroy", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:empty_bpobj", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/org.illumos:lz4_compress", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.joyent:multi_vdev_crash_dump", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:spacemap_histogram", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:enabled_txg", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:hole_birth", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:zpool_checkpoint", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:spacemap_v2", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:extensible_dataset", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:bookmarks", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.joyent:filesystem_limits", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:embedded_data", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:livelist", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:log_spacemap", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/org.open-zfs:large_blocks", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/org.zfsonlinux:large_dnode", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/org.illumos:sha512", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/org.illumos:skein", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/org.illumos:edonr", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:redaction_bookmarks", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:redacted_datasets", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:bookmark_written", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:device_removal", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.delphix:obsolete_counts", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/org.zfsonlinux:userobj_accounting", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.datto:bookmark_v2", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.datto:encryption", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/org.zfsonlinux:project_quota", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/org.zfsonlinux:allocation_classes", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/com.datto:resilver_defer", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/org.openzfs:device_rebuild", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/org.freebsd:zstd_compress", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
newfstatat(AT_FDCWD, "/sys/module/zfs/features.pool/org.openzfs:draid", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f50c415a000
ioctl(3, ZFS_IOC_OBJSET_STATS, 0x7fff469477e0) = 0
ioctl(3, ZFS_IOC_POOL_STATS, 0x7fff46944140) = 0
munmap(0x7f50c415a000, 266240)          = 0
brk(0x562fa00ca000)                     = 0x562fa00ca000
ioctl(3, ZFS_IOC_POOL_GET_PROPS, 0x7fff46943b60) = 0
openat(AT_FDCWD, "/proc/self/mounts", O_RDONLY|O_CLOEXEC) = 5
newfstatat(5, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0
read(5, "proc /proc proc rw,nosuid,nodev,"..., 1024) = 1024
read(5, "latime,pagesize=2M 0 0\ndebugfs /"..., 1024) = 1024
read(5, "316 squashfs ro,nodev,relatime 0"..., 1024) = 1024
read(5, "o,inode64,logbufs=8,logbsize=256"..., 1024) = 1023
read(5, "", 1024)                       = 0
close(5)                                = 0
ioctl(3, ZFS_IOC_OBJSET_STATS, 0x7fff46941950) = 0
newfstatat(AT_FDCWD, "/mnt/p", {st_mode=S_IFDIR|0755, st_size=2, ...}, AT_SYMLINK_NOFOLLOW) = 0
statfs("/mnt/p", {f_type=ZFS_SUPER_MAGIC, f_bsize=131072, f_blocks=3658903, f_bfree=2639514, f_bavail=2639514, f_files=677052778, f_ffree=675715632, f_fsid={val=[0x6c27214, 0xca4975]}, f_namelen=255, f_frsize=131072, f_flags=ST_VALID|ST_RELATIME}) = 0
newfstatat(AT_FDCWD, "/mnt/p", {st_mode=S_IFDIR|0755, st_size=2, ...}, 0) = 0
mount("tank/enc/p", "/mnt/p", "zfs", MS_NOATIME, "defaults,noatime,dev,exec,rw,sui"...) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLDstrace: Process 107446 attached
, child_tidptr=0x7f50c419da90) = 107446
[pid 107446] set_robust_list(0x7f50c419daa0, 24 <unfinished ...>
[pid 107444] wait4(107446,  <unfinished ...>
[pid 107446] <... set_robust_list resumed>) = 0
[pid 107446] openat(AT_FDCWD, "/dev/null", O_WRONLY|O_CLOEXEC) = 5
[pid 107446] dup2(5, 1)                 = 1
[pid 107446] dup2(5, 2)                 = 2
[pid 107446] execve("/usr/sbin/exportfs", ["/usr/sbin/exportfs", "-ra"], 0x7fff4694b5a8 /* 22 vars */) = 0
[pid 107446] brk(NULL)                  = 0x557168d03000
[pid 107446] arch_prctl(0x3001 /* ARCH_??? */, 0x7ffcd8d74020) = -1 EINVAL (Invalid argument)
[pid 107446] access("/etc/ld.so.preload", R_OK) = 0
[pid 107446] openat(AT_FDCWD, "/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3
[pid 107446] newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0
[pid 107446] close(3)                   = 0
[pid 107446] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
[pid 107446] newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=253704, ...}, AT_EMPTY_PATH) = 0
[pid 107446] mmap(NULL, 253704, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e7d771000
[pid 107446] close(3)                   = 0
[pid 107446] openat(AT_FDCWD, "/usr/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
[pid 107446] read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\200\0\0\0\0\0\0"..., 832) = 832
[pid 107446] pread64(3, "\4\0\0\0@\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0"..., 80, 792) = 80
[pid 107446] pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\7\310\371[O2Q\320\205P!z\330\241\363\20"..., 68, 872) = 68
[pid 107446] newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=154040, ...}, AT_EMPTY_PATH) = 0
[pid 107446] mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e7d76f000
[pid 107446] mmap(NULL, 131472, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7e7d74e000
[pid 107446] mprotect(0x7f7e7d755000, 81920, PROT_NONE) = 0
[pid 107446] mmap(0x7f7e7d755000, 61440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f7e7d755000
[pid 107446] mmap(0x7f7e7d764000, 16384, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f7e7d764000
[pid 107446] mmap(0x7f7e7d769000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a000) = 0x7f7e7d769000
[pid 107446] mmap(0x7f7e7d76b000, 12688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7e7d76b000
[pid 107446] close(3)                   = 0
[pid 107446] openat(AT_FDCWD, "/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
[pid 107446] read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`|\2\0\0\0\0\0"..., 832) = 832
[pid 107446] pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
[pid 107446] pread64(3, "\4\0\0\0@\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0"..., 80, 848) = 80
[pid 107446] pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0K@g7\5w\10\300\344\306B4Zp<G"..., 68, 928) = 68
[pid 107446] newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=2150424, ...}, AT_EMPTY_PATH) = 0
[pid 107446] pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
[pid 107446] mmap(NULL, 1880536, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7e7d582000
[pid 107446] mmap(0x7f7e7d5a8000, 1355776, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f7e7d5a8000
[pid 107446] mmap(0x7f7e7d6f3000, 311296, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x171000) = 0x7f7e7d6f3000
[pid 107446] mmap(0x7f7e7d73f000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bc000) = 0x7f7e7d73f000
[pid 107446] mmap(0x7f7e7d745000, 33240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7e7d745000
[pid 107446] close(3)                   = 0
[pid 107446] mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e7d57f000
[pid 107446] arch_prctl(ARCH_SET_FS, 0x7f7e7d57f740) = 0
[pid 107446] mprotect(0x7f7e7d73f000, 12288, PROT_READ) = 0
[pid 107446] mprotect(0x7f7e7d769000, 4096, PROT_READ) = 0
[pid 107446] mprotect(0x557167853000, 4096, PROT_READ) = 0
[pid 107446] mprotect(0x7f7e7d7dd000, 8192, PROT_READ) = 0
[pid 107446] munmap(0x7f7e7d771000, 253704) = 0
[pid 107446] set_tid_address(0x7f7e7d57fa10) = 107446
[pid 107446] set_robust_list(0x7f7e7d57fa20, 24) = 0
[pid 107446] rt_sigaction(SIGRTMIN, {sa_handler=0x7f7e7d755b70, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f7e7d761870}, NULL, 8) = 0
[pid 107446] rt_sigaction(SIGRT_1, {sa_handler=0x7f7e7d755c10, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f7e7d761870}, NULL, 8) = 0
[pid 107446] rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
[pid 107446] prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
[pid 107446] getpid()                   = 107446
[pid 107446] rt_sigaction(SIGUSR1, {sa_handler=0x55716784c780, sa_mask=[USR1], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f7e7d5beda0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
[pid 107446] rt_sigaction(SIGUSR2, {sa_handler=0x55716784c780, sa_mask=[USR2], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f7e7d5beda0}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
[pid 107446] newfstatat(AT_FDCWD, "/etc/nfs.conf", {st_mode=S_IFREG|0644, st_size=1422, ...}, 0) = 0
[pid 107446] openat(AT_FDCWD, "/etc/nfs.conf", O_RDONLY) = 3
[pid 107446] flock(3, LOCK_SH)          = 0
[pid 107446] lseek(3, 0, SEEK_END)      = 1422
[pid 107446] lseek(3, 0, SEEK_SET)      = 0
[pid 107446] brk(NULL)                  = 0x557168d03000
[pid 107446] brk(0x557168d24000)        = 0x557168d24000
[pid 107446] read(3, "#\n# This is a general configurat"..., 1422) = 1422
[pid 107446] close(3)                   = 0
[pid 107446] openat(AT_FDCWD, "/etc/nfs.conf.d", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
[pid 107446] openat(AT_FDCWD, "/var/lib/nfs/export-lock", O_RDWR|O_CREAT, 0666) = 3
[pid 107446] fcntl(3, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_CUR, l_start=0, l_len=0}) = 0
[pid 107446] openat(AT_FDCWD, "/etc/exports", O_RDONLY) = 4
[pid 107446] newfstatat(4, "", {st_mode=S_IFREG|0644, st_size=439, ...}, AT_EMPTY_PATH) = 0
[pid 107446] read(4, "# /etc/exports - exports(5) - di"..., 512) = 439
[pid 107446] read(4, "", 512)           = 0
[pid 107446] close(4)                   = 0
[pid 107446] openat(AT_FDCWD, "/etc/exports.d", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
[pid 107446] newfstatat(4, "", {st_mode=S_IFDIR|0755, st_size=2, ...}, AT_EMPTY_PATH) = 0
[pid 107446] getdents64(4, 0x557168d039f0 /* 2 entries */, 32768) = 48
[pid 107446] getdents64(4, 0x557168d039f0 /* 0 entries */, 32768) = 0
[pid 107446] close(4)                   = 0
[pid 107446] openat(AT_FDCWD, "/var/lib/nfs/.etab.lock", O_RDWR|O_CREAT, 0600) = 4
[pid 107446] fcntl(4, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0
[pid 107446] openat(AT_FDCWD, "/var/lib/nfs/etab.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 5
[pid 107446] close(5)                   = 0
[pid 107446] openat(AT_FDCWD, "/var/lib/nfs/etab.tmp", O_RDONLY) = 5
[pid 107446] openat(AT_FDCWD, "/var/lib/nfs/etab", O_RDONLY) = 6
[pid 107446] read(5, "", 4096)          = 0
[pid 107446] read(6, "", 4096)          = 0
[pid 107446] close(5)                   = 0
[pid 107446] close(6)                   = 0
[pid 107446] unlink("/var/lib/nfs/etab.tmp") = 0
[pid 107446] close(4)                   = 0
[pid 107446] openat(AT_FDCWD, "/proc/net/rpc/auth.unix.ip/flush", O_RDWR) = -1 ENOENT (No such file or directory)
[pid 107446] openat(AT_FDCWD, "/proc/net/rpc/auth.unix.gid/flush", O_RDWR) = -1 ENOENT (No such file or directory)
[pid 107446] openat(AT_FDCWD, "/proc/net/rpc/nfsd.fh/flush", O_RDWR) = -1 ENOENT (No such file or directory)
[pid 107446] openat(AT_FDCWD, "/proc/net/rpc/nfsd.export/flush", O_RDWR) = -1 ENOENT (No such file or directory)
[pid 107446] fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_CUR, l_start=0, l_len=0}) = 0
[pid 107446] close(3)                   = 0
[pid 107446] exit_group(0)              = ?
[pid 107446] +++ exited with 0 +++
<... wait4 resumed>[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 107446
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=107446, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
ioctl(3, ZFS_IOC_LOG_HISTORY, 0x7fff46947e50) = -1 EPERM (Operation not permitted)
close(3)                                = 0
close(4)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++
``
@mtippmann mtippmann added the Type: Defect Incorrect behavior (e.g. crash, hang) label Jul 22, 2021
@mtippmann
Copy link
Author

mtippmann commented Jul 22, 2021 •
edited
Loading

additional data:

~ » cat /proc/cmdline
initrd=\intel-ucode.img initrd=\initramfs-linux-zen.img zfs=tank/enc/ROOT/arch rw scsi_mod.use_blk_mq=1 zfs_force=1 apparmor=1 lsm=lockdown,yama,apparmor mds=off mitigations=off systemd.unified_cgroup_hierarchy=1 audit=1 threadirqs transparent_hugepage=madvise nmi_watchdog=0
~ » cat /etc/modprobe.d/zfs.conf
options zfs zfs_vdev_max_active=32
options zfs zfs_txg_timeout=5
options zfs zfs_vdev_scrub_min_active=1
options zfs zfs_vdev_scrub_max_active=2
# sync write
options zfs zfs_vdev_sync_write_min_active=4
options zfs zfs_vdev_sync_write_max_active=4
# sync reads (normal)
options zfs zfs_vdev_sync_read_min_active=4
options zfs zfs_vdev_sync_read_max_active=4
# async reads : prefetcher
options zfs zfs_vdev_async_read_min_active=1
options zfs zfs_vdev_async_read_max_active=4
# async write : bulk writes
options zfs zfs_vdev_async_write_min_active=1
options zfs zfs_vdev_async_write_max_active=4
# scheduler 
options zfs zfs_vdev_scheduler=none

options zfs metaslab_lba_weighting_enabled=0
options zfs zfs_dirty_data_max_percent=15
options zfs zio_taskq_batch_pct=25
options zfs zfs_sync_taskq_batch_pct=25
options zfs zfs_prefetch_disable=1
options zfs zfs_arc_sys_free=1000000000
options zfs zfs_abd_scatter_enabled=0

@mtippmann
Copy link
Author

more digging: zfs set mountpoint=somethingnew tank/enc/p allows mounting the dataset - zfs unmount / zfs mount work fine after that until rebooting, after reboot without changing the mountpoint mounting fails silent again.

@tkiapril
Copy link

tkiapril commented Sep 15, 2021 •
edited
Loading

This happens to me too, with kernel version 5.13.13-arch1-1 and zfs version zfs-2.1.0-1 and zfs-kmod-2.1.0-1. What's weird is that if I use the dkms version, it WILL mount the encrypted dataset on creation with canmount=noauto (with zfs mount pool/dataset,) but won't do so on reboot. However, the module version won't mount the dataset whatsoever.

EDIT: Also present in zfs-2.1.99-448_gb9ec4a15e zfs-kmod-2.1.99-1 (commit r7099)
EDIT 2: it is also not possible to mount encrypted zvols (with key loaded). Apparently exporting the block device via iSCSI works though.

@rincebrain rincebrain mentioned this issue Sep 29, 2021
Closed
@mtippmann mtippmann mentioned this issue Nov 12, 2021
Closed
@Carbon401
Copy link

Carbon401 commented Nov 15, 2021 •
edited
Loading

Are you by any chance using zfs root like me? I've just installed test system on a flash drive with ext4 and there is no issue. If I boot from the zfs root which is unencrypted btw, the issue is there. There could be a connection. Also I don't have nested datasets with different encpryption keys, just one encrypted dataset with children sharing the same key.

@kescherCode
Copy link

Similar issue, to the point where I believe this is the same issue:
Mounting an encrypted dataset inside of an unencrypted dataset (on an unencrypted dataset) fails silently.

data/kescher: normal dataset created with default options, no encryption.
data/kescher/test: encrypted dataset with passphrase. key loaded before attempting mount, obviously.

Result: data/kescher/test does not mount after creation nor after running zfs mount data/kescher/test, but exits with exit code 0 and no output, as if the mount was successful.

Not a ZFS root, the pool is mounted at /mnt/data.

@RichardJActon
Copy link

Ah good it's not just me I was quite confused by this. I'm also running zfs on root. Versions: zfs-2.1.1-1, zfs-kmod-2.1.1-1 with kernel 5.14.18-1-MANJARO. I saw the same thing as @mtippmann where setting the mountpoint works until reboot. I was experimenting with this nice little rust tool shavee https://ashu.io/projects/shavee/ that lets you use a Yubikey to generate an encryption key when I noticed this behaviour.

@kescherCode
Copy link

On the machine I tried mounting encrypted datasets anywhere after rebooting, checking journalctl -xe shows that right after trying to mount, the mount for those datasets is immediately stopped.

@bloatmode
Copy link

bloatmode commented Mar 2, 2022 •
edited
Loading

I'm having the same issue on kernel 5.15.25-1-lts. I don't have / on the encrypted dataset, it's supposed to get mounted in a directory in my home folder.

@bloatmode
Copy link

bloatmode commented Mar 3, 2022 •
edited
Loading

I ended up using this script as a workaround, after creating two empty folders zfs1/ and zfs2/ in my home directory and a file testfile.txt in the root directory of the dataset:

#!/bin/bash

set -x

sudo zfs set canmount=on pool-raidz/bloatmode
sudo zfs set mountpoint=/home/bloatmode/zfs1 pool-raidz/bloatmode
sudo zfs mount -l pool-raidz/bloatmode
sleep 0.1

if [ -f "/home/bloatmode/zfs1/testfile.txt" ]
then
	ln -snf /home/bloatmode/zfs1 /home/bloatmode/zfs
else
	sudo zfs set mountpoint=/home/bloatmode/zfs2 pool-raidz/bloatmode
	sudo zfs mount -l pool-raidz/bloatmode
	ln -snf /home/bloatmode/zfs2 /home/bloatmode/zfs
fi

sudo zfs set canmount=noauto pool-raidz/bloatmode

Notice the sleep call, it was necessary because the dataset would get mounted for a fraction of a second (I don't know exactly for how long) so the if statement would always return true.
The only reason I'm setting canmount is because the computer is used by other people and with canmount=on it would ask for a password on boot. canmount=off would immediately unmount the dataset.

Update: after some days of using the above script I noticed that sometimes after a few hours the dataset is silently unmounted, but it can be remounted using the script. I also removed the last line from the script and used a systemd service instead, which sets canmount=off before zed.service is stopped. However even with canmount set to on while the computer is on, the dataset still gets unmounted silently every once in a while.

@mtippmann mtippmann mentioned this issue Jul 2, 2022
Open
@DHowett
Copy link

DHowett commented Sep 10, 2022

A cool new symptom appears! My encrypted datasets are mounted in the last mountpoint they used, rather than the current set one.

# zfs get mountpoint acentauri/vault
NAME             PROPERTY    VALUE   SOURCE
acentauri/vault  mountpoint  /mnt/a  -

# zfs inherit mountpoint acentauri/vault

# zfs get mountpoint acentauri/vault              
NAME             PROPERTY    VALUE                 SOURCE
acentauri/vault  mountpoint  /mnt/acentauri/vault  inherited from acentauri

# zfs mount acentauri/vault

# mount|grep vault
acentauri/vault on /mnt/a type zfs (rw,relatime,xattr,posixacl)
                   ^^^^^^

@SvenMb
Copy link

SvenMb commented Sep 23, 2022 •
edited
Loading

Hi,
since I had the same problem and was debugging this for nearly two days...

This seems to be a systemd problem when running root zfs, systemd creates a .mount and because of this it executes an immediate unmount of the newly mounted zfs dataset.

You should see the instant systemd unmounts in /var/log/daemon.log like this (or use journalctl):

Sep 22 11:19:20 s740deb systemd[1]: Unmounting /var/lib/docker/compose/srv...
Sep 22 11:19:20 s740deb systemd[1]: var-lib-docker-compose-srv.mount: Deactivated successfully.

So after the first mount you just do ¨systemctl daemon-reload" and it will work in future just as it should.

(I also added the datasets to fstab (with noauto), just to be sure.)

regards
Sven

@mtippmann
Copy link
Author

@SvenMb thank you! Can confirm that it works fine after masking the mount unit:

got this in the logs:

Sep 23 23:09:14 systemd[1]: Unmounting /home/mt/.p...
Sep 23 23:09:14 systemd[1]: home-mt-.p.mount: Deactivated successfully.
Sep 23 23:09:14 systemd[1]: Unmounted /home/mt/.p.

~ » systemctl mask home-mt-.p.mount
Created symlink /etc/systemd/system/home-mt-.p.mount → /dev/null.
~ » sudo zfs mount -l rpool/enc/p
~ » sudo zfs mount -l rpool/enc/p
cannot mount 'rpool/enc/p': filesystem already mounted

This also explains why mount() returns 0 and works - now what the hell is going on with systemd here? why does changing the mountpoint works? Is this something that warrants a bug report for systemd?

@mtippmann
Copy link
Author

Maybe systemd is not at fault here - could this be a bug in zfs-mount-generator?

@luismcv
Copy link

luismcv commented Oct 10, 2022

After having this problem and seeing here that it was somewhat related to systemd the easiest workaround that I've found is doing

sudo systemctl start dpool-ARCHIVE-Personal.mount

instead of

sudo zfs mount -l dpool/ARCHIVE/Personal

Mind that for the systemd command the mountpoint has to be used instead of the dataset name, if different.

@luismcv
Copy link

luismcv commented Oct 10, 2022

Actually, I've just noticed that @mtippmann's way is more convenient. I masked the unit:

sudo systemctl mask dpool-ARCHIVE-Personal.mount

and can use zfs mount even after reboots.

@almereyda almereyda mentioned this issue Oct 10, 2022
Closed
@marramgg
Copy link

I experienced this yesterday. This started happening after I enabled zfs-mount-generator for one of my two pools. The one I enabled it for started experiencing this issue. The other one allowed me to mount things without any issue.

In sum: created file /etc/zfs/zfs-list.cache/POOL (where POOL is my pool name). Toggled a property in my pool and enabled/started the ZFS Event Daemon.

An alternative solution to masking the generated .mount file is simply to remove the /etc/zfs/zfs-list.cache/POOL file and restarting zed. The .mount units should be removed.

@olmari
Copy link

olmari commented Nov 3, 2023 •
edited
Loading

On Debian 12.2 I am currently facing exactly same issue according to description...

I have ZFS native encrypted dataset with canmount=noauto, when I mount it manually by zfs mount it appears briefly and then disappears, rinse, repeat... Changing mountpoint helps for current session, but not after reboot... Methods of @luismcv mentioned seems to work.. But indeed something is fishy somewhere.

I am willing to help debug this some, if I get clear commands to run and report :)

ADD:
Kernel: 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux
zfs-2.1.11-1
zfs-kmod-2.1.11-1

@tinuzz
Copy link

tinuzz commented Jan 18, 2024

If the dataset is encrypted and zfs-mount-generator is enabled, the generated mount unit gets a strong dependency on the generated service that loads the encryption key:

BindsTo=zfs-load-key@data-enc0.service

BindsTo has the effect that the depending unit (the mount) will be stopped, if the bound service is stopped. In other words: if the zfs-load-key@ service for the mount has not been started, filesystem will be immediately unmounted by systemd.

So it appears that the actual fix for this problem is to load encryption keys not by running zfs load-key on the command line, but by running systemctl start zfs-load-key@<encryption root>. After that, the service is active and the mount will succeed.

Does that make sense?

@a1ad
Copy link

a1ad commented Jan 21, 2024

@tinuzz Thanks, was battling the same issue. It doesn't make a lot of sense from a user point of view.

zfs load-key
zfs mount ...

Would make a lot more sense :-)

@tinuzz
Copy link

tinuzz commented Jan 21, 2024 •
edited
Loading

Well, there's always man zfs-mount-generator, that describes many ways to influence the generated units, or skip them altogether.

@genevieve-me
Copy link

genevieve-me commented Dec 21, 2024 •
edited
Loading

I can confirm that systemctl start zfs-load-key-poolname-fsname.service to start that generated service worked for me on Ubuntu 24.04. I'm not sure what triggered this bug, as it didn't occur on boot, and ZFS had previously been loading keys and mounting the FS for me automatically via generated systemd services after I ran:

sudo mkdir /etc/zfs/zfs-list.cache/
sudo touch /etc/zfs/zfs-list.cache/tank
sudo zfs set canmount=on tank/data

When this was occurring, I did not see any failed systemd services either, and sudo zfs mount -av produced no output.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests