ZED needs to know the size of a device

[FransUrbo]

I have a pool with disks of two sizes - 1.5TB and 3TB disks. The 1.5TB disks will eventually be replaced, but for now two VDEVs with 3x3TB and 3 VDEVs with 3x1.5TB. I have two spare 1.5TB disks, so I added them as spares.

Unfortunately, at the moment, ZED is unable to figure out which disk to use when replacing a failed device, it does a round robin attempt.

Playing with this on a test environment, I came up with:

# Given a <device>, return the size
vdev_size() {
    local VDEV=`basename $1`

    for dir in /dev /dev/disk/by-*; do
        if [[ -L $dir/$VDEV ]]; then
            blockdev --getsize64  $dir/$VDEV
        fi
    done
}

And then in the for spare ... loop:

                        spare_size=$(vdev_size ${SPARE_VDEV_FOUND})
                        orig_size=$(vdev_size ${ZEVENT_VDEV_PATH%-part*})

The orig_size will never (?) contain a value though, because the device have been removed.

So ZED needs to receive a ZEVENT_VDEV_SIZE which we could compare with, and if ...., well that's as far as I've got.

[behlendorf]

Good point. At the moment we assume any of the hot spares can be used to replace any other. I think we have two reasonable options here.

1. As you suggested we can easily include a ZEVENT_VDEV_SIZE in the event and update the script accordingly. See commit 904ea27 for how to go about this. Or,

2. When posting the event we could restrict the ZEVENT_SPARE_PATHS and ZEVENT_SPARE_GUIDS to only include devices which are large enough.

The first option feels like a more generic fix. But I think there's a decent argument to be made that there's no reason to include hot spares which can't be used in the posted spares list. That would keep the scripts simple which I think is a good thing.

[FransUrbo]

2. When posting the event we could restrict the ZEVENT_SPARE_PATHS and ZEVENT_SPARE_GUIDS to only include devices which are large enough.

So in my case, i can't/sholdn't use 1.5TB disks as spares, I'd need >= 3TB?

I'm not againt that philosohy (there's a reason those two were replaced :).

But... They still work and have a lot more to offer... Feels ... 'saad' just to throw them away :D.

But regardless, knowing how big the failed disk is, still seem prudent - double checking that the spare is big enough before doing the replace (instead of 'willy-nilly' attempt the replace and then possibly have a failure...)?

So the code is mostly there anyway...

[behlendorf]

So in my case, i can't/sholdn't use 1.5TB disks as spares, I'd need >= 3TB?

You can add the 1.5TB drives as spares, but they can't be used to replace a 3TB drive failure. They'd only work for your other 1.5TB drives in the pool. I was suggesting restricting the spares list to drives which could be used as a replacement for the failed device.

[FransUrbo]

@behlendorf Remove tag "Bug" and instead add "Feature"?

[behlendorf]

Done, that's reasonable.

[FransUrbo]

Where would the vdev size be? I've tried vdev_stat_t vs_rsize and vdev_stat_t vs_space, both is 0...

[FransUrbo]

There's seems to be an even better candidate in struct vdev (vdev_psize), but that's also 0...

[behlendorf]

@FransUrbo You want vdev_asize, vdev_min_asize, vdev_max_asize, and vdev_ashift. Exposing vdev_psize while you're at it might also prove useful.

[FransUrbo]

@FransUrbo You want vdev_asize, vdev_min_asize, vdev_max_asize, and vdev_ashift.

All of them?
Exposing vdev_psize while you're at it might also prove useful.

That I didn't understand. I tried using that, but it only contains '0'.

[behlendorf]

Might as well. You should only need vdev_asize and vdev_max_asize but there others may prove useful and there's little harm in making them available to the scripts.

[FransUrbo]

Might as well. You should only need vdev_asize and vdev_max_asize

So how do I get the size from these?
but there others may prove useful and there's little harm in making them available to the scripts.

Sure, no problem.

[FransUrbo]

Turns out it's not me. It's also (not) in zpool list -v:

NAME   SIZE  ALLOC   FREE   FRAG    CAP  DEDUP  HEALTH  ALTROOT
test  23.8G  5.57G  18.2G    22%    23%  1.00x  ONLINE  -
  mirror  7.94G  2.45G  5.49G    30%         -
    ata-VBOX_HARDDISK_VB1f45d95c-477fd9c6      -      -      -      -         -
    ata-VBOX_HARDDISK_VBa23b735b-e12dbf0c      -      -      -      -         -
  mirror  15.9G  3.13G  12.7G    18%         -
    ata-VBOX_HARDDISK_VB00c23763-14860ab9      -      -      -      -         -
    ata-VBOX_HARDDISK_VBd15b7714-c746e03e      -      -      -      -         -

So the code is probably working, I just don't seem to have the possibility to test it further.

[loli10K]

Sorry to revive such an old issue, but i have and old pool that could suffer from this:

History for 'pool':
2011-05-14.13:33:30 zpool create pool mirror /dev/sdb /dev/sda
2011-05-14.13:34:00 [internal pool create txg:6] pool spa 23; zfs spa 23; zpl 4; uts nas.local 2.6.18-238.el5 #1 SMP Thu Jan 13 16:24:47 EST 2011 i686

A lot has changed in ZED (with 976246f cmd/zed/zed.d/io-spare.sh is gone) but the logic is to always attempt every spare:

https://github.com/zfsonlinux/zfs/blob/ccad64314ab55963c1a0731ff62c8cf6c6976f28/cmd/zed/agents/zfs_retire.c#L254-L274

This seems to work just fine. From a test box (DISK[N] are 10GB virtio disks, DISKSMALL[N] 5GB disks):

root@linux:~# zpool status $POOLNAME 
  pool: testpool
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
	repaired.
  scan: resilvered 60K in 0h0m with 0 errors on Sun Nov 12 10:10:23 2017
config:

	NAME                   STATE     READ WRITE CKSUM
	testpool               DEGRADED     0     0     0
	  mirror-0             DEGRADED     0     0     0
	    spare-0            DEGRADED     0     0     0
	      virtio-DISK0     FAULTED     32   340     0  too many errors
	      virtio-DISK2     ONLINE       0     0     0
	    virtio-DISK1       ONLINE       0     0     0
	  mirror-1             ONLINE       0     0     0
	    virtio-DISKSMALL0  ONLINE       0     0     0
	    virtio-DISKSMALL1  ONLINE       0     0     0
	spares
	  virtio-DISKSMALL2    AVAIL   
	  virtio-DISKSMALL3    AVAIL   
	  virtio-DISKSMALL4    AVAIL   
	  virtio-DISK2         INUSE     currently in use
	  virtio-DISKSMALL5    AVAIL   
	  virtio-DISKSMALL6    AVAIL   
	  virtio-DISKSMALL7    AVAIL   

errors: No known data errors
root@linux:~# 

ZED tries to replace the failing disk with every spare:

Diagnosis Engine: error event 'ereport.fs.zfs.io'
Diagnosis Engine: error event 'ereport.fs.zfs.io'
Diagnosis Engine: error event 'ereport.fs.zfs.io'
Diagnosis Engine: timer fired (0x7f8758036a10)
Diagnosis Engine: solving fault 'fault.fs.zfs.vdev.io'

zed_fault_event:
	uuid: 441d5c0a-cfc7-49b5-897d-8ddcfc70f452
	class: fault.fs.zfs.vdev.io
	code: ZFS-8000-FD
	certainty: 100
	scheme: zfs
	pool: 15706503133311786235
	vdev: 4628339226959795810 

Retire Agent: zfs_retire_recv: 'list.suspect'
Retire Agent: matched vdev 4628339226959795810
Diagnosis Engine: case solved (441d5c0a-cfc7-49b5-897d-8ddcfc70f452)
Diagnosis Engine: removing timer (0x7f8758036a10)
Retire Agent: zpool_vdev_fault: vdev 4628339226959795810 on 'testpool'
Retire Agent: zpool_vdev_replace 'virtio-DISK0' with spare 'virtio-DISKSMALL2-part1'
Retire Agent: zpool_vdev_replace 'virtio-DISK0' with spare 'virtio-DISKSMALL3-part1'
Retire Agent: zpool_vdev_replace 'virtio-DISK0' with spare 'virtio-DISKSMALL4-part1'
Retire Agent: zpool_vdev_replace 'virtio-DISK0' with spare 'virtio-DISK2-part1'
Diagnosis Engine: resource event 'resource.fs.zfs.statechange'
Retire Agent: zfs_retire_recv: 'resource.fs.zfs.statechange'

While it's true this can be optimized to only attempt "valid" spares i think we have another, more important, issue to address: i took the previous example and changed virtio-DISK2 phys block size from 512b to 4K, also bumping its size to 11GB:

Snippet from strace -tf -e trace=ioctl zed -Ffv:

Diagnosis Engine: error event 'ereport.fs.zfs.io'
[pid  1325] 10:18:46 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = -1 ENOMEM (Cannot allocate memory)
[pid  1325] 10:18:46 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = 0
Diagnosis Engine: error event 'ereport.fs.zfs.io'
[pid  1325] 10:18:46 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = -1 ENOMEM (Cannot allocate memory)
[pid  1325] 10:18:46 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = 0
Diagnosis Engine: error event 'ereport.fs.zfs.io'
[pid  1325] 10:18:46 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = -1 ENOMEM (Cannot allocate memory)
[pid  1325] 10:18:46 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = 0
Diagnosis Engine: error event 'ereport.fs.zfs.io'
[pid  1325] 10:18:46 ioctl(5, 0x5a81Process 1522 attached
Diagnosis Engine: timer fired (0x7f0924036c90)
Diagnosis Engine: solving fault 'fault.fs.zfs.vdev.io'

zed_fault_event:
	uuid: 8d1cee0a-9ec8-498e-b0f9-22cff8c6a608
	class: fault.fs.zfs.vdev.io
	code: ZFS-8000-FD
	certainty: 100
	scheme: zfs
	pool: 15745788201175468001
	vdev: 14357789399624531578 

Retire Agent: zfs_retire_recv: 'list.suspect'
 <unfinished ...>
[pid  1327] 10:18:48 ioctl(5, 0x5a04, 0x7f0929a706b0) = -1 EEXIST (File exists)
[pid  1327] 10:18:48 ioctl(5, 0x5a05, 0x7f0929a70680) = -1 ENOMEM (Cannot allocate memory)
[pid  1327] 10:18:48 ioctl(5, 0x5a05, 0x7f0929a70680) = 0
[pid  1327] 10:18:48 ioctl(5, 0x5a27, 0x7f0929a70640) = 0
Retire Agent: matched vdev 14357789399624531578
[pid  1327] 10:18:48 ioctl(5, 0x5a0d <unfinished ...>
[pid  1325] 10:18:48 <... ioctl resumed> , 0x7ffd9d19eb20) = 0
Diagnosis Engine: case solved (8d1cee0a-9ec8-498e-b0f9-22cff8c6a608)
Diagnosis Engine: removing timer (0x7f0924036c90)
[pid  1522] 10:18:48 +++ exited with 0 +++
[pid  1325] 10:18:48 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = 0
[pid  1325] 10:18:48 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = 0
[pid  1325] 10:18:48 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = 0
[pid  1325] 10:18:48 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = 0
[pid  1325] 10:18:48 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = 0
[pid  1325] 10:18:48 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = 0
[pid  1325] 10:18:48 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = 0
[pid  1327] 10:18:48 <... ioctl resumed> , 0x7f0929a70790) = 0
[pid  1325] 10:18:48 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = 0
Retire Agent: zpool_vdev_fault: vdev 14357789399624531578 on 'testpool'
Retire Agent: zpool_vdev_replace 'virtio-DISK0' with spare 'virtio-DISKSMALL2-part1'
[pid  1327] 10:18:48 ioctl(5, 0x5a0e <unfinished ...>
[pid  1325] 10:18:48 ioctl(5, 0x5a81, 0x7ffd9d19eb20) = 0
[pid  1325] 10:18:48 ioctl(5, 0x5a81 <unfinished ...>
[pid  1327] 10:18:48 <... ioctl resumed> , 0x7f0929a706f0) = -1 EOVERFLOW (Value too large for defined data type)
Retire Agent: zpool_vdev_replace 'virtio-DISK0' with spare 'virtio-DISKSMALL3-part1'
[pid  1327] 10:18:48 ioctl(5, 0x5a0e, 0x7f0929a706f0) = -1 EOVERFLOW (Value too large for defined data type)
Retire Agent: zpool_vdev_replace 'virtio-DISK0' with spare 'virtio-DISKSMALL4-part1'
[pid  1327] 10:18:48 ioctl(5, 0x5a0e, 0x7f0929a706f0) = -1 EOVERFLOW (Value too large for defined data type)
Retire Agent: zpool_vdev_replace 'virtio-DISK0' with spare 'virtio-DISK2-part1'
[pid  1327] 10:18:48 ioctl(5, 0x5a0e, 0x7f0929a706f0) = -1 EDOM (Numerical argument out of domain)
Retire Agent: zpool_vdev_replace 'virtio-DISK0' with spare 'virtio-DISKSMALL5-part1'
[pid  1327] 10:18:48 ioctl(5, 0x5a0e, 0x7f0929a706f0) = -1 EOVERFLOW (Value too large for defined data type)
Retire Agent: zpool_vdev_replace 'virtio-DISK0' with spare 'virtio-DISKSMALL6-part1'
[pid  1327] 10:18:48 ioctl(5, 0x5a0e, 0x7f0929a706f0) = -1 EOVERFLOW (Value too large for defined data type)
Retire Agent: zpool_vdev_replace 'virtio-DISK0' with spare 'virtio-DISKSMALL7-part1'
[pid  1327] 10:18:48 ioctl(5, 0x5a0e, 0x7f0929a706f0) = -1 EOVERFLOW (Value too large for defined data type)
Diagnosis Engine: resource event 'resource.fs.zfs.statechange'
Retire Agent: zfs_retire_recv: 'resource.fs.zfs.statechange'

The pool remains without redundancy, because we attempted to replace "virtio-DISK0" (512b, 10GB) with "virtio-DISK2" (4K, 11GB) without specifying a custom ashift value:

Retire Agent: zpool_vdev_replace 'virtio-DISK0' with spare 'virtio-DISK2-part1'
[pid  1327] 10:18:48 ioctl(5, 0x5a0e, 0x7f0929a706f0) = -1 EDOM (Numerical argument out of domain)

Replacing the disk manually works if we override ashift value from the command line:

root@linux:~# zpool status
  pool: testpool
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
	repaired.
  scan: scrub repaired 0B in 0h0m with 0 errors on Sun Nov 12 10:18:48 2017
config:

	NAME                   STATE     READ WRITE CKSUM
	testpool               DEGRADED     0     0     0
	  mirror-0             DEGRADED     0     0     0
	    virtio-DISK0       FAULTED     22   471     0  too many errors
	    virtio-DISK1       ONLINE       0     0     0
	  mirror-1             ONLINE       0     0     0
	    virtio-DISKSMALL0  ONLINE       0     0     0
	    virtio-DISKSMALL1  ONLINE       0     0     0
	spares
	  virtio-DISKSMALL2    AVAIL   
	  virtio-DISKSMALL3    AVAIL   
	  virtio-DISKSMALL4    AVAIL   
	  virtio-DISK2         AVAIL   
	  virtio-DISKSMALL5    AVAIL   
	  virtio-DISKSMALL6    AVAIL   
	  virtio-DISKSMALL7    AVAIL   

errors: No known data errors
root@linux:~# zpool remove testpool virtio-DISK2
root@linux:~# zpool replace testpool virtio-DISK0 virtio-DISK2
cannot replace virtio-DISK0 with virtio-DISK2: new device has a different optimal sector size; use the option '-o ashift=N' to override the optimal size
root@linux:~# zpool replace testpool virtio-DISK0 virtio-DISK2 -o ashift=9
root@linux:~# zpool status
  pool: testpool
 state: ONLINE
  scan: resilvered 0B in 0h0m with 0 errors on Sun Nov 12 10:25:20 2017
config:

	NAME                   STATE     READ WRITE CKSUM
	testpool               ONLINE       0     0     0
	  mirror-0             ONLINE       0     0     0
	    virtio-DISK2       ONLINE       0     0     0
	    virtio-DISK1       ONLINE       0     0     0
	  mirror-1             ONLINE       0     0     0
	    virtio-DISKSMALL0  ONLINE       0     0     0
	    virtio-DISKSMALL1  ONLINE       0     0     0
	spares
	  virtio-DISKSMALL2    AVAIL   
	  virtio-DISKSMALL3    AVAIL   
	  virtio-DISKSMALL4    AVAIL   
	  virtio-DISKSMALL5    AVAIL   
	  virtio-DISKSMALL6    AVAIL   
	  virtio-DISKSMALL7    AVAIL   

errors: No known data errors
root@linux:~# 

Tested on 62df1bc.