Permanent errors <metadata>:<0x714> #14693

Redsandro · 2023-03-30T15:45:50Z

Redsandro
Mar 30, 2023

I'm trying to use some single disks as offline backup using a SATA to USB 3 dock. While this is not the main raison d'être for ZFS, I figured this is a good use of copies=3. This is how I set up the disks:

for OPTION in \
    mountpoint=none \
    copies=3 \
    compression=lz4 \
    xattr=sa \
    acltype=posixacl \
    atime=off \
    relatime=off \
    redundant_metadata=most
do sudo zfs set $OPTION $POOL
done

While filling the pools with data, I've ended up with "Permanent errors" three times on two different disks now. I'm getting the following message from zpool:

$ sudo zpool status -v
  pool: backup32
 state: ONLINE
status: One or more devices has experienced an error resulting in data
	corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
	entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub in progress since Thu Mar 30 16:21:51 2023
	5.77M scanned at 844K/s, 148K issued at 21.2K/s, 1.95T total
	0B repaired, 0.00% done, no estimated completion time
config:

	NAME                                          STATE     READ WRITE CKSUM
	backup32                                      ONLINE       0     0     0
	  ata-Hitachi_000000000000000_00000000000000  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        <metadata>:<0x0>
        <metadata>:<0x1>
        <metadata>:<0x714>
        backup32:<0x0>

Regarding: "Restore the file in question if possible. Otherwise restore the entire pool from backup;" first, I can't figure out what the "file in question" is. I consider the message not helpful in that regard, and perhaps it can be improved. Second, this is happening near the end of filling a brand new pool, so there is no backup.

I have destroyed the pool and restarted for the third time now, this time on a different disk. The process of filling the pool takes hours, so I decided to see if this behavior is indicative of something. After searching, I found that this Q/A was most similar. However, my setup is really very simple. No external log/cache. No multi-disk. No ZFS send. I'm trying to find out:

If I can clear this error somehow and just lose 1 file in my backup, which will get repaired next time I use rsync;
if this is a coincidence or a pattern; and
if the problem is with ZFS, the disks, the dock, or the PC, or perhaps with a specific file it is trying to copy.

Although the disks had not failed in the past, they do come from an older multi-disk ZFS pool and were replaced merely as a precaution because they are 5 years old, they may be prone to age-based errors. I just want to use them as offline single disk backups with some internal redundancy (copies=3) if I can, rather than discarding them as e-waste.

Unfortunately I didn't record the exact error logs from previous attempts, but they were the same format: <metadata>:<0xSomething>.

sudo zdb -dd backup32 0x0
Dataset mos [META], ID 0, cr_txg 4, 2.79M, 499 objects

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
         0    2   128K    16K   168K     512   976K   25.56  DMU dnode

$ sudo zdb -dd backup32 0x1
Dataset mos [META], ID 0, cr_txg 4, 2.79M, 498 objects

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
         1    1   128K    16K  13.5K     512    32K  100.00  object directory

$ sudo zdb -dd backup32 0x174
Dataset mos [META], ID 0, cr_txg 4, 2.79M, 498 objects

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
zdb: dmu_object_info() failed, errno 2
$ sudo zdb -dd backup32 0x174
Dataset mos [META], ID 0, cr_txg 4, 2.81M, 500 objects

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
zdb: dmu_object_info() failed, errno 2
$ sudo zdb -ddd backup32 0x174
Dataset mos [META], ID 0, cr_txg 4, 1.90M, 333 objects

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
zdb: dmu_object_info() failed, errno 2

I don't know what it means, but the values on that last one keep changing.

Answered by IvanVolosyuk

Mar 30, 2023

One more interesting question - do you use ZFS encryption? It might have weird interaction with 3 copies.

View full answer

rincebrain · 2023-03-30T19:52:21Z

rincebrain
Mar 30, 2023
Collaborator

Are you seeing any IO errors at all, or just suddenly errors on disk?

You might want to not set redundant_metadata=most when you're setting copies=3, that seems a bit counterproductive.

4 replies

Redsandro Mar 30, 2023
Author

2/3 times no IO errors at all, one time a few read errors, which I disregarded as a fluke.

You might want to not set redundant_metadata=most when you're setting copies=3, that seems a bit counterproductive.

Thank you for the advise, ~~I had not considered this~~. I assumed that copies=3 includes 3 copies of metadata, but for offline backup purposes, it may not make sense to set this to anything other than all. I will change this setting to all.

Redsandro Mar 30, 2023
Author

Update: The resilver scrub I started to see if it would make the pool state clearable just suspended minutes before being finished, and now there is a ton of checksum errors. I have not observed this before, but this is the first time I'm attempting a scrub rather than a destroy and recreate.

  pool: backup32
 state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC
  scan: scrub in progress since Thu Mar 30 16:21:51 2023
	1.95T scanned at 93.4M/s, 1.85T issued at 88.8M/s, 1.95T total
	0B repaired, 95.14% done, 00:18:35 to go
config:

	NAME                                          STATE     READ WRITE CKSUM
	backup32                                      ONLINE       0     0     0
	  ata-Hitachi_000000000000000_00000000000000  ONLINE      11     0   687
errors: List of errors unavailable: pool I/O is currently suspended

This amount of checksum errors makes no sense to me. I am starting to wonder if the problem is actually with the AMD ZEN Chipset similar to #11741. I guess I should try this whole thing again on an Intel chipset to be sure (but I really don't want to play this game again).

rincebrain Mar 30, 2023
Collaborator

I would agree that copies=3 probably implies metadata copies=3, since the metadata stuff is defined as being "copies on top of the amount set up to the maximum possible", but I figured I'd suggest it because it's ambiguous which should win of a setting to reduce that number and a setting to increase it.

Redsandro Mar 30, 2023
Author

I agree it is definitely ambiguous, and since the errors mention metadata specifically, I should just set them to all so I'll be swimming in precautionary copies, if that's what it takes.

rincebrain · 2023-03-30T20:50:48Z

rincebrain
Mar 30, 2023
Collaborator

Nor would I envy you having to test that. I'm not really sure what ZFS can do better if that's the case though, at the point where the hardware doesn't error, it's unclear what to do differently... That said, I was about to ask what distro/kernel/openzfs version, so I still will.

…

On Thu, Mar 30, 2023, 4:34 PM Redsandro ***@***.***> wrote: *Update:* The resilver scrub I started to see if it would make the pool state clearable just suspended minutes before being finished, and now there is a ton of checksum errors. I have not observed this before, but this is the first time I'm attempting a scrub rather than a destroy and recreate. pool: backup32 state: SUSPENDED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC scan: scrub in progress since Thu Mar 30 16:21:51 2023 1.95T scanned at 93.4M/s, 1.85T issued at 88.8M/s, 1.95T total 0B repaired, 95.14% done, 00:18:35 to go config: NAME STATE READ WRITE CKSUM backup32 ONLINE 0 0 0 ata-Hitachi_000000000000000_00000000000000 ONLINE 11 0 687 errors: List of errors unavailable: pool I/O is currently suspended — Reply to this email directly, view it on GitHub <#14693 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUI7KVX6WLNXLC5G5CIB3W6XU35ANCNFSM6AAAAAAWNPRRBE> . You are receiving this because you commented.Message ID: ***@***.***>

1 reply

Redsandro Mar 30, 2023
Author

Yes, of course, I should have said that from the start. Linux Mint flavor Ubuntu 22.04.1 kernel 5.19.0-38-generic 64 bit with zfs-2.1.5-1ubuntu6~22.04.1 and zfs-kmod-2.1.5-1ubuntu6.

It will probably take some time before I have the hardware setup and mental peace to retry this workflow on an Intel chipset. Let's call it plan C. If there are no other perspectives, that's what needs to be done.

IvanVolosyuk · 2023-03-30T23:53:58Z

IvanVolosyuk
Mar 30, 2023

One more interesting question - do you use ZFS encryption? It might have weird interaction with 3 copies.

…

On Fri, 31 Mar 2023, 10:08 Redsandro, ***@***.***> wrote: Yes, of course, I should have said that from the start. Linux Mint flavor Ubuntu 22.04.1 kernel 5.19.0-38-generic 64 bit with zfs-2.1.5-1ubuntu6~22.04.1 and zfs-kmod-2.1.5-1ubuntu6. It will probably take some time before I have the hardware setup and mental peace to retry this workflow on an Intel chipset. Let's call it plan C. If there are no other perspectives, that's what needs to be done. — Reply to this email directly, view it on GitHub <#14693 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HMQ5TDXFYWOS5YGBXDW6YHAHANCNFSM6AAAAAAWNPRRBE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

8 replies

rincebrain Mar 31, 2023
Collaborator

2.1.8 is the one with a bug around that, actually. So, uh, don't use that.

cf. the entire changelog of 2.1.9 being "reverted that" and a Linux 6.2 backport.

Redsandro Mar 31, 2023
Author

On a side note, does this mean that copies=2 is sort of busted for encrypted pools, because you can have all the copies you want, but only a single metadata block is used for the encryption necessities for that block? It would imply a single point of failure, perhaps technically necessary, but semantically unintuitive.

rincebrain Mar 31, 2023
Collaborator

I think you're misunderstanding what I described.

Entries for blocks in metadata tables have 3 fields for where they are, essentially. One of those fields got repurposed for storing encryption metadata, so you can only have up to 2 copies of any encrypted block, because [gestures].

That doesn't mean the block containing that entry only has 1 copy.

Redsandro Apr 1, 2023
Author

I think my misunderstanding is with ZFS itself. I'd need a visual schematic to really understand how the number of blocks, tables, entries, copies and metadatas are related to each other and how they change according to copies=, redundant_metadata= and encryption=. That said, today I'm trying a fresh batch using the previously recommended copies=2 and redundant_metadata=all. If it fails again, it would be a strong indicator that the problem is with the hardware, either the dock for the disk, or the ZEN chipset.

Redsandro Aug 18, 2023
Author

I can now safely confirm over a longer period of observation that copies=3 was indeed the problem. Everything works as expected with copies=2 in all cases.

Misleadingly it may not happen until the disk is 2/3rds filled, so I wanted to mark this as answered so that others may save some time.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Permanent errors <metadata>:<0x714> #14693

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 13 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Permanent errors <metadata>:<0x714> #14693

Redsandro Mar 30, 2023

Replies: 3 comments · 13 replies

rincebrain Mar 30, 2023 Collaborator

Redsandro Mar 30, 2023 Author

Redsandro Mar 30, 2023 Author

rincebrain Mar 30, 2023 Collaborator

Redsandro Mar 30, 2023 Author

rincebrain Mar 30, 2023 Collaborator

Redsandro Mar 30, 2023 Author

IvanVolosyuk Mar 30, 2023

rincebrain Mar 31, 2023 Collaborator

Redsandro Mar 31, 2023 Author

rincebrain Mar 31, 2023 Collaborator

Redsandro Apr 1, 2023 Author

Redsandro Aug 18, 2023 Author

Redsandro
Mar 30, 2023

Replies: 3 comments 13 replies

rincebrain
Mar 30, 2023
Collaborator

Redsandro Mar 30, 2023
Author

Redsandro Mar 30, 2023
Author

rincebrain Mar 30, 2023
Collaborator

Redsandro Mar 30, 2023
Author

rincebrain
Mar 30, 2023
Collaborator

Redsandro Mar 30, 2023
Author

IvanVolosyuk
Mar 30, 2023

rincebrain Mar 31, 2023
Collaborator

Redsandro Mar 31, 2023
Author

rincebrain Mar 31, 2023
Collaborator

Redsandro Apr 1, 2023
Author

Redsandro Aug 18, 2023
Author