Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

L2ARC marked as DEGRADED, reboot needed to clear the error #16437

Closed
shodanshok opened this issue Aug 12, 2024 · 1 comment
Closed

L2ARC marked as DEGRADED, reboot needed to clear the error #16437

shodanshok opened this issue Aug 12, 2024 · 1 comment
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@shodanshok
Copy link
Contributor

System information

Type Version/Name
Distribution Name Rocky
Distribution Version 9.4
Kernel Version 5.14.0-427.28.1.el9_4.x86_64
Architecture x86_64
OpenZFS Version 2.1.15

Describe the problem you're observing

After upgrading to the latest kernel and ZFS 2.1.15 version (coming from ZFS 2.1.14) and rebooting, L2ARC was restored but marked as DEGRADED in arc_summary output. On further inspection, the issue was caused by a single l2_cksum_bad (ie: a checksum error on L2ARC) albeit zpool status reported no checksum error at all.

In order to clear the DEGRADED mark I tried to remove and re-add the cache device, to no avail. The only solution was to reboot the server, which reset the l2_cksum_bad counter. From what I can see here, no other means (short of a ZFS module remove or full reboot) exists to clear such counters:

zfs/cmd/arc_summary

Lines 862 to 864 in 9c56b8e

l2_errors = int(arc_stats['l2_writes_error']) +\
int(arc_stats['l2_cksum_bad']) +\
int(arc_stats['l2_io_error'])

This seems problematic because:

  • no real checksum error was reported by zpool status
  • reboot to clear a transient / dubious error seems way too much, especially in the light of commands as zpool clear

Describe how to reproduce the problem

I don't know how the reproduce l2_cksum_bad without zpool checksum errors. The impossibility to clear l2_cksum_bad (or other similar counters as l2_io_error) without a reboot or ZFS module remove/reimport seems self-evident enough.

Include any warning/errors/backtraces from the system logs

Nothing relevant was found in dmesg output.

@shodanshok shodanshok added the Type: Defect Incorrect behavior (e.g. crash, hang) label Aug 12, 2024
shodanshok added a commit to shodanshok/zfs that referenced this issue Aug 13, 2024
Fixes openzfs#16437

This patch introduces a means to reset some ARC statistics without
rebooting the machine or reimporting the zfs module. Please note
that it currently only resets l2_cksum_bad and l2_io_error

Signed-off-by: Gionatan Danti <g.danti@assyoma.it>
@shodanshok shodanshok mentioned this issue Aug 13, 2024
13 tasks
shodanshok added a commit to shodanshok/zfs that referenced this issue Aug 13, 2024
Fixes openzfs#16437

This patch introduces a means to reset some ARC statistics without
rebooting the machine or reimporting the zfs module. Please note
that it currently only resets l2_cksum_bad and l2_io_error

Signed-off-by: Gionatan Danti <g.danti@assyoma.it>
@shodanshok
Copy link
Contributor Author

Closing, because:

  • arc_summary L2ARC HEALTHY / DEGRADED status is going to be remove, as per ARC statistics reset #16441 (comment)
  • L2ARC devices are not scrubbed or checked for errors the same as data drive (this probably means that zpool status READ / WRITE / CKSUM columns have no meaning for cache devices).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant