ZFS corruption related to snapshots post-2.0.x upgrade #12014
Comments
|
Two other interesting tidbits... When I do the reboot after this issue occurs, the mounting of the individual zfs datasets is S L O W. Several seconds each, and that normally just flies by. After scrubbing, it is back to normal speed of mounting. The datasets that have snapshot issues vary with each one. Sometimes it's just one, sometimes many. But var is almost always included. (Though its parent, which has almost no activity ever, also is from time to time, so that's odd.) |
|
Same symptoms here, more or less. See also issue #11688. |
|
I also have the symptom with the corrupted snapshots, without kernel panics so far. So far it only affected my Debian system with Linux 5.10 and zfs 2.0.3 (I've turned the server off for today, I can check the exact versions tomorrow). Also, while the system has the 2.0.3 zfs utils + module, the pool is still left on 0.8.6 format. I wasn't able to execute On the corrupted system, after I got the mail from ZED, I manually ran a scrub at first, after which the I've rebooted the server into an Ubuntu 20.10 live with zfs 0.8.4-1ubuntu11 The errors didn't seem to affect the data on the zvols (all 4 affected snapshots are of zvols). The zvols are used as disks for VMs with ext4 on them. I have two other Ubuntu 21.04 based Systems with zfs-2.0.2-1ubuntu5 which are not affected until now. However, they have their pools already upgraded to 2. All are snapshotted with sanoid and have the datasets encrypted.
EDIT:
EDIT 2:
|
|
I'm seeing this too, on Ubuntu 21.04, also using zfs encryption I have znapzend running, and it makes a lot of snapshots. Sometimes, some of them are bad, and can't be used (for example, attempting to send them to a replica destination fails). I now use the In the most recent case (this morning) I had something like 4300 errors (many more than I'd seen previously). There are no block-level errors (read/write/cksum). They're cleared after destroying the affected snapshots and scrubbing (and maybe a reboot, depending on .. day?) Warning! Speculation below:
|
|
@jgoerzen Can you
In my case, #11688 (which you already reference), I've discovered that rebooting "heals" the snapshot -- at least using the patchset I mentioned there |
|
I'll be glad to. Unfortunately, I rebooted the machine yesterday, so I expect it will be about a week before the problem recurs. It is interesting to see the discussion today in #11688. The unique factor about the machine that doesn't work for me is that I have encryption enabled. It wouldn't surprise me to see the same thing here, but I will of course wait for it to recur and let you know. |
|
Hello @aerusso, The problem recurred over the weekend and I noticed it this morning. Unfortunately, the incident that caused it had already expired out of the It should be noted that my hourly snap/send stuff runs at 17 minutes past the hour, so that may explain this timestamp correlation. zpool status reported: Unfortunately I forgot to attempt to do a So I think that answers the question. After a reboot but before a scrub, the |
|
I have similar symptoms, on an encrypted single-ssd ubuntu 21.04 boot pool, using stock zfs from ubuntu's repos. Deleting the affected snapshots and scrubbing previously cleared the errors, but on reoccurence, repeated scrubbing (without deleting them) caused a deadlock. My system has ECC memory, so it's probably not RAM related.
|
|
@cbreak-black Was there a system restart between the occurrence of the corrupted snapshot and the problems? Restarting has "fixed" this symptom for me (though you will need to scrub twice for the message to disappear, I believe). I have a suspicion that this may be a version of #10737 , which has an MR under way there. The behavior I am experiencing could be explained by that bug (syncoid starts many I'm holding off on trying to bisect this issue (at least) until testing that MR. (And all the above is conjecture!) |
|
@aerusso No, without a restart I got into the scrub-hang, and had to restart hard. Afterwards, the scrub finished, and several of the errors vanished. The rest of the errors vanished after deleting the snapshots and scrubbing again. |
|
Can I join the club too? #10019 |
|
@InsanePrawn I can't seem to find commit 4d5b4a33d in any repository I know of (and neither can github, apparently, either). However, in your report you say this was a "recent git master" and the commit I'm currently betting on being guilty is da92d5c, which was committed in November of the previous year, so I can't use your data point to rule out my theory! Also, it sounds like you didn't have any good way to reproduce the error --- however, you were using a test pool. Compared to my reproduction strategy (which is just, turn my computer on and browse the web, check mail, etc.) it might be easier to narrow in on a test case (or might have been easier a year and a half ago, when this was all fresh). Anyway, if you have any scripts or ideas of what you were doing that caused this besides "snapshots being created and deleted every couple minutes", it might be useful too. (I already tried lots of snapshot creations and deletions during fio on several datasets in a VM). |
|
Yeah, idk why I didn't go look for the commit in my issue - luckily for us, that server (and pool; it does say yolo, but it's my private server's root pool. it's just that i won't cry much if it breaks; originally due to then unreleased crypto) and the git repo on it still exist. Looks like 4d5b4a33d was two systemd-generator commits by me after 610eec4 |
|
FWIW the dataset the issue appeared on was an empty filesystem (maybe a single small file inside) dataset that had snapshots (without actual fs activity) taken in quick intervals (somewhere between 30s and 5m intervals) in parallel with a few (5-15) other similarly empty datasets. The pool is a raidz2 on 3.5" spinning SATA disks. Edit: Turns out the dataset also still exists, the defective snapshot however does not anymore. I doubt that's helpful? |
|
@InsanePrawn Does running the zrepl workload reproduce the bug on 2.0.5 (or another recent release?) I don't think the snapshot is terribly important --- unless you're able to really dig into it with zdb (which I have not developed sufficient expertise to do). Rather, I think it's the workload, hardware setup, and (possibly, but I don't understand the mechanism at all) the dataset itself. Encryption also is a common theme, but that might just affect the presentation (i.e., there's no MAC to fail in the unencrypted, unauthenticated, case). Getting at |
|
I've since added redundancy to my pool (it's now a mirror with two devices), and disabled autotrim. The snapshot corruption still happens. Still don't know what is causing it. And I also don't know if the corruption happens when creating the snapshot, and only later gets discovered (when I try to zfs send the snapshots), or if snapshots get corrupted some time in between creation and sending. |
|
@cbreak-black Can you enable the all-debug.sh ZEDlet, and put the temporary directory somewhere permanent (i.e., not the default of This will get the output of I'll repeat this here: if anyone gets me a reliable reproducer on a new pool, I have no doubt we'll be able to solve this in short order. |
|
Just mentioning here that we saw this on TrueNAS 12.0-U5 with OpenZFS 2.0.5 as well -- see #11688 (comment) for our story. |
|
Since I don't see anyone mentioning it here yet, #11679 contains a number of stories about the ARC getting confused when encryption is involved and, in a very similar looking illumos bug linked from there, eating data at least once. |
|
@gamanakis Nope, I'm not using raw (-w). |
|
it's present in v2.1.1 as well: |
|
@aerusso you wrote that da92d5c may be the cause of this issue. My workstation at work panics after a couple of days and I need to reset it. Could you provide a branch of 2.1.1 with this commit reverted (as revert causes merge conflicts I can't fix myself) so I could test if the machine no longer crashes? |
|
@phreaker0 Unfortunately, the bug that da92d5c introduced (#10737) was fixed by #12299, which I believe is present in all maintained branches now. It does not fix #11688, (which I suspect is the same as this bug). I'm currently running 0.8.6 on Linux 5.4.y, and am hoping to wait out this bug (I don't have a lot of time right now, or for the foreseeable future). But, If you have a reliable reproducer (or a whole lot of time) you could bisect while running 5.4 (or some other pre-5.10 kernel). I can help anyone who wants to do that. If we can find the guilty commit, I have no doubt this can be resolved. |
@robn link from issue 13755 to some more recent point in this issue 12014? |
|
I Now use a different approach with sanoid/syncoid. I use sendoptions to recusive send the snapshots. Now i don't have errors on the enrypted pool. Its now running 6 days without errors. before i always had errors within 5 days. I will keep you informed if it keeps running without errors. |
|
Guys, I can confirm. If i use the following i get encryption errors within the 5 days. syncoid --recursive But when i use it like this the errors are not happening. syncoid --sendoptions="R" I am using it now to send it in raw mode. So now the errors are not happening anymore. |
|
Just throwing this out there for those of you doing automated tests (@HankB and @Germano0) and are trying to reduce the amount of time it takes to reproduce the issue: |
Agreed. The script I wrote eventually goes 5 deep for nested data sets with a total of 35 in my present test.
And writes files throughout the pool (using I loop each pass with 750s sleep in between with the syncoid command and modify script in their own loops. Since the syncoid and modify script take different times to complete, that results in un-synchronized overlap between the two. I'm open to other suggestions, but in the interest of reproducible results, I can't make too many changes at this point. I'd suggest that someone interested in additional strategies try them either on available H/W or VMs and this can reduce the overall time to get an answer. best, Edit: I'm adding a daily scrub to the testing process (manually invoked.) It occurs to me that the corruption could be happening and just not detected. |
|
I also have the symptom with the corrupted snapshots, but with one kernel panic 😒: Sadly the system with this specific RAIDZ3 and encryption configuration is quite new. Its an very nasty bug, its corrupts the snapshots (or other metadata) slowly and at some time there it forces you to scrub and (forcefully sadly reboot) the system... |
|
On Tue, Feb 4, 2025 at 2:11 PM Michael Kirgus ***@***.***> wrote:
How can I assist to solve the problem?
Hi Michael,
I don't know what data to collect that would be helpful. My focus is on
identifying steps that reliably reproduce the issue in an environment that
can be shared. (I can reproduce this on my laptop in a matter of hours but
don't care to share the contents of my storage with the world.) To that end
I'm working on scripts to populate a pool with synthetic data and
subsequently modify it and then use `syncoid` to send the pool to other
storage. It has been a little frustrating and it is taking a lot of time. I
can usually provoke the corruption within two days but when no corruption
occurs, I need to run a *lot* longer to insure the corruption is not
occurring. My most recent test was with the kernel and ZFS versions listed
in this issue and that ran for over 10 days and over 1100 `syncoid`
invocations and did not result in corruption.
At the moment I'm revisiting my methods to see if I can improve on the
ability to reveal the issue. If you have any ideas for conditions that tend
to cause this (if you're experiencing the same issue) Feel free to share
them with me.
You can review my efforts at https://github.com/HankB/provoke_ZFS_corruption.
Feel free to contribute with informative issues or PRs. Better yet, if you
have any spare H/W available, try duplicating my tests (or adapting the
synthetic data and processing to your typical work flow.)
I'm not a kernel dev or ZFS expert by any stretch of the imagination so
don't be afraid to suggest something because you think I already thought of
it.
best,
… Message ID: ***@***.***>
--
Beautiful Sunny Winfield
|
|
it seems plausible that the issue is some kind of lock or resource contention, so setting up conditions that provoke or exacerbate that presumably may make it easier to reproduce. I know both of these were true in my pool, back when i saw the problem
Also perhaps artificially slow pool storage via vm config. |
|
On Tue, Feb 4, 2025 at 8:40 PM Daniel Carosone ***@***.***> wrote:
it seems plausible that the issue is some kind of lock or resource
contention, so setting up conditions that provoke or exacerbate that
presumably may make it easier to reproduce.
I agree.
I know both of these were true in my pool, back when i saw the problem
- new recursive snapshots taken while the previous is still sending;
try a short snapshot interval, limited send bandwidth
- multiple subtrees getting snapshots; test/a test/b test/c all
getting "independent" snapshots and sends, in the same pool
Also perhaps artificially slow pool storage via vm config.
I'm running on bare metal. I'd be delighted if someone would pick up the
mantle on testing on a VM.
With my latest test I have reduced the time to provoke corruption to less
than a day. Results can be viewed at
https://hankb.github.io/provoke_ZFS_corruption/tests/2025-02-03_methodology/results/
The only potential contention I can spot is a 'stir' (fairly heavy disk I/O
to a lot of files) completing about 2s before the next `sanoid` run. I
think that 2 seconds is not long enough to flush all data to the drive. I
suppose what I should do is stop the timer that normally starts the sanoid
snapshots and kick sanoid off manually (or perhaps not use `sanoid` and
just create snapshots directly.) Tomorrow.
best,
… Message ID: ***@***.***>
--
Beautiful Sunny Winfield
|
|
I have one pool that hasn't had this issue in 6 days (and from what I can see from the logs didn't have in the previous "restart cycle"). The only difference other than it having less disk I/O is that it has recordsize 128k while the others have recordsize 1M. I suggest taking snapshots every minute instead and why not also run syncoid and stir concurrently and continuously. |
|
I finally found my old tests under another ticket: #11688 (comment) Among other things I wrote:
And there is a test script that quickly produces the issue: What I don't remember, though, is whether the script was run on a freshly restarted and scrubbed dataset or not. |
|
@siilike Thanks for the comments and link to the other issue.
I agree and that's what I intend to explore with my next test. I also plan to modify the scripts to halt further operations as soon as corruption is identified.
I'll give that a try as well. I particularly like sending the data to |
|
When setting up syncoid/sanoid, I also observed issues with fast syncoid pulls and added a wait time of 10 seconds between datasets. Might be totally not correlated, as I am not using pull_snapshots.sh#!/bin/bash
################################################################################
#
# Shell script to pull zfs snapshots with syncoid using a list of key:value pairs
# Works with sendoptions=R, where syncoid's --recursive flag cannot be used
#
# IMPORTANT: Use bash, not sh! e.g.: > bash pull_snapshots.sh
#
################################################################################
# Exit as soon as a command fails
set -e
# Accessing an empty variable will yield an error
set -u
FILE_SNAPSHOTS="$1"
# FILE_SNAPSHOTS=snapshots.txt
pull_syncoid() {
ZFS_SOURCE=$1
ZFS_TARGET=$2
# we use </dev/null to redirect standard input for the syncoid (ssh) command
# which might otherwise consume the while loop's input
syncoid --no-sync-snap --no-privilege-elevation \
--sshkey /home/recvuser/.ssh/id_ed25519 \
--sshconfig=/home/recvuser/.ssh/config \
--sendoptions=Rw \
'proxmox':"$ZFS_SOURCE" "$ZFS_TARGET"
# optionally add --debug above;
}
while IFS=: read -u 3 -r zfs_source zfs_target; do
# skip commented lines
[[ "$zfs_source" =~ ^[[:space:]]*# ]] && continue
# skip empty lines
[ -z "$zfs_source" ] && continue
echo "$(date +%Y-%m-%d) $(date +%H:%M) Pulling: $zfs_source -> $zfs_target"
pull_syncoid "$zfs_source" "$zfs_target";
# wait 2 seconds between pulls
sleep 10
done 3< "$FILE_SNAPSHOTS"This script is called via #!/bin/bash
# Exit as soon as a command fails
set -e
# Accessing an empty variable will yield an error
set -u
# To avoid having to type the absolute path to a command
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin"
# If started as root, then re-start as user "mastodon":
if [ "$(id -u)" -eq 0 ]; then
echo "Running as root. This should never be reached.";
exit 1
fi
echo "This runs as user $(id -un)";
# prints "recvuser"
TODAY=$(date -I)
if [[ ( -f /home/recvuser/maintenance) ]]; then
echo "Maintenance mode, aborting autostart."
exit 1
fi
echo "Starting pull snapshots at ${TODAY}"
cd ~
bash pull_snapshots.sh "snapshots.txt"
echo "Checking Scrub Status.."
zpool status -v
echo "All Done. $(date +%Y-%m-%d) $(date +%H:%M).";
echo "Goodbye.";
# check if scrub is currently running;
# do nothing if it is, shutdown otherwise
if [[ $(zpool status | grep -q "scrub in progress") || ( -f /home/recvuser/maintenance) ]]; then
echo "Scrub still running or maintenance mode, not shutting down"
else
sudo /sbin/shutdown
fi;I don't think I observed the error, although I sometimes get
I thought this was related to an unstable internet connection, as resume always works and neither sending nor receiving pools are corrupted. Might not be the issue described here (?). |
|
In my case TrueNAS does the replication and snapshotting with his #middleware, but in the end it comes to changes at the metadata and I/O which has to be handled by ZFS and the hardware somehow.
Yes, it is "random"...but i had errors on datasets with static configuration data which are not touched at all and had errors.
I want to mention that all my pools are SSD only with an LSI HBA...the iowait of the system is always under 5 % or so... I will try to setup an test environment with a encrypted pool and 2 virtual systems to replicate between each of them. |
|
I tried to reproduce the issue with the script, but it did not work, so I likely ran it when the pool had gone bad already. |
|
@siilike What pool settings are used in your environment? I started with this (this is default setting in TrueNAS): I wonder if the sync property can make an difference.
I executed the test script twice, but it seems not to corrupt any metadata so far. |
|
As far as I understand, the issue is that syncoid does something that triggers this bug in a particular pool. Other pools on the same machine are not affected. The script I posted works for triggering the creation of invalid snapshots in a pool that has already gone bad. But it does not cause the pool to go bad i.e. it does not trigger the actual bug. |
I think the root cause is the zfs send in combination with recursive snapshots. If an snapshot that was recursive created and then an child dataset was send. If in this situation one of his parents are send, modified, deleted or (enumerated?), this seems to trigger this metadata corruption. Sadly i am not an ZFS developer, but the productive system was running fine for years without the encryption. |
Just confirming: we saw this issue with pyznap, and never used syncoid or sanoid.
Just confirming: we never did any recursive snapshots. Also confirming: new replication to another server without encryption enabled eliminated the problem. |
|
Suggestions to run my test operations have borne fruit in that the first test produced corruption in about 15 minutes. Current results at https://hankb.github.io/provoke_ZFS_corruption/ I've also mentioned this in the thread at https://zfsonlinux.topicbox.com/groups/zfs-discuss/Ta136a4d5eb5a9af8-M23a47f6202c782813cda2d84/reproducing-corruption-related-to-native-encryption |
Thank you for this; I'm getting the impression that there has been a serious correlation!=causation problem around this bug recently, no matter how many times I remind people that syncoid doesn't hit any APIs, it literally just orchestrates underlying OpenZFS commands! =) |
Mr Salter, I'm continuing to use I'm running the snapshot command + a process that modifies random files in a tight loop. Another process runs best, |
|
@HankB I was playing with your scripts and the following occurred: Trying to delete the "%recv" snapshot, created by "receive -s" which syncoid uses by default, led to the same errors (of note: no kernel panic). After a couple of scrubs and rebooting the VM the pool was ok. It might be worth noting that syncoid uses "receive -s" by default, which got me into thinking perhaps this may be the culprit. @wohali from what I can tell pyznap does not use "receive -s" by default. Do you know if you were using this feature? |
|
@gamanakis I experienced the snapshot corruption on an encrypted zfs dataset regularly, and I only used pyznap with mostly-default settings. So it seems unlikely that |
|
@gamanakis Thank you for your help.
I've seen the I've looked for and not seen I/O errors. If present that would cast doubt that the problem is a ZFS bug and rather point to I/O errors instead.
I've never bothered to try to recover as my interest is in provoking the corruption reproducibly and quickly.
I don't see how something on the receiving end could cause this as it's decoupled from the send. Once the stream leaves the I need to add a script to loop the script that manages snapshots and provide better instructions now that I'm settling on a procedure. I also believe I have provoked the corruption on FreeBSD and need to investigate that and report if that seems likely. |
|
Is there anything I can do to help you? |
|
@Germano0 Yes, probably. I will, of course, defer to the ZFS and kernel devs for any direction they would like this to take. My desire was to see if there was a way to reproducibly provoke corruption as the first step in tracking down the regression. My repo (https://github.com/HankB/provoke_ZFS_corruption) is a bit rough but provides that capability. I'm working to improve that, particularly the instructions. The next step is to determine which commit introduced the regression via git bisect. I have not yet started that. Anyone following my process will likely find errors and omissions and things that are just not clear. Those should probably be referred to me at my repo and not here. It would certainly help to have others try to reproduce my results. I'll repeat my offer to either start another issue relevant to this issue in OpenZFS or continue discussion at my repo if anyone thinks this is unnecessarily cluttering this issue. |
It was likely enabled, yes. I don't recally for sure, but we move enough data that restarting a full transfer would be very bad. I agree that I don't see how something on the receiving end could trigger this bug, though. |
|
When I was seeing it with |
System information
Describe the problem you're observing
Since upgrading to 2.0.x and enabling crypto, every week or so, I start to have issues with my zfs send/receive-based backups. Upon investigating, I will see output like this:
Of note, the
<0xeb51>is sometimes a snapshot name; if Izfs destroythe snapshot, it is replaced by this tag.Bug #11688 implies that zfs destroy on the snapshot and then a scrub will fix it. For me, it did not. If I run a scrub without rebooting after seeing this kind of
zpool statusoutput, I get the following in very short order, and the scrub (and eventually much of the system) hangs:However I want to stress that this backtrace is not the original cause of the problem, and it only appears if I do a scrub without first rebooting.
After that panic, the scrub stalled -- and a second error appeared:
I have found the solution to this issue is to reboot into single-user mode and run a scrub. Sometimes it takes several scrubs, maybe even with some reboots in between, but eventually it will clear up the issue. If I reboot before scrubbing, I do not get the panic or the hung scrub.
I run this same version of ZoL on two other machines, one of which runs this same kernel version. What is unique about this machine?
I made a significant effort to rule out hardware issues, including running several memory tests and the built-in Dell diagnostics. I believe I have rules that out.
Describe how to reproduce the problem
I can't at will. I have to wait for a spell.
Include any warning/errors/backtraces from the system logs
See above
Potentially related bugs
arc_buf_destroyis in silent corruption for thousands files gives input/output error but cannot be detected with scrub - at least for openzfs 2.0.0 #11443. The behavior described there has some parallels to what I observe. I am uncertain from the discussion what that means for this.The text was updated successfully, but these errors were encountered: