cannot resolve path '/dev/disk/by-vdev/c1d14-part1': 2 #1646
Comments
|
In fact we do wait 1 second for udev to create the partition You could try increasing the delay here, although 1 second already sounds like an awfully long time. https://github.com/zfsonlinux/zfs/blob/master/cmd/zpool/zpool_vdev.c#L1051 |
|
Yeah, but obviously not long enough to handle the added time for the udev/vdev-thingies. I'm guessing that this will take even longer time on a machine with lots and lots of devices (this machine has 24 disk slots). The comment block for libzfs/libzfs_pool.c:zpool_label_disk_wait() actually states: Depending on the udev rules this may take a few seconds. So waiting just 1 second seems to contradict that comment ;) Anyhow, changing that timeout from 1 second to 10 seconds should solve the problem for now. An alternative could be to call the "udevadm settle" command, but I guess you'd stil have to wait for a bit for the partiton scan to trigger the udev event(s). Btw, instead of doing partition-wait for each disk device it would be faster to first partition all disks and then start waiting for the partition links/devices to show up, but I guess you already knew that :-) |
|
Hmm. This might be a bigger problem than I thought... I'm now doing some live testing, and I yanked a drive while doing heavy IO and replaced it with another drive. I then tried doing zpool replace pool drive to sync up the new one, but I constantly get the "cannot resolve path" error message. The thing is that when the system is maxed out IO-wise creating a partition and doing the rescan can take a LONG time: So after zpool replace has errored out it took 15 seconds for udev to finish its job! I'm more and more leaning towards that the correct solution is doing something like: But of course this poses problems on older distros like Centos/RHEL 5 that has a separate udevsettle command that doesn't grok the --exit-if-exists argument. |
|
@ZNikke It would be useful to run down exactly why udev is taking so long to complete for you. It typically takes just a few milliseconds per device,, under load it may take slightly longer but shouldn't be that bad. I've tested using systems with 100+ devices and 1 second has always been enough. That said, I'm happy to take a patch to do this in parallel. We could partition them all and then wait say 30 seconds. |
|
Oh, that's easy. You just have to do some violent testing.
In my zpool replace issue above, the workaround was to stop all IO and then retry the zpool replace. However, on a loaded disk server (think a loaded Lustre backend thrashed to the max) silencing the IO to be able to perform a zpool replace operation isn't ideal... So yes, on an idle system with quick/nicely behaving controllers things are rather quick. Add IO load and not-so quick IO controllers and then you're in a completely different ballpark :/ I think the main issue has shifted to having zpool replace behaving in a reliable manner, parallelism for zpool create would be nice but honestly if zpool create takes a while doesn't matter much... Just increasing the timeout is the "easy fix", but will it be good enough? If there was a config knob so it could be tuned for ill-behaving systems then the default could be set to a couple of seconds for sane systems and we could just bump it in a local config to whatever is needed to get the job done. |
|
@ZNikke Crappy hardware and drivers will certainly will stress things more! Regarding the We could simply increase it since for well behaved systems you should never reach the timeout and on badly behaved systems you will want to wait longer. The case where this could bite us is for system with broken udev rules resulting in the partition links never being created. In that case it would take longer to get the failure but that's perhaps acceptable. So then my question for you and your spectacularly bad (and busy) hardware is, how long do we need to wait? Can you play around with this and determine what a reasonable timeout is. I'll be the first to admit the current value was picked out of thin air and just happens to work well for most situations. |
|
Just looking at my previous comment, it took about 15 seconds for udevadm settle to return, so I'd say that a 20 second timeout should cover it for this particularly crappy controller when loaded. And yes, more testing shows that it's performing even worse than we feared so this rig probably won't be useful for more than pathetic worst-case tests... I couldn't imagine something worse than this will see real-life use, but then someone reminded me that there are people out there with droves of USB drives hanging off a single USB hub... So, go for broke with a 30 second timeout? One could even add a note that the flashy solution would be to use "udevadm settle" or an equivalent API. |
|
I experienced the same problem on my environment. Sometimes creating zpool is successfully sometimes not so I suspect that 1 second timeout is too small: root@59944118: |
|
Hi, the same problem. If I use sdb, sdc, sdd etc. all works fine, but 'by-id' names are unusable for me with ZFSonLinux. Best regards, |
When creating a new pool, or adding/replacing a disk in an existing pool, partition tables will be automatically created on the devices. Under normal circumstances it will take less than a second for udev to create the expected device files under /dev/. However, it has been observed that if the system is doing heavy IO concurrently udev may take far longer. If you also throw in some cheap dodgy hardware it may take even longer. To prevent zpool commands from failing due to this the default wait time for udev is being increased to 30 seconds. This will have no impact on normal usage, the increase timeout should only be noticed if your udev rules are incorrectly configured. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#1646
|
@ZNikke @arturpzol @marcinkuk Can one of you please verify the patches proposed in #1783 resolves this issue for you. |
|
Thank you for the fix but it seems that issue is not in timeout udev value: root@44354374:~# time zpool create test1 /dev/disk/by-id/scsi-2796444756e523656 /dev/disk/by-id/scsi-27851597232454b44 /dev/disk/by-id/scsi-278386c465471514c /dev/disk/by-id/scsi-26d7a547273593576 -f real 0m33.180s root@44354374: real 0m2.362s Do you think that it can be issue in udev tool? |
|
I tested again the issue with the newest version of Debian Wheezy (previously test was performed on old Debian) and issue is repeatable with clean ZFS 0.6.2: root@4435437:~# while true; do zpool create test1 /dev/disk/by-id/scsi-2796444756e523656 /dev/disk/by-id/scsi-27851597232454b44 /dev/disk/by-id/scsi-278386c465471514c /dev/disk/by-id/scsi-26d7a547273593576 -f && zpool destroy test1 ; done but with fix for 30 seconds timeout problem is not repeatable so I think that I had two issues: first was connected with timeout and second was connected with udev tool/old Debian system. |
|
Thanks for the feedback. It sounds like in general this will help make things more robust. |
When creating a new pool, or adding/replacing a disk in an existing pool, partition tables will be automatically created on the devices. Under normal circumstances it will take less than a second for udev to create the expected device files under /dev/. However, it has been observed that if the system is doing heavy IO concurrently udev may take far longer. If you also throw in some cheap dodgy hardware it may take even longer. To prevent zpool commands from failing due to this the default wait time for udev is being increased to 30 seconds. This will have no impact on normal usage, the increase timeout should only be noticed if your udev rules are incorrectly configured. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#1646
|
Hello all, iam trying to create lxc on ubuntu 14.04.i executed |
|
Still encounter this issue in 0.6.5.6 on Centos 7 ... has this patch been reverted? |
|
Similar issue with Debian 10 where, immediately after formatting with Adding a one-second delay prior to |
When doing zpool create on a new device using its vdev name, ie a name specified in vdev_id.conf, the zpool create command doesn't seem to wait for udev to actually create the partition symlink before trying to use it.
Doing ls -l afterwards on the path in the error message shows that it does exist, and retrying the command either succeeds or complains on the next unpartitioned disk device.
Ie, successive invocations can look like this:
As the workaround is to merely retry until it succeeds this is a low priority issue, albeit annoying and probably easy to fix if (either wait/retry on error or do whatever udev magic that exist for it to do its thing before trying to use the created partitions).
Environment: Ubuntu 12.04 using the zfs-native/stable PPA (zfsutils 0.6.1-1~precise).
The text was updated successfully, but these errors were encountered: