0.6.5.1 still has problems with large rsync's and xattrs #3822
Comments
|
dmesg output: [19857.719842] perf interrupt took too long (2503 > 2500), lowering kernel.perf_event_max_sample_rate to 50000 |
|
Just realized not all of my stack traces were appearing in the comment above. Here's the rest of the output: [23018.216637] INFO: task BackupPC_nightl:11059 blocked for more than 120 seconds. |
|
Yes, it's the same problem as #3808. |
|
Closing as duplicate of #3808 to avoid confusion. |
As described in the comment above arc_reclaim_thread() it's critical that the reclaim thread be careful about blocking. Just like it must never wait on a hash lock, it must never wait on a task which can in turn wait on the CV in arc_get_data_buf(). This will deadlock, see issue openzfs#3822 for full backtraces showing the problem. To resolve this issue arc_kmem_reap_now() has been updated to use the asynchronous arc prune function. This means that arc_prune_async() may now be called while there are still outstanding arc_prune_tasks. However, this isn't a problem because arc_prune_async() already keeps a reference count preventing multiple outstanding tasks per registered consumer. Functionally, this behavior is the same as the counterpart illumos function dnlc_reduce_cache(). Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#3808 Issue openzfs#3822
As described in the comment above arc_reclaim_thread() it's critical that the reclaim thread be careful about blocking. Just like it must never wait on a hash lock, it must never wait on a task which can in turn wait on the CV in arc_get_data_buf(). This will deadlock, see issue #3822 for full backtraces showing the problem. To resolve this issue arc_kmem_reap_now() has been updated to use the asynchronous arc prune function. This means that arc_prune_async() may now be called while there are still outstanding arc_prune_tasks. However, this isn't a problem because arc_prune_async() already keeps a reference count preventing multiple outstanding tasks per registered consumer. Functionally, this behavior is the same as the counterpart illumos function dnlc_reduce_cache(). Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tim Chase <tim@chase2k.com> Issue #3808 Issue #3834 Issue #3822
As described in the comment above arc_reclaim_thread() it's critical that the reclaim thread be careful about blocking. Just like it must never wait on a hash lock, it must never wait on a task which can in turn wait on the CV in arc_get_data_buf(). This will deadlock, see issue #3822 for full backtraces showing the problem. To resolve this issue arc_kmem_reap_now() has been updated to use the asynchronous arc prune function. This means that arc_prune_async() may now be called while there are still outstanding arc_prune_tasks. However, this isn't a problem because arc_prune_async() already keeps a reference count preventing multiple outstanding tasks per registered consumer. Functionally, this behavior is the same as the counterpart illumos function dnlc_reduce_cache(). Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tim Chase <tim@chase2k.com> Issue #3808 Issue #3834 Issue #3822
My CentOS 7 server running 0.6.5.1 is still having issues with large rsync jobs. The server itself doesn't lock-up (I can still log in and look around), but the rsync jobs hang and once that happens other I/O to the ZFS pools (such as a simple 'ls') also hangs. I'll paste the stack traces from 'dmesg' below.
This is possibly related to issue #3808. It's the same server running the same BackupPC jobs at least. The stack trace is different though, likely because I upgraded from 0.6.5 to 0.6.5.1.
The text was updated successfully, but these errors were encountered: