deadlock because of zpl_evict_inode requiring more memory

[avg-I]

One of our test loads can easily put a system in a deadlocked state where a machine can be pinged but pretty much nothing else works. It seems that this is caused by the following chain of calls:
any call to allocate pages while there is a page shortage -> try_to_free_pages -> super_cache_scan -> prune_icache_sb -> destroy_inode -> zpl_inode_destroy -> iput-> zpl_evict_inode -> zfs_rmnode -> arc_read -> spl_kmem_cache_alloc -> alloc_pages_current.
All active threads seem to be in the above state.

[avg-I]

Output of foreach active bt: https://gist.github.com/avg-I/10f4384486eecf70c903

[behlendorf]

Setting the following module options may help with the memory pressure and prevent the issue. They're planned to be become the defaults in the next tag so it would be nice to know if it helps your workload.

options spl spl_kmem_cache_slab_limit=16384
options spl spl_kmem_cache_reclaim=0

Additionally, the patches proposed in #2573 might help by making the last iput in the destroy path asynchronous. They're designed to resolve an entirely different issue but you still might see some benefit.

[ioquatix]

I've seen this happen recently.

[behlendorf]

@avg-I has this been resolved in newer versions of ZoL or is it still and issue?

[behlendorf]

Closing as stale. There have been several recent fixes to master which should further improve things.