Optimize vmem_alloc() retry path

For performance reasons the reworked kmem code maps vmem_alloc() to
kmalloc_node() for allocations less than spa_kmem_alloc_max.  This
allows for more concurrency in the system and less contention of
the virtual address space.  Generally, this is a good thing.

However, in the case when the kmalloc_node() fails it makes little
sense to retry it using kmalloc_node() again.  It will likely fail
in exactly the same way.  A smarter strategy is to abandon this
optimization and retry using spl_vmalloc() which is very likely
to succeed.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ned Bass <bass6@llnl.gov>
Closes openzfs#428

module/spl/spl-kmem.c

@@ -149,6 +149,7 @@ inline void *
 spl_kmem_alloc_impl(size_t size, int flags, int node)
 {
 	gfp_t lflags = kmem_flags_convert(flags);
+	int use_vmem = 0;
 	void *ptr;
 
 	/*
@@ -182,7 +183,7 @@ spl_kmem_alloc_impl(size_t size, int flags, int node)
 		 * impact performance so frequently manipulating the virtual
 		 * address space is strongly discouraged.
 		 */
-		if (unlikely(size > spl_kmem_alloc_max)) {
+		if ((size > spl_kmem_alloc_max) || use_vmem) {
 			if (flags & KM_VMEM) {
 				ptr = spl_vmalloc(size, lflags, PAGE_KERNEL);
 			} else {
@@ -195,6 +196,15 @@ spl_kmem_alloc_impl(size_t size, int flags, int node)
 		if (likely(ptr) || (flags & KM_NOSLEEP))
 			return (ptr);
 
+		/*
+		 * For vmem_alloc() and vmem_zalloc() callers retry immediately
+		 * using spl_vmalloc() which is unlikely to fail.
+		 */
+		if ((flags & KM_VMEM) && (use_vmem == 0))  {
+			use_vmem = 1;
+			continue;
+		}
+
 		if (unlikely(__ratelimit(&kmem_alloc_ratelimit_state))) {
 			printk(KERN_WARNING
 			    "Possible memory allocation deadlock: "