Convert zfs_mg_noalloc_threshold to a module parameter and document

The parameter was added as illumos issue 4081 which was committed to
zfsonlinux in ac72fac.  This patch
documents the parameter and allows for it to be set as a module parameter.

Signed-off-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#2483

man/man5/zfs-module-parameters.5

@@ -729,6 +729,32 @@ Disable meta data compression
 Use \fB1\fR for yes and \fB0\fR for no (default).
 .RE
 
+.sp
+.ne 2
+.na
+\fBzfs_mg_noalloc_threshold\fR (int)
+.ad
+.RS 12n
+Defines a threshold at which metaslab groups should be eligible for
+allocations.  The value is expressed as a percentage of free space
+beyond which a metaslab group is always eligible for allocations.
+If a metaslab group's free space is less than or equal to the
+the threshold, the allocator will avoid allocating to that group
+unless all groups in the pool have reached the threshold.  Once all
+groups have reached the threshold, all groups are allowed to accept
+allocations.  The default value of 0 disables the feature and causes
+all metaslab groups to be eligible for allocations.
+
+This parameter allows to deal with pools having heavily imbalanced
+vdevs such as would be the case when a new vdev has been added.
+Setting the threshold to a non-zero percentage will stop allocations
+from being made to vdevs that aren't filled to the specified percentage
+and allow lesser filled vdevs to acquire more allocations than they
+otherwise would under the old \fBzfs_mg_alloc_failures\fR facility.
+.sp
+Default value: \fB0\fR.
+.RE
+
 .sp
 .ne 2
 .na

module/zfs/metaslab.c

@@ -2120,4 +2120,8 @@ MODULE_PARM_DESC(metaslab_debug_load, "load all metaslabs during pool import");
 module_param(metaslab_debug_unload, int, 0644);
 MODULE_PARM_DESC(metaslab_debug_unload,
 	"prevent metaslabs from being unloaded");
+
+module_param(zfs_mg_noalloc_threshold, int, 0644);
+MODULE_PARM_DESC(zfs_mg_noalloc_threshold,
+	"percentage of free space for metaslab group to allow allocation");
 #endif /* _KERNEL && HAVE_SPL */