[SPARK-21305][ML][MLLIB]Add options to disable multi-threading of native BLAS #18551

mpjlu · 2017-07-06T08:04:38Z

What changes were proposed in this pull request?

Many ML/MLLIB algorithms use native BLAS (like Intel MKL, ATLAS, OpenBLAS) to improvement the performance.
Many popular Native BLAS, like Intel MKL, OpenBLAS, use multi-threading technology, which will conflict with Spark. Spark should provide options to disable multi-threading of Native BLAS.

https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded
https://software.intel.com/en-us/articles/recommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications

How was this patch tested?

The existing UT.

srowen · 2017-07-06T09:47:00Z

Hm, can we try to set this with System.setenv anywhere in the code or will it be too late by the time this triggers? probably.

I think this is OK, but should be documented also in places in the docs that cover BLAS and netlib.

mpjlu · 2017-07-06T10:45:35Z

Thanks, @srowen . I have updated the doc.
I also validated the current option in spark-env.sh, it works.
Thanks.

SparkQA · 2017-07-06T10:59:46Z

Test build #79272 has finished for PR 18551 at commit d0d94e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MLnick · 2017-07-06T13:18:42Z

docs/ml-guide.md

@@ -61,6 +61,9 @@ To configure `netlib-java` / Breeze to use system optimised binaries, include
 project and read the [netlib-java](https://github.com/fommil/netlib-java) documentation for your
 platform's additional installation instructions.

+The most popular native BLAS such as [Intel MKL](https://software.intel.com/en-us/mkl), OpenBLAS, are based on multi-threading, which will conflict with Spark.


I think we need more detail on "conflict with Spark". Explain how exactly

MLnick · 2017-07-06T13:20:11Z

conf/spark-env.sh.template

@@ -61,3 +61,7 @@
 # - SPARK_IDENT_STRING  A string representing this instance of spark. (Default: $USER)
 # - SPARK_NICENESS      The scheduling priority for daemons. (Default: 0)
 # - SPARK_NO_DAEMONIZE  Run the proposed command in the foreground. It will not output a PID file.
+# Options for native BLAS, like Intel MKL, OpenBLAS, and so on.
+# You shoud enable these options if use native BLAS (SPARK-21305).


"if use" -> "if using"

"(see SPARK-21305)"

MLnick · 2017-07-06T13:20:29Z

docs/ml-guide.md

@@ -61,6 +61,9 @@ To configure `netlib-java` / Breeze to use system optimised binaries, include
 project and read the [netlib-java](https://github.com/fommil/netlib-java) documentation for your
 platform's additional installation instructions.

+The most popular native BLAS such as [Intel MKL](https://software.intel.com/en-us/mkl), OpenBLAS, are based on multi-threading, which will conflict with Spark.


Link for OpenBLAS?

MLnick · 2017-07-06T13:21:04Z

docs/ml-guide.md

@@ -61,6 +61,9 @@ To configure `netlib-java` / Breeze to use system optimised binaries, include
 project and read the [netlib-java](https://github.com/fommil/netlib-java) documentation for your
 platform's additional installation instructions.

+The most popular native BLAS such as [Intel MKL](https://software.intel.com/en-us/mkl), OpenBLAS, are based on multi-threading, which will conflict with Spark.
+To use multi-threading based native BLAS, you must set it to use single thread first (SPARK-21305).


Perhaps "If using a native BLAS based on multi-threading, for best performance you must ..."

Provide a URL link for SPARK-21305

SparkQA · 2017-07-06T13:29:38Z

Test build #79276 has finished for PR 18551 at commit adfa6f1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-06T13:35:28Z

Test build #79277 has finished for PR 18551 at commit 08e900b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-06T13:47:44Z

Test build #79278 has finished for PR 18551 at commit 7ef8b80.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-06T18:55:43Z

Test build #79289 has finished for PR 18551 at commit a4d4f50.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2017-07-07T07:43:04Z

docs/ml-guide.md

@@ -61,6 +61,11 @@ To configure `netlib-java` / Breeze to use system optimised binaries, include
 project and read the [netlib-java](https://github.com/fommil/netlib-java) documentation for your
 platform's additional installation instructions.

+The most popular native BLAS such as [Intel MKL](https://software.intel.com/en-us/mkl), [OpenBLAS](http://www.openblas.net), are based on multi-threading. 
+For example, when OpenBLAS is loaded, it will create a thread pool with `MAX_CPU_NUMBER` threads, and the threads are using spinlock by default, which will conflict with Spark.


Is it always worse? for example can it multi-thread a single computation? it's possible that's advantageous if the other tasks on the machine aren't CPU-intensive. But probably counterproductive if all the tasks are CPU intensive.

Maybe we can soften the language slightly, to say you might get better performance by setting these to 1.

Also I don't think users will know what spinlock or MAX_CPU_NUMBER is (not a Spark value?)

Also, I don't think we ever explain here what MKL or OpenBLAS is, in any docs. You might briefly mention that this is explained in the netlib docs.

Hi @srowen , it is not always worse, here is one test result: https://issues.apache.org/jira/browse/SPARK-21305?focusedCommentId=16076152&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16076152
I will soften the language slightly. And give two links about recommended settings for calling native blas. How do you think about it?
https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded
https://software.intel.com/en-us/articles/recommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications

felixcheung · 2017-07-08T21:04:57Z

shouldn't we not suggest to disable multithreading completely but instead say to make sure the number of core assigned to Spark and number of threads to native BLAS add up to the total number of CPU core available on the worker?

mpjlu · 2017-07-10T02:00:29Z

hi @felixcheung , I have tested one case, write a single thread java program, and call native blas. The performance is much better to disable native blas multi-threading (the total program performance). This is because JVM itself is multi-threading.
For JVM env, maybe we cannot set "the number of core assigned to Spark and number of threads to native BLAS add up to the total number of CPU core available on the worker". I didn't do enough test for this.
Thanks.

srowen · 2017-07-10T06:26:17Z

@mpjlu Felix is making a slightly different point. Let's say Spark is actually configured to use 2 CPUs per task. The right setting is not 1 thread per task but 2. Using 1 would almost surely not be faster.

mpjlu · 2017-07-10T06:40:20Z

hi @srowen , I understand Felix's point. I mean if you only have 1 task in C/C++, and 2 CPUs, setting native BLAS to use 2 CPUs will be faster. But in JVM env, even you only have one task, and 2 CPUs, but there is still some JVM system threads, setting native BLAS to use 2 CPUs maybe not faster.

srowen · 2017-07-10T06:43:55Z

I doubt the other background tasks will consume a whole additional CPU for every task. You're saying that on a 16-core machine with 8 tasks, the other 8 CPUs are probably nearly fully utilized? I just can't see it. In any event, I think that's why it's best to state this as something to look at, and the principle here -- match MKL/BLAS cores to task cores -- rather than say "always set to 1".

mpjlu · 2017-07-10T07:05:25Z

Hi @srowen , Thanks very much for your review.
I will revise the document of this PR to soften the language.
According to my profiling data, I guess, when the native BLAS is loaded (or when a multi-threading subprogram is called), the native BLAS it will create a thread pool (If you have 16 cores, by default it will create 15 threads, these threads will exist until the end of the job, not the same life time as subprogram, but the same life time as your job), this 15 threads each will consume a whole CPU.
Thanks.

SparkQA · 2017-07-11T07:04:55Z

Test build #79506 has finished for PR 18551 at commit 141d92d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

mpjlu · 2017-07-11T08:33:40Z

retest this please

srowen · 2017-07-11T10:54:13Z

docs/ml-guide.md

@@ -61,6 +61,12 @@ To configure `netlib-java` / Breeze to use system optimised binaries, include
 project and read the [netlib-java](https://github.com/fommil/netlib-java) documentation for your
 platform's additional installation instructions.

+The most popular native BLAS such as [Intel MKL](https://software.intel.com/en-us/mkl), [OpenBLAS](http://www.openblas.net), are based on multi-threading, which will conflict with Spark.


I would say "can use multiple threads in a single operation, which can conflict with Spark's execution model."

srowen · 2017-07-11T11:02:51Z

docs/ml-guide.md

@@ -61,6 +61,12 @@ To configure `netlib-java` / Breeze to use system optimised binaries, include
 project and read the [netlib-java](https://github.com/fommil/netlib-java) documentation for your
 platform's additional installation instructions.

+The most popular native BLAS such as [Intel MKL](https://software.intel.com/en-us/mkl), [OpenBLAS](http://www.openblas.net), are based on multi-threading, which will conflict with Spark.
+
+If using a native BLAS based on multi-threading, you might get better performance to set it to use single thread first ([SPARK-21305](https://issues.apache.org/jira/browse/SPARK-21305)). 


I might say, "Configuring these BLAS implementations to use a single thread for operations may actually improve performance (see ...). It is usually optimal to match this to the number of cores each Spark task is configured to use, which is 1 by default and typically left at 1."

srowen · 2017-07-11T11:03:27Z

docs/ml-guide.md

+
+If using a native BLAS based on multi-threading, you might get better performance to set it to use single thread first ([SPARK-21305](https://issues.apache.org/jira/browse/SPARK-21305)). 
+
+Please reference the recommended settings for [Intel MKL](https://software.intel.com/en-us/articles/recommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications) and [OpenBLAS](https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded).


Maybe: "Please refer to resources like the following to understand how to configure the number of threads these BLAS implementations use: ..."

Done, thanks very much. @srowen

SparkQA · 2017-07-11T11:34:44Z

Test build #79513 has finished for PR 18551 at commit 141d92d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-11T14:46:14Z

Test build #79521 has finished for PR 18551 at commit e01a613.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

mpjlu · 2017-07-11T15:02:40Z

retest this please

SparkQA · 2017-07-11T18:04:20Z

Test build #79525 has finished for PR 18551 at commit e01a613.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2017-07-12T10:02:20Z

Merged to master

disable multi-threading in spark-env.sh

d0d94e1

Peng Meng added 2 commits July 6, 2017 18:40

revise the doc

adfa6f1

revise the doc

08e900b

add link

7ef8b80

MLnick reviewed Jul 6, 2017

View reviewed changes

add OpenBlas multi-threading introduction

a4d4f50

srowen reviewed Jul 7, 2017

View reviewed changes

add recommended setting for native blas link

141d92d

srowen reviewed Jul 11, 2017

View reviewed changes

fix ml-guide.md

e01a613

srowen approved these changes Jul 11, 2017

View reviewed changes

asfgit closed this in 5ed134e Jul 12, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-21305][ML][MLLIB]Add options to disable multi-threading of native BLAS #18551

[SPARK-21305][ML][MLLIB]Add options to disable multi-threading of native BLAS #18551

mpjlu commented Jul 6, 2017

srowen commented Jul 6, 2017

mpjlu commented Jul 6, 2017

SparkQA commented Jul 6, 2017

MLnick Jul 6, 2017

MLnick Jul 6, 2017

MLnick Jul 6, 2017

MLnick Jul 6, 2017 •

edited

Loading

MLnick Jul 6, 2017

SparkQA commented Jul 6, 2017

SparkQA commented Jul 6, 2017

SparkQA commented Jul 6, 2017

SparkQA commented Jul 6, 2017

srowen Jul 7, 2017

mpjlu Jul 10, 2017

felixcheung commented Jul 8, 2017

mpjlu commented Jul 10, 2017

srowen commented Jul 10, 2017

mpjlu commented Jul 10, 2017

srowen commented Jul 10, 2017

mpjlu commented Jul 10, 2017

SparkQA commented Jul 11, 2017

mpjlu commented Jul 11, 2017

srowen Jul 11, 2017

srowen Jul 11, 2017

srowen Jul 11, 2017

mpjlu Jul 11, 2017

SparkQA commented Jul 11, 2017

SparkQA commented Jul 11, 2017

mpjlu commented Jul 11, 2017

SparkQA commented Jul 11, 2017

srowen commented Jul 12, 2017


		If using a native BLAS based on multi-threading, you might get better performance to set it to use single thread first ([SPARK-21305](https://issues.apache.org/jira/browse/SPARK-21305)).

		Please reference the recommended settings for [Intel MKL](https://software.intel.com/en-us/articles/recommended-settings-for-calling-intel-mkl-routines-from-multi-threaded-applications) and [OpenBLAS](https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded).

[SPARK-21305][ML][MLLIB]Add options to disable multi-threading of native BLAS #18551

[SPARK-21305][ML][MLLIB]Add options to disable multi-threading of native BLAS #18551

Conversation

mpjlu commented Jul 6, 2017

What changes were proposed in this pull request?

How was this patch tested?

srowen commented Jul 6, 2017

mpjlu commented Jul 6, 2017

SparkQA commented Jul 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MLnick Jul 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jul 6, 2017

SparkQA commented Jul 6, 2017

SparkQA commented Jul 6, 2017

SparkQA commented Jul 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

felixcheung commented Jul 8, 2017

mpjlu commented Jul 10, 2017

srowen commented Jul 10, 2017

mpjlu commented Jul 10, 2017

srowen commented Jul 10, 2017

mpjlu commented Jul 10, 2017

SparkQA commented Jul 11, 2017

mpjlu commented Jul 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jul 11, 2017

SparkQA commented Jul 11, 2017

mpjlu commented Jul 11, 2017

SparkQA commented Jul 11, 2017

srowen commented Jul 12, 2017

MLnick Jul 6, 2017 •

edited

Loading