[SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors #27360

zhengruifeng · 2020-01-25T13:37:17Z

What changes were proposed in this pull request?

1, stack input vectors to blocks (like ALS/MLP);
2, add new param blockSize;
3, add a new class InstanceBlock
4, standardize the input outside of optimization procedure;

Why are the changes needed?

1, reduce RAM to persist traing dataset; (save ~40% in test)
2, use Level-2 BLAS routines; (12% ~ 41% faster, without native BLAS)

Does this PR introduce any user-facing change?

a new param blockSize

How was this patch tested?

existing and updated testsuites

zhengruifeng · 2020-01-25T13:41:33Z

testCode:

import org.apache.spark.ml.classification._
import org.apache.spark.storage.StorageLevel


var df = spark.read.format("libsvm").load("/data1/Datasets/a9a/a9a").withColumn("label", (col("label")+1)/2)
df.persist(StorageLevel.MEMORY_AND_DISK)
df.count

(0 until 8).foreach{ _ => df = df.union(df) }
df.count

new LinearSVC().setMaxIter(10).fit(df) // warm up

val svc = new LinearSVC().setMaxIter(100)
val start = System.currentTimeMillis; val model = svc.fit(df); val end = System.currentTimeMillis; end - start


val svc = new LinearSVC().setMaxIter(100).setFitIntercept(false)
val start = System.currentTimeMillis; val model = svc.fit(df); val end = System.currentTimeMillis; end - start

result:
this PR:
RAM:1418.9M
Duration: 396524(fitIntercept=true), 324944(fitIntercept=false)

MASTER:
RAM: 2.3G
Duration: 446354(fitIntercept=true), 441961(fitIntercept=false)

Native-BLAS is NOT used in above tests, maybe future performance gain can be obtained by setting appropriate Native BLAS.

zhengruifeng · 2020-01-25T13:49:10Z

mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala

@@ -220,8 +227,22 @@ class LinearSVC @Since("2.2.0") (
        None
      }

-      val getAggregatorFunc = new HingeAggregator(bcFeaturesStd, $(fitIntercept))(_)
-      val costFun = new RDDLossFunction(instances, getAggregatorFunc, regularization,
+      val standardized = instances.map {


Move the standardization outside of HingeAggregator, so that no longer need to standardize input in each iter.

SparkQA · 2020-01-25T16:03:49Z

Test build #117398 has finished for PR 27360 at commit 6dcb9eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait HasBlockSize extends Params
class HasBlockSize(Params):

SparkQA · 2020-01-25T16:32:21Z

Test build #117400 has finished for PR 27360 at commit c51c37b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2020-01-25T18:16:01Z

OK, I would generally support it if it's not a huge change, optimizes performance, and doesn't change behavior / correctness.

zhengruifeng · 2020-01-26T00:41:54Z

I added InstanceBlock as a common class, since I think it will also works in LoR, LiR, and other algs.
I add some testsuite to check correctness.
Thanks for reviewing! @srowen

SparkQA · 2020-01-26T03:11:58Z

Test build #117404 has finished for PR 27360 at commit ea83dc1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-01-26T03:24:37Z

Test build #117407 has finished for PR 27360 at commit 7676b8e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

zhengruifeng · 2020-01-26T04:06:45Z

retest this please

SparkQA · 2020-01-26T06:58:29Z

Test build #117410 has finished for PR 27360 at commit 7676b8e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-01-26T08:05:02Z

Test build #117416 has finished for PR 27360 at commit 487ee0f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-01-26T08:05:02Z

Test build #117418 has finished for PR 27360 at commit 2f84a3d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

zhengruifeng · 2020-01-26T09:04:11Z

retest this please

SparkQA · 2020-01-26T12:02:08Z

Test build #117420 has finished for PR 27360 at commit 2f84a3d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

init init fix py fix fix fix fix fix

nit py py

SparkQA · 2020-01-27T17:24:26Z

Test build #117444 has finished for PR 27360 at commit cb91306.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-01-28T11:44:58Z

Test build #117474 has finished for PR 27360 at commit 819227e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zhengruifeng · 2020-01-28T12:56:38Z

Merged to master.

### What changes were proposed in this pull request? Revert #27360 #27396 #27374 #27389 ### Why are the changes needed? BLAS need more performace tests, specially on sparse datasets. Perfermance test of LogisticRegression (#27374) on sparse dataset shows that blockify vectors to matrices and use BLAS will cause performance regression. LinearSVC and LinearRegression were also updated in the same way as LogisticRegression, so we need to revert them to make sure no regression. ### Does this PR introduce any user-facing change? remove newly added param blockSize ### How was this patch tested? reverted testsuites Closes #27487 from zhengruifeng/revert_blockify_ii. Authored-by: zhengruifeng <ruifengz@foxmail.com> Signed-off-by: zhengruifeng <ruifengz@foxmail.com>

### What changes were proposed in this pull request? Revert apache#27360 apache#27396 apache#27374 apache#27389 ### Why are the changes needed? BLAS need more performace tests, specially on sparse datasets. Perfermance test of LogisticRegression (apache#27374) on sparse dataset shows that blockify vectors to matrices and use BLAS will cause performance regression. LinearSVC and LinearRegression were also updated in the same way as LogisticRegression, so we need to revert them to make sure no regression. ### Does this PR introduce any user-facing change? remove newly added param blockSize ### How was this patch tested? reverted testsuites Closes apache#27487 from zhengruifeng/revert_blockify_ii. Authored-by: zhengruifeng <ruifengz@foxmail.com> Signed-off-by: zhengruifeng <ruifengz@foxmail.com>

zhengruifeng commented Jan 25, 2020

View reviewed changes

zhengruifeng added 9 commits January 27, 2020 22:21

init

be6c18b

init init fix py fix fix fix fix fix

use BLAS

f952e0f

nit py py

doc

0ef0946

add testsuites and reuse array in HingeAgg

bb57a84

nit

1c1ce65

avoid @transient tuple assignment

9f8d3b3

nit

b09a343

nit

db60d68

nit, reuse a vec among blocks

cb91306

zhengruifeng force-pushed the blockify_svc branch from 2f84a3d to cb91306 Compare January 27, 2020 14:21

nit

819227e

zhengruifeng closed this in 96d2727 Jan 28, 2020

zhengruifeng deleted the blockify_svc branch January 28, 2020 12:56

zero323 mentioned this pull request Jan 28, 2020

Sync with changes merged after 6502c66025718bf45e0e2ee12398b7b92da41a0c zero323/pyspark-stubs#315

Closed

14 tasks

zhengruifeng mentioned this pull request Jan 29, 2020

[SPARK-30659][ML][PYSPARK] LogisticRegression blockify input vectors #27374

Closed

zhengruifeng restored the blockify_svc branch February 6, 2020 08:03

This was referenced Feb 7, 2020

Revert "[SPARK-30662][ML][PYSPARK] ALS/MLP extend HasBlockSize" #27485

Closed

Revert "[SPARK-30642][SPARK-30659][SPARK-30660][SPARK-30662]" #27486

Closed

Revert "[SPARK-30642][SPARK-30659][SPARK-30660][SPARK-30662]" #27487

Closed

This was referenced Apr 26, 2020

[SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors #28349

Closed

[SPARK-31454][ML] An optimized K-Means based on DenseMatrix and GEMM #28229

Closed

zero323 mentioned this pull request May 18, 2020

[SPARK-30642] LinearSVC blockify input vectors zero323/pyspark-stubs#406

Closed

zhengruifeng deleted the blockify_svc branch February 25, 2021 10:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors #27360

[SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors #27360

zhengruifeng commented Jan 25, 2020 •

edited

Loading

zhengruifeng commented Jan 25, 2020 •

edited

Loading

zhengruifeng Jan 25, 2020

SparkQA commented Jan 25, 2020

SparkQA commented Jan 25, 2020

srowen commented Jan 25, 2020

zhengruifeng commented Jan 26, 2020

SparkQA commented Jan 26, 2020

SparkQA commented Jan 26, 2020

zhengruifeng commented Jan 26, 2020

SparkQA commented Jan 26, 2020

SparkQA commented Jan 26, 2020

SparkQA commented Jan 26, 2020

zhengruifeng commented Jan 26, 2020

SparkQA commented Jan 26, 2020

SparkQA commented Jan 27, 2020

SparkQA commented Jan 28, 2020

zhengruifeng commented Jan 28, 2020

[SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors #27360

[SPARK-30642][ML][PYSPARK] LinearSVC blockify input vectors #27360

Conversation

zhengruifeng commented Jan 25, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

zhengruifeng commented Jan 25, 2020 • edited Loading

zhengruifeng Jan 25, 2020

Choose a reason for hiding this comment

SparkQA commented Jan 25, 2020

SparkQA commented Jan 25, 2020

srowen commented Jan 25, 2020

zhengruifeng commented Jan 26, 2020

SparkQA commented Jan 26, 2020

SparkQA commented Jan 26, 2020

zhengruifeng commented Jan 26, 2020

SparkQA commented Jan 26, 2020

SparkQA commented Jan 26, 2020

SparkQA commented Jan 26, 2020

zhengruifeng commented Jan 26, 2020

SparkQA commented Jan 26, 2020

SparkQA commented Jan 27, 2020

SparkQA commented Jan 28, 2020

zhengruifeng commented Jan 28, 2020

zhengruifeng commented Jan 25, 2020 •

edited

Loading

zhengruifeng commented Jan 25, 2020 •

edited

Loading