[SPARK-7578] [ml] [doc] User guide for spark.ml Normalizer, IDF, StandardScaler #6127

jkbradley · 2015-05-13T19:03:01Z

Added user guide sections with code examples.
Also added small Java unit tests to test Java example in guide.

CC: @mengxr

AmplabJenkins · 2015-05-13T19:07:10Z

Merged build triggered.

AmplabJenkins · 2015-05-13T19:07:18Z

Merged build started.

SparkQA · 2015-05-13T19:09:16Z

Test build #32635 has started for PR 6127 at commit 8a2b5cd.

SparkQA · 2015-05-13T20:52:09Z

Test build #32635 has finished for PR 6127 at commit 8a2b5cd.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class JavaNormalizerSuite
- public class JavaStandardScalerSuite

AmplabJenkins · 2015-05-13T20:52:14Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-13T20:52:15Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32635/
Test PASSed.

AmplabJenkins · 2015-05-19T17:22:10Z

Build triggered.

AmplabJenkins · 2015-05-19T17:22:16Z

Build started.

SparkQA · 2015-05-19T17:24:13Z

Test build #33085 has started for PR 6127 at commit 45245e8.

SparkQA · 2015-05-19T19:49:07Z

Test build #33085 has finished for PR 6127 at commit 45245e8.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- public class JavaNormalizerSuite
- public class JavaStandardScalerSuite

AmplabJenkins · 2015-05-19T19:49:12Z

Build finished. Test PASSed.

AmplabJenkins · 2015-05-19T19:49:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33085/
Test PASSed.

…unit tests

jkbradley · 2015-05-21T20:09:44Z

@mengxr Just rebased off of master, so this should be ready to review (with fewer PRs for the ml-features.md file now).

AmplabJenkins · 2015-05-21T20:12:11Z

Merged build triggered.

AmplabJenkins · 2015-05-21T20:12:19Z

Merged build started.

SparkQA · 2015-05-21T20:14:28Z

Test build #33274 has started for PR 6127 at commit f16bcec.

SparkQA · 2015-05-21T20:25:56Z

Test build #33274 has finished for PR 6127 at commit f16bcec.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-21T20:26:00Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-21T20:26:00Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33274/
Test FAILed.

SparkQA · 2015-05-21T22:34:34Z

Test build #848 has started for PR 6127 at commit f16bcec.

SparkQA · 2015-05-22T00:19:15Z

Test build #848 has finished for PR 6127 at commit f16bcec.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mengxr · 2015-05-22T00:56:58Z

docs/ml-features.md

+val idf = new IDF().setInputCol("rawFeatures").setOutputCol("features")
+val idfModel = idf.fit(featurizedData)
+val rescaledData = idfModel.transform(featurizedData)
+rescaledData.select("features", "label").take(3).foreach(println)


minor: take(3) -> collect()

I thought about that, but was worried users would copy this into their own code and try to print out a huge dataset. What do you think?

mengxr · 2015-05-22T00:57:12Z

mllib/src/test/java/org/apache/spark/ml/feature/JavaHashingTFSuite.java

+    IDF idf = new IDF().setInputCol("rawFeatures").setOutputCol("features");
+    IDFModel idfModel = idf.fit(featurizedData);
+    DataFrame rescaledData = idfModel.transform(featurizedData);
+    for (Row r : rescaledData.select("features", "label").take(3)) {


take(3) -> collect()?

jkbradley · 2015-05-22T01:52:24Z

@mengxr Updated. Thanks!

AmplabJenkins · 2015-05-22T01:57:12Z

Merged build triggered.

AmplabJenkins · 2015-05-22T01:57:18Z

Merged build started.

SparkQA · 2015-05-22T01:59:17Z

Test build #33310 has started for PR 6127 at commit cd47f4b.

SparkQA · 2015-05-22T03:44:31Z

Test build #33310 has finished for PR 6127 at commit cd47f4b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class JavaNormalizerSuite
- public class JavaStandardScalerSuite

AmplabJenkins · 2015-05-22T03:44:35Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-22T03:44:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33310/
Test PASSed.

…dardScaler Added user guide sections with code examples. Also added small Java unit tests to test Java example in guide. CC: mengxr Author: Joseph K. Bradley <joseph@databricks.com> Closes #6127 from jkbradley/feature-guide-2 and squashes the following commits: cd47f4b [Joseph K. Bradley] Updated based on code review f16bcec [Joseph K. Bradley] Fixed merge issues and update Python examples print calls for Python 3 0a862f9 [Joseph K. Bradley] Added Normalizer, StandardScaler to ml-features doc, plus small Java unit tests a21c2d6 [Joseph K. Bradley] Updated ml-features.md with IDF (cherry picked from commit 2728c3d) Signed-off-by: Xiangrui Meng <meng@databricks.com>

mengxr · 2015-05-22T06:00:13Z

LGTM. Merged into master and branch-1.4. Thanks!

…dardScaler Added user guide sections with code examples. Also added small Java unit tests to test Java example in guide. CC: mengxr Author: Joseph K. Bradley <joseph@databricks.com> Closes #6127 from jkbradley/feature-guide-2 and squashes the following commits: cd47f4b [Joseph K. Bradley] Updated based on code review f16bcec [Joseph K. Bradley] Fixed merge issues and update Python examples print calls for Python 3 0a862f9 [Joseph K. Bradley] Added Normalizer, StandardScaler to ml-features doc, plus small Java unit tests a21c2d6 [Joseph K. Bradley] Updated ml-features.md with IDF

…dardScaler Added user guide sections with code examples. Also added small Java unit tests to test Java example in guide. CC: mengxr Author: Joseph K. Bradley <joseph@databricks.com> Closes apache#6127 from jkbradley/feature-guide-2 and squashes the following commits: cd47f4b [Joseph K. Bradley] Updated based on code review f16bcec [Joseph K. Bradley] Fixed merge issues and update Python examples print calls for Python 3 0a862f9 [Joseph K. Bradley] Added Normalizer, StandardScaler to ml-features doc, plus small Java unit tests a21c2d6 [Joseph K. Bradley] Updated ml-features.md with IDF

jkbradley force-pushed the feature-guide-2 branch from 8a2b5cd to 45245e8 Compare May 19, 2015 17:17

jkbradley added 3 commits May 21, 2015 13:08

Updated ml-features.md with IDF

a21c2d6

Added Normalizer, StandardScaler to ml-features doc, plus small Java …

0a862f9

…unit tests

Fixed merge issues and update Python examples print calls for Python 3

f16bcec

jkbradley force-pushed the feature-guide-2 branch from 45245e8 to f16bcec Compare May 21, 2015 20:09

mengxr reviewed May 22, 2015
View reviewed changes

Updated based on code review

cd47f4b

jkbradley closed this May 22, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-7578] [ml] [doc] User guide for spark.ml Normalizer, IDF, StandardScaler #6127

[SPARK-7578] [ml] [doc] User guide for spark.ml Normalizer, IDF, StandardScaler #6127

jkbradley commented May 13, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 13, 2015

SparkQA commented May 13, 2015

SparkQA commented May 13, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 19, 2015

AmplabJenkins commented May 19, 2015

SparkQA commented May 19, 2015

SparkQA commented May 19, 2015

AmplabJenkins commented May 19, 2015

AmplabJenkins commented May 19, 2015

jkbradley commented May 21, 2015

AmplabJenkins commented May 21, 2015

AmplabJenkins commented May 21, 2015

SparkQA commented May 21, 2015

SparkQA commented May 21, 2015

AmplabJenkins commented May 21, 2015

AmplabJenkins commented May 21, 2015

SparkQA commented May 21, 2015

SparkQA commented May 22, 2015

mengxr May 22, 2015

jkbradley May 22, 2015

mengxr May 22, 2015

jkbradley commented May 22, 2015

AmplabJenkins commented May 22, 2015

AmplabJenkins commented May 22, 2015

SparkQA commented May 22, 2015

SparkQA commented May 22, 2015

AmplabJenkins commented May 22, 2015

AmplabJenkins commented May 22, 2015

mengxr commented May 22, 2015

[SPARK-7578] [ml] [doc] User guide for spark.ml Normalizer, IDF, StandardScaler #6127

[SPARK-7578] [ml] [doc] User guide for spark.ml Normalizer, IDF, StandardScaler #6127

Conversation

jkbradley commented May 13, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 13, 2015

SparkQA commented May 13, 2015

SparkQA commented May 13, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 19, 2015

AmplabJenkins commented May 19, 2015

SparkQA commented May 19, 2015

SparkQA commented May 19, 2015

AmplabJenkins commented May 19, 2015

AmplabJenkins commented May 19, 2015

jkbradley commented May 21, 2015

AmplabJenkins commented May 21, 2015

AmplabJenkins commented May 21, 2015

SparkQA commented May 21, 2015

SparkQA commented May 21, 2015

AmplabJenkins commented May 21, 2015

AmplabJenkins commented May 21, 2015

SparkQA commented May 21, 2015

SparkQA commented May 22, 2015

mengxr May 22, 2015

Choose a reason for hiding this comment

jkbradley May 22, 2015

Choose a reason for hiding this comment

mengxr May 22, 2015

Choose a reason for hiding this comment

jkbradley commented May 22, 2015

AmplabJenkins commented May 22, 2015

AmplabJenkins commented May 22, 2015

SparkQA commented May 22, 2015

SparkQA commented May 22, 2015

AmplabJenkins commented May 22, 2015

AmplabJenkins commented May 22, 2015

mengxr commented May 22, 2015