Spark 2.4 support #402

tovbinm · 2019-09-04T05:13:24Z

Related issues
We are currently on Spark 2.3, but 2.4 is out

Describe the proposed solution
Upgrade to Spark 2.4

Describe alternatives you've considered
N/A

Additional Context
This branch to be parked here until we are ready to upgrade to Spark 2.4

This reverts commit 95a77b1.

codecov · 2019-09-04T05:33:58Z

Codecov Report

Merging #402 into master will decrease coverage by 5.31%.
The diff coverage is 95.38%.

@@            Coverage Diff             @@
##           master     #402      +/-   ##
==========================================
- Coverage    86.9%   81.59%   -5.32%     
==========================================
  Files         340      341       +1     
  Lines       11450    11507      +57     
  Branches      364      374      +10     
==========================================
- Hits         9951     9389     -562     
- Misses       1499     2118     +619

Impacted Files	Coverage Δ
...sforce/op/stages/impl/feature/Transmogrifier.scala	`95.82% <100%> (-2.23%)`	⬇️
...c/main/scala/com/salesforce/op/ModelInsights.scala	`93.06% <100%> (+0.31%)`	⬆️
...m/salesforce/op/utils/spark/OpVectorMetadata.scala	`85.93% <85.71%> (+0.48%)`	⬆️
...om/salesforce/op/SensitiveFeatureInformation.scala	`97.14% <97.14%> (ø)`
...ce/op/stages/impl/feature/TextLenTransformer.scala	`100% <0%> (ø)`	⬆️
...op/stages/impl/feature/TimePeriodTransformer.scala	`100% <0%> (ø)`	⬆️
...ala/com/salesforce/op/utils/tuples/RichTuple.scala	`100% <0%> (ø)`	⬆️
...force/op/stages/impl/feature/OpIndexToString.scala	`100% <0%> (ø)`	⬆️
...alesforce/op/aggregators/TimeBasedAggregator.scala	`100% <0%> (ø)`	⬆️
...com/salesforce/op/test/TestOpWorkflowBuilder.scala	`100% <0%> (ø)`	⬆️
... and 62 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c64eb51...d79bcc3. Read the comment docs.

… made to decision tree pruning in Spark 2.4. If nodes are split, but both child nodes lead to the same prediction then the split is pruned away. This updates the test so this doesn't happen for feature 'b'

gerashegalov · 2020-01-29T19:11:19Z

README.md

@@ -128,7 +128,7 @@ Start by picking TransmogrifAI version to match your project dependencies from t

 | TransmogrifAI Version                                 | Spark Version | Scala Version | Java Version |
 |-------------------------------------------------------|:-------------:|:-------------:|:------------:|
-| 0.6.2 (unreleased, master)                            |      2.3      |      2.11     |      1.8     |
+| 0.7.0 (unreleased, master)                            |      2.4      |      2.11     |      1.8     |


TODO for a future PR: I really hope we start making strides to automating this stuff. We should have here placeholders that are populated from build. Even if it's some quick and dirty version of sed script

build.gradle

gerashegalov · 2020-01-30T08:33:27Z

cli/build.gradle

@@ -69,20 +69,16 @@ task copyTemplates(type: Copy) {
        fileName.replace(".gradle.template", ".gradle")
    }
    expand([
-            databaseHostname: 'db.company.com',


do we absolutely need to change this file ?

this was a cleanup, why not?

because the PR is big enough, and for easier maintenance I hoped to keep it focussed on the Spark upgrade

Sure, this has no material difference for the change. You always get a bonus of such with my PRs 😛

gerashegalov · 2020-01-30T08:34:08Z

cli/src/main/scala/com/salesforce/op/cli/SchemaSource.scala

@@ -138,7 +138,7 @@ case class AutomaticSchema(recordClassName: String)(dataFile: File) extends Sche
        case Some(actualType) =>
          val newSchema = Schema.create(actualType)
          val schemaField =
-            new Schema.Field(field.name, newSchema, "auto-generated", orgSchemaField.defaultValue)
+            new Schema.Field(field.name, newSchema, "auto-generated", orgSchemaField.defaultVal())


do we need this change? defaultValue is deprecated but still there, technically not required to change until it's dropped entirely

I am keeping the existing functionality as is. We can change the behavior separately.

core/src/main/scala/com/salesforce/op/stages/impl/classification/OpLinearSVC.scala

core/src/main/scala/com/salesforce/op/stages/impl/classification/OpXGBoostClassifier.scala

core/src/test/scala/com/salesforce/op/ModelInsightsTest.scala

core/src/test/scala/com/salesforce/op/stages/impl/classification/OpClassifierModelTest.scala

tovbinm · 2020-01-30T17:36:17Z

@gerashegalov thanks for reviewing. I hope we can get this merged and released soon. Especially because I don't even see an option now to download Spark lower than 2.4 from the official website, so our users might struggle to run current version of TransmogrifAI.

gerashegalov

LGETM

gerashegalov · 2020-01-31T00:38:04Z

@nicodv feel free to address the comments that make sense now (or not) and merge.

Helw150 · 2020-02-13T22:52:07Z

@tovbinm Do you have an ETA on merging this? I'm using Spark 2.4, but would love to get started doing a proposed POC with TransmogrifAI so am waiting excitedly!

tovbinm · 2020-02-14T02:22:23Z

I bumped Spark to 2.4.5 (it was just recently released). Should we bump other dependencies as well, e.g. mleap?

nicodv · 2020-02-15T00:52:50Z

@tovbinm , could you please revert the commits related to Spark 2.4.5? We're planning to do an internal release of TMog on 2.4.4 soon, and don't want to redo any testing / take on additional risk (even if it's just a patch release).

We can do a separate upgrade of TMog to Spark 2.4.5 soon after that in a separate PR.

This reverts commit b3c0a74.

This reverts commit f4ab3fd.

This reverts commit 50d9dfb.

This reverts commit 3417972.

This reverts commit df38bcc.

tovbinm · 2020-02-15T01:28:57Z

@nicodv ok

leahmcguire · 2020-02-19T20:18:48Z

Woohooo!

tovbinm · 2020-02-19T20:24:01Z

Made my day!

salesforce-cla · 2020-10-27T15:01:31Z

Thanks for the contribution! It looks like @Jauntbox is an internal user so signing the CLA is not required. However, we need to confirm this.

salesforce-cla · 2020-10-27T15:01:31Z

Thanks for the contribution! Before we can merge this, we need @wsuchy to sign the Salesforce.com Contributor License Agreement.

Revert "Revert back to Spark 2.3 (#399)"

77e2229

This reverts commit 95a77b1.

tovbinm added work in progress DO NOT MERGE labels Sep 4, 2019

tovbinm requested review from gerashegalov, Jauntbox, leahmcguire and wsuchy as code owners September 4, 2019 05:13

tovbinm changed the title ~~Spark 2.4~~ Spark 2.4 support Sep 4, 2019

tovbinm mentioned this pull request Sep 25, 2019

XGBoost error code 255 #181

Closed

tovbinm and others added 20 commits October 14, 2019 13:36

Update to Spark 2.4.3 and XGBoost 0.90

485fcd5

special double serializer fix

12f5333

fix serialization

a4ee986

fix serialization

47703fa

docs

ea4b11f

fixed missng value for test

ab81e31

meta fix

c017869

Updated DecisionTreeNumericMapBucketizer test to deal with the change…

99ea7e1

… made to decision tree pruning in Spark 2.4. If nodes are split, but both child nodes lead to the same prediction then the split is pruned away. This updates the test so this doesn't happen for feature 'b'

fix params meta test

daa2672

FIxed failing xgboost test

7d3ebb7

ident

5661640

cleanup

8852d69

added dataframe reader and writer extensions

2ab8924

added const

7c8b988

cherrypick fixes

d98c8a9

added xgboost params + update models to use public predict method

cce0d8f

blarg

8afbae7

double ser test

e8770f6

update mleap and spark testing base

afffc56

Update README.md

ed43719

gerashegalov reviewed Jan 30, 2020

View reviewed changes

Merge branch 'master' into revert-399-mt/revert-spark-2.4

e0c164d

gerashegalov approved these changes Jan 31, 2020

View reviewed changes

leahmcguire approved these changes Feb 13, 2020

View reviewed changes

tovbinm added 5 commits February 13, 2020 18:08

Spark 2.4.5

b3c0a74

Spark 2.4.5

f4ab3fd

Spark 2.4.5

50d9dfb

Update OpTitanicSimple.ipynb

3417972

Update OpIris.ipynb

df38bcc

tovbinm added 5 commits February 14, 2020 17:27

Revert "Spark 2.4.5"

3af27f3

This reverts commit b3c0a74.

Revert "Spark 2.4.5"

66afd79

This reverts commit f4ab3fd.

Revert "Spark 2.4.5"

39ff755

This reverts commit 50d9dfb.

Revert "Update OpTitanicSimple.ipynb"

09c5f2b

This reverts commit 3417972.

Revert "Update OpIris.ipynb"

d79bcc3

This reverts commit df38bcc.

nicodv merged commit fbad63d into master Feb 19, 2020

nicodv deleted the revert-399-mt/revert-spark-2.4 branch February 19, 2020 20:18

nicodv removed the ready for review label Feb 20, 2020

nicodv mentioned this pull request Jun 11, 2020

0.7.0 release #481

Merged

salesforce-cla bot added cla:missing and removed cla:signed labels Oct 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spark 2.4 support #402

Spark 2.4 support #402

tovbinm commented Sep 4, 2019 •

edited

Loading

codecov bot commented Sep 4, 2019 •

edited

Loading

gerashegalov Jan 29, 2020 •

edited

Loading

gerashegalov Jan 30, 2020

tovbinm Jan 30, 2020

gerashegalov Jan 30, 2020 •

edited

Loading

tovbinm Jan 30, 2020

gerashegalov Jan 30, 2020

tovbinm Jan 30, 2020

tovbinm commented Jan 30, 2020 •

edited

Loading

gerashegalov left a comment

gerashegalov commented Jan 31, 2020

Helw150 commented Feb 13, 2020

tovbinm commented Feb 14, 2020

nicodv commented Feb 15, 2020

tovbinm commented Feb 15, 2020

leahmcguire commented Feb 19, 2020

tovbinm commented Feb 19, 2020

salesforce-cla bot commented Oct 27, 2020

salesforce-cla bot commented Oct 27, 2020

Spark 2.4 support #402

Spark 2.4 support #402

Conversation

tovbinm commented Sep 4, 2019 • edited Loading

codecov bot commented Sep 4, 2019 • edited Loading

Codecov Report

gerashegalov Jan 29, 2020 • edited Loading

Choose a reason for hiding this comment

gerashegalov Jan 30, 2020

Choose a reason for hiding this comment

tovbinm Jan 30, 2020

Choose a reason for hiding this comment

gerashegalov Jan 30, 2020 • edited Loading

Choose a reason for hiding this comment

tovbinm Jan 30, 2020

Choose a reason for hiding this comment

gerashegalov Jan 30, 2020

Choose a reason for hiding this comment

tovbinm Jan 30, 2020

Choose a reason for hiding this comment

tovbinm commented Jan 30, 2020 • edited Loading

gerashegalov left a comment

Choose a reason for hiding this comment

gerashegalov commented Jan 31, 2020

Helw150 commented Feb 13, 2020

tovbinm commented Feb 14, 2020

nicodv commented Feb 15, 2020

tovbinm commented Feb 15, 2020

leahmcguire commented Feb 19, 2020

tovbinm commented Feb 19, 2020

salesforce-cla bot commented Oct 27, 2020

salesforce-cla bot commented Oct 27, 2020

tovbinm commented Sep 4, 2019 •

edited

Loading

codecov bot commented Sep 4, 2019 •

edited

Loading

gerashegalov Jan 29, 2020 •

edited

Loading

gerashegalov Jan 30, 2020 •

edited

Loading

tovbinm commented Jan 30, 2020 •

edited

Loading