spark.executor.memoryspark.driver.cores512m, 2g).
+ Number of cores to use for the driver process, only in cluster mode.
512m, 2g).
spark.executor.memory512m, 2g).
+ spark.driver.maxResultSize| Property Name | Default | Meaning | ||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
spark.driver.extraJavaOptions |
+ (none) | ++ A string of extra JVM options to pass to the driver. For instance, GC settings or other logging. + | +||||||||||||||||||||||||||||||||||||||||||||||
spark.driver.extraClassPath |
+ (none) | ++ Extra classpath entries to append to the classpath of the driver. + | +||||||||||||||||||||||||||||||||||||||||||||||
spark.driver.extraLibraryPath |
+ (none) | ++ Set a special library path to use when launching the driver JVM. + | +||||||||||||||||||||||||||||||||||||||||||||||
spark.executor.extraJavaOptions |
(none) | diff --git a/docs/ml-guide.md b/docs/ml-guide.md index 1c2e27341473b..be178d7689fdd 100644 --- a/docs/ml-guide.md +++ b/docs/ml-guide.md @@ -3,13 +3,16 @@ layout: global title: Spark ML Programming Guide --- -Spark ML is Spark's new machine learning package. It is currently an alpha component but is potentially a successor to [MLlib](mllib-guide.html). The `spark.ml` package aims to replace the old APIs with a cleaner, more uniform set of APIs which will help users create full machine learning pipelines. +`spark.ml` is a new package introduced in Spark 1.2, which aims to provide a uniform set of +high-level APIs that help users create and tune practical machine learning pipelines. +It is currently an alpha component, and we would like to hear back from the community about +how it fits real-world use cases and how it could be improved. -MLlib vs. Spark ML: - -* Users can use algorithms from either of the two packages, but APIs may differ. Currently, `spark.ml` offers a subset of the algorithms from `spark.mllib`. Since Spark ML is an alpha component, its API may change in future releases. -* Developers should contribute new algorithms to `spark.mllib` and can optionally contribute to `spark.ml`. See below for more details. -* Spark ML only has Scala and Java APIs, whereas MLlib also has a Python API. +Note that we will keep supporting and adding features to `spark.mllib` along with the +development of `spark.ml`. +Users should be comfortable using `spark.mllib` features and expect more features coming. +Developers should contribute new algorithms to `spark.mllib` and can optionally contribute +to `spark.ml`. **Table of Contents** @@ -686,17 +689,3 @@ Spark ML currently depends on MLlib and has the same dependencies. Please see the [MLlib Dependencies guide](mllib-guide.html#Dependencies) for more info. Spark ML also depends upon Spark SQL, but the relevant parts of Spark SQL do not bring additional dependencies. - -# Developers - -**Development plan** - -If all goes well, `spark.ml` will become the primary ML package at the time of the Spark 1.3 release. Initially, simple wrappers will be used to port algorithms to `spark.ml`, but eventually, code will be moved to `spark.ml` and `spark.mllib` will be deprecated. - -**Advice to developers** - -During the next development cycle, new algorithms should be contributed to `spark.mllib`, but we welcome patches sent to either package. If an algorithm is best expressed using the new API (e.g., feature transformers), we may ask for developers to use the new `spark.ml` API. -Wrappers for old and new algorithms can be contributed to `spark.ml`. - -Users will be able to use algorithms from either of the two packages. The main difficulty will be the differences in APIs between the two packages. - diff --git a/docs/mllib-collaborative-filtering.md b/docs/mllib-collaborative-filtering.md index 2094963392295..ef18cec9371d6 100644 --- a/docs/mllib-collaborative-filtering.md +++ b/docs/mllib-collaborative-filtering.md @@ -192,12 +192,11 @@ We use the default ALS.train() method which assumes ratings are explicit. We eva recommendation by measuring the Mean Squared Error of rating prediction. {% highlight python %} -from pyspark.mllib.recommendation import ALS -from numpy import array +from pyspark.mllib.recommendation import ALS, Rating # Load and parse the data data = sc.textFile("data/mllib/als/test.data") -ratings = data.map(lambda line: array([float(x) for x in line.split(',')])) +ratings = data.map(lambda l: l.split(',')).map(lambda l: Rating(int(l[0]), int(l[1]), float(l[2]))) # Build the recommendation model using Alternating Least Squares rank = 10 @@ -205,10 +204,10 @@ numIterations = 20 model = ALS.train(ratings, rank, numIterations) # Evaluate the model on training data -testdata = ratings.map(lambda p: (int(p[0]), int(p[1]))) +testdata = ratings.map(lambda p: (p[0], p[1])) predictions = model.predictAll(testdata).map(lambda r: ((r[0], r[1]), r[2])) ratesAndPreds = ratings.map(lambda r: ((r[0], r[1]), r[2])).join(predictions) -MSE = ratesAndPreds.map(lambda r: (r[1][0] - r[1][1])**2).reduce(lambda x, y: x + y)/ratesAndPreds.count() +MSE = ratesAndPreds.map(lambda r: (r[1][0] - r[1][1])**2).reduce(lambda x, y: x + y) / ratesAndPreds.count() print("Mean Squared Error = " + str(MSE)) {% endhighlight %} @@ -217,7 +216,7 @@ signals), you can use the trainImplicit method to get better results. {% highlight python %} # Build the recommendation model using Alternating Least Squares based on implicit ratings -model = ALS.trainImplicit(ratings, rank, numIterations, alpha = 0.01) +model = ALS.trainImplicit(ratings, rank, numIterations, alpha=0.01) {% endhighlight %} diff --git a/docs/mllib-guide.md b/docs/mllib-guide.md index efd7dda310712..39c64d06926bf 100644 --- a/docs/mllib-guide.md +++ b/docs/mllib-guide.md @@ -35,16 +35,20 @@ MLlib is under active development. The APIs marked `Experimental`/`DeveloperApi` may change in future releases, and the migration guide below will explain all changes between releases. -# spark.ml: The New ML Package +# spark.ml: high-level APIs for ML pipelines -Spark 1.2 includes a new machine learning package called `spark.ml`, currently an alpha component but potentially a successor to `spark.mllib`. The `spark.ml` package aims to replace the old APIs with a cleaner, more uniform set of APIs which will help users create full machine learning pipelines. +Spark 1.2 includes a new package called `spark.ml`, which aims to provide a uniform set of +high-level APIs that help users create and tune practical machine learning pipelines. +It is currently an alpha component, and we would like to hear back from the community about +how it fits real-world use cases and how it could be improved. -See the **[spark.ml programming guide](ml-guide.html)** for more information on this package. - -Users can use algorithms from either of the two packages, but APIs may differ. Currently, `spark.ml` offers a subset of the algorithms from `spark.mllib`. +Note that we will keep supporting and adding features to `spark.mllib` along with the +development of `spark.ml`. +Users should be comfortable using `spark.mllib` features and expect more features coming. +Developers should contribute new algorithms to `spark.mllib` and can optionally contribute +to `spark.ml`. -Developers should contribute new algorithms to `spark.mllib` and can optionally contribute to `spark.ml`. -See the `spark.ml` programming guide linked above for more details. +See the **[spark.ml programming guide](ml-guide.html)** for more information on this package. # Dependencies diff --git a/docs/programming-guide.md b/docs/programming-guide.md index 5e0d5c15d7069..2443fc29b4706 100644 --- a/docs/programming-guide.md +++ b/docs/programming-guide.md @@ -913,7 +913,7 @@ for details.|||||||||||||||||||||||||||||||||||||||||||||||
| cogroup(otherDataset, [numTasks]) | - When called on datasets of type (K, V) and (K, W), returns a dataset of (K, Iterable<V>, Iterable<W>) tuples. This operation is also called groupWith. |
+ When called on datasets of type (K, V) and (K, W), returns a dataset of (K, (Iterable<V>, Iterable<W>)) tuples. This operation is also called groupWith. |
||||||||||||||||||||||||||||||||||||||||||||||
| cartesian(otherDataset) | @@ -1316,7 +1316,35 @@ For accumulator updates performed inside actions only, Spark guarantees t will only be applied once, i.e. restarted tasks will not update the value. In transformations, users should be aware of that each task's update may be applied more than once if tasks or job stages are re-executed. +Accumulators do not change the lazy evaluation model of Spark. If they are being updated within an operation on an RDD, their value is only updated once that RDD is computed as part of an action. Consequently, accumulator updates are not guaranteed to be executed when made within a lazy transformation like `map()`. The below code fragment demonstrates this property: +||||||||||||||||||||||||||||||||||||||||||||||||
spark.driver.cores |
+ 1 | +
+ Number of cores used by the driver in YARN cluster mode.
+ Since the driver is run in the same JVM as the YARN Application Master in cluster mode, this also controls the cores used by the YARN AM.
+ In client mode, use spark.yarn.am.cores to control the number of cores used by the YARN AM instead.
+ |
+||||||||||||||||||||||||||||||||||||||||||||||
spark.yarn.am.cores |
+ 1 | +
+ Number of cores to use for the YARN Application Master in client mode.
+ In cluster mode, use spark.driver.cores instead.
+ |
+||||||||||||||||||||||||||||||||||||||||||||||
spark.yarn.am.waitTime |
100000 | diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md index 729045b81a8c0..be8c5c2c1522e 100644 --- a/docs/sql-programming-guide.md +++ b/docs/sql-programming-guide.md @@ -1333,9 +1333,9 @@ import org.apache.spark.sql._
| ByteType | byte or Byte | - DataType.ByteType + DataTypes.ByteType | ||
| ShortType | short or Short | - DataType.ShortType + DataTypes.ShortType | ||
| IntegerType | int or Integer | - DataType.IntegerType + DataTypes.IntegerType | ||
| LongType | long or Long | - DataType.LongType + DataTypes.LongType | ||
| FloatType | float or Float | - DataType.FloatType + DataTypes.FloatType | ||
| DoubleType | double or Double | - DataType.DoubleType + DataTypes.DoubleType | ||
| DecimalType | java.math.BigDecimal |
- DataType.DecimalType
+ DataTypes.createDecimalType() + DataTypes.createDecimalType(precision, scale). |
||
| StringType | String | - DataType.StringType + DataTypes.StringType | ||
| BinaryType | byte[] | - DataType.BinaryType + DataTypes.BinaryType | ||
| BooleanType | boolean or Boolean | - DataType.BooleanType + DataTypes.BooleanType | ||
| TimestampType | java.sql.Timestamp | - DataType.TimestampType + DataTypes.TimestampType | ||
| DateType | java.sql.Date | - DataType.DateType + DataTypes.DateType | ||
| ArrayType | java.util.List |
- DataType.createArrayType(elementType) + DataTypes.createArrayType(elementType) Note: The value of containsNull will be true - DataType.createArrayType(elementType, containsNull). + DataTypes.createArrayType(elementType, containsNull). |
||
| MapType | java.util.Map |
- DataType.createMapType(keyType, valueType) + DataTypes.createMapType(keyType, valueType) Note: The value of valueContainsNull will be true. - DataType.createMapType(keyType, valueType, valueContainsNull) + DataTypes.createMapType(keyType, valueType, valueContainsNull) |
||
| StructType | org.apache.spark.sql.api.java.Row |
- DataType.createStructType(fields) + DataTypes.createStructType(fields) Note: fields is a List or an array of StructFields. Also, two fields with the same name are not allowed. |
@@ -1458,7 +1459,7 @@ please use factory methods provided in
The value type in Java of the data type of this field (For example, int for a StructField with the data type IntegerType) | - DataType.createStructField(name, dataType, nullable) + DataTypes.createStructField(name, dataType, nullable) |