Spark and Spark History 2.3.1-2.2.1-2

Updates

• Updated libmesos version with critical bug fix, MESOS-8171.

Documentation

• Added a page documenting results from scale testing of Spark on DC/OS.

Spark and Spark History 2.3.0-2.2.1-2

New Features

Spark

• Added secrets support in Driver, so a secret can be disseminated to the executors. (SPARK-22131)
• Added Kerberos ticket renewal. (SPARK-21842)
• Added Mesos sandbox URI to Dispatcher UI. (SPARK-13041)
• Added support for Driver<->Executor TLS with file-based secrets.
• Added support for Driver<->Executor SASL (RPC endpoint authentication and encryption), via file-based secrets.
• Added --executor-auth-secret as a shortcut for Driver<->Executor Spark SASL (RPC endpoint authentication and encryption) configuration.
• Added CLI command to generate a random secret.
• Enabled native BLAS for MLLib.
• Added configuration to deploy Dispatcher on UCR (default is Docker).
• Instead of setting the krb5.conf as a base64-encoded blob, the user can now specify service.security.kerberos.kdc.[port|hostname] and service.security.kerberos.realm directly in options.json. The behavior with the base64-encoded blob remains the same, and will overwrite the new configs.

History Server

• Added Kerberos support for integration with a Kerberized HDFS. See documentation for configuration instructions.
• Made the user configurable, defaults to root.

Updates

Spark

• Updated JRE version to 8u152 JCE.
• Changed the default user to root. (Breaking change)

Bug fixes

• First delegation token renewal time is not 75% of renewal time. (SPARK-22583)
• Fixed supervise mode with checkpointing. (SPARK-22145)
• Added support for older SPARK_MESOS_KRB5_CONF_BASE64 environment variable.
• The spark CLI has "shortcut" command-line args, that are translated into spark.config=setting configurations downstream (such as spark.executor.memory). Fixed a bug where a user sets the configuration directly and is overwritten with the default value for the shortcut argument.

Breaking Changes

• Changed the default user to root, in both the Dispatcher and History Server.
• To configure Kerberos in options.json, a new property service.security.kerberos.enabled must be set to true. This applies to both the Dispatcher and History Server.
• Removed the security.ssl properties from options.json. These properties are no longer needed for the new Go-based CLI.
• Removed --dcos-space option from the CLI. Access to secrets is determined by the Spark Dispatcher service name. See the Spark Security doc page for more information about where to place secrets.

Spark and Spark History 2.3.1-2.2.1-2-beta

NOTICE

This is a beta release of the DC/OS Spark framework. It contains multiple improvements as well as new features that are to be considered of beta quality. Do not operate this version in production.

Improvements

• Upgraded to Spark 2.2.1

Bug Fixes

• Fixed the security.kerberos.krb5conf media type in the spark-history config
• Fixed a stale jar link in the "Install and Customize" doc page

Spark and Spark-History 2.3.0-2.2.0-2-beta

NOTICE

This is a beta release of the DC/OS Spark framework. It contains multiple improvements as well as new features that are to be considered of beta quality. Do not operate this version in production.

Improvements

• Added --executor-auth-secret as a shortcut for Spark SASL (Executor authentication and BlockTransferService encryption) configuration.
• Changed the default user to root. (Breaking change)
• Instead of setting the krb5.conf as a base64-encoded blob, the user can now specify service.security.kerberos.kdc.[port|hostname] and service.security.kerberos.realm directly in options.json. The old behavior with the base64-encoded blob remains the same, and will overwrite the new configs.

Bug Fixes

• The spark CLI has "shortcut" command-line args, that are translated into spark.config=setting configurations downstream (such as spark.executor.memory). Fixed a bug where a user sets the configuration directly and is overwritten with the default value for the shortcut argument.

Tests

• Changes to allow integration tests to run in strict mode.

Documentation

• Added worked examples (walkthroughs) for setting up Spark securely.
• Added docs on using Mesos Quota to manage resources in job scheduling.
• Added instructions to Install docs describing how to install in strict mode.

Breaking Changes

• Changed the default user to root, in both the Dispatcher and History Server.
• To configure Kerberos in options.json, a new property service.security.kerberos.enabled must be set to true. This applies to both the Dispatcher and History Server.
• Removed the security.ssl properties from options.json. These properties are no longer needed for the new Go-based CLI.

Spark-History 2.1.0-2.2.1-1

Improvements

• Upgraded Spark to 2.2.1.

Spark-history 2.1.0-2.2.0-1

Improvements

• Upgraded docker image to 2.1.0-2.2.0-1-hadoop-2.6.
• Wrapped the config properties in a "service" object. (See "Breaking Changes")

Bug Fixes

• Fixed field names in marathon.json. Installing from the DC/OS UI now works.

Breaking Changes

• The configuration properties have been wrapped in a new "service" object. If installing from the CLI, please update your options.json file accordingly. See the Spark history server docs for an example.

Spark 2.1.0-2.2.1-1

Improvements

• Upgraded Spark to 2.2.1.

Spark 2.2.0-2.2.0-2-beta

NOTICE

This is a beta release of the DC/OS Spark framework. It contains multiple improvements as well as new features that are to be considered of beta quality. Do not operate this version in production.

Improvements

• Added secrets support in Driver. (SPARK-22131)
• Added Kerberos ticket renewal. (SPARK-21842)
• Added Mesos sandbox URI to Dispatcher UI. (SPARK-13041)
• Updated JRE version to 8u152 JCE.
• Added support for Driver<->Executor TLS with file-based secrets.
• Added support for Driver<->Executor SASL (RPC endpoint authentication and encryption), via file-based secrets.
• Added CLI command to generate a random secret.
• Enabled native BLAS for MLLib.
• Added configuration to deploy Dispatcher on UCR (default is Docker).

Bug fixes

• First delegation token renewal time is not 75% of renewal time. (SPARK-22583)
• Fixed supervise mode with checkpointing. (SPARK-22145)
• Added support for older SPARK_MESOS_KRB5_CONF_BASE64 environment variable.

Tests

• Added integration test that reads / writes to a Kerberized HDFS.
• Added integration test that reads / writes to a Kerberized Kafka.
• Added integration test of checkpointing and supervise.

Documentation

• Updated naming of DC/OS.
• Updated docs links in package post-install notes.
• Updated Kerberos docs.
• Documented running Spark Streaming jobs with Kerberized Kafka.
• Documented nobody limitation on certain OSes.

Spark 2.1.0-2.2.0-1

Improvements

• Changed image to run as user nobody instead of root by default. (#189)

Bug fixes

• Configuration to allow custom Dispatcher docker image. (#179)
• CLI breaks with multiple spaces in submit args. (#193)

Documentation

• Updated HDFS endpoint in hdfs doc page.
• Added checkpointing instructions. (#181)
• Updated custom docker image support policy. (#200)

Spark 2.0.1-2.2.0-1

Improvements

• Exposed isR and isPython spark run args

Bug fixes

• Allowed for application args to have arguments without equals sign
• Fixed docs link in Universe package description