Spark 2.2.0-2.2.0-2-beta

NOTICE

This is a beta release of the DC/OS Spark framework. It contains multiple improvements as well as new features that are to be considered of beta quality. Do not operate this version in production.

Improvements

• Added secrets support in Driver. (SPARK-22131)
• Added Kerberos ticket renewal. (SPARK-21842)
• Added Mesos sandbox URI to Dispatcher UI. (SPARK-13041)
• Updated JRE version to 8u152 JCE.
• Added support for Driver<->Executor TLS with file-based secrets.
• Added support for Driver<->Executor SASL (RPC endpoint authentication and encryption), via file-based secrets.
• Added CLI command to generate a random secret.
• Enabled native BLAS for MLLib.
• Added configuration to deploy Dispatcher on UCR (default is Docker).

Bug fixes

• First delegation token renewal time is not 75% of renewal time. (SPARK-22583)
• Fixed supervise mode with checkpointing. (SPARK-22145)
• Added support for older SPARK_MESOS_KRB5_CONF_BASE64 environment variable.

Tests

• Added integration test that reads / writes to a Kerberized HDFS.
• Added integration test that reads / writes to a Kerberized Kafka.
• Added integration test of checkpointing and supervise.

Documentation

• Updated naming of DC/OS.
• Updated docs links in package post-install notes.
• Updated Kerberos docs.
• Documented running Spark Streaming jobs with Kerberized Kafka.
• Documented nobody limitation on certain OSes.