spark

An additinal state for the Salt Stack hadoop-formula , which installs apache spark on the hadoop_master

Relies on hadoop-formula/hadoop/settings.sls. This is intended to be added to the formula directory like so:

git clone https://github.com/mraymond77/spark-formula
cd spark-formula
mv spark /srv/salt/formulas/hadoop-formula/hadoop/

hadoop.spark will then be available as a state.

hadoop.spark

Installs a prebuilt spark directory and creates a link to /usr/lib/spark

Spark is installed on the hadoop_master only, and no further configuration is needed. Spark in yarn mode uses hadoop configuration files to identify cluster members. For other modes, see the spark documentation.

Similar to the hadoop.hdfs state, this requires the hadoop roles be defined in /etc/salt/grains

roles:
  - hadoop_slave

and

roles:
  - hadoop_master

Version settings in the hadoop pillar

A key for spark should be added to the hadoop pillar to specify a version. Otherwise, it defaults to spark-1.5.1-bin-hadoop2.6

spark:
  version: 1.5.1-bin-hadoop2.6