• A Scala tool which helps deploying Apache Spark stand-alone cluster on EC2 and submitting job.
• Currently supports Spark 2.0.0+.
• There are two modes when using spark-deployer: SBT plugin mode and embedded mode.

Here are the basic steps to run a Spark job (all the sbt commands support TAB-completion):

1. Set the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
2. Prepare a project with structure like below:

project-root
├── build.sbt
├── project
│   └── plugins.sbt
└── src
    └── main
        └── scala
            └── mypackage
                └── Main.scala

3. Add one line in project/plugins.sbt:

addSbtPlugin("net.pishen" % "spark-deployer-sbt" % "3.0.2")

4. Write your Spark project's build.sbt (Here we give a simple example):

name := "my-project-name"
 
scalaVersion := "2.11.8"
 
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.0.0" % "provided"
)

5. Write your job's algorithm in src/main/scala/mypackage/Main.scala:

package mypackage
 
import org.apache.spark._
 
object Main {
  def main(args: Array[String]) {
    //setup spark
    val sc = new SparkContext(new SparkConf())
    //your algorithm
    val n = 10000000
    val count = sc.parallelize(1 to n).map { i =>
      val x = scala.math.random
      val y = scala.math.random
      if (x * x + y * y < 1) 1 else 0
    }.reduce(_ + _)
    println("Pi is roughly " + 4.0 * count / n)
  }
}

6. Enter sbt, and build a config by:

> sparkBuildConfig

(Most settings have default values, just hit Enter to go through it.)

7. Create a cluster with 1 master and 2 workers by:

> sparkCreateCluster 2

8. See your cluster's status by:

> sparkShowMachines

9. Submit your job by:

> sparkSubmit

10. When your job is done, destroy your cluster with

> sparkDestroyCluster

• To build config with different name or build a config based on old one:

  > sparkBuildConfig <new-config-name>
  > sparkBuildConfig <new-config-name> from <old-config-name>
  

  All the configs are stored as .deployer.json files in the conf/ folder. You can modify it if you know what you're doing.

• To change the current config:

  > sparkChangeConfig <config-name>
  

• To submit a job with arguments or with a main class:

  > sparkSubmit <args>
  > sparkSubmitMain mypackage.Main <args>
  

• To add or remove worker machines dynamically:

  > sparkAddWorkers <num-of-workers>
  > sparkRemoveWorkers <num-of-workers>
  

If you don't want to use sbt, or if you would like to trigger the cluster creation from within your Scala application, you can include the library of spark-deployer directly:

libraryDependencies += "net.pishen" %% "spark-deployer-core" % "3.0.2"

Then, from your Scala code, you can do something like this:

import sparkdeployer._

// build a ClusterConf
val clusterConf = ClusterConf.build()

// save and load ClusterConf
clusterConf.save("path/to/conf.deployer.json")
val clusterConfReloaded = ClusterConf.load("path/to/conf.deployer.json")

// create cluster and submit job
val sparkDeployer = new SparkDeployer()(clusterConf)

val workers = 2
sparkDeployer.createCluster(workers)

val jar = new File("path/to/job.jar")
val mainClass = "mypackage.Main"
val args = Seq("arg0", "arg1")
sparkDeployer.submit(jar, mainClass, args)

sparkDeployer.destroyCluster()

• Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY should also be set.
• You may prepare the job.jar by sbt-assembly from other sbt project with Spark.
• For other available functions, check SparkDeployer.scala in our source code.

spark-deployer uses slf4j, remember to add your own backend to see the log. For example, to print the log on screen, add

libraryDependencies += "org.slf4j" % "slf4j-simple" % "1.7.14"

Yes, just specify the ami id when running sparkBuildConfig. The image should be HVM EBS-Backed with Java 7+ installed. You can also run some commands before Spark start on each machine by editing the preStartCommands in json config. For example:

"preStartCommands": [
  "sudo bash -c \"echo -e 'LC_ALL=en_US.UTF-8\\nLANG=en_US.UTF-8' >> /etc/environment\"",
  "sudo apt-get -qq install openjdk-8-jre",
  "cd spark/conf/ && cp log4j.properties.template log4j.properties && echo 'log4j.rootCategory=WARN, console' >> log4j.properties"
]

When using custom ami, the root device should be your root volume's name (/dev/sda1 for Ubuntu) that can be enlarged by disk size settings in master and workers.

Yes, just change the tgz url when running sparkBuildConfig, the tgz will be extracted as a spark/ folder in each machine's home folder.

Assuming your security group id is sg-abcde123, the basic settings is:

Change to the config you want to upgrade, and run sparkUpgradeConfig to build a new config based on settings in old one. If this doesn't work or you don't mind rebuilding one from scratch, it's recommended to directly create a new config by sparkBuildConfig.

You can change it by add the following line to your build.sbt:

sparkConfigDir := "path/to/my-config-dir"

• Please report issue or ask on gitter if you meet any problem.
• Pull requests are welcome.