java - How to set amount of Spark executors?

Question

How could I configure from Java (or Scala) code amount of executors having SparkConfig and SparkContext? I see constantly 2 executors. Looks like spark.default.parallelism does not work and is about something different.

I just need to set amount of executors to be equal to cluster size but there are always only 2 of them. I know my cluster size. I run on YARN if this matters.

question from:https://stackoverflow.com/questions/26168254/how-to-set-amount-of-spark-executors

Answer 1

You could also do it programmatically by setting the parameters "spark.executor.instances" and "spark.executor.cores" on the SparkConf object.

Example:

SparkConf conf = new SparkConf()
      // 4 executor per instance of each worker 
      .set("spark.executor.instances", "4")
      // 5 cores on each executor
      .set("spark.executor.cores", "5");

The second parameter is only for YARN and standalone mode. It allows an application to run multiple executors on the same worker, provided that there are enough cores on that worker.