Spark
(since EVL 2.0)
In case jar file is specified (i.e. file with mask *.jar), it invokes:
$EVL_SPARK_SUBMIT <spark_submit_options> <jar_file> --name <name>
where EVL_SPARK_SUBMIT is ‘spark-submit’ by default.
When other than jar file is used, then it firstly build the code by ‘$EVL_SPARK_BUILD’, which is by default ‘sbt’, and then run such jar file in above manner.
Synopsis
Spark ( <jar_file> | <scala_source> ) [--name <name>] evl spark ( <jar_file> | <scala_source> ) [--name <name>] [--verbose] evl spark ( --help | --usage | --version )
Options
Standard options:
- --help
-
print this help and exit
- --usage
-
print short usage information and exit
- -v, --verbose
-
print to stderr info/debug messages of the component
- --version
-
print version and exit
Examples
Run already built scala code in YARN:
export EVL_SPARK_SUBMIT="--master yarn --executor-memory 2G --conf spark.executor.memoryOverhead=4G" Spark agregate_something.jar --name aggregate_something
Run scala code in YARN:
export EVL_SPARK_SUBMIT="--master yarn --executor-memory 2G --conf spark.executor.memoryOverhead=4G" Spark agregate_something.scala --name aggregate_something