EVL – ETL Tool


Products, services and company names referenced in this document may be either trademarks or registered trademarks of their respective owners.

Copyright © 2017–2022 EVL Tool, s.r.o.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts.

Table of Contents

Spark

(since EVL 2.0)

In case jar file is specified (i.e. file with mask *.jar), it invokes:

$EVL_SPARK_SUBMIT <spark_submit_options> <jar_file> --name <name>

where EVL_SPARK_SUBMIT is ‘spark-submit’ by default.

When other than jar file is used, then it firstly build the code by ‘$EVL_SPARK_BUILD’, which is by default ‘sbt’, and then run such jar file in above manner.

Synopsis

Spark
  ( <jar_file> | <scala_source> ) [--name <name>]

evl spark
  ( <jar_file> | <scala_source> ) [--name <name>]
  [--verbose]

evl spark
  ( --help | --usage | --version )

Options

Standard options:

--help

print this help and exit

--usage

print short usage information and exit

-v, --verbose

print to stderr info/debug messages of the component

--version

print version and exit

Examples

Run already built scala code in YARN:

export EVL_SPARK_SUBMIT="--master yarn --executor-memory 2G
                         --conf spark.executor.memoryOverhead=4G"
Spark agregate_something.jar --name aggregate_something

Run scala code in YARN:

export EVL_SPARK_SUBMIT="--master yarn --executor-memory 2G
                         --conf spark.executor.memoryOverhead=4G"
Spark agregate_something.scala --name aggregate_something