EVL – ETL Tool

Settings

EVL installation resides (usually) in

/opt/EVL-2.5

To initiate EVL for current user, run

/opt/EVL-2.5/bin/evl init

which add an .evlrc file into your $HOME folder and add sourcing it into $HOME/.bashrc.

Then one can check, add or modify some settings in .evlrc, for example variable EVL_ENV. These settings are top level settings for given user. (Later there are project.sh files in each project, to set project-wide variables.)

After that EVL is ready to use. Good to start is to create new project with some sample data, jobs and workflows:

evl project sample my_first_sample

Compiler

Mappings are compiled either by GCC or Clang. It depends on environment variable EVL_COMPILER, these two values are possible:

EVL_COMPILER=gcc
EVL_COMPILER=clang

If this variable is not set, then on Linux systems is GCC by default, and on Windows and Mac it is Clang.

GCC must be at least in the version 7.4 and Clang at least 6.0.

When Clang would be the option, one can replace gcc/c++ packages by clang above in installation instructions.

Project

EVL project is a directory with EVL jobs, workflow, data structure definitions, mappings, etc. Each project is intended for a group of jobs and workflows which are grouped somehow from the business point of view. So completely unrelated processings, which share nothing, would be good to place in separate projects.

project.sh

Each project has project.sh file inside. This file contains project-wide settings. If the project is created by evl project new or by evl project sample command, then project.sh would contain a good set of variables to start with.

Path to log and tmp folders are handled by environment variables:

EVL_PROJECT_LOG_DIR

path to folder where log files are stored,

EVL_PROJECT_TMP_DIR

path to folder where temporary files are stored.

By default are these variables set to:

EVL_PROJECT_LOG_DIR="$EVL_LOG_PATH/<project_name>"
EVL_PROJECT_TMP_DIR="$EVL_TMP_PATH/<project_name>"

These values can be override in project.sh file.

Project directory structure

In each project directory, there would be these files and folders:

.comp_bin/

contains subdirectories of compiled components of each job (one subfolder equals to one evl file).

doc/

generated and/or custom documentation of the project.

evc/*.evc

EVL custom component definition files.

evd/*.evd

EVL data definition files.

evs/*.evs

EVL job structure files (job definition itself, sometimes called a graph of ETL process).

evm/*.evm

EVL mapping files for components Aggreg, Join and Map.

ews/*.ews

EVL workflow structure files.

run/*.evl

files, called by evl run command, which specify the variables for the job structure, i.e. it contains parameters of the job described by evs file.

workflow/*.ewf

EVL workflow parameter files for ews files.

project.sh

This file is interpreted as bash shell script at the beginning of each job in the project. Usually contains all project wide variables.