EVL – ETL Tool

Products, services and company names referenced in this document may be either trademarks or registered trademarks of their respective owners.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts.

Wait

(since EVL 2.1)

It waits for successful finish of all previous components of an EVL job (in an EVS file) or all previous ‘Run’ commands in an EVL workflow (in an EWS file). So it acts similar way as standard Bash ‘wait’ command.

Once all previous components/parts of the job/workflow successfully finish the job/workflow continue further.

EVS is EVL job structure definition file, for details see ‘man 5 evl-evs’.

EWS is EVL workflow structure definition file, for details see ‘man 5 evl-ews’.

EVL job (EVS file):

If any component of an EVL job fails, it immediately cancel the whole job.

There is neither argument nor option for ‘Wait’ command in EVL job, i.e. in EVS file. More precisely: all arguments are ignored, so it can be used for comments.

EVL workflow (EWS file):

If any invoked EVL job/workflow (by ‘Run’ command) fails, then cancel only given ‘Run’ command and continue in processing others. Once all ‘Run’ commands finish (either successfully or fail), then ‘Wait’ command fails unless ‘EVL_WAIT_FAIL’ is set to 0.

When no <job>, <workflow> or <script> is specified, further processing will wait <time> for all previously fired parts (i.e. ‘Run’ commands) and fail if at least one of them fails.

When no <time> is specified by ‘--time’ option, then wait at most ‘$EVL_WAIT_TIME’, which is by default 10 hours.

<time> can be secified in seconds, minutes, hours or days, so suffix ‘s’, ‘m’, ‘h’ or ‘d’ need to be specified to the number. If no unit is specified, seconds are assumed.

When <job>, <workflow> or <script> is specified, it waits for successful run of such job/workflow/script of the current project and of the current <odate>. Different <project> and/or <odate> can be specified by the particular options.

Synopsis

Wait
  ( <job>.evl | <workflow>.ewf | <script>.sh )
  [-p|--project=<project>] [-o|--odate=<odate> | -y|--yesterday]
  [-t|--time=<time>]

Wait
  [-m|--myself [-o|--odate=<odate> | -y|--yesterday]]
  [-t|--time=<time>]

evl wait
  ( --help | --usage | --version )

Options

Standard options:

--help: print this help and exit
--usage: print short usage information and exit
--version: print version and exit

Environment Variables

EVL_WAIT_FAIL=1 ¶: EVL workflow only. whether or not to fail the whole workflow when the ‘Run’ command fails, so when zero is set, the workflow continue regardless task failures
EVL_WAIT_INTERVAL=2s ¶: EVL workflow only. the time interval between each check for ‘Wait’ command. It can be specified in seconds, minutes, hours or days, so suffix ‘s’, ‘m’, ‘h’ or ‘d’ need to be specified to the number. If no unit is specified, seconds are assumed.
EVL_WAIT_TIME=10h ¶: EVL workflow only. maximal amount of time to wait for a previous ‘Run’ commands to finish. It can be specified in seconds, minutes, hours or days, so suffix ‘s’, ‘m’, ‘h’ or ‘d’ need to be specified to the number. If no unit is specified, seconds are assumed.

Examples

EVL job

Run EVL job in two steps:

Read  file.json INPUT   evd/file.evd --text-input
Map   INPUT     MAPPED  evd/file.evd evd/stage.evd evm/file.evm
Write MAPPED    stage.parquet        evd/stage.evd

Wait "For for creation of parquet file."

Run   "impala \"refresh table stage;\""

End

EVL workflow

Wait for myself (i.e. the same workflow yesterday) to finish:
```
Wait --myself
```
or the inother words:
```
Wait --myself --yesterday
```
or shortly also:
```
Wait -my
```

Wait for all jobs to finish, but at most 1 hour, then fail:

Run file2stage.bills.evl update.bills.evl
Run file2stage.invoices.evl
Wait 1h

Wait (forever) for successful run of the job ‘file2stage_example.evl’, then continue:
```
Wait file2stage.example.evl
```
Wait (at most 1 day) for the job ‘export_job.evl’ before continue:
```
Wait 1d export_job.evl
```
Wait (at most 120 minutes) for the job ‘sftp_billing.evl’ of the different project ‘billing’ (of current ODATE):
```
Wait 120m sftp_billing.evl billing
```
Wait (at most 300 seconds) for the job ‘sftp_billing.evl’ of the different project ‘billing’ of the 20230215}:
```
Wait 300 sftp_billing.evl billing 20230215
```
Wait (at most 6 hours) for previous run of current ‘EWF’ job. (Variable ‘ODATE_MINUS1’ has to be set by you.):
```
# Wait for previous ODATE 
Wait 6h $EWF_JOB_NAME.ewf $ODATE_MINUS1
```