EVL – ETL Tool

Products, services and company names referenced in this document may be either trademarks or registered trademarks of their respective owners.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts.

Release Notes

Versions numbering: EVL x.y.z

x – major release, i.e. big changes must happen to advance this number

y – minor releases, i.e. introduce new features

z – bugfixes

Overview

Version 1.0 (2017/07): Initial version of “classical” ETL tool with HDFS support.
New components: Aggreg, Cat, Comp, Cut, Filter, Join, Map, Read, Sort, Tee, Trash, Write.
New commands: Mv, Mkdir.
Version 1.1 (2017/11): Read/Write-Avro (in flat format only), Read/Write-Kafka, Read/Write-Teradata. Validation functions within mapping: ‘is_valid_*’.
New components: Head, Readavro, Readkafka, ReadTD, Tail, Validate, Writeavro, Writekafka, WriteTD.
Version 1.2 (2018/01): Nested fields, Partitioning/parallelism, Read/Write JSON/Avro with full nested fields support.
New utilities: evd_remove_comments, evd_to_avro_scheme.
New components: Assign, Depart, Gather, Merge, Partition, Readjson, Watcher, Writejson.
Version 1.3 (2018/03): Read/Write XML, Records generator.
New components: Generate, Readxml, Writexml.
New command: Cp.
Version 2.0 (2018/11): EVL Job Manager renamed to EVL Worklow, man pages, Read/Write PostgreSQL, standalone components, ‘in_to_out()’ enhanced, Shared lookup tables, Read/Write Parquet, sort within a group.
New utilities: evl_increment_run_id – unique EVL run ID.
New components: Echo, Lookup, Readparq, ReadPG, RunPG, Sortgroup, Writeparq, WritePG.
New commands: Fr, Ls, Rm, Spark.
Version 2.1 (2019/04): Microservices introduced, Read/Write Oracle, Write XLSX and QVX, Docker image available.
New components: ReadOra, RunOra, WriteOra, Writeqvx, Writexlsx.
New commands: Mail, Test.
Version 2.2 (2019/10): EVL Manager – graphical web UI, output to Qlikview/Qliksense’s QVD file,
New utilities: csv2evd, csv2qvd.
New components: Readxls, Readxlsx, Tac, Wc, Writeqvd.
Version 2.3 (2020/04): EVL Manager enhancements, Read and Write supports also sftp://, gs://, s3://, Clang compiler can be used, debug mode.
New utilities: qvd_header.
New components: Readqvd.
New commands: Chmod.
Version 2.4 (2020/10): EVL Manager – Microsevices integration, crontab generation, anonymize IBAN, IP addresses manipulation, MySQL/MariaDB, password handling.
New components: Readmysql, Runmysql, Writemysql.
New commands: cancel, crontab, info, manager, Sleep.
New utilities: json2evd, qvd2evd.
Version 2.5 (2021/04): Read and Write supports also Samba and database URIs, randomize IBAN, EVD manipulation components.
New components: Readevd, Writeevd.
Version 2.6 (2021/10): New options for Readqvd to speed up reading subset of QVD file, secret password handling.
New mapping functions: rsa_encrypt(), rsa_decrypt(), str_to_base64(), base64_to_str().
New commands: key generate, Rmdir.
New utilities: pg2evd.
Version 2.7 (2022/04): Read and Write supports also Google Drive: sdrive://.
New utilities: evd2sql, guess-timestamp-format.
New commands: Touch.
New components: Readsqlite, Runsqlite, Writesqlite.
Version 2.8 (2023/04): New math functions: abs(), ceil(), floor(), max(), min(), pow(), round(), sqrt(), trunc()
New mapping functions: fail(), is_equal(), log(), warn()
New data types: time, interval.
New component: Wc.

EVL – ETL Tool

Table of Contents

Release Notes

Overview