Running incremental ETL from the XMDBtool Client

The existing ETL batch file in the XMDBTool Client cannot update the last run dates in the alex_xmdbtool_etl table. Technically, the full ETL runs and the subsequent incremental ETL runs in the XMDBTool ETL Background Process, and updates the alex_xmdbtool_etl table with the last run dates. After the full ETL is run, the next time ETL can be run with only one parameter, incremental=true. The incremental ETL receives the start date (which is the date of the previous ETL run) from the table and the end date is the current date. After the incremental ETL is run, the bkg process updates the table with the current date (which is considered as the start date for the next ETL run).

However, this feature is missing when the ETL runs on the client side. The tables are not updated when the full ETL and the incremental ETL are run on the client side. So, when the incremental ETL is run, we must always specify the start date.

Starting from 10.1 release, when ETL is run (full ETL as well as the incremental ETL) from the xmdbtool on the client side, can access the alex_xmdbtool_etl table and update the last run date. This process is similar to the one in which ETL is run as the background process. The incremental ETL can also run with incremental=true parameter only. This process is similar to the incremental ETL background process.

For example: run_etl.bat -incremental true

This command runs the incremental ETL process, starting from the previous run date (received from the database) to the current date. Also, after the process is completed, the table is updated with the current date.

You can also specify the date range (startDate and endDate) to run the ETL process. When dates are specified in the command line, add a % between the data and the time part:

run_etl.bat -toFile C:\etl\etl.sql -startDate 01-01-2000%00:00:00 -endDate01-01-2016%00:00:00