ETL Process Debugging

The ETL Process Debugging design is modified by introducing conditions to hold or exit the ETL process.

From the version 10.2.3, the done.txt file is no more generated to flag the end of the ETL Process. Instead, the alex_xmdbtool_etl table has a new entry with the script_type = DONE. The new entry in the table is done as soon as the ETL thread is started with the startDate = <the ETL thread start Date>,script_type = <DONE> and is_active_script=0. The entry is updated with the endDate=<The ETL end date> and is_active_script=1 as soon as the ETL thread ends.

Assume that in the ETL thread, after the ETL starts, the calling thread monitors these two parameters as signals to the ETL process status:

  • The database entry for the script_type=<DONE> and is_active_script=1
  • The ETL Thread ID

If the DB entry with the script_type=DONE (that was created when the ETL thread started - with is_active_script=0) is updated with the is_active_script=1 and if the status is yes, then the ETL thread status (true if alive, else false) is checked.

  • If the first status is true (db entry with script_type=DONE and is_active_script=1) and the second status is false(ETL thread is not alive) - Indicates a perfect exit and the system assumes that the ETL has ended properly.
  • If the first status is true (db entry with script_type=DONE and is_active_script=1) and the second status is true(ETL thread is alive) - Indicates the ETL has ended properly but the ETL thread is still alive (which is not possible, but the ETL status flags off that the ETL has ended properly).
  • If the first status is false (db entry with script_type=DONE and is_active_script=0) and the second status is false(ETL thread is not alive) - Indicates that the db is not updated, but the ETL thread is dead. This indicates a mid-way crash where the ETL has NOT ended properly and the background process must exit with the appropriate message.
  • If the first status is false (db entry with script_type=DONE and is_active_script=0) and the second status is true(ETL thread is alive) - Indicates the ETL thread is alive and it is still processing. In this case, the system waits and monitors the ETL results.