Incremental load start and end scripts

The incremental load process is implemented as a wrapper around the job stream. It consists of two scripts in the Source space: IncrementalLoadStart and IncrementalLoadEnd.

IncrementalLoadStart is the first step in the orchestration process and is included in the Start publishing group. IncrementalLoadEnd is designed to be the last step and is included in the End publishing group. These scripts populate a fact and hierarchy that serve as the history of all workflow executions, including the duration of the whole process.

In general, this is the logic behind IncrementalLoadStart and IncrementalLoadEnd:

• For Initial load, you insert a record in the fact with an In Progress status, the extraction start and end dates, the date and time when the script was run, and the next start date. These are being computed by the script based on the value of the different extraction variables.

  After you run the IncrementalLoadEnd at the end of the workflow, it will update the status of the inserted record to Successful along with other information, such as the end time and the duration of the workflow execution.

• For incremental loads, these scenarios are being handled by IncrementalLoadStart:
  • If the previous run is successful, IncrementalLoadStart will use the next start date computed by the previous run, and then it will insert a record in the table.
  • If the previous run failed, IncrementalLoadStart will set the status of the record to Failed from In Progress, and it will insert a new record with the same values as the previous run.

See the examples and steps for extracting and publishing data sources in KB2153208.

The status of the published domains and sub-domains are shown in the Incremental Load Logs dashboard. See Admin collection.