Incremental loading

Incremental load is the processing of loading only new or updated records from data sources into the hierarchies and facts. This process typically runs with higher efficiency than full loads which are data extractions that load all data. Incremental loading is implemented in the Source space and run in the workflow of the M3 Analytics solution.

These terms are used in incremental loading:

• Load Frequency

  Describes how often the load processes are run. Load processes are usually run daily but you can also run it weekly, monthly, or at any time interval.

• Change Data Capture

  This method is used to detect changes to a source record.

• Extraction Window

  The duration of the extraction period. The value, in minutes, that is added to the load start time to calculate the end date and time. This should be specified in the M3A_ExtractWindowMin variable.

• Job Stream or Workflow

  A series of jobs, which run in a controlled stream, the order and parallelism of which are controlled by defining dependencies between jobs. For example, one job can depend on one or more other jobs completing first.

• Full Load

  All data is extracted from the sources and the data warehouse is rebuilt. This is often called drop and load.