Incremental loading

Incremental load is the processing of loading only new or updated records from data sources into the hierarchies and facts. This process typically runs with higher efficiency than full loads which are data extractions that load all data. Incremental loading is implemented in the Source space and run in the workflow of the M3 Analytics solution.

These terms are used in incremental loading:

  • Load Frequency

    Describes how often the load processes are run. Load processes are usually run daily but you can also run it weekly, monthly, or at any time interval.

  • Change Data Capture

    This method is used to detect changes to a source record.

  • Extraction Window

    The duration of the extraction period. The value, in minutes, that is added to the load start time to calculate the end date and time. This should be specified in the M3A_ExtractWindowMin variable.

  • Job Stream/Workflow

    A series of jobs, which run in a controlled stream, the order and parallelism of which are controlled by defining dependencies between jobs. For example, one job can depend on one or more other jobs completing first.

  • Full Load

    All data is extracted from the sources and the data warehouse is re-built. This is often called drop and load.