Incremental load

You can configure the AnySQL modeler to use the selected columns for loading new or changed rows from the source object.

These are the supported column data types:

Date-time
Date
Time
String
Number

The ordering logic for each data type must be taken into account. For example using String comparison 100 < 20 and 100 > 020.

The data in the selected column must be incremental. Each new value must be larger than the previous value. If both of the conditions in the list are true, then AnySQL can exceptionally miss some records:

The column is not fully incremental, for example the datetime datatype has two records in the same millisecond.
The duplicated record is available in the DB with a time delay larger than few millisecond.

To prevent loading data that is not yet complete, specify a value for Time Offset. You can use the Time Offset with Date and Date-Time data types if the Enable Improved Data Type Normalization check box is selected.

For the Time Offset option to work properly, ensure that the time zone is set correctly. Otherwise, the time offset can be longer or records can be unnoticed. For the data types that contain the time zone information, the correct time zone is used automatically. For data types stored without the time zone information, manually specify the time zone of the date time that is stored in the source database.

You can define several columns that must be tracked. The result contains the lines where the maximum value in one of these columns was increased from the last load.

Additionally, you can configure the parent-child relationship between two columns to use incremental load. For example, in case date and time are stored in separate columns.