Partial importing from Data Lake
M3 stores data in Data Lake by using the variationNumber value as the epoch (LMTS) timestamp. When you run , you update the ImportData table (Year column) with the epoch time to record the last import time. TemplateType = 11 is the configuration for Import Data Lake.
You can then create this partial import feature by adding WHERE clause but the user can still change the clause. During Import Data Lake, the application service replaces [LASTIMPORTTIME] with the current value in the ImportData table (Year column). This behavior keeps the WHERE clause continuously updated with each import from Data Lake.
These are some example scenarios:
- Compass query without a WHERE clause: WHERE variationNumber > [LASTIMPORTTIME]
- Compass query with a WHERE clause: WHERE variationNumber > [LASTIMPORTTIME] AND (earlier WHERE clause + “)” )
Note: You can use [LASTIMPORTTIME] for other
columns, not only for variationNumber.
When extracting the WHERE clause, ensure that other keywords do
not follow the clause, such as ORDER BY and
HAVING.