Creating a workflow for a full extract and process of the table in source & model space

Use these instructions to do a full load of a table in the source space which will be extended in the SX-Dimension-Extension space. When you do a full load, the extended column are updated to all rows in the model space.

  1. In Birst, select the [Tenant]-CSD-Source space.
  2. Next to Modeler, click Connect.
  3. Select the CSD Infor Data Lake.
  4. Hover your mouse over the object, click the down arrow and then click Edit.
  5. Scroll to the bottom of the Query text box.
  6. Replace Extract Start Date variable in the WHERE lastModified statement with 2019-01-01T00:00:00.000Z.
    Example before the change:
    WHERE lastModified >= 'V{InforCSD_oeel_ExtractStartDate}'
    Example after the change:
    WHERE lastModified >= '2019-01-01T00:00:00.000Z'
  7. Click Save.
  8. Repeat steps 1-7 for additional sources.
  9. Select Modeler > Admin 2.0.
  10. Click Orchestration.
  11. Click Create Workflow.
  12. For Step 1, click Add task to step.
  13. From the Task Type drop down, select Extract.
  14. Next to the Space field, click the icon to view the list of spaces.
  15. Select [Tenant]-SX-Source.
  16. In the Connection field, select CSD Infor Data Lake.
  17. Click Select and then click the extract group in the list. For example, OEEL_EXT.
  18. Click Add new step.
  19. For Step 2, click Add task to step.
  20. Select Publish from the Task Type.
  21. Select the [Tenant]-SX-Source space and click Select.
  22. Select the respective publishing group.
    Press Ctrl to select multiple groups.
    Note: To determine the table processing and publishing group, in Birst Classic Admin, select Define sources > Manage source. Select the source and then click the Properties tab.
  23. Click Add new step.
  24. For Step 3, click Add task to step.
  25. Select Publish from the Task Type.
  26. Select the respective model space. For example, [Tenant]-SX-Model-Sales. Click Select.
  27. Select the respective publishing group. For example, 300-SalesLine.
  28. Assign a name to the workflow and click Save.
  29. Hover your mouse over the new workflow and select Actions > Run Workflow.
  30. When you receive an email indicating the workflow ran successfully, repeat steps 1-7 to restore the Extract Start Date variable to its original value.
    This ensures the incremental extract does not keep extracting full loads from Data Lake.
  31. After the extract and processing is run successfully, open Birst Visualizer in the Consumer space and select the extended attribute or measure from the Default Subject Area to validate it.