Publish Data Bundles

The Publish Data Bundles task exports selected records from the Infor Operations and Regulations database to the Infor Data Lake.

If the Streaming Ingestion feature toggle is enabled, the task streams records from Operations and Regulations to Infor Data Fabric. This is a newer and more efficient way to publish records to the Data Lake.

If the Streaming Ingestion feature toggle is not enabled, the older ION ingestion method is used to publish data from Operations and Regulations. In this case, additional configuration is required in ION

See ION configuration.

The Data Lake aggregates data from different systems. For example, an instance of Operations and Regulations might have an interface to an instance of Infor CloudSuite Financials and Supply Management. If you publish data from both systems to the Data Lake, then you can use the Data Lake for analytics that encompass the entire integrated solution.

The Data Lake contains various kinds of data objects that can use different formats, such as XML and JSON, as well as files such as PDFs and CSV files. The records that are published from Operations and Regulations are stored in the Data Lake as JSON objects. The schemas for the JSON objects are stored in the Data Catalog.

Records from the Operations and Regulations database are exported in packages called data bundles. For example, the Bill data bundle includes bills and related records such as bill runs, line items, and bill types. This is the general process for publishing data bundles:

Add metadata to the Data Catalog.
Get the records to be exported from the tables in the selected data bundles.
The process is incremental, meaning that only new data is exported. Processing history tables are used to determine what has changed since the last export.
Convert each record to a JSON object.
Infor Messaging Service (IMS) sends the JSON objects to the Data Lake.
The first time a data bundle is published, the task adds database triggers to keep track of any records that are deleted from the tables in that bundle.
When records are deleted in Operations and Regulations, the triggers add information about the deleted records to the DeletedRecords table. Then, each time the task is run, the same records are marked as deleted in the Data Lake. The deleted records are then removed from the DeletedRecords table.

Note: Columns with these data types are excluded when records are published to the Data Lake:

bit
geometry
longvarchar max
varbinary

For example, some of the Operations and Regulations GIS tables, such as GISDATA and IPSDATA, include geometry columns. If you publish these tables you will see all of the other columns in the Data Lake, but the geometry columns are not included.

The exclusion of longvarchar max means that COMMENTS columns in Operations and Regulations are not included.

Infor Operations and Regulations setup

In Operations and Regulations, you use the Data Bundle Setup form to define data bundles. A set of standard Infor data bundles is provided, and you can also add custom data bundles.

This table lists the Operations and Regulations database tables that store setup information for this task:


Table	Comments
Tables under Core.DataLake	These tables store setup information such as data bundle definitions.

This table describes the Operations and Regulations Configuration Editor settings that affect this task:


Configuration	Comments
MAXRECORDS node in Hansen8 configuration	The Datalake Upload Limit attribute specifies the maximum number of records to publish in each batch. The defsult is 1000. If the Streaming Ingestion feature toggle is enabled, the Datalake Upload Limit attribute should be set to 2000 for best performance. Setting this value to less than 2000 will degrade performance.

Invocation

This task can be scheduled to run automatically, or it can be invoked manually from the Batch Manager.

If the Data Bundle Schedules feature toggle is enabled, you can add and view Publish Data Bundles tasks from the Data Bundle Setup form.

For the parameters, you must select the data bundles that you want to publish. The Batch Manager shows a check box for each available data bundle. See the Data Bundle Setup page for a list of the tables in each bundle.

Data output

The published data in the Data Lake is available for consuming applications such as Birst. You can also access the Data Lake through Infor OS.

This table lists the Operations and Regulations database tables in which this task adds or updates records:


Table	Comments
CORE_DATALAKE.BUNDLEPROCESSINGHISTORY	Stores processing history data for specific data bundles.
CORE_DATALAKE.DELETEDRECORDS	Keeps track of records that have been deleted in Operations and Regulations so they can be marked as deleted in the Data Lake.
CORE_DATALAKE.DELETETRIGGERS	Stores information about the delete triggers that the task adds to the database.
CORE_DATALAKE.FACETPROCESSINGHISTORY	Stores processing history data for specific tables. (A facet is equivalent to a table.)

This table lists the Operations and Regulations forms that you can use to review the results of this task:


Form	Comments
Data Bundle Setup	Use the Publish History tab to review previous exports. If the Data Bundle Schedules feature toggle is enabled, you can add and view Publish Data Bundles tasks from the Data Bundle Setup form.