Publish Data Bundles

The Publish Data Bundles task exports selected records from the Infor Public Sector database to the Infor Data Lake. For example, you might want to publish billing account data for use in Birst analytics.

The Data Lake aggregates data from different systems. For example, an instance of Infor Public Sector might have an interface to an instance of Infor CloudSuite Financials and Supply Management. If you publish data from both systems to the Data Lake, you can use the Data Lake for analytics that encompass the entire integrated solution.

The Data Lake contains various kinds of data objects that can use different formats, such as XML and JSON, as well as files such as PDFs and CSV files. The records that are published from Infor Public Sector are stored in the Data Lake as JSON objects. The schemas for the JSON objects are stored in the Data Catalog.

Records from the Infor Public Sector database are exported in packages called data bundles. For example, the Bill data bundle includes bills and related records such as bill runs, line items, and bill types. This is the general process for publishing data bundles:

  1. Add metadata to the Data Catalog.
  2. Get the records to be exported from the tables in the selected data bundles.

    The process is incremental, meaning that only new data is exported. Processing history tables are used to determine what has changed since the last export.

  3. Convert each record to a JSON object.
  4. Infor Messaging Service (IMS) sends the JSON objects to the Data Lake.
  5. The first time a data bundle is published, the task adds database triggers to keep track of any records that are deleted from the tables in that bundle.

    When records are deleted in Infor Public Sector, the triggers add information about the deleted records to the DeletedRecords table. Then, each time the task is run, the same records are marked as deleted in the Data Lake. The deleted records are then removed from the DeletedRecords table.

Note: Columns with these data types are excluded when records are published to the Data Lake:
  • bit
  • geometry
  • longvarchar max
  • varbinary

For example, some of the Infor Public Sector GIS tables, such as GISDATA and IPSDATA, include geometry columns. If you publish these tables you will see all of the other columns in the Data Lake, but the geometry columns are not included.

The exclusion of longvarchar max means that COMMENTS columns in Infor Public Sector are not included.

This task requires setup in both Infor Public Sector and ION Desk.

Infor Public Sector setup

In Infor Public Sector, you use the Data Bundle Setup form to define data bundles. A set of standard Infor data bundles is provided, and you can also add custom data bundles.

This table lists the Infor Public Sector database tables that store setup information for this task:

Table Comments
Tables under Core.DataLake These tables store setup information such as data bundle definitions.

This table describes the Infor Public Sector Configuration Editor settings that affect this task:

Configuration Comments
MAXRECORDS node in Hansen8 configuration The Datalake Upload Limit attribute specifies the maximum number of records to publish in each batch.

ION Desk setup

In ION Desk you must configure your IPS IMS connection point and create a Data Lake flow.

To configure the IPS IMS connection point, you must add the documents that you want to publish. These documents represent the tables that are included in each data bundle. For example, if you're publishing the Account data bundle then you must add IPS_BILLING_ACCOUNT, IPS_BILLING_ACCTGROUP, IPS_BILLING_SENDBILLTO, and so on.

These documents are defined in the Data Catalog in ION Desk. The documents for the standard data bundles are available by default, and the documents for custom data bundles are added when you create the data bundles in Infor Public Sector.

See Adding documents to the IPS IMS connection point.

Next you must create a Data Lake flow that uses the IPS IMS connection point to send data from Infor Public Sector to the Data Lake.

See Creating the Data Lake flow.

See these documents for more information about ION and Data Lake:

  • Infor ION Desk User Guide - Cloud Edition
  • Infor ION Development Guide - Cloud Edition

Invocation

This task can be scheduled to run automatically, or it can be invoked manually from the Batch Manager.

For the parameters, you must select the data bundles that you want to publish. The Batch Manager shows a check box for each available data bundle. See the Data Bundle Setup page for a list of the tables in each bundle.

Data output

The published data in the Data Lake is available for consuming applications such as Birst. You can also access the Data Lake directly through ION Desk. Use the Data Lake node in the ION Desk menu.

This table lists the Infor Public Sector database tables in which this task adds or updates records:

Table Comments
CORE_DATALAKE.BUNDLEPROCESSINGHISTORY Stores processing history data for specific data bundles.
CORE_DATALAKE.DELETEDRECORDS Keeps track of records that have been deleted in Infor Public Sector so they can be marked as deleted in the Data Lake.
CORE_DATALAKE.DELETETRIGGERS Stores information about the delete triggers that the task adds to the database.
CORE_DATALAKE.FACETPROCESSINGHISTORY Stores processing history data for specific tables. (A facet is equivalent to a table.)

This table lists the Infor Public Sector forms that you can use to review the results of this task:

Form Comments
Data Bundle Setup Use the Publish History tab to review previous exports.