Data Publishing Parameters (ttdpm5130m000)

Use this session to define the parameters for publishing to Data Lake or Infor OS.

For details on how to configure publishing in hybrid deployment, see the Infor LN Enterprise Server Library (Secured) (On-premises) and select Administrator > Enterprise Server > Configuration with Infor OS (Cloud) > Configuring LN for publishing to Data Lake.

Note: If the micro batching parameters are changed, the data publishers must be restarted to take effect.

Buttons

These buttons are available:

Check IMS Connection: Checks if publishing to IMS with the current connection parameters is possible.
This button is only valid for the IMS Connection parameters.
Check Catalog Connection: Checks if publishing to Data Catalog with the current connection parameters is possible.
This button is only valid for the Data Catalog Connection parameters.
Check Ingestion Connection: Checks if publishing to Data Lake with the current connection parameters is possible.
This button is only valid for the Batch Ingestion Connection and Streaming Ingestion Connection parameters.

Field Information

Publication Method for Initial Load

You can select one of these methods:

Data Lake Ingestion

The messages are published directly to Data Lake through the Data Lake Batch Ingestion API and not through ION. In this way, it is not required to model a data flow in ION.

If this option is selected, the Message Size (in MB) field in the Publish Data (ttdpm5205m000) session is automatically increased to 50 MB. This is because the Data Lake Batch Ingestion API can handle larger messages than IMS.
ION Messaging Service

Data is published from LN to ION. You must model document flows in ION to distribute the data to its final destination.

Publication Method for Changes

You can select one of these methods:

Data Lake Ingestion

The messages are published directly to Data Lake using the Data Lake Streaming Ingestion API and not through ION. In this way, it is not required to model a data flow in ION.
ION Messaging Service

Data is published from LN to ION. You must model document flows in ION to distribute the data to its final destination.

The changes are not published immediately, but they are batched into larger messages during a user-defined period of time before publishing. This is called micro-batching. The period during which the changes are batched and the message size are specified in the Micro-Batching Parameters in the current session.

Note:

The available options for the Publication Method for Initial Load and the Publication Method for Changes are interdependent. You cannot select Data Lake Ingestion for changes if the publication method for the initial load is ION Messaging Service.
The preferred way of publishing the initial and incremental load to Data Lake is through the Data Lake Ingestion APIs.
The ION Messaging Service remains available for LN data publishing for the time being but will be phased out in the future.
If the LN data must be replicated to a location other than Data Lake, then the LN data must first be sent to Data Lake from where the data can be distributed to other locations.

Logical IDs

Logical ID

The logical ID of the ION Messaging Service.

The logical ID is also required for Data Lake Ingestion, in which case it is used to identify the origin of the messages - it is part of the message envelope.

Local Logical ID

The destination Logical ID in ION. This is used in destinations other than Data Lake.

IMS Connection

These connection parameters are used to publish LN data to ION.

Select OAuth version 1.0 or 2.0.

Select OAuth 1.0 if the LN server directly communicates with the Infor OS server. Usually this is the case in full on-premises environments.

Select OAuth 2.0 if the integration is set up using the API Gateway.

URL

The URL of the IMS endpoint without /service/.

Infor OS on premises:

The full URL is displayed in the ION Connection Point details, although the server name is referred to as <ion>. Replace <ion> with the FQDN and the port number of the Infor OS router or the load balancer pointing to it.

Example:

https://inforos.domain.com:9543/api/ion/messaging
Infor OS CE:

See the Infor LN Enterprise Server Library (Secured) (On-premises) and select Administrator > Enterprise Server > Configuration with Infor OS (Cloud).

OAuth Consumer Key

The OAuth consumer key to connect to the ION Messaging Service. This applies if OAuth version 1.0 is selected.

You can retrieve the OAuth 1.0 credentials during the creation of the IMS Connection Point in ION.

OAuth Secret Key

The OAuth secret key to connect to the ION Messaging Service. This applies if OAuth version 1.0 is selected.

You can retrieve the OAuth 1.0 credentials during the creation of the IMS Connection Point in ION.

OAuth 2.0 Parameter Set

The OAuth 2.0 Parameter Set. This applies if OAuth version 2.0 is selected.

Specify the parameter set or zoom to the OAuth 2.0 Parameter Sets (ttaad0108m000) session to select an OAuth 2.0 parameter set.

Data Catalog Connection

These connection parameters are used to publish schemes (meta data) to Data Catalog.

Select OAuth version 1.0 or 2.0.

Select OAuth 1.0 if the LN server directly communicates with the Infor OS server. Usually this is the case in full on-premises environments.

Select OAuth 2.0 if the integration is set up using the API Gateway.

URL

The Data Catalog endpoint URL.

Infor OS on premises:

The full URL is https://<ion>/datacatalog/v1/document/json. Replace <ion> with the FQDN and the port number of the Infor OS router or the load balancer pointing to it.

Example:

https://inforos.domain.com:9543/datacatalog/v1/document/json
Infor OS CE:

See the Infor LN Enterprise Server Library (Secured) (On-premises) and select Administrator > Enterprise Server > Configuration with Infor OS (Cloud).

OAuth Consumer Key

The OAuth consumer key to connect to Data Catalog. This applies if OAuth version 1.0 is selected.

The OAuth 1.0 credentials can be generated in the ION Grid Management under Security – OAuth Credentials. The required roles are grid-run-as-user and datacatalog-admin.

OAuth Secret Key

The OAuth secret key to connect to Data Catalog. This applies if OAuth version 1.0 is selected.

OAuth 2.0 Parameter Set

The OAuth 2.0 Parameter Set. This applies if OAuth version 2.0 is selected.

Specify the parameter set or zoom to the OAuth 2.0 Parameter Sets (ttaad0108m000) session to select an OAuth 2.0 parameter set.

Batch Ingestion Connection

These connection parameters are used to publish data (initial load) to Data Lake through the Data Lake Batch Ingestion API.

For the integration from LN on premises you must use OAuth 2.0, because the integration is set up using the API Gateway.

URL:

The Data Lake Batch Ingestion API endpoint URL.

See the Infor LN Enterprise Server Library (Secured) (On-premises) and select Administrator > Enterprise Server > Configuration with Infor OS (Cloud).

OAuth 2.0 Parameter Set

This applies if OAuth version 2.0 is selected.

Specify the parameter set or zoom to the OAuth 2.0 Parameter Sets (ttaad0108m000) session to select an OAuth 2.0 parameter set.

Streaming Ingestion Connection

These connection parameters are used to publish data (changes) to Data Lake through the Data Lake Streaming Ingestion API.

For the integration from LN on premises you must use OAuth 2.0, because the integration is set up using the API Gateway.

Web Socket URL:

The Data Lake Streaming Ingestion API web socket URL.

See the Infor LN Enterprise Server Library (Secured) (On-premises) and select Administrator > Enterprise Server > Configuration with Infor OS (Cloud).

OAuth 2.0 Parameter Set

This applies if OAuth version 2.0 is selected.

Specify the parameter set or zoom to the OAuth 2.0 Parameter Sets (ttaad0108m000) session to select an OAuth 2.0 parameter set.

Tenant ID

The ID of the tenant to which the data is sent.

Micro-Batching Parameters

Micro-batching is used to reduce data fragmentation and optimize storage in Data Lake, which helps enhancing system performance when retrieving data objects or querying Data Lake using Compass queries.

Micro-batching is done by grouping data into batches by table. The batch size is determined by the time interval specified in the Maximum Delay field and the message size specified in the Minimum Message Size field.

A JSON message containing LN data is published if the maximum delay time has passed or the minimum message size has been reached.

Micro-batching is mandatory for the publication of changes to prevent too much data fragmentation and to give a better performance in Data Lake.

The initial load process automatically groups all data belonging to one table to a size-limited message. Therefore, micro-batching is not applicable for either of the publication methods for the initial load.

Is Enabled

If this check box is selected, JSON messages containing LN data are not generated and sent by transaction, but the transactions are grouped by table into larger messages.

If this check box is cleared, micro-batching is not enabled. Data Lake Batch ingestion cannot be used as publication method for the incremental load (that is, the changes). In this case, the changes are published through IMS.

Note: If the micro batching parameters are changed, the data publishers must be restarted to take effect.

Maximum Delay

The time interval during which data is published. The time interval is expressed in minutes. The recommended interval is 10 minutes. For example, if you specify 10 minutes, a message containing data is published once every ten minutes, unless the minimum message size has been reached before the time interval has passed.

A message is generated and sent if the maximum delay time has passed or if the minimum message size has been reached, it depends on which of these conditions is met first.

Minimum Message Size

The minimum message size. The message size is expressed in MBs. The recommended message size is 5 MB. A message is published when the data volume has increased up to the specified message size, unless the maximum delay time has passed before the minimum message size has been reached.

A message is generated and sent if the maximum delay time has passed or if the minimum message size has been reached, it depends on which of these conditions is met first.

If the ION Messaging Service is the publication method for changes, the standard message size is set to 5 MB and this field is disabled.

Number of Transformers

The number of transformers per publisher bshell. The transformers combine the data from individual transactions into micro-batched data, including the determination of the calculated fields.

Number of Micro-Batch Publishers

The number of micro-batch publishers per publisher bshell. The micro-batch publishers publish the messages from LN to IMS or Data Lake.

Periodical Reset

The frequency of the reset, in which data with status Error is reset. After the reset, publishing is retried. The recommended frequency is 30 minutes.