Streaming Ingestion

Streaming Ingestion is a Data Fabric ingestion method to continuously collect and publish data events to Data Fabric in real time. Data events are occurrences or actions that generate or change data in a system. In data processing and storage, data events are typically represented as discrete data units, such as records or messages.

Streaming Ingestion enables data event processing in real time through Stream Pipelines. After processing data events, the data is stored in the Data Lake data objects. Combining those technologies allows for an end-to-end data movement in real time.

The Streaming Ingestion method is adopted by Infor data publishers and is available for third-party data sources to use through the Infor Data Fabric API Suite in API Gateway.

Streaming Ingestion is based on the WebSocket protocol, which is a standardized protocol for the real-time communication between a client and a server. With WebSocket, a persistent connection is established between a client and a server. The connection allows data to be sent and received in real time.

For more details about WebSocket protocols and how to use them in your applications, see the online WebSocket Protocol Standard specifications and documentation.

For details how to authorize your applications to use the Streaming Ingestion endpoint, see the API Gateway documentation.

After you establish the connection with the Streaming Ingestion endpoint, you can start streaming your data events.

Note: If the connection is closed, you must reconnect to resume streaming the remaining data.

Typically, an API connection remains open until one of these conditions is met:

  • No data is sent within 5 minutes, which causes the connection to time out.
  • The size of a streamed message exceeds 6 megabytes.
  • An internal server error occurs.

Data is streamed as individual Streaming Ingestion JSON messages. Each message contains a single data event.