Streaming Ingestion

Streaming Ingestion is a Data Fabric ingestion method to continuously collect and publish data events to Data Fabric in real time. Data events are occurrences or actions that generate or change data in a system. In data processing and storage, data events are typically represented as discrete data units, such as records or messages.

When the Streaming Ingestion service receives data events, the service starts two processing paths in parallel.

In the first path, the service checks whether Stream Pipelines are configured for the data object. If a stream pipeline exists, the service sends the events to the pipeline immediately. The service does not wait until Data Lake stores the data. In the second path, the Streaming Ingestion service groups incoming events into micro-batches. The service then stores the micro-batches as data objects in Data Lake. Data Lake provides durable storage and supports consumption by other Data Fabric services. Both processing paths run concurrently. For information about how Stream Pipelines process events that the Streaming Ingestion service receives, see Streaming data from Data Fabric.

The Streaming Ingestion method is adopted by Infor data publishers and is available for third-party data sources to use through the Infor Data Fabric API Suite in API Gateway.

Streaming Ingestion is based on the WebSocket protocol, which is a standardized protocol for the real-time communication between a client and a server. With WebSocket, a persistent connection is established between a client and a server. The connection allows data to be sent and received in real time.

For more details about WebSocket protocols and how to use them in your applications, see the online WebSocket Protocol Standard specifications and documentation.

For details how to authorize your applications to use the Streaming Ingestion endpoint, see the API Gateway documentation.

After you establish the connection with the Streaming Ingestion endpoint, you can start streaming your data events.

Note: If the connection is closed, you must reconnect to resume streaming the remaining data.

Typically, an API connection remains open until one of these conditions is met:

  • No data is sent within 5 minutes, which causes the connection to time out.
  • The size of a streamed message exceeds 6 megabytes.
  • An internal server error occurs.

Data is streamed as individual Streaming Ingestion JSON messages. Each message contains a single data event.