Stream connection point

With a Stream connection point you can connect to Amazon Kinesis Data Streams, a real-time data streaming service that enables real-time analytics. You can send any document type that is supported by ION.

How to define Amazon Kinesis Data Streams is not part of this guide.

In Data Flow you can use Stream connection point to send input documents to the Amazon Kinesis Data Streams. A document that is delivered to the stream connection point is handled in this way:

  1. Batches up to 20 ION documents are sent to the Stream connection point for processing.
  2. Each ION document is compressed (deflated) and Base64-encoded.

    The deflate compression usually decreases the document size to 15% - 30% of input document. These rations depend on document content and can vary. On top of compression the base64-encoding is applied, resulting in increase of the document size by 33%. The expected kinesis record size is therefore in range of 20% - 40% original document.

  3. The document is wrapped into a JSON structure together with the message headers.

    The JSON wrapper structure:

    {
  {
    "documentName": "<string>",
    "messageId": "<string>",
    "fromLogicalId": "<string>",
    "toLogicalId": "<string>",
    "document": {
        "encoding": "<encoding enum>",
        "characterSet": "UTF-8",
        "value": "<deflated base64 string>"
    },
    "additionalProperties": {
        "accountingEntity": "<string>",
        "location": "<string>",
        "documentId": "<string>",
        "variationId": <long>,
        "revisionId": "string",
        "batchId": "string",
        "batchSequence": <long>,
        "batchSize": <long>,
        "batchRevision": <long>,
        "batchAbortIndicator": <boolean[true,false]>,
        "instances": <long>,
        "source": "<string>",
        "custom_<anyUnqValue>": "<string>"
    }
}
  4. Each JSON structure is packed into the Kinesis record format.

    Each record is given a random partition key, which should result in the records being distributed to all available Kinesis shards.

    Records that have a compressed size of more than 1 MB cannot be handled by the Stream Connector. A Confirm BOD is created for such records.

  5. All available Kinesis records are added into a Kinesis request until the size reaches the maximum.

    The maximum size is defined based on a several available shards in Amazon Kinesis Data Streams. For each shard the max size is increased by 1 MB. Up to a maximum size of 5 MB.

    The number of shards is retrieved at the time of activation. In case the shard count changes, it is required to reactivate the data flow to retrieve the updated information.

With ION the delivery of records to the Amazon Kinesis Data Streams is guaranteed. Infor cannot ensure that the data is available from the Kinesis Stream if the document is not retrieved on time. By default, the Kinesis data stream stores records from 24 hours up to 168 hours.

On top of the encoding that ION applies, an extra base64 encoding is done by Kinesis Data Streams. However on the receiving side the Kinesis Data Streams decoding is not done.

To get to the document data, you must take action in this order:

  1. Retrieve the Kinesis record.
  2. Extract the data part from the Kinesis record.
  3. Base64 decode the data part (JSON wrapper).
  4. Read the ION Message headers.
  5. Base64 decode and inflate the ION document.
  6. Read the ION Document.