Data object creation
Data objects are typically created by a source system or Data Lake.
Data object creation by a source system
A data publisher provides data replication modules that accumulate transactions from a
specific period of time. The transactions are stored within a data object, which is later
published to Data Lake. If the data is determined to be replicated
to Data Lake by, for example, a schedule or optimal data object
size, the data publisher can use the Data Lake Flows in ION Connect or Data Lake's Batch API for a direct
upload to Data Lake. The Channel property in
Atlas and the storage APIs provide information on the ingestion
origin.
Data object creation by Data Lake
When a data publisher streams transactional events in real time with Data Fabric's Streaming Ingestion API, the transactions are accumulated and then micro-batched by Data Lake. Data Lake generates data objects after 5 MB of data are reached or every 5 minutes, whichever happens first. Until then, the transactions are not available in Data Lake, but can be processed immediately, in real time with Infor Stream Pipelines.