Choose appropriate methods of ingestion

Select the technology that best align with your product's solution needs.

For example, if your application is BOD-capable, or you are building an integration footprint into your solution, adopt ION as the integration method for sending data to Data Lake.

To benefit from direct ingestion capabilities, you can adopt the Data Lake's batch ingestion API, which allows for direct transfer of larger file sizes.

This table shows the use cases to help you choose the appropriate method of ingestion:

Method Use case
Batch Ingestion

Use this method in these cases:

  • Your data source is an application that can use REST API to send data objects.
  • You want to send archived data objects.
  • The data to send is created or read in a batches, such as:
    • Publishing data on request.
    • Publishing an initial load.
    • Publishing batch-based business workstreams, for example MRP.
Data Lake Flows (in ION Desk)

Use this method in these cases:

  • Your data source is a connection point in ION.
  • Your data must be transformed before ingesting to Data Lake, for example with ION Scripting or enrichment activities within Data Flows.
  • Multiple application subscribers require published replication data in addition to Data Lake.
Data Loader Use this method when migrating on-premises data stored within database tables, views, and materialized views to Data Lake.
Note: Do not send the same data objects through both the batch ingestion method and Data Lake flows in parallel as it can lead to data duplication in Data Lake.