Data Preparation

Applications publishing data to the Data Lake are typically expected to micro-batch data to optimize data publishing and query performance.

As a best practice, applications can develop condition-based logic that micro-batches data for a pre-determined amount of time. Or if a certain amount of data is buffered. There is no file size limitation imposed within the service. But it is best practice that files are no smaller than 5 MB and no larger than 50 MB. This reduces risks associated with network interruptions and improves overall processing performance of downstream consumers that can directly interface with raw data objects.

Data objects must be sent compressed using ZLIB compressed data format with DEFLATE compression method. Requests containing uncompressed files or files compressed with a different method are rejected.

You can ingest any supported format of data. We recommend sending data objects in flat Newline-delimited JSON format.