Reconciliation of streamed data
For streamed data, when a manifest has been published to the Data Ledger service, a
reconciliation process is initiated by the Ledger after approximately 15 minutes.
This 15-minute buffer allows completing any ongoing content transfers and delivery retries before measurements are taken. This facilitates accuracy and effectiveness of the reporting process. After the reconciliation process is completed, a transaction card is published to the Ledger. In the Ledger, you can expand the card to view details. The details help identify misalignments in the information provided by the source application and Data Lake, and how and which measurements were reported during the reconciliation process.
This table shows the header attributes on each transaction card:
| Attribute | Description |
|---|---|
| Interval start | The timestamp of when the reconciliation fixed interval has started and the application starts capturing ingestion statistics. |
| Interval end | The timestamp of when the reconciliation fixed interval has ended and the application has finished capturing ingestion statistics. |
| Application | A system of record that is associated with the Ledger transaction and streaming data. |
| Data object | The name of a data object that is reconciled. |
| Classification | A conclusion that results in a match or mismatch identification for a transaction card. Classification is determined by whether all measurements between an application and Data Lake are aligned. |
This table shows what details are included on a transaction card:
| Detail | Description |
|---|---|
| Checksum verification | The result of the checksum CRC32 comparison between summarized records that are sent by an application and summarized records in Data Lake. |
| Instance Count Application | The number of records for the defined interval that are sent by an application and included in reconciliation. |
| Instance Count Data Lake | The number of records for the defined interval that are ingested in Data Lake and included in reconciliation. |
| Row Count Application | The number of unique records for a specific timestamp in the source application. |
| Row Count Compass | The number of unique records in Data Lake for the timestamp provided in the Row Count by the source application. |