Reconciliation of data sent in batches
When an application publishes a transaction to the Ledger service, the Ledger begins a reconciliation process approximately 5 minutes after receipt from an application. This buffer permits any in-flight content and delivery retries to conclude before measurements are taken for accuracy and effectiveness of reporting. After a card is published to the Ledger, the card can be expanded to explore additional details. These details can be used to determine whether there is a misalignment between any of the information provided and then reconciled within Data Lake.
This table shows header attributes on each card:
Attribute | Description |
---|---|
Interval start |
The timestamp from which an application has begun a measurement window capturing delivery statistics and counters for replicated data objects. |
Interval end |
The timestamp from which an application has concluded a measurement window for delivery statistics and counters for replicated data. |
Application |
The application, or system-of-record, associated with the Ledger transaction and replicated data objects. |
Data object |
The name of the data object being reconciled. |
Time in transit |
The average time during which data objects included in this reconciliation are in transit from a source application to Data Lake. |
Classification |
A conclusion resulting in a Match or Mismatch identification for the card. This is determined by whether all of the measurements between an application and Data Lake are in alignment. It depends if differences between statistics from application and Data Lake are equal to 0 or not. |
Cards can be expanded to explore additional details identifying how and what measurements were reported during the reconciliation process.
This table shows what details are included on a card:
Attribute | Descpription |
---|---|
Data object Sent |
The number of objects with specific name sent within defined interval from application to Data Lake. |
Data object Ingested |
The number of objects with specific name and source_publication_date between start and end of defined interval, ingested to Data Lake. |
Instance Count Application |
The number of records for all the objects sent by application included in reconciliation. |
Instance Count Data Lake |
The number of records for all the objects ingested in Data Lake and included in reconciliation. |
Row Count Application |
The number of unique records for specific table and time. |
Row Count Compass |
The number of unique records for specific data object and timestamp. |