Replication validation

General requirements

• The validation is only available for columnar or Data Lake replication of a business class's current data.
• The replication being validated must be complete, not actively running or failed.
• For Data Lake, the replication must have an export format of DataLake or IMS, with the destination being DataLake. This functionality uses DMP Compass to query DataLake records, which is not available in IGS. To validate that the connection to Compass is healthy, navigate to Async Administrator > ION Components > ION Connections. The Compass connection should be visible and in healthy state.

Condition requirements

If the replication set has a condition and contains references to fields other than persistent fields that are resident on the business class, such as derived fields, user fields and related fields, overriding the replication set condition is required. To override, use either no condition or a condition that meets the requirements.

Checksum field requirements

Checksum fields are used for the initial checking for mismatches.

• You must use a numeric data type for checksum fields.
• You can specify multiple checksum fields.

• Define checksum fields as either the same field in both the columnar or Data Lake table and the rowstore table, or as a field in the columnar or Data Lake table that maps to a delivered, persistent rowstore field. You can apply conditions to the rowstore field matchup. These examples show how to apply this guideline:

  • Columnar or Data Lake has derived Credit and Debit fields in the replication set. The derived fields are based on a persistent delivered field called Amount. The Credit derived field returns Amount when < 0. Debit returns Amount when > 0. You can add checksum fields for Credit and Debit, and specify a rowstore field of Amount for each, adding a condition to each to apply to the Row Store field to make it match.
  • Columnar or Data Lake has the Amount field. You can add two checksum fields, one with a condition < 0 and another > 0. The rowstore field must be persistent, delivered meaning not a user field, and neither user field nor a related field.
• If the checksum field can contain both negative and positive values, you need to add two fields, one with a condition < 0 and another > 0 since the values are added to together to compute the checksum value.
• Checksum field conditions can only be a single condition and contain only persistent fields that are resident on the business class. Related fields and user fields are not allowed.
• Up to 10 Check Sum values are allowed, provided that one is a global Instance Count. Otherwise, only 9 are allowed, as one is reserved for the implicit Instance Count.

Recommendations

• We recommend that the replication set business class is one that contains a delivered, meaning not a user field, Date field to use for date chunking. This field must always be populated and remain unchanged after replication.
  • Since this field is used to compute date chunks, records with blank values are excluded from validation.
  • If this value changes and replication is outdated at the time of validation, undetectable mismatches can occur.
  • If you do not use a date field or the business class has no date field, you can still run the validation by selecting the Validate All Data option. However, the validation fails if the number of records to process exceeds the setting for the maximum number of records to use for chunking. The default is 1.5 million.
• We recommend to run the validation when the replication is up to date before other changes have been made. If the replication is not up to date, the validation attempts to account for this where possible. This is dependent on checksum fields being audited but this also affects performance.
• We recommend that you run validations for business classes that have a unique ID field and that field is included in the replication set.

• Validation is accomplished by first breaking data into date chunks so the number of records to be read and processed is limited to a manageable number. Dates are grouped together in ranges up to the maximum record count. Smaller date ranges are suggested. The range is always the specified date, or computed date based on the offset option chosen, to current.

  The date field used for chunking should be included in a non-virtual, non-conditional set. The process attempts to find a set with this field to use for rowstore queries. If the field is in more than one set, the process selects the set where this field is nearest the top.

Validation process details

• For each date chunk, an instance count and the checksum field aggregation values are retrieved from columnar or Data Lake. If any checksum values have conditions applied to columnar or Data Lake, additional aggregation calls are made for each unique condition.
• The rowstore is then queried.
• If the update stamp is after the validation start time and the business class is audited, then the record is looked up using the Object Persistence Framework or the OPF and the as-of date, so values and deleted state is determined when possible. This has a performance impact. For this reason, we recommend that the validation is done shorty after a replication completes and before data changes are made. A few records would have minimal impact. Large volumes of changed records can have a significant impact.
• Checksum values are computed and the record's UID or keys are hashed. If all checksum values match, then the chunk is valid and the process moves to the next date chunk.
• If there are any mismatches, then it queries non-aggregated records from columnar or Data Lake. If it does not find a match in the rowstore, a mismatch record is created. If a match is found and there were any conditional checksum values, it looks up the record using OPF with the validation start time as an as-of date.
• If there was a replication set condition and it was overridden, a check is done to see if the record would have been excluded had the condition been applied. If so, a note is made. Other checks are made for deleted state and notes made of any discrepancies. If an indicator is used, checks are done on that and discrepancies are noted.
• The rowstore map is checked for any remaining records that had no columnar or Data Lake match. If the replication set has a condition and it has been overridden, the rowstore record is looked up using OPF with the validation start time as an as-of date, and a check is done to see if the record would have been excluded had the condition been applied. If so, this is a false positive and the record is not considered a mismatch.
  Note: There are possibilities that the validation process would not detect a mismatch that exists. For example, a field's value is 10, replication is done, the value is changed to 20, then replication is done again but the record is missed. If validation was done now, it would detect a mismatch. However, if a retroactive change was made to change the record back to 10 and a replication was not done, the validation would not detect a mismatch even though it exists. This is a known case that would not be detected. This happens because initial checks are done using DBI for performance. Audit checks are only done if a mismatch is detected at this point. However, in this case the next replication corrects the data assuming the record isn’t missed again.