Creating replication sets

Business Class

Select a business class.

If the Delivered check box is selected, you cannot change fields in the business class that are already included in the replication set, but you can still add fields that are not yet included. You can add user or custom fields through the Custom Fields tab. This check box is non-editable.

Custom Fields

For delivered replication sets, those with a name ending in "_ST", a Custom Fields tab is available for adding custom fields. Click Create and then select the field to add. Specify a Replicate Name that don't conflict with current or future fields in the business class. For example, add a prefix or suffix to ensure the name is unique.

If the delivered replication set is updated in a future release, the custom fields are persisted in the replication set definition. The custom fields and the delivered ones are sent within the same payload when the replication occurs.

Note: When any new fields are added to the replication set, you must register the schema again.

Inactive

Indicates if the business class for this replication set is active or inactive. Inactive classes are not replicated.

Replicate Class

To avoid duplicate data, you must provide an alternate name in the replicated data for the business class. We recommended to prefix the name with a short text string not used in any other replication set. See the description of the Name Prefix field.

Columnar Flatten Hierarchy

For columnar tables, select this check box to flatten any child or parent hierarchies in the selected business class. Flattening enables the tables to be more efficiently consumed by Birst. These columnar tables are stored in Landmark and their names end in _CFH.

Number of Hierarchy Levels

Select the number of hierarchy levels to flatten. Specify a number that is slightly higher than the number of hierarchies needed to capture the data that you need. Specifying a number of levels that is unnecessarily high impacts performance.

Always Rebuild

Select this check box to always rebuild the columnar table on each replication. If you do not select this check box, replication is incremental. A full rebuild occurs when hierarchy changes require it.

Use Columnar Variation Pattern

If the replication set's export format is columnar, select this check box to improve the performance of incremental replications to a columnar table if the table holds a large amount of data and you must do frequent incremental replications. With this selected, updated records are inserted into the columnar table with a variation id without deleting the previous record. Only the latest variation of any record is read in Landmark views that are created on the columnar table. When data is replicated using the columnar variation pattern, the appropriate columnar table is queried, and the last variation ID for the current record is used by external views.

Note: If a replication set was replicated with the Use Columnar Variation Pattern check box selected and you later clear the check box, you must reset the business class record and re‑replicate the replication set. If you select the Use Columnar Variation Pattern check box after replication, no reset or re‑replication is required. The correct incremental variation is automatically displayed in the columnar view list.

Create Blank Dimension Rows

Select this check box to enable hierarchy processing by adding blank dimension records to the column store during replication. When selected, blank rows are generated for each combination of Primary Set Sort fields, ensuring that blank dimension records are included during a full replication. This option is available only for columnar replication. If additional blank dimension rows are required, you must reset and replicate the Replication Class again to generate the additional rows. For proper hierarchy processing in the GL Report Writer, ensure to include blank dimension records for each upper-level key set.

Disable Incremental Replication

Select this check box to replicate all records each time the replication is run, even if the records have not changed. You can only use this option if the business class has an update stamp. We recommend that you do not disable incremental replication for large data sets. For business classes without an update stamp, you must select this option. When doing a full replication, all records are in creates status. When doing incremental replication, you can expect a mix of creates and updates statuses.

Re-Replicate On Any Change

If this check box is selected, the incremental replication is enabled, and the indicator replication is disabled, any change to a record replicates all records for the replication class. If the replication is for columnar, the table is dropped and recreated.

Additional Business Class

This field is displayed if you select the Re-Replicate On Any Change check box. If you specify a business class in this field, any change in this business class triggers a full replication for the replication class. If the replication is for columnar, the table is dropped and recreated.

Override Replication Set Time Storage Resolution

To override the value set for the replication set, select a different time unit.

Purge Data Lake With Full Replication

Select this check box to purge Data Lake when a full replication occurs. This option is available only for replication sets with data destinations of IMS, Data Lake, or Streaming. A replication is considered a full replication when you reset a replication set or replication class, or when you select the Disable Incremental Replication check box on a replication class.

Note: If this field is selected, purging occurs during replication. Data Lake tables are cleared first, before a full replication of data is performed again to Data Lake. To run this action, ensure that your actor profile has permission to purge the replication class in Data Lake.

Run Data Lake Purge As User

Select the actor who runs the Data Lake purge. If no actor is selected, the actor profile you are signed in with is used. The selected actor must have valid permission to delete tables in the Data Lake.

Data To Extract

Data type includes these options:

• Current Data
• Audit Log Entries: Valid only for business classes that implement auditing.
• Future Effective Entries: Valid only for business classes that implement effective dating.
• Audit and Future Entries: Both audit log and future effective entries in the same replication set.

Use Transaction ID for Related Data

When selected, this option enables related fields in a replication of audit and effective data to retrieve data using the transaction ID instead of relying only on the audit entry timestamp. This ensures that related data from the same transaction is included in the replication, despite any timestamp differences in the audit entry. This option is available if Audit Log Entries, Future Effective Entries, or Audit and Future Entries option is selected in the Data to Extract field.

Include Source Entry Action

If you select Default, the source entry action field is included in the replication data. For other values, the action field is not included. You can select true or false to have an explicit choice.

Earliest Change Date to Include

Use this field to set a date so that the replication of the business class includes only data that has been created or modified on or after that date.

See Setting an earliest data inclusion date and backfilling data.

Backfill Through

After you have performed a replication for a business class where you have set a date in the Earliest change date to include field, this field replaces the Earliest change date to include field and shows the date back to which data was replicated.

Multi Thread Full Replications

Select this check box to enable the multi-threading of full replications. This setting is applied to the initial replication of a replication set that is incrementally updated in the future. You cannot use this setting on replication sets that are for backfilling of data.

If you select this, you must specify the number of threads and the field to use to partition the data. You can only select the Multi Thread Field value from the available numeric, integer, date, or timestamp fields, which date and timestamp field values are converted internally to numeric to enable their use. The number of threads determine the number of partitions of the data and the number of async jobs to process the replication. When selecting a field for multi-threading, the best fields to select are those included in the primary index and capable of distributing data as evenly as possible across threads. For example, an auto-sequenced key value is ideal.

UniqueID is used as a multi-thread option for full replications, ensuring records are distributed as evenly as possible across async threads for optimal performance. To use UniqueID as a multi-thread option, select RepSetBC as the business class and select the Multi Thread Full Replications check box. The Number of Threads and Multi Thread Field fields are then available for configuration.

Note: Ensure that the replication set is running a full replication, not incremental.

Number of Threads

Specify the number of threads.

Multi Thread Field

Specify UniqueID.

Enable Indicator Replication

For business classes that support indicator replication, this check box is selected by default. When selected, it causes incremental replication to be applied only to records where the indicator field value equals the replicate when value specified in the business class's LPL. After replication, the indicator field value is changed to the set to value, which is also specified in the LPL, and other actions may apply. The use of the indicator field means that incremental replication is not done based on the update stamp on records, but rather on how the application has defined that business class. Because of the way that databases use the update stamp to index and process records, using the indicator field instead can result in much better replication performance for high-volume business classes.

Indicator Field

Some business classes that support indicator replication have more than one indicator field. Use this field to specify which indicator field this replication set uses. One advantage to specifying different indicator fields for different replication sets that include the same business class, is that only one replication set can include that business class if it had only one indicator field available. With multiple indicator fields for a business class, it is possible to include that business class in different replication sets that serve different purposes. For backwards capability, any replication sets that were defined with only the Enable Indicator Replication check box selected uses the first indicator field defined in the business class LPL.

Create Only

When this check box is selected, any updated records are sent as Creates. Deleted records are still sent as Deletes. Selecting this check box can improve replication performance, particularly for columnar replications for large business classes where the records are expected to be immutable once they have been replicated. The business class must meet these conditions:

• The business class records include an update stamp.
• The indicator pattern should specify as index update stamp set.
• The indicator field has auditing disabled or the business class is not audited.

If the indicator field were audited, replication would cause the update stamp to be changed, which in turn would hinder performance by causing otherwise unneeded reads of the records. If the conditions for the business class are met and the Create Only check box is selected, then the incremental replication replicates records where both the update stamp indicates they are new or modified and the replication when value for the indicator field is matched.

Ignore Indicator On Full Replication

When this check box is selected, for the initial replication or after a reset, any records where the indicator field matches either the replicate when value or the set to value are picked up. Records that are not already set to the set to value are updated, enabling re-replication without having to reset the indicator on all records. When this check box is not selected, only records that match the replicate when value are picked up for both full and incremental replications.

Partition Table

Select this check box to partition a columnar table.

Hybrid Replication

Select this check box to perform incremental replication on older records and full replication on newer records.

Partition Field is Immutable

Select this check box to improve replication performance if the partition field does not change.

Use Alpha as Julian Date

Select this check box if the partition field to use is an Alpha field that contains Julian Date values.

Partition Field

Select the field to use for partitioning the columnar table.

Partition Range

If the partition field is a date field, select the partition range by day, month, or year.

Historical Partition Value

If the partition field is a state field, specify a field value as the Historical Partition Value. Records in this partition value are replicated incrementally, and all other values are fully replicated.

Number of Partitions to Fully Replicate

Specify the number of most recent partitions to fully replicate. Records older than this number are replicated incrementally.

When you select the type of persistent fields to add, you can also choose to explode compound fields and to replace periods with underscores in the field names. Compound fields are always exploded during replication but selecting the Explode Compound Fields check box enables you to see the exploded field names in the list of fields and also to set different replication field names for the members of the compound field. If you select the Explode Compound Fields check box, the Use SQL Name check box is displayed. Selecting this causes the replication names to be auto-generated from the SQL name. This action is useful, for example, for avoiding issues with excessively long field names.

If the replication set data is used where periods are not allowed, the use of periods in the LPL compound field names could be a problem. To avoid this issue, you can use the Replace Periods With Underscores check box.

You can also use the Fields tab to modify the list of fields, such as by adding them individually or deleting them. If you add fields to a business class in a delivered replication set, ensure that you add an alternate name through the Replicate Name field. Use the Related Replicates tab to add related records. If the related record has changed, the business class record replicates even if it has not changed. You can use this tab if you selected Current Data in the Data To Extract field.

Module

Select one of the application modules to include business classes associated with that module.

Owned By

Select All, Application, or Framework to indicate whether to include business classes that belong to the application, framework, or both. If a module is selected, there is no need to select a value for this field.

Name Is Like

Specify a character string, including asterisks to indicate any character to select the business classes with matching names.

Name Prefix

Specify a prefix to avoid duplicates in the replicated data. We recommend this if multiple replication sets include the same business class. This prefix must differ from any prefixes used by any replication sets delivered by a Landmark application.

Automatically add fields?

Select the types of persistent fields to add automatically to the replication sets, and select whether to add cube dimension values for persistent fields.

Explode Compound Fields

Compound fields are always exploded during replication but selecting the Explode Compound Fields check box enables you to see the exploded field names in the list of fields and also to set different replication field names for the members of the compound field. If you select the Explode Compound Fields check box, the Use SQL Name check box is displayed. Selecting this causes the replication names to be auto-generated from the SQL name. This action is useful, for example, for avoiding issues with excessively long field names.

Replace Periods With Underscores

If the replication set data is used where periods are not allowed, the use of periods in the LPL compound field names could be a problem. To avoid this issue, you can use the Replace Periods With Underscores check box.