Setting an earliest data inclusion date and backfilling data

When you replicate data, you may encounter two situations that impact the performance and usefulness of the replication. First, you may not want to include data from the earliest time that data exists in the business class. For example, for your business's analytical needs, you may want only data going back six months whereas the business class contains data going back several years. Second, when business classes have a large amount of data, the initial replication of that data can be resource and time intensive. To address these situations, data replications includes two features.

  • First, to avoid replicating data from earlier time periods, you can set an "earliest change date to include". Replication then includes only data that has been created or modified on or after that date.
  • Second, to avoid long-running replication processes, you can perform the initial replication using an "earliest change date to include" value, and then use the Backfill data action to incrementally replicate data for earlier time periods than the "earliest change date to include" value.
Note: You cannot use the Backfill data action on all business classes. It is only available if the business class has a create stamp and update stamp, does NOT implement the Incremental Replication pattern, the replication is for current data (not audit or effective dated transactions), and at least the initial replication has been done.

See the following steps for details on how to use the Earliest change date to include field and the Backfill data action. This example assumes that you want to use both the Earliest change date to include field and the Backfill data action.

  1. In the Landmark Administration Console, select Database > Replication.
  2. In the Replication Sets pane on the Maintain tab, click Create.
  3. Specify a name for the replication set, select the export format and output destinations, and select or specify values for any optional fields. Click Save.
  4. Add one or more business classes to the replication set. For more information on adding business classes, see Creating replication sets. For this example process, set the Earliest change date to include field to a past date. How far in the past this date is depends on how far back you want to replicate data and on how much data the business class contains. If the business class contains an extremely large amount of data, then the date should be closer to the present.
  5. Save your changes and register the replication set schema.
  6. On the Replication Sets form, select the replication set to schedule and click Replicate. For information on starting a replication, see Starting and scheduling data replication.
  7. After the replication is complete, select the replication set again in the Replication Sets form and then on the Business Classes tab select a business class for which to run the Backfill data action. The Backfill data action is on a business class by business class basis, so you must select each business class individually for which you want to do the data backfilling.
  8. For the selected business class, click the ellipsis button and select Backfill data.
  9. On the Backfill Data form, provide a value for one of the two following fields (not both):
    Backfill From
    Use this field to set a date that is earlier than the currently stored value for the Earliest change date to include field. That value is displayed in a Backfill Through on the Backfill Data form. When the Backfill data action runs, the data for the business class that was modified or created in between the two dates will be replicated. After the Backfill data action is complete, the backfill from date will be stored as the value for a Backfill Through field that replaces the Earliest change date to include field on the business class form. This backfill option is usually best if you want to backfill data for only one or a few time periods, or if you want to maintain careful control of the backfilling for a business class. In this case, you would schedule the action to run only once.
    Offset Days
    The number of days before the currently stored value for the Backfill Through field to replicate data for. After the Backfill data action is complete, the Backfill Through field's value will be modified to be earlier by the offset number of days. This option is usually best if you want to repeatedly run the Backfill data option for a business class, such as backfilling repeatedly for 30-day offsets until you have replicated the earliest data records. In this case, you would schedule the action to run at regular intervals. After the earliest data has been replicated in this scenario, you should disable the action request.
  10. Click Ok to run the action or click Schedule to schedule the action to run.