Setting an earliest data inclusion date and backfilling data

When you replicate data, you may encounter two situations that impact the performance and usefulness of the replication. First, you may not want include data from the earliest time that data exists in the business class. For example, for your business's analytical needs, you may want only data going back six months whereas the business class contains data going back several years. Second, when business classes have a large amount of data, the initial replication of that data can be resource and time intensive. To address these situations, data replications includes two features.

  • First, to avoid replicating data from earlier time periods, you can set an "earliest change date to include". Replication then includes only data that has been created or modified on or after that date.
  • Second, to avoid long-running replication processes, you can perform the initial replication using an "earliest change date to include" value, and then use the Backfill data action to incrementally replicate data for earlier time periods than the "earliest change date to include" value.
Note: You cannot use the Backfill data action on all business classes. It is only available if the business class has a create stamp and update stamp, does NOT implement the Incremental Replication pattern, the replication is for current data (not audit or effective dated transactions), and at least the initial replication has been done.

See the following steps for details on how to use the Earliest change date to include field and the Backfill data action. This example assumes that you want to use both the Earliest change date to include field and the Backfill data action.

  1. Navigate to the Replication Sets form. This form is available in the Landmark Administration Console under the Database tab or you can locate it in the Infor Rich Client by searching for ReplicationSet.
  2. Click Create, then specify a name for the replication set, select the export format and ouput destinations, and select or specify values for any optional fields. Click Save.
  3. Add one or more business classes to the replication set. For more information on adding business classes, see Creating replication sets. For this example process, set the Earliest change date to include field to a past date. How far in the past this date is depends on how far back you want to replicate data and on how much data the business class contains. If the business class contains an extremely large amount of data, then the date should be closer to the present.
  4. Save your changes and register the replication set schema.
  5. On the Replication Sets form, select the replication set to schedule and click Replicate. For information on starting a replication, see Starting and scheduling data replication.
  6. After the replication is complete, select the replication set again in the Replication Sets form and then on the Business Classes tab select a business class for which to run the Backfill data action. The Backfill data action is on a business class by business class basis, so you must select each business class individually for which you want to do the data backfilling.
  7. For the selected business class, select Actions > Backfill data.
  8. On the Backfill Data form, provide a value for one of the two following fields (not both):
    Backfill From
    Use this field to set a date that is earlier than than the currently stored value for the Earliest change date to include field. When the Backfill data action runs, the data for the business class that was modified or created in between the two dates will be replicated. After the Backfill data action is complete, the backfill from date will be stored as the value for the Earliest change date to include field. This option is usually best if you want to backfill data for only one or few time periods or if you want to maintain careful control of the backfilling for a business class. In this case, you would schedule the action to run only once.
    Offset Days
    The number of days before the currently stored value for the Earliest change date to include field to replicate data for. After the Backfill data action is complete, the Earliest change date to include field's value will be modified to be earlier by the offset number of days. This option is usually best if you want to repeatedly run the Backfill data option for a business class, such as backfilling repeatedly for 30-day offsets until you have replicated the earliest data records. In this case, you would schedule the action to run at regular intervals. After the earliest data has been replicated in this scenario, you should disable the action request.
  9. Click OK to run the action or click Schedule to schedule the action to run.