Managing changes in object schemas

When a pipeline includes a Data Object activity that is followed directly by a Table activity, a mapping is automatically created between the object's properties and the destination table columns. The mapping remains static when the pipeline is running. If the object schema changes during the pipeline run, for example, new properties are added, the changes are not reflected in the pipeline until the pipeline is stopped and the mapping is refreshed.

Stopping the pipeline prompts the Table activity to detect and include the added properties in the mapping. If corresponding columns are missing in the destination table, you must manually create them to complete the mapping and resume processing.

To manage schema changes and prevent disruptions, we recommend that you include a Select Properties activity in the pipeline. Through that activity, you control when new object properties are added. Added object properties are not selected by default in the Select Properties activity configuration. This enables you to plan their inclusion, including the table schema updates and the pipeline reactivation to process the added properties.

The Select Properties activity makes schema changes intentional and controlled by ensuring that new properties are streamed only when explicitly selected.