Management of object schema modifications

The pipeline models that subscribe to the Data Catalog object reflect modifications done to the object schema.

When you create a pipeline, a mapping is established between the selected Data Catalog object and the destination table. Based on the mapping, the running pipeline processes incoming values into appropriate table columns. When running, the pipeline continues to use the mapping model even if the object schema has changed and the values for new fields have been published. The pipeline does not process those new values because they do not belong to the existing mapping model.

To retrieve an updated object schema and include the new object properties, you must stop the pipeline and refresh the model. To include the new properties, perform these steps:

  1. Open the table activity of the table that has the mapping.
  2. In the toolbar, click the Refresh icon.
  3. Review the mapping for unmapped object properties.
  4. Create table columns for each unmapped property.
  5. Refresh the pipeline and review the mapping again.
  6. When all the properties are mapped, save and start the pipeline.