Handling schema changes in the source
Stream Pipelines does not propagate source schema changes automatically. Each schema change requires manual updates.
New columns
Pipelines continue to deliver data based on the existing pipeline model. Pipelines do not include new source columns automatically.
To include a new column:
- Stop the pipeline.
- Click to add the new column to the pipeline model.
- Add the corresponding column to the destination table.
- Restart the pipeline.
New tables
Each source object requires a separate pipeline. When a source application adds a new object, create a new pipeline and a corresponding destination table.
Column renames
Data Fabric does not support column renaming. Data Catalog, Data Lake, and Stream Pipelines treat a renamed column as the removal of an existing column and the addition of a new column. To manage schema changes, add new columns alongside existing columns, deprecate columns that are no longer required, and use compatible data-type changes.