Data administration stored procedures
To automatically synchronize updates to the Data Catalog metadata definitions with Compass, select , click the icon in the upper right corner of the panel, and enable the feature. When this feature is enabled, the Compass object definition is updated automatically. Consequently, there is no need to manually run data administration procedures, such as running clear table statements when metadata changes happen.
If you have not frequently run data administration procedures before when metadata changed, the data from Data Lake is cached by Compass according to the cached metadata.
When the updated object definition does not contain information from the historical data, we recommend that you clear the Compass data for the specific object from the time when those metadata changes were introduced. Thus, the values for the changed columns are included.
See the Infor OS User and Administration Documentation Library (Cloud) and select .
You can update Data Lake data through the purge process and the process to mark objects as corrupt.
Compass is automatically updated in these scenarios:
- Object metadata is updated in Data Catalog when the feature is enabled.
- Data is purged from Data Lake in the Atlas or Purge user interfaces.
Compass is not automatically updated in these scenarios:
- Object metadata is updated in Data Catalog when the feature is disabled.
- Data is marked as corrupt.
- Data is purged from Data Lake through the purge APIs.
Use the Compass stored procedures to perform administrative
operations on the Compass data storage, object definitions, and
object views. Use the stored procedures to clear tables or clear data when the Data Lake data is purged or marked as corrupt. Use the Clear Table
stored procedure with the true
parameter to resolve partition issues.
true
parameter for large volume tables, the data clearing may take
longer for the first subsequent query. This is because all the Data Lake data is processed for a table. We recommend that you clear
data for a specific period in which to restore data in Compass.The Administration stored procedure to clear table removes the Compass object definition and variation handling views. The Clear Table procedure can clear table data object definitions with or without removing converted data. The data and object definitions are recreated the next time a query is processed for the affected data objects.