Data administration stored procedures

Data Catalog is the source of metadata definitions for Compass data objects, data object properties, and logic for variation handling.

To automatically synchronize updates to the Data Catalog metadata definitions with Compass, select Data Catalog > Object Schemas, click the icon in the upper right corner of the panel, and enable the Metadata Synchronization feature. When this feature is enabled, the Compass object definition is updated automatically. Consequently, there is no need to manually run data administration procedures, such as running clear table statements when metadata changes happen.

If you have not frequently run data administration procedures before when metadata changed, the data from Data Lake is cached by Compass according to the cached metadata.

Note: If you enabled Metadata Synchronization, NULL values can be displayed in the new columns that may be included with the next metadata synchronization.

When the updated object definition does not contain information from the historical data, we recommend that you clear the Compass data for the specific object from the time when those metadata changes were introduced. Thus, the values for the changed columns are included.

See the Infor OS User and Administration Documentation Library (Cloud) and select ION > Data Catalog > Object schemas > Notify Compass of schema updates.

You can update Data Lake data through the purge process and the process to mark objects as corrupt.

Compass is automatically updated in these scenarios:

  • Object metadata is updated in Data Catalog when the Metadata Synchronization feature is enabled.
  • Data is purged from Data Lake in the Atlas or Purge user interfaces.

Compass is not automatically updated in these scenarios:

  • Object metadata is updated in Data Catalog when the Metadata Synchronization feature is disabled.
  • Data is marked as corrupt.
  • Data is purged from Data Lake through the purge APIs.

Use the Compass stored procedures to perform administrative operations on the Compass data storage, object definitions, and object views. Use the stored procedures to clear tables or clear data when the Data Lake data is purged or marked as corrupt. Use the Clear Table stored procedure with the true parameter to resolve partition issues.

Note: When you use the Clear Data and Clear Table procedures with the true parameter for large volume tables, the data clearing may take longer for the first subsequent query. This is because all the Data Lake data is processed for a table. We recommend that you clear data for a specific period in which to restore data in Compass.

The Administration stored procedure to clear table removes the Compass object definition and variation handling views. The Clear Table procedure can clear table data object definitions with or without removing converted data. The data and object definitions are recreated the next time a query is processed for the affected data objects.