Flexible data formats

Compass data for numbers, including integers and big integers, dates and timestamps are stored in a general format for Compass queries. The general format is designed to be more accommodating of decimal sizes and allowable values. This allows for Data Catalog data object metadata changes that do not require data to be reloaded into Compass data storage.

Data Catalog metadata for integers, big integers and decimals are all stored in Compass data storage as decimal (38,15). For example, if the Data Catalog metadata definition for a “quantity” is initially defined as a data type of integer. Values in the payload, though, include decimal values. The decimal values are stored in the general format of decimal (38,15). Later, the metadata definition is updated to a number data type with 2 decimal places. The Compass data does not have to be cleared and reloaded since the data is already stored as a decimal. A query for an integer value that has a decimal value in the Data Lake results in a NULL value. If the value is used in an aggregation or numeric function, the general data type is used.

Dates and timestamps, in the default format and Data Catalog available formats, are stored as timestamps in UTC

When data is converted to Compass data storage, if the general data type matches the payload values, the data is stored in the “general” format columns. If the data cannot be stored as a number or datetime, the value is unparseable. The unparseable value is stored in Compass as a string.