Flexible data formats

Compass data for numbers, including integers and big integers, dates and timestamps are stored in a general format for Compass queries. The general format is more accommodating of decimal sizes and allowable values. This accommodates for Data Catalog data object metadata changes that do not require data to be reloaded into Compass data storage.

Data Catalog metadata for integers, big integers and decimals are all stored in Compass data storage as decimal (38,15). For example, if the Data Catalog metadata definition for a quantity is initially defined as a data type of integer. Values in the payload include decimal values. The decimal values are stored in the general format of decimal (38,15). Later, the metadata definition is updated to a number data type with 2 decimal places. The Compass data does not have to be cleared and reloaded because the data is already stored as a decimal. A query for an integer value that has a decimal value in theData Lake results in a NULL value. If the value is used in an aggregation or numeric function, then the general data type is used.

Dates and timestamps, in the default format and Data Catalog available formats, are stored as timestamps in UTC

When data is converted to Compass data storage, and the general data type matches the payload values. The data is stored in the general format columns. If the data cannot be stored as a number or datetime, then the value is unparsable. The unparsable value is stored in Compass as a string.