Data object properties
Commonly used object properties in Data Lake are the Object ID, Object Name, and the Stored Date timestamp.
These are some examples of the ways in which these object properties are used:
- In Purge, you specify a date range based on the Stored Date, and provide Object Names. And you can optionally provide the Logical ID.
- In Compass you can query, and request to show, additional properties in the response.
For example, the
dataObjectID
andlastModified
columns which relate to the Object ID and Stored Date object properties - In the Data Lake
/dataobjects
API, you can use the object properties to filter the API response. You can use the/byfilter
Filter API to retrieve a specific set of data objects.
This table shows the object properties that are available for data objects in Data Lake:
Property Name | Description | Data Type |
---|---|---|
Object ID | Unique identifier for the raw data object. | String |
Index Date |
Date and time when the object became visible in Data Lake after it was stored. |
Date & Time |
Stored On |
Date and time when the object was physically stored in Data Lake. |
Date & Time |
Object Type |
The data format of the data object name as defined in Data Catalog. |
String |
Compressed Size |
Compressed size of the raw data object. |
Number |
Uncompressed Size |
The uncompressed size of the raw data object. Value is 0 if the data object was sent compressed. |
Number |
Encoding |
Type of character encoding for the stored data in the raw object. |
String |
Source Event Date |
Date and time when a messaging or delivery interface marked delivery of a data object. |
Date & Time |
Compression Type |
Compression type that is applied on the stored object. Currently it is always DEFLATED. |
String |
Corrupt |
Boolean property indicating if the data object is prevented from further processing. The property value can be changed with marking the data object as corrupt. When set to true, /dataobjects/byfilter API does not retrieve the object by default. This can be overridden. Compass does not query the data in the data object. |
Boolean |
Logical ID |
The logical ID of the application that provided the data. |
String |
Message ID |
The unique message identifier as set by the provider of the data. |
String |
Object Name |
The associative identity of a data object as defined and referenced within Data Catalog. |
String |
Instances |
The number of instances within the raw data object. For example, the number of records that are stored in a Newline-delimited JSON or DSV object. |
Number |
Document ID |
The Document ID of the business object, if the data object contains a single business object or record. |
String |
Accounting Entity |
The accounting entity of the data, if the data belongs to an accounting entity. |
String |
Location |
The location of the data, if applicable, if the data belongs to a location. |
String |
Source |
The source of the data, for example the file name where the data object came from. |
String |
Revision ID |
The revision ID of the data, in case the data object contains a single business object or record and has a revision ID. |
String |
Variation ID |
The variation ID of the data, in case the data object contains a single business object or record and has a variation ID. |
Long |
Batch ID |
If the data object belongs to a batch of multiple data objects, then this is the ID of the batch. |
String |
Batch Sequence |
If the data object belongs to a batch of multiple data objects, then this is the sequence number of the data object in the batch. |
Long |
Batch Size |
If the data object belongs to a batch of multiple data objects, then this is the total number of data objects in the batch. |
Long |
Batch Revision |
When the data object belongs to a batch of multiple data objects, and a new revision of the batch was sent. Then this is the revision ID of the batch. |
Long |
Batch Abort Indicator |
If the data object belongs to a batch of multiple data objects,then this indicates whether the batch was canceled before it was completed. |
Boolean |
Custom Keys | A list of custom property names, if applicable. The keys order matches the order for the values in the custom properties . |
Array of string values |
Custom Properties |
The list of values for the customer properties as listed in the custom keys. The value order matches the order for the keys in the custom keys. |
Array of string values |
Source Publication Date |
Date and time when the data object was published by a source application. |
Date & Time |
Time in Transit | The time in which a data object spent transiting from a source application to receipt in Data Lake. | Long Value |
Channel |
How the data was ingested into Data Lake. A data object is added in Data Lake through ION = ION. For example by IMS, AnySQL, File Connector, Database connector, JMS queue, orAPI Gateway. A data object is added to Data Lake through IoT = IoT A data object is added to Data Lake through batch ingestion = Bulk or Batch A data object is added to Data Lake through streaming ingestion = Stream A data object is added to Data Lake through Pipelines = Stream |
String |
Archived | Boolean property indicating whether the data object is archived or not. |