Data object properties

The object properties are used in several Data Lake features to filter out a set of data objects to apply an action to.

Commonly used object properties in Data Lake are the Object ID, Object Name, and the Stored Date timestamp.

These are some examples of the ways in which these object properties are used:

  • In Purge, you specify a date range based on the Stored Date, and provide Object Names. And you can optionally provide the Logical ID.
  • In Compass you can query, and request to show, additional properties in the response. For example, the dataObjectID and lastModified columns which relate to the Object ID and Stored Date object properties
  • In the Data Lake /dataobjects API, you can use the object properties to filter the API response. You can use the /byfilter Filter API to retrieve a specific set of data objects.

This table shows the object properties that are available for data objects in Data Lake:

Property Name Description Data Type
Object ID Unique identifier for the raw data object. String
Index Date

Date and time when the object became visible in Data Lake after it was stored.

Date & Time
Stored On

Date and time when the object was physically stored in Data Lake.

Date & Time
Object Type

The data format of the data object name as defined in Data Catalog.

String
Compressed Size

Compressed size of the raw data object.

Number
Uncompressed Size

The uncompressed size of the raw data object. Value is 0 if the data object was sent compressed.

Number
Encoding

Type of character encoding for the stored data in the raw object.

String
Source Event Date

Date and time when a messaging or delivery interface marked delivery of a data object.

Date & Time
Compression Type

Compression type that is applied on the stored object. Currently it is always DEFLATED.

String
Corrupt

Boolean property indicating if the data object is prevented from further processing. The property value can be changed with marking the data object as corrupt.

When set to true, /dataobjects/byfilter API does not retrieve the object by default. This can be overridden.

Compass does not query the data in the data object.

Boolean
Logical ID

The logical ID of the application that provided the data.

String
Message ID

The unique message identifier as set by the provider of the data.

String
Object Name

The associative identity of a data object as defined and referenced within Data Catalog.

String
Instances

The number of instances within the raw data object. For example, the number of records that are stored in a Newline-delimited JSON or DSV object.

Number
Document ID

The Document ID of the business object, if the data object contains a single business object or record.

String
Accounting Entity

The accounting entity of the data, if the data belongs to an accounting entity.

String
Location

The location of the data, if applicable, if the data belongs to a location.

String
Source

The source of the data, for example the file name where the data object came from.

String
Revision ID

The revision ID of the data, in case the data object contains a single business object or record and has a revision ID.

String
Variation ID

The variation ID of the data, in case the data object contains a single business object or record and has a variation ID.

Long
Batch ID

If the data object belongs to a batch of multiple data objects, then this is the ID of the batch.

String
Batch Sequence

If the data object belongs to a batch of multiple data objects, then this is the sequence number of the data object in the batch.

Long
Batch Size

If the data object belongs to a batch of multiple data objects, then this is the total number of data objects in the batch.

Long
Batch Revision

When the data object belongs to a batch of multiple data objects, and a new revision of the batch was sent. Then this is the revision ID of the batch.

Long
Batch Abort Indicator

If the data object belongs to a batch of multiple data objects,then this indicates whether the batch was canceled before it was completed.

Boolean
Custom Keys A list of custom property names, if applicable.

The keys order matches the order for the values in the custom properties .

Array of string values
Custom Properties

The list of values for the customer properties as listed in the custom keys. The value order matches the order for the keys in the custom keys.

Array of string values
Source Publication Date

Date and time when the data object was published by a source application.

Date & Time
Time in Transit The time in which a data object spent transiting from a source application to receipt in Data Lake. Long Value
Channel One of these options through which data is ingested into Data Lake:
  • Stream for Streaming Ingestion
  • Batch for Batch Ingestion
  • ION for data that arrives at Data Lake through ION Data Lake flows with the Ingest to Data Lake activity.
String
Archived Boolean property indicating whether the data object is archived or not.