Setting Up Incremental Extracts from Eloqua
Incremental extracts pick up only the data that has changed since the last extraction. Use incremental extracts when you do not want to extract all the data each time.
To set up incremental extracts from Eloqua
| 1. | In the Admin module, select the Define Sources tab. |
| 2. | Click Manage Sources. |
| 3. | Confirm that the Eloqua source contains a Date Modified column in the Columns tab and that its Analyze by Date box is checked. |
| 4. | Select the Eloqua source in the data sources panel. |
| 5. | Click the Properties tab. |
| 6. | Check the
Rows in Data Source are Transactions box then click Save. If a grain has been defined for this source, the Rows in Data Source are Transactions setting will treat rows as transactions, meaning that Birst will not apply a Load_ID prefix when a measure from this grain is selected. See Setting Transaction and Snapshot Policies for a Data Source for more information about this setting. After changing this setting, the next time you perform an Eloqua extraction, a full extraction will be performed. After that, all future extracts will be incremental, meaning only the data that has changed since the previous extract will be extracted. |
Important Notes About Incremental Extractions from Eloqua
Incremental updates can occur only on the following Eloqua objects: Account, Activity, Campaign, Contact, Email, Form, Form_FormField, FormData, and LandingPage. Custom objects cannot be incrementally updated.
The Eloqua Connector does not support incremental updates for data sources created using the Add Object feature.
The Campaign object has two sub-objects, Campaign_Element and Campaign_Element_OutputTerminal, that do not have their own datetime fields. These objects can be extracted independently or with Campaign.
If Campaign and Campaign sub-objects are assigned to the same extraction group then Campaign sub-objects can be incrementally updated.
If Campaign sub-objects are in a different extraction group from the Campaign object, then Campaign sub-objects cannot be incrementally updated because they do not have the necessary datetime field.
If you select Campaign sub-objects and not the Campaign object then Birst will extract all the rows.
Incremental extraction is based on the Max Last
Modified (or Last Created) DateTime of the data being extracted. Eloqua
stores datetime in its database accurate to millisecond, however data
returned by its API is only accurate to the second. For example, a datetime
value accessed via the API such as 2014-06-17
04:49:02 AM is not exactly what Eloqua stores in its database;
that could be 2014-06-17 04:49:02.
0000001 AM or 2014-06-17
04:49:02. 9970000 AM. As a result:
Birst filters should not use = or != for Eloqua datetime expressions. Use >=, >, <, <= for Eloqua datetime in a filter
Some data may be duplicated during incremental extraction. Entries with time = Max Last Modified (or Last Created) will be redundant. In this case, set up the "Bulk Insert and Delete Measure Record" advanced option in the Manage Source tab to delete the duplicate rows.
See Also