Extracting Data Using REST

Prior to setting up data extraction using REST familiarize yourself with the latest implementation notes at About the REST Connector.

 

To connect to REST services

  1. On the Home page select a space or create a new one. This is the space to which the data will be extracted.
  2. Go to Admin - Define Sources - Application Connectors. Alternatively, click the Use Extractors link on the Admin Navigation page.
  3. Click New in the Connections panel.
  4. In Connection Details name the connection.
  5. Select REST as the Connector Type.
  6. Set the Connector API Version.

For new spaces, select the latest API version (highest version number). 

For existing spaces, test the latest API version before changing from the default API version to the latest API version.
Birst determines which version is the default. If the selected API version is no longer available in Birst, the default API version will be automatically assigned. If the default API version is selected, when Birst changes to a new API version as the default, existing spaces will be automatically upgraded to the new default API version.

  1. Enter the REST endpoint information, the REST base URL that connects to the RESTful service.
  2. Enter the login information. Choose between OAuth 2.0, Basic, or No authentication, depending on the service.
a. If there is to be no authentication, select None and move on to step 9.
b. If the service requires Basic authentication, enter the user name and password, and optionally upload a self-signed SSL certificate.
Tips:

Depending on the service, the username may not always be an email address. For example it could be "myname@mycompany.com" or "myname".

The user account must have permissions to use the application's Web Services.

c. If the service requires OAuth 2.0, select how the token is to be generated, either Resource Owner, Client Credentials, Implicit Grant, Authorization Code, or None.

None: Use this if you have the OAuth2 access token for the API. Enter the Access Token. Optionally enter the Refresh Token, Client Id, Refresh URL, and Client Secret, and Scope, to be able to refresh the access token on expiry.

Authorization Code: Enter the Access Token URL, Authorization URL, Client Id, Client Secret, and Scope. Save the connection and click Generate Test Token. The connector redirects to the application's authorization page in a new tab or page where you must authorize the application to access the data. On authorization, the page redirects back to Birst with a success or failure message. On success, close the tab/page and go back to the Birst.

Implicit Grant: Enter the Authorization URL, Client Id and Scope. Save the connection and click Generate Test Token. The connector redirects to the application's authorization page in a new tab or page where you must authorize the application to access the data. On authorization, the page redirects back to Birst with a success or failure message. On success, close the tab/page and go back to the Birst.

Client Credentials: Enter the Access Token URL, Client Id, Client Secret, and Scope. Then click Generate Test Token.

Resource Owner: Enter the Access Token URL, Client Id, Owner Username and Password, Client Secret, and Scope. Then click Generate Test Token.

If the API requires the access token to be part of the URL, click Add access token to URL.

Some applications require that the callback URL be authorized within the application. Use the Callback URL displayed for Authorization Code and Implicit Grant type OAuth2 connection details for this.

If the API authentication method does not match any of the above scenarios and requires a key or signature to be passed in headers/query parameters, you can add that as part of specifying extraction objects in the next section.

  1. Optionally, upload a self-signed SSL certificate if the API uses the HTTPS protocol with a self-signed certificate.

  2. Optionally, specify a Source File Prefix. This alphanumeric string will be prepended to the data source names so that you can easily recognize them and differentiate objects that may have similar names from different data sources. For example, use the string "RESTNW" to easily identify REST Northwind objects.
  3. Click Save Connection. The Data Sources panel appears under the Connections panel.  This panel will list the extraction objects you define in the next steps.

To specify extraction objects

Extraction objects help you organize what data is extracted for the connection.

1. Click Add Object. The dialog prompts you for information needed to create a new Data Sources extraction object. The Build Request tab has labels, request type, URL parameters, and mapping information. The Advanced tab has pagination settings.
2. In the Build Request tab, enter the Object Label to use in Birst.
Important: Do not use the same name as an existing saved object from this or any other connector. It must be unique within a Birst space. Using a source file prefix can help to differentiate similarly named objects.
If you edit an object and change the object label, a new object will be created with this name.
3. Select the Request Type, either GET or POST.
4. Enter the URL Resource path. For example, Northwind.svc/Customers.
5. Enter the Header, if needed.
6. Enter the URL query parameters, if any.
7. For a POST request type, enter the type of POST parameters. For the x-www-urlencoded type, enter the Key and the Value. Click Add (+) for additional Key/Value pairs. For raw, enter the raw POST body.
8. Optionally, select the Action on Failure. By default if there are errors the connector will stop processing. You can change it to Retry once, or to Ignore errors and continue processing.
9. Click Get Columns.
Birst populates the Columns list and Mapping field.

The Columns list contains the objects in the source represented as a JSON hierarchy. The names are referred to as the raw name or "technical name".

The Mapping field contains the name of each object in the source, an equal sign, then the name it will be mapped to in Birst. By default the Mapping field is called the display name or "friendly name".
10. Click the Select Columns tab to check only the objects you need for reporting and uncheck the ones you don't need.  



Important:
When there are complex JSON responses with multiple levels, you may need to use multiple Data Source extraction objects to extract the different objects. Repeat the steps to create an extraction object and give each one a different label, keeping the other information the same. Then change the Columns list for each. For example, create a different extract object for Northwind.svc/Customer, Northwind.svc/Employees, etc.
11.  Edit the Mapping field list to delete the objects you do not need. It should contain only the mappings for the objects checked in the Columns list.
12. Optionally, change the name of the column in Birst by changing the friendly name after the equal sign.  
Tip: After the first time that Birst extracts the column and lists it in the Columns tab of the Manage Sources page, do not change the column name in the Mapping field again. If you do so, a new result will be created and the old one will remain listed as a placeholder. If your intention is to change the name for display purposes, you should use the Manage Sources - Columns tab.
13. Click Validate to validate the syntax in Birst. 
14. In the Set Paging Parameters tab, specify the pagination types. See REST Connector Pagination.
15. Click Save. Birst adds it to the list of objects to be extracted in the Data Sources table, ready for extraction. 

Tip: To edit the object, click on the hyperlinked name.
16. Optionally, select the row and enter the name of one or more extract groups, separated by commas, in the Extract Group Name(s) field. Extract groups make it possible to extract specific objects and process them separately from other objects.  See Creating Extract Groups.  

Important: Use the same extract group for Issue and its sub-objects.

Click Save after adding extract group names.
17. Click Extract to upload the selected objects to Birst. The Extract Groups dialog prompts you to select the extract group or groups to use. If you did not define extract groups it defaults to All.

Selecting All in the Extract Group dialog box will extract all selected objects, including those that do not belong to extract groups.
18. Click Continue. Birst displays an indicator to show that the extraction is in progress.

Depending on the size of the data objects, the extraction may take a few minutes. When the extraction completes Birst lists the date and time of the extraction in the Data Sources table of the Show Selected Objects view.

Tips:

You can safely log out of Birst while the extraction takes place and log back in later.

To cancel the extraction, click the Cancel Extract link under the progress bar. 

Click the Last Extract Log link to view the logs from the data extraction and check for errors. 

To remove an object from the Data Sources list, select the row and click Remove on the right.

Extracting Objects Requiring Multiple Calls

Some REST APIs require multiple calls with dynamic parameters to extract a complete object. For example, JIRA provides a REST API for extracting all Boards. To extract all Sprint Objects, they provide a REST API which takes boardId as input and provides a list of Sprints for the given Board. Therefore to extract all sprints requires multiple calls to the JIRA Sprint API, one for each boardID.

The REST Connector supports multiple calls, also called multiple endpoints, using Birst variables with single or multiple values. The multiple values are indicated by M[] notation for Resource and URL Parameters in the REST Connector Edit Object definition. For example, BoardID = M[V{BoardID} where the BoardID variable could be either a constant or logical query and could return single or multiple values.

Example Syntax:

BoardID = M[V{boardID}]

BoardID = M[10,20,30,40]

Important: Use the same extract group for objects that are dependent upon one another. Otherwise the extraction may fail.

Examples Using Variables

Example Using Values from Other Objects as Dynamic Parameters

Provide a parameter in URL Params that will be populated with dynamic parameter values for each API call. The syntax is:

/API_path/M[parameter_name]

Then go to the Set Dynamic Parameters tab and enter the Parameter_Name, select the REST Connector Object from the dropdown, then the Column, and enter a test value to be used for validation. Click Add to repeat for each dynamic parameter.

Click Validate to verify the syntax and dynamic parameters using test values.

Next Steps

Look at the raw data in Manage Sources page. Iterate the extraction process, if needed, to get the data you expect.  

Process the data in the Process Data - Process New Data page. See Processing Data.

Set up scheduling in the Process Data - Schedule Data Processing page. See Scheduling Extraction and Data Processing for Cloud Applications.