REST Connector Pagination

REST API calls often return many results. For that reason, APIs are designed to be paginated, to break up the results into sections are easier to handle.

The REST Connector supports commonly-used types of pagination:

Paging Parameter

Offset Parameter

Cursor Pagination

Fully Baked URL Pagination

Partial Resource URL

Paging Parameter Pagination

Paging Parameter pagination accepts a single number page number in the request query parameters.

Examples:
https://api.example.com/accounts?page=4
https://api.example.com/accounts?page=1&count=100
https://anotherapi.example.com/accounts?p=4&limit=10

Enter the following settings:

Paging parameter: The parameter name is typically "page". This may not always be the case, so read the API documentation and find the correct parameter name.

From: The page number for the first page you want to extract. By default it is 1.

To: The page number for the last page you want to extract. By default all available pages are extracted.

Detect new fields at runtime: In some cases when the data varies among pages or over time, the Columns tree view does not fully reflect the JSON structures of all pages. Select this option to update the tree view at runtime.

Max concurrent requests: The number of concurrent request during runtime. By default it is 1. Increase the value of this field to improve performance. The maximum allowed is 5.

Response from last request: Displays the root information from the last time you clicked Get Columns. This view can help you select the best pagination type.

Offset Parameter Pagination

The Offset Parameter pagination style is similar to the syntax for looking up multiple database records. There is both a limit and an offset query parameter. The limit is the maximum number of items to return. The offset is the starting position of the query in relation to the complete set of items.

Examples:
https://api.example.com/accounts?offset=0&limit=100
https://api.example.com/accounts?offset=1500&limit=100  
https://anotherapi.example.com/accounts?startAt=100&count=10  

Enter the following settings:

Offset Parameter: The parameter name for the offset number.

From: The start value for the offset number. By default it is 0.

To: The end value for the offset number. By default all available pages are extracted.

Page Size Parameter: The page size parameter name.

Page Size: The number of items returned per page.

Detect new fields at runtime: In some cases when the data varies among pages or over time, the Columns tree view does not fully reflect the JSON structures of all pages. Select this option to update the tree view at runtime.

Max concurrent requests: The number of concurrent request during runtime. By default it is 1. Increase the value of this field to improve performance. The maximum allowed is 5.

Response from last request: Displays the root information from the last time you clicked Get Columns. This view can help you select the best pagination type.

Cursor Pagination

Cursor-based pagination uses a cursor indicator to page through the result set. Cursor pagination uses forward (next page cursor) and reverse (previous page cursor) controls. The cursor parameter indicates the starting position of the query relation to the complete set of items. This mechanism is similar to the offset parameter, however the cursor value is encoded (or hashed) and included in JSON response while offset is numeric and can be manually entered.

Example:

The parameter name "next" in this example is not necessarily the actual field name. Read the API documentation to find the correct parameter name.

Page request:
https://api.example.com/accounts

Response:

{
"accounts":[ ... ], 
 "next":"qe3Ws15jAb67leEh", 
 "count":25 }

Next page request:

https://api.example.com/accounts?offset=qe3Ws15jAb67leEh

Enter the following settings:

Next page cursor field: Field to use as the cursor field.

Cursor Parameter: The name for the cursor parameter.  

Detect new fields at runtime: In some cases when the data varies among pages or over time, the Columns tree view does not fully reflect the JSON structures of all pages. Select this option to update the tree view at runtime.

Max concurrent requests: The number of concurrent request during runtime. By default it is 1. Increase the value of this field to improve performance. The maximum allowed is 5.

Response from last request: Displays the root information from the last time you clicked Get Columns. This view can help you select the cursor.

 

Fully Baked URL Pagination

Fully Baked URL Pagination is similar to the other styles, however instead of building a URL query for all pages, it includes a full URL representing the next result set of data (next page) in each page data (not exist or null if it represents last page of data).

Example:

The parameter name "next" in this example is not necessarily the actual field name. Read the API documentation to find the correct parameter name.

Page request:
https://api.example.com/accounts

Response:

{
"accounts":[ ... ], 
 "next_page":"https://api.example.com/accounts?offset=qe3Ws15jAb67leEh", 
 "count":25 
}

Enter the following settings:

Next page cursor field: Field to use as the cursor field.

Detect new fields at runtime: In some cases when the data varies among pages or over time, the Columns tree view does not fully reflect the JSON structures of all pages. Select this option to update the tree view at runtime.

Max concurrent requests: The number of concurrent request during runtime. By default it is 1. Increase the value of this field to improve performance. The maximum allowed is 5.

Response from last request: Displays the root information from the last time you clicked Get Columns. This view can help you select the Next page cursor field.