This topic describes the data types and parameters supported by RestAPI Reader and how to configure it by using the codeless user interface (UI) or code editor. Before you create a Data Integration node, you can refer to this topic to understand the data types and parameters that you must configure for RestAPI Reader to extract data from data sources.

Background information

RestAPI Reader allows you to read data from RESTful APIs. RestAPI Reader obtains data from a RESTful API URL and sends the data to a writer by using a transport protocol. RestAPI Reader supports JSON-formatted data that is returned by RESTful APIs and can extract data of the following types from the returned data: INT, BOOLEAN, DATE, DOUBLE, FLOAT, LONG, and STRING.

Data types

Data type Value of the type field in column
Integer LONG and INT
String STRING
Floating point DOUBLE and FLOAT
Boolean BOOLEAN
Date and time DATE

Parameters

To implement data integration, you must add a data source and configure it as the source or destination. You must also configure the data that you want to integrate and the data types. During data integration, RestAPI Reader extracts data from the source, and RestAPI Writer writes data to the destination.

The following table describes the parameters that you must configure when you use RestAPI Reader to extract data from a RestAPI data source.
Note The following parameters are configured when you add data sources and configure Data Integration nodes.
Parameter Description Required Default value
url The URL of the RESTful API. Yes No default value
dataMode The method used by RestAPI Reader to extract data from the JSON-formatted data returned by a RESTful API. Valid values:
  • oneData: Extract one data record.
  • multiData: Extract a JSON array to transfer multiple data records to a writer.
 Yes No default value
responseType The format of the data returned by the RESTful API. Only the JSON format is supported. Yes json
column The columns to read. The type field specifies the data type of a column. The name field specifies the JSON-formatted path where the column is located. You can specify the column parameter in the following format:

"column":[{"type":"long","name":"a.b" // Query data in the a.b path.},{"type":"string","name":"a.c"// Query data in the a.c path.}]

For the column parameter, you must specify the type and name fields.

 Yes No default value
dataPath The path in which the JSON data record or JSON array is queried. No No default value
method The request method. Valid values: get and post. Yes No default value
customHeader The header information transferred to the RESTful API. No No default value
parameter The parameter information transferred to the RESTful API.
  • If the method parameter is set to get, set the value to abc=1&def=1.
  • If the method parameter is set to post, specify JSON parameters.
 No No default value
dirtyData The processing mechanism when no data is found in the specified JSON-formatted path of column.
  • dirty: if a specific data record cannot be found in the specified JSON-formatted path, the data record is marked as dirty data.
  • null: if a specific data record cannot be found in the specified JSON-formatted path, column is set to null.
Yes dirty
requestTimes The number of times RestAPI Reader requests data from the RESTful API.
  • single: only once
  • multiple: multiple times
 Yes single
requestParam If the requestTimes parameter is set to multiple, you must specify a parameter to be included in each repeatedly transmitted request. If you specify the pageNumber parameter, RestAPI Reader passes the pageNumber parameter to the RESTful API based on the settings of the startIndex, endIndex, and step parameters. No No default value
startIndex The start point of requests. The data at the start point is also requested. No No default value
endIndex The end point of requests. The data at the end point is also requested. No No default value
step The step at which requests are sent. No No default value
authType The authentication method. Valid values:
  • Basic Auth: basic authentication

    If the data source API supports username and password-based authentication, you can select Basic Auth and configure the username and password to be used for authentication. During data integration, the username and password are transferred to the RESTful API URL for authentication. The data source is connected only after the authentication succeeds.

  • Token Auth: token-based authentication

    If the data source API supports token-based authentication, you can select Token Auth and configure a fixed token value to be used for authentication. During data integration, the token is contained in the request header, such as {"Authorization":"Bearer TokenXXXXXX"}, and transferred to the RESTful API URL for authentication. The data source is connected only after the authentication succeeds.

  • Aliyun API Signature: Alibaba Cloud API signature-based authentication

    If the following conditions are met, you can select Aliyun API Signature and configure the AccessKey ID and AccessKey secret to be used for authentication: The data source you want to connect is an Alibaba Cloud service and the API of this service supports AccessKey pair-based authentication.

No No default value
authUsername/authPassword The username and password used for basic authentication. No No default value
authToken The token used for token-based authentication. No No default value
accessKey/accessSecret The account information used for Alibaba Cloud API signature-based authentication. No No default value

Configure RestAPI Reader by using the codeless UI

  1. Configure a data source.
    Configure a data source for Source.RestAPI data source
    Parameter Description
    Connection Select RestAPI from the left-side drop-down list and a data source from the right-side drop-down list.
    Request Method This parameter corresponds to the method parameter that is described in the Parameters section.
    Data Structure This parameter corresponds to the dataMode parameter that is described in the Parameters section.
    json path to store data This parameter corresponds to the dataPath parameter that is described in the Parameters section.
    Response data format This parameter corresponds to the responseType parameter that is described in the Parameters section. Only the JSON format is supported.
    Dirty data This parameter corresponds to the dirtyData parameter that is described in the Parameters section. The default value is Set dirty data.
    header This parameter corresponds to the customHeader parameter that is described in the Parameters section.
    Request parameters This parameter corresponds to the parameters parameter that is described in the Parameters section.
    Number of requests This parameter corresponds to the requestTimes parameter that is described in the Parameters section.
    StartIndex/Step/EndIndex These parameters correspond to the startIndex, step, and endIndex parameters that are described in the Parameters section.

Configure RestAPI Reader by using the code editor

The following code shows a configuration example:
{
    "type":"job",
    "version":"2.0",
    "steps":[
        {
            "stepType":"restapi",
            "parameter":{
                "url":"http://127.0.0.1:5000/get_array5",
                "dataMode":"oneData",
                "responseType":"json",
                "column":[
                    {
                        "type":"long",
                        "name":"a.b"  // Query data in the a.b path.
                    },
                    {
                        "type":"string",  // Query data in the a.c path.
                        "name":"a.c"
                    }
                ],
                "dirtyData":"null",
                "method":"get",
                "defaultHeader":{
                    "X-Custom-Header":"test header"
                },
                "customHeader":{
                    "X-Custom-Header2":"test header2"
                },
                "parameters":"abc=1&def=1"
            },
            "name":"restapireader",
            "category":"reader"
        },
        {
            "stepType":"stream",
            "parameter":{

            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":""
        },
        "speed":{
            "throttle":false,  // Specifies whether to enable throttling. The value false indicates that throttling is disabled, and the value true indicates that throttling is enabled. The concurrent parameter takes effect only if throttle is set to true.
            "concurrent":1  // The maximum number of concurrent jobs.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}