This topic describes the data types and parameters that are supported by RestAPI Reader and how to configure RestAPI Reader by using the codeless user interface (UI) and code editor. Before you create a Data Integration node, you can read this topic to familiarize yourself with the data types and parameters that you must set for RestAPI Reader.

Background information

RestAPI Reader reads data from the responses returned by RESTful APIs, converts the data into that of the data types supported by Data Integration, and then sends the converted data to a writer. RestAPI Reader can read data of the following data types from the JSON-formatted responses returned by RESTful APIs: INT, BOOLEAN, DATE, DOUBLE, FLOAT, LONG, and STRING.

Data types

Category Data Integration data type
Integer LONG and INT
String STRING
Floating point DOUBLE and FLOAT
Boolean value BOOLEAN
Date and time DATE

Parameters

Before you perform data integration, you must add a data source and configure it as the source or destination. You must also configure the data that you want to integrate and the data types. During data integration, RestAPI Reader reads data from the source, and a writer writes data to the destination.

The following table describes the parameters that you can set when you use RestAPI Reader to read data from a RestAPI data source.
Note You can set the parameters that are described in the following table when you add a RestAPI data source and configure a Data Integration node.

You cannot set scheduling parameters for RestAPI Reader.

Parameter Description Required Default value
url The URL of the RESTful API. Yes No default value
dataMode The method that RestAPI Reader uses to read data from the JSON-formatted response returned by the RESTful API. Valid values:
  • oneData: RestAPI Reader extracts one data record.
  • multiData: RestAPI Reader extracts a JSON array and transfers multiple data records to a writer.
Yes No default value
responseType The format of the response returned by the RESTful API. Only the JSON format is supported. Yes JSON
column The names of the fields from which you want to read data. The type parameter specifies the data type of a field. The name parameter specifies the JSON-formatted path where the field is located. You can set the column parameter in the following format:

"column":[{"type":"long","name":"a.b" // Query data in the a.b path.},{"type":"string","name":"a.c"// Query data in the a.c path.}]

You must specify the type and name parameters for each field.

Yes No default value
dataPath The path in which the JSON-formatted data record or JSON array is queried. No No default value
method The request method. Valid values: get and post. Yes No default value
customHeader The header information transferred to the RESTful API. No No default value
parameters The parameter information transferred to the RESTful API.
  • If the method parameter is set to get, set this parameter to abc=1&def=1.
  • If the method parameter is set to post, specify JSON parameters.
No No default value
dirtyData The processing mechanism to take effect when no data is found in the specified JSON-formatted path of the column parameter. Valid values:
  • dirty: If a specific data record cannot be found in the specified JSON-formatted path, this data record is marked as a dirty data record.
  • null: If a specific data record cannot be found in the specified JSON-formatted path, the column parameter is set to null.
Yes dirty
requestTimes The number of times RestAPI Reader requests to read data from the response returned by the RESTful API. Valid values:
  • single: only once
  • multiple: multiple times
Yes single
requestParam If the requestTimes parameter is set to multiple, you must specify a parameter in each repeatedly transmitted request. For example, if you specify the pageNumber parameter, RestAPI Reader passes the pageNumber parameter to the RESTful API based on the settings of the startIndex, endIndex, and step parameters. No No default value
startIndex The start point of requests. The data at the start point is also requested. No No default value
endIndex The end point of requests. The data at the end point is also requested. No No default value
step The step at which requests are sent. No No default value
authType The authentication method. Valid values:
  • Basic Auth: basic authentication

    If the data source supports username- and password-based authentication, you can select Basic Auth and configure the username and password that can be used for authentication. During data integration, the username and password are transferred to the RESTful API URL for authentication. The data source is connected only after the authentication is successful.

  • Token Auth: token-based authentication

    If the data source supports token-based authentication, you can select Token Auth and configure a fixed token value that can be used for authentication. During data integration, the token is contained in the request header, such as {"Authorization":"Bearer TokenXXXXXX"}, and transferred to the RESTful API URL for authentication. The data source is connected only after the authentication is successful.

  • Aliyun API Signature: Alibaba Cloud API signature-based authentication

    If the following conditions are met, you can select Aliyun API Signature and configure the AccessKey ID and AccessKey secret that can be used for authentication: The data source that you want to connect is an Alibaba Cloud service, and the API of this service supports AccessKey pair-based authentication.

No No default value
authUsername/authPassword The username and password used for basic authentication. No No default value
authToken The token used for token-based authentication. No No default value
accessKey/accessSecret The AccessKey pair used for Alibaba Cloud API signature-based authentication. No No default value

Configure RestAPI Reader by using the codeless UI

  1. Configure data sources.
    Configure Source and Target for the synchronization node. RestAPI data source
    Parameter Description
    Connection The name of the data source from which you want to read data. Select RestAPI from the left-side drop-down list and the name of a data source that you configured from the right-side drop-down list.
    Request Method This parameter is equivalent to the method parameter that is described in the preceding section.
    Data Structure This parameter is equivalent to the dataMode parameter that is described in the preceding section.
    json path to store data This parameter is equivalent to the dataPath parameter that is described in the preceding section.
    Response data format This parameter is equivalent to the responseType parameter that is described in the preceding section. Only the JSON format is supported.
    Dirty data This parameter is equivalent to the dirtyData parameter that is described in the preceding section. Default value: Set dirty data.
    Header This parameter is equivalent to the customHeader parameter that is described in the preceding section.
    Request parameters This parameter is equivalent to the parameters parameter that is described in the preceding section.
    Number of requests This parameter is equivalent to the requestTimes parameter that is described in the preceding section.
    StartIndex/Step/EndIndex These parameters are equivalent to the startIndex, step, and endIndex parameters that are described in the preceding section.
  2. Configure field mappings.
    Fields in the source on the left have a one-to-one mapping with fields in the destination on the right. You can click Add to add a field. To remove an added field, move the pointer over the field and click the Remove icon. Field mappings
    Operation Description
    Map Fields with the Same Name Click Map Fields with the Same Name to establish mappings between fields with the same name. The data types of the fields must match.
    Map Fields in the Same Line Click Map Fields in the Same Line to establish mappings between fields in the same row. The data types of the fields must match.
    Delete All Mappings Click Delete All Mappings to remove the mappings that are established.
    Auto Layout Click Auto Layout to sort the fields based on specific rules.
    Change Fields Click the Change Fields icon. In the Change Fields dialog box, you can manually edit the fields in the source table. Each field occupies a row. The first and the last blank rows are included, whereas other blank rows are ignored.
    Add Click Add to add a field. Take note of the following rules when you add a field:
    • You can enter constants. Each constant must be enclosed in single quotation marks ('), such as 'abc' and '123'.
    • You can use scheduling parameters, such as ${bizdate}.
    • You can enter functions that are supported by relational databases, such as now() and count(1).
    • If the field that you entered cannot be parsed, the value of Type for the field is Unidentified.
  3. Configure channel control policies. Channel control
    Parameter Description
    Expected Maximum Concurrency The maximum number of concurrent threads that the synchronization node can use to read data from the source or write data to the destination. You can configure the concurrency for the synchronization node on the codeless UI.
    Bandwidth Throttling Specifies whether to enable bandwidth throttling. You can enable bandwidth throttling and specify a maximum transmission rate to prevent heavy read workloads on the source. We recommend that you enable bandwidth throttling and set the maximum transmission rate to an appropriate value based on the configurations of the source.
    Dirty Data Records Allowed The maximum number of dirty data records allowed.
    Distributed Execution This parameter is not supported for synchronization nodes that use RestAPI Reader.

Configure RestAPI Reader by using the code editor

In the following sample code, a synchronization node is configured to read data from the response returned by a RESTful API:
{
    "type":"job",
    "version":"2.0",
    "steps":[
        {
            "stepType":"restapi",
            "parameter":{
                "url":"http://127.0.0.1:5000/get_array5",
                "dataMode":"oneData",
                "responseType":"json",
                "column":[
                    {
                        "type":"long",
                        "name":"a.b"  // Query data in the a.b path.
                    },
                    {
                        "type":"string",  // Query data in the a.c path.
                        "name":"a.c"
                    }
                ],
                "dirtyData":"null",
                "method":"get",
                "defaultHeader":{
                    "X-Custom-Header":"test header"
                },
                "customHeader":{
                    "X-Custom-Header2":"test header2"
                },
                "parameters":"abc=1&def=1"
            },
            "name":"restapireader",
            "category":"reader"
        },
        {
            "stepType":"stream",
            "parameter":{

            },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":""
        },
        "speed":{
            "throttle":true,  // Specifies whether to enable bandwidth throttling. A value of false specifies that bandwidth throttling is disabled, and a value of true specifies that bandwidth throttling is enabled. The mbps parameter takes effect only if the throttle parameter is set to true. 
            "concurrent":1,  // The maximum number of concurrent threads allowed.  
            "mbps":"12"// The maximum transmission rate.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}
Take note of the following information when you configure RestAPI Reader by using the code editor:
After RestAPI Reader sends an HTTP or HTTPS request, a JSON-formatted response is returned. The dataPath parameter is used to specify the path in which the JSON-formatted data record or JSON array is queried. The following part provides two sample responses:


In the following sample response, a JSON array is returned for the DATA parameter that contains the business data.
{
    "HEADER": {
        "BUSID": "bid1",
        "RECID": "uuid",
        "SENDER": "dc",
        "RECEIVER": "pre",
        "DTSEND": "202201250000"
    },
    "DATA": [
        {
            "SERNR": "sernr1"
        },
        {
            "SERNR": "sernr2"
        }
    ]
}

To extract multiple data records from the JSON array and transfer the data records to a writer, you must specify the column parameter in the "column": [ "SERNR" ] format, the dataMode parameter in the "dataMode": "multiData" format, and the dataPath parameter in the "dataPath": "DATA" format.


In the following sample response, a JSON object is returned for the content.DATA parameter that contains the business data.
{
    "HEADER": {
        "BUSID": "bid1",
        "RECID": "uuid",
        "SENDER": "dc",
        "RECEIVER": "pre",
        "DTSEND": "202201250000"
    },
    "content": {
        "DATA": {
            "SERNR": "sernr2"
        }
    }
}

To extract one data record from the JSON object and transfer the data record to a writer, you must specify the column parameter in the "column": [ "SERNR" ] format, the dataMode parameter in the "dataMode": "oneData" format, and the dataPath parameter in the "dataPath": "content.DATA" format.