This topic describes the parameters that are supported by RestAPI Writer and how to configure RestAPI Writer by using the codeless user interface (UI) and code editor.

Background information

RestAPI Writer can send requests to RESTful APIs to write data. RestAPI Writer receives data from a reader, generates JSON-formatted data based on the setting of the column parameter, and then sends requests to RESTful APIs to transfer the data.

Parameters

To implement data integration, you must add a data source and configure it as the source or destination. You must also configure the data that you want to integrate and the data types. During data integration, a reader extracts data from the source, and a writer writes data to the destination.

The following table describes the parameters that you must configure when you use RestAPI Writer to write data to a RestAPI data source.
Note You must configure the parameters that are described in the following table when you add a RestAPI data source and configure a data synchronization node.

Scheduling parameters are not supported for data synchronization nodes that use RestAPI Writer.

Parameter Description Required Default value
url The URL of the RESTful API. Yes No default value
dataMode The format in which RESTful Writer transfers JSON-formatted data.
  • oneData: RestAPI Writer transfers one data record in each request.
  • multiData: RestAPI Writer transfers multiple data records in each request. The number of requests is determined by the number of tasks generated by the reader.
Yes No default value
column The columns to which you want to write the generated JSON-formatted data. The type field specifies the data type of a column. The name field specifies the JSON-formatted path where the column is stored. You can also specify the column parameter in the following format:

"column":[{"type":"long","name":"a.b" // Store data in the a.b path.},{"type":"string","name":"a.c"// Store data in the a.c path.}]

Note For the column parameter, you must specify the type and name fields.
Yes No default value
dataPath The path that is used to store the JSON-formatted data. No No default value
method The request method. Valid values: post and put. Yes No default value
customHeader The header information transferred to the RESTful API. No No default value
authType The authentication method. Valid values:
  • Basic Auth: basic authentication

    If the data source API supports username and password-based authentication, you can select Basic Auth and configure the username and password to be used for authentication. During data integration, the username and password are transferred to the RESTful API URL for authentication. The data source is connected only after the authentication succeeds.

  • Token Auth: token-based authentication

    If the data source API supports token-based authentication, you can select Token Auth and configure a fixed token value to be used for authentication. During data integration, the token is contained in the request header, such as {"Authorization":"Bearer TokenXXXXXX"}, and transferred to the RESTful API URL for authentication. The data source is connected only after the authentication succeeds.

  • Aliyun API Signature: Alibaba Cloud API signature-based authentication

    If the following conditions are met, you can select Aliyun API Signature and configure the AccessKey ID and AccessKey secret to be used for authentication: The data source you want to connect is an Alibaba Cloud service and the API of this service supports AccessKey pair-based authentication.

No No default value
authUsername/authPassword The username and password used for basic authentication. No No default value
authToken The token used for token-based authentication. No No default value
accessKey/accessSecret The AccessKey pair used for Alibaba Cloud API signature-based authentication. No No default value
batchSize The maximum number of data records that can be transferred in each request when the dataMode parameter is set to multiData. Yes 512

Configure RestAPI Writer by using the codeless UI

  1. Configure data sources.
    Configure Source and Target for the synchronization node. RestAPI data source
    Parameter Description
    Connection Select RestAPI from the left-side drop-down list and a data source from the right-side drop-down list in the Target section.
    Request Method This parameter is equivalent to the method parameter that is described in the preceding section.
    Data structure of request parameters This parameter is equivalent to the dataMode parameter that is described in the preceding section.
    BatchSize This parameter is equivalent to the batchSize parameter that is described in the preceding section.
    json path to store data This parameter is equivalent to the dataPath parameter that is described in the preceding section.
    Header This parameter is equivalent to the customHeader parameter that is described in the preceding section.
  2. Configure channel control policies. Channel control
    Parameter Description
    Expected Maximum Concurrency The maximum number of parallel threads that the synchronization node uses to read data from the source or write data to the destination. You can configure the parallelism for the synchronization node on the codeless UI.
    Bandwidth Throttling Specifies whether to enable bandwidth throttling. You can enable bandwidth throttling and specify a maximum transmission rate to prevent heavy read workloads on the source. We recommend that you enable bandwidth throttling and set the maximum transmission rate to an appropriate value based on the configurations of the source.
    Dirty Data Records Allowed The maximum number of dirty data records allowed.
    Distributed Execution This parameter is not supported for synchronization nodes that use RestAPI Writer.

Configure RestAPI Reader by using the code editor

In the following code, a synchronization node is configured to write data to a RestAPI data source: 
{
    "type":"job",
    "version":"2.0",
    "steps":[
        {
            "stepType":"stream",
            "parameter":{

            },
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"restapi",
            "parameter":{
                "url":"http://127.0.0.1:5000/writer1",
                "dataMode":"oneData",
                "responseType":"json",
                "column":[
                    {
                        "type":"long", // Store data in the a.b path.
                        "name":"a.b"
                    },
                    {
                        "type":"string", // Store data in the a.c path.
                        "name":"a.c"
                    }
                ],
                "method":"post",
                "defaultHeader":{
                    "X-Custom-Header":"test header"
                },
                "customHeader":{
                    "X-Custom-Header2":"test header2"
                },
                "parameters":"abc=1&def=1",
                "batchSize":256
            },
            "name":"restapiwriter",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0" // The maximum number of dirty data records allowed. 
        },
        "speed":{
            "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "concurrent":1, // The maximum number of parallel threads. 
            "mbps":"12"// The maximum transmission rate.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}