This topic describes the parameters that are supported by RestAPI Writer and how to configure RestAPI Writer by using the codeless user interface (UI) and code editor.
Background information
RestAPI Writer can send requests to RESTful APIs to write data. RestAPI Writer receives data from a reader, generates JSON-formatted data based on the setting of the column parameter, and then sends requests to RESTful APIs to transfer the data.
Parameters
To implement data integration, you must add a data source and configure it as the source or destination. You must also configure the data that you want to integrate and the data types. During data integration, a reader extracts data from the source, and a writer writes data to the destination.
Scheduling parameters are not supported for data synchronization nodes that use RestAPI Writer.
Parameter | Description | Required | Default value |
---|---|---|---|
url | The URL of the RESTful API. | Yes | No default value |
dataMode | The format in which RESTful Writer transfers JSON-formatted data.
|
Yes | No default value |
column | The columns to which you want to write the generated JSON-formatted data. The type
field specifies the data type of a column. The name field specifies the JSON-formatted
path where the column is stored. You can also specify the column parameter in the
following format:
"column":[{"type":"long","name":"a.b" // Store data in the a.b path.},{"type":"string","name":"a.c"// Store data in the a.c path.}] Note For the column parameter, you must specify the type and name fields.
|
Yes | No default value |
dataPath | The path that is used to store the JSON-formatted data. | No | No default value |
method | The request method. Valid values: post and put. | Yes | No default value |
customHeader | The header information transferred to the RESTful API. | No | No default value |
authType | The authentication method. Valid values:
|
No | No default value |
authUsername/authPassword | The username and password used for basic authentication. | No | No default value |
authToken | The token used for token-based authentication. | No | No default value |
accessKey/accessSecret | The AccessKey pair used for Alibaba Cloud API signature-based authentication. | No | No default value |
batchSize | The maximum number of data records that can be transferred in each request when the dataMode parameter is set to multiData. | Yes | 512 |
Configure RestAPI Writer by using the codeless UI
- Configure data sources.
Configure Source and Target for the synchronization node.
Parameter Description Connection Select RestAPI from the left-side drop-down list and a data source from the right-side drop-down list in the Target section. Request Method This parameter is equivalent to the method parameter that is described in the preceding section. Data structure of request parameters This parameter is equivalent to the dataMode parameter that is described in the preceding section. BatchSize This parameter is equivalent to the batchSize parameter that is described in the preceding section. json path to store data This parameter is equivalent to the dataPath parameter that is described in the preceding section. Header This parameter is equivalent to the customHeader parameter that is described in the preceding section. - Configure channel control policies.
Parameter Description Expected Maximum Concurrency The maximum number of parallel threads that the synchronization node uses to read data from the source or write data to the destination. You can configure the parallelism for the synchronization node on the codeless UI. Bandwidth Throttling Specifies whether to enable bandwidth throttling. You can enable bandwidth throttling and specify a maximum transmission rate to prevent heavy read workloads on the source. We recommend that you enable bandwidth throttling and set the maximum transmission rate to an appropriate value based on the configurations of the source. Dirty Data Records Allowed The maximum number of dirty data records allowed. Distributed Execution This parameter is not supported for synchronization nodes that use RestAPI Writer.
Configure RestAPI Reader by using the code editor
{
"type":"job",
"version":"2.0",
"steps":[
{
"stepType":"stream",
"parameter":{
},
"name":"Reader",
"category":"reader"
},
{
"stepType":"restapi",
"parameter":{
"url":"http://127.0.0.1:5000/writer1",
"dataMode":"oneData",
"responseType":"json",
"column":[
{
"type":"long", // Store data in the a.b path.
"name":"a.b"
},
{
"type":"string", // Store data in the a.c path.
"name":"a.c"
}
],
"method":"post",
"defaultHeader":{
"X-Custom-Header":"test header"
},
"customHeader":{
"X-Custom-Header2":"test header2"
},
"parameters":"abc=1&def=1",
"batchSize":256
},
"name":"restapiwriter",
"category":"writer"
}
],
"setting":{
"errorLimit":{
"record":"0" // The maximum number of dirty data records allowed.
},
"speed":{
"throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true.
"concurrent":1, // The maximum number of parallel threads.
"mbps":"12"// The maximum transmission rate.
}
},
"order":{
"hops":[
{
"from":"Reader",
"to":"Writer"
}
]
}
}