Create External Data Source Configuration
Debugging
Authorization information
The following table shows the authorization information corresponding to the API. The authorization information can be used in the Action policy element to grant a RAM user or RAM role the permissions to call this API operation. Description:
- Operation: the value that you can use in the Action element to specify the operation on a resource.
- Access level: the access level of each operation. The levels are read, write, and list.
- Resource type: the type of the resource on which you can authorize the RAM user or the RAM role to perform the operation. Take note of the following items:
- For mandatory resource types, indicate with a prefix of * .
- If the permissions cannot be granted at the resource level,
All Resourcesis used in the Resource type column of the operation.
- Condition Key: the condition key that is defined by the cloud service.
- Associated operation: other operations that the RAM user or the RAM role must have permissions to perform to complete the operation. To complete the operation, the RAM user or the RAM role must have the permissions to perform the associated operations.
| Operation | Access level | Resource type | Condition key | Associated operation |
|---|---|---|---|---|
| gpdb:ModifyStreamingJob | create | *DBInstance acs:gpdb:{#regionId}:{#accountId}:dbinstance/{#DBInstanceId} |
| none |
Request parameters
| Parameter | Type | Required | Description | Example |
|---|---|---|---|---|
| RegionId | string | No | Region ID. Note
You can call the DescribeRegions API to view available region IDs.
| cn-beijing |
| JobId | long | Yes | Job ID. | 1 |
| DBInstanceId | string | Yes | Instance ID | ModifyStreamingJob |
| JobDescription | string | No | Job description. | test-job |
| SrcColumns | array | No | Source data field list. | |
| string | No | Source data field key. | src_column_1 | |
| DestColumns | array | No | Target data table mapping field list. | |
| string | No | Mapping field key in the target table. | dest_column_1 | |
| Account | string | No | Account name. | test-account |
| Password | string | No | Password. | pwd123 |
| DestDatabase | string | No | Target database name. | dest-db |
| DestSchema | string | No | Target schema. | dest-schema |
| DestTable | string | No | Target table name. | dest-table |
| WriteMode | string | No | The write mode. Valid values:
Enumeration Value:
| INSERT/UPDATE/MERGE |
| TryRun | boolean | No | Specifies whether to test the real-time job. Valid values:
Default value: false. | true |
| JobConfig | string | No | Job configuration file, required for professional mode. | DATABASE: adbpgss_test USER: adbpgss_test PASSWORD: adbpgssTest HOST: gp-xxx-master.gpdb.rds-aliyun-pre.rds.aliyuncs.com PORT: 5432 KAFKA: INPUT: SOURCE: BROKERS: broker1:9092,broker2:9092,broker3:9092 TOPIC: testtopic FALLBACK_OFFSET: earliest KEY: COLUMNS: - NAME: customer_id TYPE: int FORMAT: delimited DELIMITED_OPTION: DELIMITER: '|' VALUE: COLUMNS: - TYPE: integer NAME: l_orderkey - TYPE: integer NAME: l_partkey - TYPE: integer NAME: l_suppkey - TYPE: integer NAME: l_linenumber - TYPE: decimal NAME: l_quantity - TYPE: decimal NAME: l_extendedprice - TYPE: decimal NAME: l_discount - TYPE: decimal NAME: l_tax - TYPE: char NAME: l_returnflag - TYPE: char NAME: l_linestatus - TYPE: date NAME: l_shipdate - TYPE: date NAME: l_commitdate - TYPE: date NAME: l_receiptdate - TYPE: text NAME: l_shipinstruct - TYPE: text NAME: l_shipmode - TYPE: text NAME: l_comment FORMAT: delimited DELIMITED_OPTION: DELIMITER: '|' ERROR_LIMIT: 10 OUTPUT: SCHEMA: adbpgss_test TABLE: write_with_insert_plaintext MODE: MERGE MATCH_COLUMNS: - l_orderkey - l_partkey - l_suppkey UPDATE_COLUMNS: - l_linenumber - l_quantity - l_extendedprice - l_discount - l_tax - l_returnflag - l_linestatus - l_shipdate - l_commitdate - l_receiptdate - l_shipinstruct - l_shipmode - l_comment MAPPING: - EXPRESSION: l_orderkey NAME: l_orderkey - EXPRESSION: l_partkey NAME: l_partkey - EXPRESSION: l_suppkey NAME: l_suppkey - EXPRESSION: l_linenumber NAME: l_linenumber - EXPRESSION: l_quantity NAME: l_quantity - EXPRESSION: l_extendedprice NAME: l_extendedprice - EXPRESSION: l_discount NAME: l_discount - EXPRESSION: l_tax NAME: l_tax - EXPRESSION: l_returnflag NAME: l_returnflag - EXPRESSION: l_linestatus NAME: l_linestatus - EXPRESSION: l_shipdate NAME: l_shipdate - EXPRESSION: l_commitdate NAME: l_commitdate - EXPRESSION: l_receiptdate NAME: l_receiptdate - EXPRESSION: l_shipinstruct NAME: l_shipinstruct - EXPRESSION: l_shipmode NAME: l_shipmode - EXPRESSION: l_comment NAME: l_comment COMMIT: MAX_ROW: 1000 MINIMAL_INTERVAL: 1000 CONSISTENCY: ATLEAST POLL: BATCHSIZE: 1000 TIMEOUT: 1000 PROPERTIES: group.id: testgroup |
| GroupName | string | No | Kafka group name | group_name |
| FallbackOffset | string | No | The fallback offset for data consumption.
Valid values:
| EARLIEST / LATEST |
| Consistency | string | No | The delivery guarantee setting. Enumeration Value:
| ATLEAST / EXACTLY |
| ErrorLimitCount | long | No | When the data in Kafka does not match the ADBPG target table, it will cause a write failure. This value is the number of error rows allowed; exceeding this will cause the task to fail. | 5 |
| MatchColumns | array | No | Match columns, usually all primary key columns of the target table. If all column values in this configuration are the same, the two rows of data are considered duplicates. | |
| string | No | Match field key. | column_1 | |
| UpdateColumns | array | No | Update columns, usually all non-primary key columns of the target table. When data duplication is determined through MatchColumns, updating the UpdateColumns column values will result in new data overwriting old data. | |
| string | No | Update field key. | column_1 |
Response parameters
Examples
Sample success responses
JSONformat
{
"RequestId": "B4CAF581-2AC7-41AD-8940-D56DF7AADF5B"
}Error codes
For a list of error codes, visit the Service error codes.