Create Hadoop data source configuration
Debugging
Authorization information
The following table shows the authorization information corresponding to the API. The authorization information can be used in the Action policy element to grant a RAM user or RAM role the permissions to call this API operation. Description:
- Operation: the value that you can use in the Action element to specify the operation on a resource.
- Access level: the access level of each operation. The levels are read, write, and list.
- Resource type: the type of the resource on which you can authorize the RAM user or the RAM role to perform the operation. Take note of the following items:
- For mandatory resource types, indicate with a prefix of * .
- If the permissions cannot be granted at the resource level,
All Resourcesis used in the Resource type column of the operation.
- Condition Key: the condition key that is defined by the cloud service.
- Associated operation: other operations that the RAM user or the RAM role must have permissions to perform to complete the operation. To complete the operation, the RAM user or the RAM role must have the permissions to perform the associated operations.
| Operation | Access level | Resource type | Condition key | Associated operation |
|---|---|---|---|---|
| gpdb:CreateHadoopDataSource | create | *DBInstance acs:gpdb:{#regionId}:{#accountId}:dbinstance/{#DBInstanceId} |
| none |
Request parameters
| Parameter | Type | Required | Description | Example |
|---|---|---|---|---|
| RegionId | string | No | Region ID. Note
You can call the DescribeRegions interface to view available region IDs.
| cn-hangzhou |
| DataSourceName | string | No | Service name. | hdfs_pxf |
| DataSourceDescription | string | No | Service description. | pxf for hdfs data source |
| DataSourceType | string | No | Type of Hadoop external table to be enabled, with values:
| HDFS |
| HadoopCreateType | string | No | External service type:
| emr |
| HadoopHostsAddress | string | No | Address and hostname of the Hadoop cluster's source node in the /etc/hosts file. | 192.168.220.128 master-1-1.c-xxx.cn-shanghai.emr.aliyuncs.com 192.168.220.129 core-1-1.c-xxx.cn-shanghai.emr.aliyuncs.com 192.168.220.130 core-1-2.c-xxx.cn-shanghai.emr.aliyuncs.com |
| HadoopCoreConf | string | No | Content string of the Hadoop core-site.xml file. | <?xml version="1.0" ?> <!-- Created at 2023-08-15 13:52:39.527 --> <configuration> <property> <name>hadoop.http.authentication.kerberos.keytab</name> <value>/etc/emr/hadoop-conf/http.keytab</value> </property> <property> <name>fs.oss.idle.timeout.millisecond</name> <value>30000</value> </property> <property> <name>fs.oss.download.thread.concurrency</name> <value>32</value> </property> </configuration> |
| HDFSConf | string | No | Content string of the Hadoop hdfs-site.xml file. This field is required when enabling an HDFS external table. | <?xml version="1.0" ?> <!-- Created at 2023-08-15 13:52:43.945 --> <configuration> <property> <name>dfs.datanode.cache.revocation.timeout.ms</name> <value>900000</value> </property> <property> <name>dfs.namenode.resource.check.interval</name> <value>5000</value> </property> </configuration> |
| HiveConf | string | No | Content string of the Hadoop hive-site.xml file. This field is required when enabling a HIVE external table. | <?xml version="1.0" ?> <!-- Created at 2023-08-15 13:52:50.646 --> <configuration> <property> <name>hive.exec.reducers.bytes.per.reducer</name> <value>256000000</value> </property> <property> <name>hive.stats.column.autogather</name> <value>false</value> </property> </configuration> |
| YarnConf | string | No | Content string of the Hadoop yarn-site.xml file. This field is required when enabling an HDFS external table. | <?xml version="1.0" ?> <!-- Created at 2023-08-15 13:53:29.021 --> <configuration> <property> <name>yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user</name> <value>hadoop</value> </property> <property> <name>yarn.scheduler.fair.dynamic.max.assign</name> <value>true</value> </property> </configuration> |
| DBInstanceId | string | Yes | Instance ID. | gp-xxxxxxx |
| MapReduceConf | string | No | Content string of the Hadoop mapred-site.xml file. This field is required when enabling an HDFS external table. | xxxxxx |
| EmrInstanceId | string | No | When HadoopCreateType=Emr, this field should contain the EMR instance ID. | c-1234567 |
Response parameters
Examples
Sample success responses
JSONformat
{
"RequestId": "2C125605-266F-41CA-8AC5-3A643D4F42C5",
"DataSourceId": 123
}Error codes
For a list of error codes, visit the Service error codes.