Set Up Hive Data Source Connections in DataWorks - DataWorks

Configure a Hive data source in DataWorks using one of three connection modes: instance mode (for EMR clusters in the same or a different Alibaba Cloud account), connection string mode (for self-managed Hive clusters), or CDH mode (for Cloudera Distribution of Hadoop clusters).

Data source type: hive
Supported ConnectionPropertiesMode values: InstanceMode, UrlMode, CdhMode

Instance mode — current account

Use this mode to connect to a Hive instance on an Alibaba Cloud EMR cluster in your own account.

Parameter	Type	Example	Required	Description
`regionId`	String	`cn-shanghai`	Yes	The region ID of the EMR cluster.
`clusterId`	String	`c-d1a993bbcd298315`	Yes	The EMR cluster instance ID.
`database`	String	`db1`	Yes	The name of the database.
`version`	String	`2.3.9`	Yes	The Hive version.
`authType`	String	`Executor`	Yes	The identity used to access Object Storage Service (OSS). Valid values: `Executor` (the executor in the development environment), `PrimaryAccount` (the Alibaba Cloud account, for production), `SubAccount` (a specified RAM user, for production), `RamRole` (a specified RAM role, for production).
`authIdentity`	String	`123123`	No	The ID of the RAM user or RAM role. Required if `authType` is `SubAccount` or `RamRole`.
`loginMode`	String	`LDAP`	Yes	The Hive login mode. Valid values: `Anonymous`, `LDAP`.
`username`	String	`xxx`	No	The username for Hive authentication. Required if `loginMode` is `LDAP`.
`password`	String	`xxx`	No	The password for Hive authentication. Required if `loginMode` is `LDAP`.
`securityProtocol`	String	`authTypeNone`	No	The security protocol. Valid values: `authTypeNone` (no authentication), `authTypeSsl` (SSL), `authTypeKerberos` (Kerberos).
`truststoreFile`	String	`1`	No	The ID of the truststore file. Required if `securityProtocol` is `authTypeSsl`.
`truststorePassword`	String	`apasara`	No	The password of the truststore file. Required if `securityProtocol` is `authTypeSsl`.
`keystoreFile`	String	`2`	No	The ID of the keystore file. Required if `securityProtocol` is `authTypeSsl`.
`keystorePassword`	String	`apasara`	No	The password of the keystore file. Required if `securityProtocol` is `authTypeSsl`.
`kerberosFileConf`	String	`123123`	No	The IDs of the Kerberos configuration files. Required if `securityProtocol` is `authTypeKerberos`.
`kerberosFileKeytab`	String	`123123`	No	The ID of the Kerberos keytab file. Required if `securityProtocol` is `authTypeKerberos`.
`principal`	String	`xxx@com`	No	The Kerberos principal. Required if `securityProtocol` is `authTypeKerberos`.
`hiveConfig`	JSON Object	`{"fs.oss.accessKeyId": "xxx"}`	No	Extended configuration properties passed to the Hive driver.
`envType`	String	`Dev`	Yes	The environment type. Valid values: `Dev` (development environment), `Prod` (production environment).

Example

{
    "clusterId": "c-xxxxxxxxx",
    "regionId": "cn-shanghai",
    "database": "db",
    "loginMode": "Anonymous",
    "version": "2.3.9",
    "authType": "Executor",
    "securityProtocol": "authTypeNone",
    "envType": "Dev"
}

Instance mode — another account

Use this mode to connect to a Hive instance on an EMR cluster owned by a different Alibaba Cloud account. This mode requires cross-account RAM role authorization. authType must be set to RamRole.

Parameter	Type	Example	Required	Description
`crossAccountOwnerId`	String	`11111`	Yes	The UID of the other Alibaba Cloud account.
`crossAccountRoleName`	String	`xx-role`	Yes	The name of the RAM role assumed by the other account.
`regionId`	String	`cn-shanghai`	Yes	The region ID of the EMR cluster.
`clusterId`	String	`c-d1a993bbcd298315`	Yes	The EMR cluster instance ID.
`database`	String	`db1`	Yes	The name of the database.
`version`	String	`2.3.9`	Yes	The Hive version.
`authType`	String	`RamRole`	Yes	Must be set to `RamRole` for cross-account access.
`loginMode`	String	`LDAP`	Yes	The Hive login mode. Valid values: `Anonymous`, `LDAP`.
`username`	String	`xxx`	No	The username for Hive authentication. Required if `loginMode` is `LDAP`.
`password`	String	`xxx`	No	The password for Hive authentication. Required if `loginMode` is `LDAP`.
`securityProtocol`	String	`authTypeNone`	No	The security protocol. Valid values: `authTypeNone` (no authentication), `authTypeSsl` (SSL), `authTypeKerberos` (Kerberos).
`truststoreFile`	String	`1`	No	The ID of the truststore file. Required if `securityProtocol` is `authTypeSsl`.
`truststorePassword`	String	`apasara`	No	The password of the truststore file. Required if `securityProtocol` is `authTypeSsl`.
`keystoreFile`	String	`2`	No	The ID of the keystore file. Required if `securityProtocol` is `authTypeSsl`.
`keystorePassword`	String	`apasara`	No	The password of the keystore file. Required if `securityProtocol` is `authTypeSsl`.
`kerberosFileConf`	String	`123123`	No	The IDs of the Kerberos configuration files. Required if `securityProtocol` is `authTypeKerberos`.
`kerberosFileKeytab`	String	`123123`	No	The ID of the Kerberos keytab file. Required if `securityProtocol` is `authTypeKerberos`.
`principal`	String	`xxx@com`	No	The Kerberos principal. Required if `securityProtocol` is `authTypeKerberos`.
`hiveConfig`	JSON Object	`{"fs.oss.accessKeyId": "xxx"}`	No	Extended configuration properties passed to the Hive driver.
`envType`	String	`Dev`	Yes	The environment type. Valid values: `Dev` (development environment), `Prod` (production environment).

Example

{
    "crossAccountOwnerId": "11111",
    "crossAccountRoleName": "xx-role",
    "clusterId": "c-xxxxxxxxx",
    "regionId": "cn-shanghai",
    "database": "db",
    "loginMode": "LDAP",
    "version": "2.3.9",
    "authType": "RamRole",
    "securityProtocol": "authTypeNone",
    "envType": "Dev"
}

Connection string mode

Use this mode to connect to a self-managed Hive cluster by specifying the HiveServer2 host and port directly.

Parameter	Type	Example	Required	Description
`address`	JSON Array	`[{"host":"127.0.0.1","port":"1234"}]`	Yes	The HiveServer2 address. Contains exactly one host-and-port pair.
`database`	String	`hive_database`	Yes	The name of the database.
`metaType`	String	`HiveMetastore`	Yes	The metadata storage type. Valid values: `HiveMetastore`, `DLF1.0`.
`metastoreUris`	String	`thrift://123:123`	Yes	The Thrift URI of the Hive Metastore, in the format `thrift://<host>:<port>`.
`version`	String	`2.3.9`	Yes	The Hive version.
`accessId`	String	`xxxxx`	No	The AccessKey ID. Required if `metaType` is `DLF1.0`.
`accessKey`	String	`xxxxx`	No	The AccessKey secret. Required if `metaType` is `DLF1.0`.
`properties`	JSON Object	`{"useSSL": "false"}`	No	Additional JDBC driver properties.
`defaultFS`	String	`xxx`	No	The default file system URI.
`loginMode`	String	`LDAP`	Yes	The Hive login mode. Valid values: `Anonymous`, `LDAP`.
`username`	String	`xxx`	No	The username for Hive authentication. Required if `loginMode` is `LDAP`.
`password`	String	`xxx`	No	The password for Hive authentication. Required if `loginMode` is `LDAP`.
`securityProtocol`	String	`authTypeNone`	No	The security protocol. Valid values: `authTypeNone` (no authentication), `authTypeSsl` (SSL), `authTypeKerberos` (Kerberos).
`truststoreFile`	String	`1`	No	The ID of the truststore file. Required if `securityProtocol` is `authTypeSsl`.
`truststorePassword`	String	`apasara`	No	The password of the truststore file. Required if `securityProtocol` is `authTypeSsl`.
`keystoreFile`	String	`2`	No	The ID of the keystore file. Required if `securityProtocol` is `authTypeSsl`.
`keystorePassword`	String	`apasara`	No	The password of the keystore file. Required if `securityProtocol` is `authTypeSsl`.
`kerberosFileConf`	String	`123123`	No	The IDs of the Kerberos configuration files. Required if `securityProtocol` is `authTypeKerberos`.
`kerberosFileKeytab`	String	`123123`	No	The ID of the Kerberos keytab file. Required if `securityProtocol` is `authTypeKerberos`.
`principal`	String	`xxx@com`	No	The Kerberos principal. Required if `securityProtocol` is `authTypeKerberos`.
`hiveConfig`	JSON Object	`{"fs.oss.accessKeyId": "xxx"}`	No	Extended configuration properties passed to the Hive driver.
`envType`	String	`Dev`	Yes	The environment type. Valid values: `Dev` (development environment), `Prod` (production environment).

Example

{
    "address": [
        {
            "host": "127.0.0.1",
            "port": 5432
        }
    ],
    "database": "db",
    "properties": {
        "connectTimeout": "2000"
    },
    "username": "aliyun",
    "password": "xxx",
    "metastoreUris": "thrift://123:123",
    "metaType": "HiveMetastore",
    "loginMode": "LDAP",
    "securityProtocol": "authTypeNone",
    "envType": "Dev"
}

CDH mode

Use this mode to connect to a Hive cluster managed by Cloudera Distribution of Hadoop (CDH).

Parameter	Type	Example	Required	Description
`clusterIdentifier`	String	`cdh_cluster`	Yes	The CDH cluster identifier.
`database`	String	`db1`	Yes	The name of the database.
`defaultFS`	String	`xxx`	No	The default file system URI.
`loginMode`	String	`LDAP`	Yes	The Hive login mode. Valid values: `Anonymous`, `LDAP`.
`username`	String	`xxx`	No	The username for Hive authentication. Required if `loginMode` is `LDAP`.
`password`	String	`xxx`	No	The password for Hive authentication. Required if `loginMode` is `LDAP`.
`securityProtocol`	String	`authTypeNone`	No	The security protocol. Valid values: `authTypeNone` (no authentication), `authTypeSsl` (SSL). Kerberos authentication (`authTypeKerberos`) is not supported in CDH mode.
`truststoreFile`	String	`1`	No	The ID of the truststore file. Required if `securityProtocol` is `authTypeSsl`.
`truststorePassword`	String	`apasara`	No	The password of the truststore file. Required if `securityProtocol` is `authTypeSsl`.
`keystoreFile`	String	`2`	No	The ID of the keystore file. Required if `securityProtocol` is `authTypeSsl`.
`keystorePassword`	String	`apasara`	No	The password of the keystore file. Required if `securityProtocol` is `authTypeSsl`.
`kerberosFileConf`	String	`123123`	No	The IDs of the Kerberos configuration files.
`kerberosFileKeytab`	String	`123123`	No	The ID of the Kerberos keytab file.
`principal`	String	`xxx@com`	No	The Kerberos principal.
`hiveConfig`	JSON Object	`{"fs.oss.accessKeyId": "xxx"}`	No	Extended configuration properties passed to the Hive driver.
`envType`	String	`Dev`	Yes	The environment type. Valid values: `Dev` (development environment), `Prod` (production environment).

Example

{
    "clusterIdentifier": "c-xxxxxxxxx",
    "database": "db",
    "ldapEnable": "LDAP",
    "authType": "Executor",
    "securityProtocol": "authTypeNone",
    "envType": "Dev"
}

Limitations

CDH mode does not support Kerberos authentication. Only authTypeNone and authTypeSsl are valid values for securityProtocol in CDH mode.
The address parameter in connection string mode accepts exactly one host-and-port pair. Multiple hosts are not supported.
authIdentity is required only when authType is SubAccount or RamRole in instance mode.