Realtime Compute for Apache Flink:Manage Hive catalogs

Hive metastore

Verify that hive.metastore.uris in hive-site.xml points to the internal or public IP address of your Hive host:

<property>
    <name>hive.metastore.uris</name>
    <value>thrift://xx.yy.zz.mm:9083</value>
    <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
</property>

If you set hive.metastore.uris to a hostname instead of an IP address, configure Alibaba Cloud DNS PrivateZone to resolve it. Without DNS resolution, Ververica Platform (VVP) returns an UnknownHostException when connecting to the metastore. See Add a DNS record to a private zone.

Alibaba Cloud DLF

Add the following properties to hive-site.xml:

<property>
  <name>hive.imetastoreclient.factory.class</name>
  <value>com.aliyun.datalake.metastore.hive2.DlfMetaStoreClientFactory</value>
</property>
<property>
  <name>dlf.catalog.uid</name>
  <value>${YOUR_DLF_CATALOG_UID}</value>
</property>
<property>
  <name>dlf.catalog.endpoint</name>
  <value>${YOUR_DLF_ENDPOINT}</value>
</property>
<property>
  <name>dlf.catalog.region</name>
  <value>${YOUR_DLF_CATALOG_REGION}</value>
</property>
<property>
  <name>dlf.catalog.accessKeyId</name>
  <value>${YOUR_ACCESS_KEY_ID}</value>
</property>
<property>
  <name>dlf.catalog.accessKeySecret</name>
  <value>${YOUR_ACCESS_KEY_SECRET}</value>
</property>

OSS

Add the following properties to hive-site.xml:

<property>
  <name>fs.oss.impl.disable.cache</name>
  <value>true</value>
</property>
<property>
  <name>fs.oss.impl</name>
  <value>org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem</value>
</property>
<property>
  <name>hive.metastore.warehouse.dir</name>
  <value>${YOUR_OSS_WAREHOUSE_DIR}</value>
</property>
<property>
  <name>fs.oss.endpoint</name>
  <value>${YOUR_OSS_ENDPOINT}</value>
</property>
<property>
  <name>fs.oss.accessKeyId</name>
  <value>${YOUR_ACCESS_KEY_ID}</value>
</property>
<property>
  <name>fs.oss.accessKeySecret</name>
  <value>${YOUR_ACCESS_KEY_SECRET}</value>
</property>
<property>
  <name>fs.defaultFS</name>
  <value>oss://${YOUR_OSS_BUCKET_DOMIN}</value>
</property>

OSS-HDFS

1. Add the following properties to hive-site.xml:

   Parameter	Required	Description
   hive.metastore.warehouse.dir	Yes	Directory where table data is stored.
   fs.oss.endpoint	Yes	OSS endpoint. See Regions and endpoints.
   fs.oss.accessKeyId	Yes	AccessKey ID. See Obtain an AccessKey pair.
   fs.oss.accessKeySecret	Yes	AccessKey secret. See Obtain an AccessKey pair.
   fs.defaultFS	Yes	Default file system for table data. Set to the HDFS endpoint of the target bucket, for example oss://oss-hdfs-bucket.cn-hangzhou.oss-dls.aliyuncs.com/.

   <property>
     <name>fs.jindo.impl</name>
     <value>com.aliyun.jindodata.jindo.JindoFileSystem</value>
   </property>
   <property>
     <name>hive.metastore.warehouse.dir</name>
     <value>${YOUR_OSS_WAREHOUSE_DIR}</value>
   </property>
   <property>
     <name>fs.oss.endpoint</name>
     <value>${YOUR_OSS_ENDPOINT}</value>
   </property>
   <property>
     <name>fs.oss.accessKeyId</name>
     <value>${YOUR_ACCESS_KEY_ID}</value>
   </property>
   <property>
     <name>fs.oss.accessKeySecret</name>
     <value>${YOUR_ACCESS_KEY_SECRET}</value>
   </property>
   <property>
     <name>fs.defaultFS</name>
     <value>oss://${YOUR_OSS_HDFS_BUCKET_DOMIN}</value>
   </property>

2. (Optional) If you plan to read Parquet-format Hive tables stored in OSS-HDFS, add the following configuration to Realtime Compute for Apache Flink:

   fs.oss.jindo.accessKeyId: ${YOUR_ACCESS_KEY_ID}
   fs.oss.jindo.accessKeySecret: ${YOUR_ACCESS_KEY_SECRET}
   fs.oss.jindo.endpoint: ${YOUR_JINODO_ENDPOINT}
   fs.oss.jindo.buckets: ${YOUR_JINDO_BUCKETS}

   For parameter details, see Write data to OSS-HDFS.

If your data is already stored in Realtime Compute for Apache Flink, skip Steps 2–3 and go directly to Create a Hive catalog.

UI method

From the console:

1. Log on to the Realtime Compute for Apache Flink console, find the workspace, and click Console in the Actions column.

2. Click Catalogs.

3. On the Catalog List page, find the catalog and click View in the Actions column.

4. Find the target database and click View in the Actions column.

5. Click Create Table.

6. On the Built-in tab of the Create Table dialog box, select a table type from the Connection Type drop-down list, select a connector type, and click Next.

7. Enter the table creation statement and configure the parameters:

   CREATE TABLE `${catalog_name}`.`${db_name}`.`${table_name}` (
     id INT,
     name STRING
   ) WITH (
     'connector' = 'hive'
   );

8. Click OK.

SQL command method

By running an SQL statement:

On the Scripts page, run the CREATE TABLE statement, then select it and click Run.

UI method

From the console:

1. Log on to the Realtime Compute for Apache Flink console, find the workspace, and click Console in the Actions column.

2. Click Catalogs.

3. In the Catalogs pane, navigate to the target table under the target database and catalog.

4. On the table details page, click Delete in the Actions column.

5. In the confirmation dialog, click OK.

SQL command method

By running an SQL statement:

On the Scripts page, run:

-- Drop the Hive table.
DROP TABLE `${catalog_name}`.`${db_name}`.`${table_name}`;

Example:

-- Drop the Hive table.
DROP TABLE `flinkexporthive`.`flinkhive`.`flink_hive_test`;