Realtime Compute for Apache Flink:Manage Hive catalogs

Prerequisites

Limitations

• Only self-managed Hive metastores are supported.

• Only Ververica Runtime (VVR) 6.x supports Hive 1.x, 2.1.x, and 2.2.x, as support for these versions is deprecated in Apache Flink 1.16 and later.

• If a Hive catalog uses Data Lake Formation (DLF) as its metastore, only Ververica Runtime (VVR) 8.0.6 or later supports creating non-Hive tables.

• Only Ververica Runtime (VVR) 8.0.6 or later supports writing data to OSS-HDFS.

Configure Hive metadata

1. Connect the Virtual Private Cloud (VPC) of the Hadoop cluster to the VPC of Realtime Compute for Apache Flink.

   Connect the VPCs using Alibaba Cloud DNS PrivateZone. For more information, see Resolver. Once the network is connected, Realtime Compute for Apache Flink uses the Hadoop cluster's configuration files to access it.

2. A Hive catalog can use either Hive MetaStore or Data Lake Formation (DLF) for metadata management. The following sections describe the required configurations.

   Hive MetaStore

   Ensure the hive.metastore.uris parameter in the hive-site.xml configuration file is configured correctly.

   <property>
       <name>hive.metastore.uris</name>
       <value>thrift://xx.yy.zz.mm:9083</value>
       <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
    </property>

   In the preceding example, xx.yy.zz.mm is the internal IP address or public IP address of the Hive MetaStore.

   Note

   If you set hive.metastore.uris to a hostname, you must configure a domain name resolution service. Otherwise, when the Flink development console remotely accesses Hive, the value of the hive.metastore.uris parameter cannot be resolved, and an UnknownHostException error is reported. For more information about how to configure a domain name resolution service, see Add a PrivateZone record.

   Data Lake Formation (DLF)

   To allow the Hive catalog to access DLF, add the following properties to the hive-site.xml configuration file.

   Note

   If your hive-site.xml file contains the dlf.catalog.akMode property, you must delete it. Otherwise, the Hive catalog cannot access DLF.

   <property>
     <name>hive.imetastoreclient.factory.class</name>
     <value>com.aliyun.datalake.metastore.hive2.DlfMetaStoreClientFactory</value>
   </property>
   <property>
     <name>dlf.catalog.uid</name>
     <value>${YOUR_DLF_CATALOG_UID}</value>
   </property>
   <property>
     <name>dlf.catalog.endpoint</name>
     <value>${YOUR_DLF_ENDPOINT}</value>
   </property>
   <property>
     <name>dlf.catalog.region</name>
     <value>${YOUR_DLF_CATALOG_REGION}</value>
   </property>
   <property>
     <name>dlf.catalog.accessKeyId</name>
     <value>${YOUR_ACCESS_KEY_ID}</value>
   </property>
   <property>
     <name>dlf.catalog.accessKeySecret</name>
     <value>${YOUR_ACCESS_KEY_SECRET}</value>
   </property>

   Parameter	Description	Remarks
   dlf.catalog.uid	Your Alibaba Cloud account ID.	For your account information, go to the User Information page.
   dlf.catalog.endpoint	The DLF service endpoint.	For more information, see Regions and endpoints. Note We recommend that you set the dlf.catalog.endpoint parameter to the VPC endpoint of DLF. For example, if your selected region is China (Hangzhou), set the dlf.catalog.endpoint parameter to dlf-vpc.cn-hangzhou.aliyuncs.com. To access DLF across VPCs, see How do I access services across VPCs?.
   dlf.catalog.region	The region of the DLF service.	For more information, see Regions and endpoints. Note Ensure this region matches the one specified in the dlf.catalog.endpoint parameter.
   dlf.catalog.accessKeyId	Your Alibaba Cloud AccessKey ID.	For more information, see Obtain an AccessKey pair.
   dlf.catalog.accessKeySecret	Your Alibaba Cloud AccessKey secret.	For more information, see Obtain an AccessKey pair.

3. A Hive catalog can store tables in Object Storage Service (OSS) or the OSS-HDFS service. The following sections describe the required configurations.

   OSS

   To allow the Hive catalog to access OSS, add the following properties to the hive-site.xml configuration file.

   <property>
     <name>fs.oss.impl.disable.cache</name>
     <value>true</value>
   </property>
   <property>
     <name>fs.oss.impl</name>
     <value>org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem</value>
   </property>
   <property>
     <name>hive.metastore.warehouse.dir</name>
     <value>${YOUR_OSS_WAREHOUSE_DIR}</value>
   </property>
   <property>
     <name>fs.oss.endpoint</name>
     <value>${YOUR_OSS_ENDPOINT}</value>
   </property>
   <property>
     <name>fs.oss.accessKeyId</name>
     <value>${YOUR_ACCESS_KEY_ID}</value>
   </property>
   <property>
     <name>fs.oss.accessKeySecret</name>
     <value>${YOUR_ACCESS_KEY_SECRET}</value>
   </property>
   <property>
     <name>fs.defaultFS</name>
     <value>oss://${YOUR_OSS_BUCKET_DOMIN}</value>
   </property>

   The following table describes the parameters.

   Parameter	Description	Remarks
   hive.metastore.warehouse.dir	The storage path for table data.
   fs.oss.endpoint	The Object Storage Service (OSS) endpoint.	For more information, see Regions and endpoints.
   fs.oss.accessKeyId	Your Alibaba Cloud AccessKey ID.	For more information, see Obtain an AccessKey pair.
   fs.oss.accessKeySecret	Your Alibaba Cloud AccessKey secret.	For more information, see Obtain an AccessKey pair.
   fs.defaultFS	The default file system for table data.	Specify your OSS bucket as the default file system. For example, oss://your-bucket-name.

   OSS-HDFS

   1. To allow the Hive catalog to access the OSS-HDFS service, add the following properties to the hive-site.xml configuration file.

      <property>
        <name>fs.jindo.impl</name>
        <value>com.aliyun.jindodata.jindo.JindoFileSystem</value>
      </property>
      <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>${YOUR_OSS_WAREHOUSE_DIR}</value>
      </property>
      <property>
        <name>fs.oss.endpoint</name>
        <value>${YOUR_OSS_ENDPOINT}</value>
      </property>
      <property>
        <name>fs.oss.accessKeyId</name>
        <value>${YOUR_ACCESS_KEY_ID}</value>
      </property>
      <property>
        <name>fs.oss.accessKeySecret</name>
        <value>${YOUR_ACCESS_KEY_SECRET}</value>
      </property>
      <property>
        <name>fs.defaultFS</name>
        <value>oss://${YOUR_OSS_HDFS_BUCKET_DOMIN}</value>
      </property>

      Parameter	Description	Remarks
      hive.metastore.warehouse.dir	The storage path for table data.
      fs.oss.endpoint	The Object Storage Service (OSS) endpoint.	For more information, see Regions and endpoints.
      fs.oss.accessKeyId	Your Alibaba Cloud AccessKey ID.	For more information, see Obtain an AccessKey pair.
      fs.oss.accessKeySecret	Your Alibaba Cloud AccessKey secret.	For more information, see Obtain an AccessKey pair.
      fs.defaultFS	The default file system for table data.	Specify your OSS bucket as the default file system. For example, oss://your-bucket-name.

   2. (Optional) To read Parquet-formatted Hive tables from the OSS-HDFS service, add the following parameters to the Realtime Compute for Apache Flink configuration file:

      fs.oss.jindo.accessKeyId: ${YOUR_ACCESS_KEY_ID}
      fs.oss.jindo.accessKeySecret: ${YOUR_ACCESS_KEY_SECRET}
      fs.oss.jindo.endpoint: ${YOUR_JINODO_ENDPOINT}
      fs.oss.jindo.buckets: ${YOUR_JINDO_BUCKETS}

      For more information about the parameters, see Write data to OSS-HDFS.

   Note

   If you use the fully managed storage feature of Realtime Compute for Apache Flink, you do not need to perform the following steps. You can proceed directly to Create a Hive catalog.

4. In the OSS console, create directories and upload the Hive configuration files and Hadoop dependencies to the target paths.

   1. Log on to the OSS console.

   2. In the left-side navigation pane, click Buckets.

   3. Click the name of the target bucket.

   4. In the oss://${bucket}/artifacts/namespaces/${ns}/ path, create a directory named ${hms}.

      For more information about how to create a directory in the OSS console, see Create directories. The following table describes the variables in the path.

      Parameter	Description
      ${bucket}	The name of the bucket used by Realtime Compute for Apache Flink.
      ${ns}	The workspace where you will use the Hive catalog.
      ${hms}	We recommend using the same name as the Hive catalog you plan to create.

      Note

      After you activate a workspace, Realtime Compute for Apache Flink automatically creates the /artifacts/namespaces/${ns}/ directory in the specified bucket to store data such as JAR packages. If you do not find this directory in the OSS console, go to the Artifacts page in the development console and upload a file to trigger directory creation.

   5. In the oss://${bucket}/artifacts/namespaces/${ns}/${hms} path, create two directories: hive-conf-dir and hadoop-conf-dir. For more information, see Create directories.

      Store the following files in the hive-conf-dir and hadoop-conf-dir directories:

      • The oss://${bucket}/artifacts/namespaces/${ns}/${hms}/hive-conf-dir/ directory stores the Hive configuration file hive-site.xml.

      • The oss://${bucket}/artifacts/namespaces/${ns}/${hms}/hadoop-conf-dir/ directory stores Hadoop configuration files, including core-site.xml, hdfs-site.xml, yarn-site.xml, and mapred-site.xml.

      After creating the directories, you can view them and copy their OSS URLs on the Files page.

   6. Upload your Hive configuration file (hive-site.xml) to the hive-conf-dir directory. For more information, see Upload objects.

   7. Upload the following configuration files to the hadoop-conf-dir directory. For more information, see Upload objects.

      • core-site.xml

      • hdfs-site.xml

      • mapred-site.xml

      • Other files, such as compressed packages used by Hive jobs.

Create a Hive catalog

After you configure the Hive metadata, you can create a Hive catalog. You can create a catalog by using the UI or by running SQL commands. We recommend that you use the UI.

Use a Hive catalog

-- Create the flink_hive_test table in the flinkhive database of the flinkexporthive catalog.
CREATE TABLE `flinkexporthive`.`flinkhive`.`flink_hive_test` (
  id INT,
  name STRING
) WITH (
  'connector' = 'hive'
);

-- Add a column to the Hive table.
ALTER TABLE `${catalog_name}`.`${db_name}`.`${table_name}` 
ADD column type-column;
-- Drop a column from the Hive table.
ALTER TABLE `${catalog_name}`.`${db_name}`.`${table_name}` 
DROP column;

-- Add the color column to the Hive table.
ALTER TABLE `flinkexporthive`.`flinkhive`.`flink_hive_test` 
ADD color STRING;
-- Drop the color column from the Hive table.
ALTER TABLE `flinkexporthive`.`flinkhive`.`flink_hive_test` 
DROP color;

INSERT INTO ${other_sink_table}
SELECT ...
FROM `${catalog_name}`.`${db_name}`.`${table_name}`;

INSERT INTO `${catalog_name}`.`${db_name}`.`${table_name}`
SELECT ... 
FROM ${other_source_table};

-- Drop the Hive table.
DROP TABLE `${catalog_name}`.`${db_name}`.`${table_name}`;

-- Drop the Hive table.
DROP TABLE `flinkexporthive`.`flinkhive`.`flink_hive_test`;

View Hive catalog

1. Go to the Catalogs page.

   1. Log in to the Realtime Compute for Apache Flink console.

   2. In the Actions column of the target workspace, click Console.

   3. Click Catalogs.

2. On the Catalog List page, check the Name and Type columns.

   Note

   To view the databases and tables in a catalog, click View in the Actions column.

Delete a Hive catalog