Realtime Compute for Apache Flink:Catalogs FAQ

Hologres Catalog job ends without consuming data in real time

Symptoms

The Flink job completes after reading existing data from Hologres and then exits. It does not wait for new records to arrive.

Cause

By default, Realtime Compute for Apache Flink reads data from Hologres in batch mode, not streaming mode.

Solution

Add the /*+ OPTIONS('binlog'='true') */ hint to your SQL query to switch to streaming mode:

INSERT INTO testfinish
SELECT
  col1,
  col2,
  col3
FROM testsource /*+ OPTIONS ('binlog' = 'true') */
   LEFT JION testdim /*+ OPTIONS ('binlog' = 'true') */
         ON testsource.co11 = testdim.'col1';

CatalogTableProvider interface error with CREATE TABLE AS

Error message

org.apache.flink.table.api.ValidationException: SQL validation failed. CREATE TABLE ... AS TABLE ... statement requires target catalog 'XXXXXX' (MongoDBCatalog) implements org.apache.flink.table.catalog.CatalogTableProvider interface. Hint: Please refer the document and use another catalog that supports schema evolution as the target catalog.

Cause

The sink table's catalog does not implement the CatalogTableProvider interface and cannot be used with the CREATE TABLE AS (CTAS) statement. For example, MongoDB catalogs do not support CTAS as a sink.

Solution

Switch to a source table and sink table that support CTAS. For the full list of supported storage systems, see CREATE TABLE AS (CTAS).

MySQL catalog queries are slow or return server exceptions

Slow query responses or server exceptions when querying a MySQL catalog are usually caused by high network latency or unstable connections between Flink and MySQL. Apply the fix that matches your situation.

Cause 1: Insufficient public bandwidth

Check whether the public bandwidth from Realtime Compute for Apache Flink to MySQL is too low. The default public bandwidth may be as low as 2 Mbit/s. Also check whether cross-region access is involved. For instructions on viewing your bandwidth, see How do I view the public bandwidth?

Cause 2: Unstable network connection

Increase the values of the interactive_timeout and wait_timeout parameters on your MySQL instance to reduce connection drop frequency.

Cause 3: Connection timeout too short

Add the 'connect.timeout'='120s' option to the WITH clause when creating a table for the MySQL catalog:

-- Add to the WITH clause of your CREATE TABLE statement
'connect.timeout' = '120s'

Cannot obtain STS token from EMR meta-service

Error message

Caused by: MetaException(message:Initialize DlfMetaStoreClient failed: Initialize DlfMetaStoreClient failed: Cannot obtain STS token from EMR meta-service. Note that AK-Mode[dlf.catalog.akMode] can only used in EMR clusters, otherwise you should config the dlf.catalog.accessKeyId and dlf.catalog.accessKeySecret explicitly.)
  at com.aliyun.datalake.metastore.hive2.ProxyMetaStoreClient.createClient(ProxyMetaStoreClient.java:91)
  at com.aliyun.datalake.metastore.hive2.ProxyMetaStoreClient.<init>(ProxyMetaStoreClient.java:71)
  ... 41 more

Cause

The hive-site.xml configuration file contains an EMR_AUTO setting for dlf.catalog.akMode. This setting only works inside E-MapReduce (EMR) clusters. Running it outside EMR causes Hive catalog initialization to fail.

Solution

Update hive-site.xml in your hive-conf-dir directory:

1. Remove the following property:

   <property>
     <name>dlf.catalog.akMode</name>
     <value>EMR_AUTO</value>
   </property>

2. Add your AccessKey ID and AccessKey secret:

   <property>
       <name>dlf.catalog.accessKeyId</name>
       <value>${AccessKeyId}</value>
     </property>
   <property>
       <name>dlf.catalog.accessKeySecret</name>
       <value>${AccessKeySecret}</value>
     </property>