Realtime Compute for Apache Flink:Parquet

Limits

• The Parquet format can be used to read and write Parquet data.

• Only the supported Object Storage Service (OSS) connector can read data in the Parquet format.

Sample code

The following sample code provides an example on how to create a table in the Parquet format by using the OSS connector.

CREATE TABLE user_behavior (
  user_id BIGINT,
  item_id BIGINT,
  category_id BIGINT,
  behavior STRING,
  ts TIMESTAMP(3),
  dt STRING
) PARTITIONED BY (dt) WITH (
 'connector' = 'filesystem',
 'path' = 'oss://<bucket>/path',
 'format' = 'parquet'
)

Parameters

The Parquet format also supports the parameters provided in ParquetOutputFormat. For example, you can configure parquet.compression=GZIP to enable the GZIP compression feature.

Data type mappings

The mappings between Parquet data types and Flink SQL data types are compatible with the mappings between Parquet data types and Apache Hive data types. The mappings between Parquet data types and Flink SQL data types defer from the mappings between Parquet data types and Apache Spark data types in the following aspects:

• The TIMESTAMP data type of Flink SQL is mapped to the INT96 data type of Parquet regardless of the precision.

• The DECIMAL data type of Flink SQL is mapped to a Parquet array that has a fixed number of bytes in length based on the precision.

The following table describes the mappings between Flink SQL data types and Parquet data types.