This topic provides an example on how to use the Avro format and describes the parameters and data type mappings of Avro.
Background information
The Avro format can be used to read and write Avro data based on the Avro structure. The Avro structure is automatically inferred from the table schema. Connectors that support the Avro format include Kafka connector, Upsert Kafka connector, and Object Storage Service (OSS) connector.
Sample code
The following sample code provides an example on how to create a table in the Avro format by using the Apache Kafka connector.
CREATE TABLE user_behavior (
user_id BIGINT,
item_id BIGINT,
category_id BIGINT,
behavior STRING,
ts TIMESTAMP(3)
) WITH (
'connector' = 'kafka',
'topic' = 'user_behavior',
'properties.bootstrap.servers' = 'localhost:9092',
'properties.group.id' = 'testGroup',
'format' = 'avro'
)Parameters
Parameter | Required | Default value | Data type | Description |
format | Yes | No default value | STRING | The format that you declare to use. If you want to use the Avro format, set this parameter to avro. |
avro.codec | No | No default value | STRING | The codec that is used to compress and decompress Avro files. This parameter takes effect only when the file system connector is used. Valid values:
|
Data type mappings
The following table describes the mappings between Flink SQL data types and Avro data types.
Flink SQL data type | Avro data type |
CHAR, VARCHAR, and STRING | STRING |
BOOLEAN | BOOLEAN |
BINARY and VARBINARY | BYTES |
DECIMAL | BYTES Note The value of this data type is a decimal number with precision. |
TINYINT | INT |
SMALLINT | INT |
INT | INT |
BIGINT | LONG |
FLOAT | FLOAT |
DOUBLE | DOUBLE |
DATE | INT Note The value of this data type is a date. |
TIME | INT Note The value of this data type is a point in time in milliseconds. |
TIMESTAMP | LONG Note The value of this data type is a timestamp in milliseconds. |
ARRAY | ARRAY |
MAP Note The elements must be of the STRING, CHAR, or VARCHAR data type. | MAP |
MULTISET Note The elements must be of the STRING, CHAR, or VARCHAR data type. | MAP |
ROW | RECORD |
Flink can read and write data of the Nullable data type. Data of the Nullable data type is mapped to Avro union(something, null). something is an Avro data type that is converted from a Flink data type.
For more information about Avro data types, see Specification.