This topic describes how to create a Log Service source table. It also describes the data definition language (DDL) statements, parameters in the WITH clause, field type mappings, and attribute fields used when you create a Log Service source table.
Introduction to Log Service
Log Service is an end-to-end data logging service. Log Service helps you collect, consume, ship, query, and analyze log data in a quick manner. It improves the operations and maintenance (O&M) efficiency, and allows you to process large amounts of data.
create table sls_source( a int, b int, c varchar ) with ( 'connector' ='sls', 'endPoint' ='<endPoint>', 'accessId' ='<yourAccessId>', 'accessKey' ='<yourAccessKey>', 'startTime' = '2017-07-05 00:00:00', 'project' ='<yourProjectName>', 'logStore' ='<yourLogStoreName>', 'consumerGroup' ='<yourConsumerGroupName>' );
- Log Service does not support the MAP data type.
- Log Service sets the fields that do not exist to null.
- We recommend that you define the fields in the source table in the same order as the fields in the table of the upstream storage system. Unordered fields are also supported.
Parameters in the WITH clause
|connector||The type of the source table.||Yes||Set the value to
|endPoint||The endpoint of Log Service.||Yes||Endpoints|
|accessId||The AccessKey ID that is used to access the database.||Yes||None.|
|accessKey||The AccessKey secret that is used to access the database.||Yes||None.|
|project||The name of the Log Service project that you want to access.||Yes||None.|
|logStore||The name of the Logstore in the Log Service project.||Yes||None.|
|startTime||The time at which logs start to be consumed.||Yes||None.|
|consumerGroup||The name of a consumer group.||No||You can specify this parameter as required. The format of the name is not fixed.|
|batchGetSize||The number of log items that are read from a log group at a time.||No||Default value: 10.
Field type mapping
The following table describes the data type mappings between Log Service and Flink fields. We recommend that you declare the mappings in a DDL statement.
|Data type of Log Service||Data type of Flink|
||The message source.|
||The message topic.|
||The time at which a log was generated.|