This topic describes the data types and parameters supported by MetaQ Reader and how to configure it by using the code editor.

Alibaba Cloud Message Queue is a professional message-oriented middleware developed by Alibaba Group. Based on technologies for building a highly available distributed cluster, Message Queue provides a complete set of cloud messaging services, including message subscription and publishing, message tracing, scheduled and delayed messages, resource statistics, and alerts. Message Queue provides asynchronous decoupling for distributed application systems and is suitable for Internet applications with large amounts of messages and high throughput. It is one of the core services used by Alibaba Group to support the Double 11 Shopping Festival.

MetaQ Reader reads real-time data from Message Queue through the Java SDK, converts the data to a format that is readable by Data Integration, and then sends the converted data to a writer.

How it works

MetaQ Reader subscribes to the real-time data from Message Queue through the Java SDK of the following version: 

<dependency>
            <groupId>com.taobao.metaq.final</groupId>
            <artifactId>metaq-client</artifactId>
            <version>4.0.1</version>
        </dependency>
        <dependency>
            <groupId>com.aliyun.openservices</groupId>
            <artifactId>ons-sdk</artifactId>
            <version>1.3.1</version>
        </dependency>

Data types

The following table lists the data types supported by MetaQ Reader.

Data Integration data type Message Queue data type
String STRING

Parameters

Parameter Description Required
accessId The AccessKey ID for accessing Message Queue. Yes
accessKey The AccessKey secret for accessing Message Queue. Yes
consumerId The consumer ID. A consumer, also known as a message subscriber, receives and consumes messages.

The consumer ID is the identifier of a type of consumers. The consumers with the same consumer ID generally receive and consume a type of messages, and use the same consumption logic.

 Yes
topicName The topic of the messages to be consumed. A topic is used to classify messages. It is the primary classifier. Yes
subExpression The subexpression used to filter messages of the topic. Yes
onsChannel The channel used for authentication when MetaQ Reader connects to Message Queue. Yes
domainNam The endpoint for connecting to Message Queue. Yes
contentType The type of the message. Valid values: singlestringcolumn, text, and json. Yes
beginOffset The offset where the sync node starts to read data. Valid values: begin and lastRead. Yes
nullCurrentOffset The offset where the sync node starts to read data when the last offset is null. Valid values: begin and current. Yes
fieldDelimiter The column delimiter used to separate message strings, such as commas (,). Control characters are supported. Example: \u0001. Yes
column The columns to be synchronized. Yes

Configure MetaQ Reader by using the code editor

In the following code, a node is configured to read data from Message Queue. 

{
    "job": {
        "content": [
            {
                "reader": {
                    "name": "metaqreader",
                    "parameter": {
                        "accessId": "xxxxxxxxxxx",
                        "accessKey": "xxxxxxxxxxxxxxxx",
                        "consumerId": "Test01",
                        "topicName": "test",
                        "subExpression": "*",
                        "onsChannel": "ALIYUN",
                        "domainName": "xxx.aliyun.com",
                        "contentType": "singlestringcolumn",
                        "beginOffset": "lastRead",
                        "nullCurrentOffset": "begin",
                        "fieldDelimiter": ",",
                        "column": [
                            "col0"
                        ],
                        "fieldDelimiter": ","
                    }
                },
                "writer": {
                    "name": "streamwriter",
                    "parameter": {
                        "print": false
                    }
                }
            }
        ]
    }
}