After you enable message subscription and configure the Kafka topic, you can develop consumer programs by using Message Queue for Apache Kafka SDK. This way, you can consume DataWorks event messages. This topic describes how to develop consumer programs by using Message Queue for Apache Kafka SDK.

Prerequisites

Limits

  • You can subscribe to OpenEvent messages only by using an SSL endpoint with PLAIN authentication of Message Queue for Apache Kafka.
  • The OpenEvent module supports only Message Queue for Apache Kafka V2.2.0. We recommend that the versions of the Message Queue for Apache Kafka client and server be consistent.

References

Message Queue for Apache Kafka used by DataWorks allows you to develop consumer programs by using SDKs for different programming languages. The supported programming languages include Java, Python, C++, Go, PHP, Ruby, Node.js, and C#. The following table provides links to the topics that describe how to develop consumer programs by using SDKs for different programming languages.

Notice
  • Before you develop a consumer program, you must obtain the Kafka configurations such as the Kafka topic, consumer group, user, and SSL endpoints from the Open Platform page in the DataWorks console.
  • If you need to specify the version of Message Queue for Apache Kafka when you configure the dependency, set the version to V2.2.0.
  • You do not need to develop a producer program. For information about the preparations and consumer group configurations required by SDKs for different programming languages, see relevant topics.
Programming language Network type Protocol Port References Demo
Java Internet and VPC SASL_SSL 9093 Send and consume messages by using an SSL endpoint with PLAIN authentication SASL_SSL/PLAIN
python Internet and VPC SASL_SSL 9093 Send and consume messages by using an SSL endpoint with PLAIN authentication SASL_SSL/PLAIN
C++ Internet and VPC SASL_SSL 9093 Use the PLAIN mechanism to send and subscribe to messages over an SSL endpoint SASL_SSL/PLAIN
Go Internet and VPC SASL_SSL 9093 Use the PLAIN mechanism to send and subscribe to messages over an SSL endpoint SASL_PLAIN
PHP Internet and VPC SASL_SSL 9093 Use the PLAIN mechanism to send and subscribe to messages over an SSL endpoint SASL_SSL/PLAIN
Ruby Internet and VPC SASL_SSL 9093 Use the PLAIN mechanism to send and subscribe to messages over an SSL endpoint SASL_SSL/PLAIN
Node.js Internet and VPC SASL_SSL 9093 Use the PLAIN mechanism to send and subscribe to messages over an SSL endpoint SASL_PLAIN
C# Internet and VPC SASL_SSL 9093 Send and subscribe to messages by using an SSL endpoint with PLAIN authentication N/A

Message parsing

The message formats vary based on the types of DataWorks events. For more information about the message formats for different types of DataWorks events, see Appendix: Message formats.