DataHub is a data distribution platform designed to process streaming data. You can publish and subscribe to applications for streaming data in DataHub and distribute the data to other platforms. DataHub allows you to analyze streaming data and build applications based on the streaming data. This topic describes how to use the serverless Spark engine of Data Lake Analytics (DLA) to access DataHub.
Prerequisites
- A project is created in DataHub. In the following procedure, DataHub is deployed in
the China (Shenzhen) region. The name of the project is spark_test, and the name of the topic is topic01.
Note The built-in SparkOnDataHub Connectors supports topics of the TUPLE type.
- Object Storage Service (OSS) is activated. For more information, see Activate OSS.
Background information
To enable the serverless Spark engine of DLA to consume data in DataHub, you must
send the locally-prepared test data to DataHub to test the connectivity between the
serverless Spark engine of DLA and DataHub. In this topic, assume that you run the
following commands to download the test code to a local directory and run the JAR
file to send data to topic01 in the spark_test project.
// Download the test code to a local directory.
wget https://spark-home.oss-cn-shanghai.aliyuncs.com/common_test/common-test-0.0.1-SNAPSHOT-shaded.jar
// Run the JAR file to send data to topic01 in the spark_test project.
java -cp /opt/jars/common-test-0.0.1-SNAPSHOT-shaded.jar com.aliyun.datahub.DatahubWrite_java spark_test topic01 xxx1 xxx2 https://dh-cn-shenzhen.aliyuncs.com
The following table describes the parameters that are used in the preceding command.Parameter | Description |
---|---|
spark_test | The name of the project in DataHub. |
topic01 | The name of the topic in the project of DataHub. |
xxx1 | The AccessKey ID that is used to access Alibaba Cloud API operations. |
xxx2 | The AccessKey secret that is used to access Alibaba Cloud API operations. |
https://dh-cn-shenzhen.aliyuncs.com | The public endpoint that is used to access DataHub in the China (Shenzhen) region. |