This topic describes how to use Spark Streaming SQL to perform data analysis and interactive development on the DataHub data source.

CREATE TABLE syntax

CREATE TABLE tbName
USING datahub
OPTIONS(propertyName=propertyValue[,propertyName=propertyValue]*);

Table schema

When you create a DataHub data table, you do not need to explicitly define the fields in the data table. Example:
spark-sql> CREATE TABLE datahub_table_test
         > USING datahub
         > OPTIONS
         > ( access.key.id = '<your access key id>',
         >   access.key.secret = '<you access key secret>',
         >   endpoint = '<your end point>',
         >   project = '<your project name>',
         >   topic = '<your topic>'
         > )

spark-sql> DESC datahub_table_test;
id  string  NULL
name string NULL
Time taken: 0.401 seconds, Fetched 2 row(s)

Parameters

Parameter Description Required
access.key.id The AccessKey ID. Yes
access.key.secret The AccessKey secret. Yes
endpoint The endpoint of the DataHub API. Yes
project The name of the DataHub project. Yes
topic The name of the DataHub topic. Yes
decimal.precision Specify this parameter if a field of the DECIMAL type is contained in the topic. No
decimal.scale Specify this parameter if a field of the DECIMAL type is contained in the topic. No