This topic describes how to use Spark Streaming SQL to perform data analysis and interactive development on the DataHub data source.
CREATE TABLE syntax
CREATE TABLE tbName
USING datahub
OPTIONS(propertyName=propertyValue[,propertyName=propertyValue]*);
Table schema
When you create a DataHub data table, you do not need to explicitly define the fields
in the data table. Example:
spark-sql> CREATE TABLE datahub_table_test
> USING datahub
> OPTIONS
> ( access.key.id = '<your access key id>',
> access.key.secret = '<you access key secret>',
> endpoint = '<your end point>',
> project = '<your project name>',
> topic = '<your topic>'
> )
spark-sql> DESC datahub_table_test;
id string NULL
name string NULL
Time taken: 0.401 seconds, Fetched 2 row(s)
Parameters
Parameter | Description | Required |
---|---|---|
access.key.id | The AccessKey ID. | Yes |
access.key.secret | The AccessKey secret. | Yes |
endpoint | The endpoint of the DataHub API. | Yes |
project | The name of the DataHub project. | Yes |
topic | The name of the DataHub topic. | Yes |
decimal.precision | Specify this parameter if a field of the DECIMAL type is contained in the topic. | No |
decimal.scale | Specify this parameter if a field of the DECIMAL type is contained in the topic. | No |