Forward data to DataHub

Updated at:
Copy as MD

Use the Data Forwarding feature of IoT Platform to forward device data to DataHub. You can then use stream computing engines such as Realtime Compute for Apache Flink and MaxCompute to process the data and generate real-time results, including charts, alerts, and statistics.

How it works

image

The Data Forwarding feature forwards device messages from an IoT Platform topic to a topic within a DataHub project. You can then use DataHub to clean, analyze, and archive the data. For more information, see What is DataHub?.

In the preceding figure:

  • Data source: For information about the supported topic message types, see Data formats.

  • SQL expression: Write an SQL expression to parse and process JSON data reported by devices. For the syntax of SQL expressions, see SQL expression.

    Note

    Binary data is forwarded directly without being parsed.

  • Data destination: The DataHub project and topic that receive device data.

Scenarios

Forward data from IoT Platform to DataHub for unified management:

  • Deliver data to downstream systems for analysis and archiving to build a clear data pipeline and unlock data value.

  • Combine DataHub with Realtime Compute to clean heterogeneous data from multiple sources into structured data in real time for further analysis.

  • Build a real-time data warehouse with layers for raw data, real-time details, and real-time summaries.

For more information, see DataHub Scenarios.

Limitations

  • IoT Platform supports forwarding data to DataHub. For detailed information about the supported regions, see Features by region.

  • You can use both the new and previous versions of the Data Forwarding feature to forward data to DataHub. For an example of how to use the new version, see Forward data to DataHub (new version).

  • The previous version of Data Forwarding does not support cloud gateway products and devices, or products and devices in MQTT-compliant instances. To forward data for these products and devices, you must use the new version of Data Forwarding. For more information, see Data Forwarding (new version).

  • For more limitations on forwarding device messages from an IoT Platform instance, see Limitations.

Prerequisites

  • You have created a DataHub project and a topic to receive data. For more information about how to use DataHub, see Project operations.

    Important

    For an Enterprise instance, the DataHub project must be in the same region as the instance.

  • Create a data forwarding rule and an SQL expression to process data. For more information, see Create a data forwarding rule.

Configure a data destination

  1. Log on to the IoT Platform console.

  2. On the Overview page, find the instance that you want to manage and click the instance ID or instance name.

  3. In the left-side navigation pane, choose Message Forwarding > Data Forwarding.

  4. Click View next to the target rule. The Data Forwarding Rule page appears.

    Important

    If the Data Forwarding (New Version) page is displayed, click Back to Previous Version in the upper-right corner, then click View next to the target rule.

  5. In the Forward Data section, click Add Operation.

  6. In the Add Operation dialog box, set the parameters and click OK.

    image

    Parameter

    Description

    Operation

    Select Send data to DataHub.

    Region

    Select the region where your DataHub project is located.

    Project

    Select a DataHub project.

    You can click Create Project to go to the DataHub console and create a project. For more information, see Project operations.

    Topic

    Select the DataHub topic that receives data.

    After you select a topic, the rules engine automatically obtains the topic's schema and maps the filtered data to it.

    Note
    • When you map data to the schema, use ${}. Otherwise, the system stores a constant in the table.

    • The schema's data types must match the data types that the rules engine processes. Otherwise, the system cannot store the data.

    You can click Create Topic to go to the DataHub console and create a DataHub topic.

    Role

    Select the RAM role that grants IoT Platform permission to write data to DataHub.

    If no RAM role exists, click Create RAM Role to create a RAM role and authorization policy in the RAM console. Create a RAM role.

  7. On the Data Forwarding page, find the rule and click Start in the Actions column.

Example

Build a device monitoring dashboard by using a big data platform

What to do next

  • View the operational logs of Device-to-Cloud Messages and Data Forwarding in the IoT Platform console. Navigate to the IoT Platform Logs tab on the Monitoring > Log Service page for your instance. For more information, see IoT Platform logs.

  • Log on to the DataHub console to view the data written to DataHub. For more information, see Quick Start (Synchronization example).

  • Synchronize data from the topic to other Alibaba Cloud products in real time or near real time to connect data across services. Supported products include MaxCompute (formerly ODPS), AnalyticDB for MySQL (ADS), ApsaraDB RDS, Tablestore, Object Storage Service (OSS), Elasticsearch, and Function Compute. For more information, see Data synchronization.

References