The DataStudio service of DataWorks allows you to create various types of nodes, such as Data Integration nodes used for data synchronization, compute engine nodes used for data cleansing, and general nodes used together with compute engine nodes to process complex logic. Compute engine nodes include ODPS SQL nodes, Hologres SQL nodes, and E-MapReduce (EMR) Hive nodes. General nodes include zero load nodes that can be used to manage multiple other nodes and do-while nodes that can run node code in loops. You can combine these types of nodes in your business to meet your different data processing requirements.
Node types supported by DataStudio
The following table describes the node types that are supported by DataStudio.
Node type | Description |
DataWorks Data Integration supports data synchronization in complex network environments. You can create a batch synchronization node to periodically synchronize offline data or create a real-time synchronization node to synchronize incremental data from a single table or database in real time. You can create this type of node on the DataStudio page. | |
DataWorks encapsulates the capabilities of compute engines. You can create and configure nodes of a compute engine type to develop data. You can enable the system to periodically schedule different types of nodes in DataWorks without the need to use complex command lines. You can create nodes of the following compute engine types: MaxCompute, Hologres, EMR, AnalyticDB for PostgreSQL, AnalyticDB for MySQL, MySQL, ClickHouse, Cloudera's Distribution including Apache Hadoop (CDH), and Platform for AI (PAI). | |
You can use general nodes and nodes of a specific compute engine type in DataWorks to process complex logic. General nodes include do-while nodes that are used to run node code in loops, for-each nodes that are used to traverse the outputs of assignment nodes in loops and judge the outputs, and branch nodes. |
Data synchronization nodes
Data synchronization nodes are used to synchronize data. The following table describes different types of data synchronization nodes.
Type | Description |
This type of node is used to periodically synchronize offline data and to synchronize data between heterogeneous data sources in complex scenarios. For information about the data source types that support batch synchronization, see Supported data source types and read and write operations. | |
This type of node is used to synchronize incremental data in real time. A real-time synchronization node uses three basic plug-ins to read, convert, and write data. These plug-ins interact with each other based on an intermediate data format that is defined by the plug-ins. For information about the data source types that support real-time synchronization, see Supported data source types and read and write operations. |
In addition to the nodes that can be created on the DataStudio page, DataWorks also allows you to create different types of synchronization tasks in Data Integration. For example, you can create a synchronization task in Data Integration that synchronizes full data at a time and then incremental data in real time or a synchronization task that synchronizes all data in a database in offline mode. For more information, see Overview of the full and incremental synchronization feature.
Compute engine nodes
In a specific workflow, you can create nodes of a specific compute engine type, use the nodes to develop data, issue the engine code to a corresponding data cleansing engine, and then run the code. The following table describes different types of compute engine nodes.
You must first activate corresponding services for your DataWorks workspace or associate required compute engines with your DataWorks workspace. For information about how to associate a compute engine with a DataWorks workspace, see Associate compute engines with a workspace and manage the compute engines.
Compute engine integrated with DataWorks | Encapsulated engine capability |
MaxCompute | |
EMR | |
CDH | |
AnalyticDB for PostgreSQL | |
AnalyticDB for MySQL | |
Hologres | |
ClickHouse | |
StarRocks | |
PAI | |
Database and others | Note To create this type of node, you do not need to associate a compute engine with your DataWorks workspace. However, you must add related data sources. |
General nodes
In a specific workflow, you can create a general node and use the node together with compute engine nodes to process complex logic. The following table describes different types of general nodes.
Scenario | Node type | Description |
Business management | A zero load node is a control node that supports dry-run scheduling and does not generate data. In most cases, a zero load node serves as the root node of a workflow and allows you to easily manage nodes and workflows. | |
Event-based trigger | You can use this type of node if you want to trigger nodes in DataWorks to run after nodes in other scheduling systems finish running. | |
You can use this type of node if you want to trigger a descendant node to run after Object Storage Service (OSS) objects are generated. | ||
You can use this type of node if you want to trigger a descendant node to run after File Transfer Protocol (FTP) files are generated. | ||
You can use this type of node to check the availability of MaxCompute partitioned tables, FTP files, and OSS objects based on check policies. If the condition that is specified in the check policy for a Check node is met, the task on the Check node is successfully run. If the running of a task depends on an object, you can use a Check node to check the availability of the object and configure the task as the descendant task of the Check node. If the condition that is specified in the check policy for the Check node is met, the task on the Check node is successfully run and then its descendant task is triggered to run. | ||
Parameter value assignment and parameter passing | You can use this type of node if you want to use the outputs parameter of an assignment node to pass the data from the output of the last row of the code for the assignment node to its descendant nodes. | |
You can use this type of node to aggregate parameters of its ancestor nodes and distribute parameters to its descendant nodes. | ||
Control | You can use this type of node to traverse the result set of an assignment node. | |
You can use this type of node to execute the logic of specific nodes in loops. You can also use this type of node together with an assignment node to generate the data that is passed to a descendant node of the assignment node in loops. | ||
You can use this type of node to route results based on logical conditions. You can also use this type of node together with an assignment node. | ||
You can use this type of node to merge the status of its ancestor nodes and prevent dry run of its descendant nodes. | ||
Others | Shell nodes support the standard Shell syntax. The interactive syntax is not supported. | |
You can use this type of node to periodically schedule and process event functions and complete integration and joint scheduling with other types of nodes. |