DataWorks:DataWorks on CDP/CDH

Charge

Description

DataWorks edition fee

Before you can develop tasks, you must activate DataWorks. If you activate DataWorks Standard Edition, Professional Edition, or Enterprise Edition, you must pay the fee for the selected edition.

Scheduling resource fee

After you develop a task, it requires scheduling resources to run on a schedule. You can use a serverless resource group (recommended) or an old-version exclusive resource group and pay the corresponding fees.

Synchronization resource fee

When you run data synchronization tasks, you need data synchronization resources in addition to scheduling resources. You can use a serverless resource group (recommended) or an old-version exclusive resource group for data integration and pay the corresponding fees.

Fee

Description

Database fees

When you synchronize data, fees may be incurred for reading data from and writing data to upstream and downstream databases.

Compute and storage fees

When you run compute engine tasks, compute and storage fees of the compute engine may be incurred.

Network service fees

When you connect DataWorks to other services over the network, network service fees may be incurred. For example, if you use Express Connect, shared bandwidth, or EIP to establish network connectivity, the corresponding service fees are incurred.

Category

Description

References

Edition selection

DataWorks Basic Edition can meet the basic requirements of CDP or CDH for data migration to the cloud, data development and scheduling, and simple Data Governance. To access more professional Data Governance and Data Security solutions, you can choose the Standard Edition, Professional Edition, or Enterprise Edition.

Feature Details by Version

Resource group selection

CDP or CDH clusters support only serverless resource groups (recommended) or old-version exclusive resource groups for scheduling.

Use serverless resource groups

Category

Description

References

Data synchronization environment preparation

Before you run data synchronization tasks based on the Hive component of a cluster, you must create a corresponding DataWorks data source for the component.

Supported data sources and synchronization solutions

Data Studio and Data Analysis environment preparation

Before you use DataWorks for periodic scheduling of compute engine tasks, you must add the cluster to DataWorks. After the cluster is added, you can use it for data development, Data Analysis, and periodic task scheduling.

Data Studio (legacy): Associate a CDH computing resource

Collaborative development environment preparation

To enable RAM users to collaborate within a workspace, perform the following operations:

• Add the RAM users who need to collaborate to the workspace as members and assign them the developer role.

• Add the workspace members to the CDP or CDH cluster environment.

Add workspace members

Module

Description

References

Data Modeling

Data modeling is the first step in end-to-end data governance. Drawing on Alibaba's data platform modeling methodology, it covers four aspects — data warehouse planning, data standards, dimensional modeling, and data metrics — to interpret business data from a business perspective, enabling consistent data understanding and communication across the organization. 

Data Modeling

Data Studio

DataWorks encapsulates the capabilities of CDP or CDH compute engines, allowing you to run CDP or CDH data synchronization and data development tasks.

• Data synchronization: Data Studio supports only some batch synchronization and real-time synchronization scenarios. For more data synchronization options, go to Data Integration.

• Data development: You can develop various tasks and configure periodic scheduling in DataWorks without using complex command lines.

• Create a CDH Hive node

• Create a CDH Spark node

• Create a CDH MR node

• Create a CDH Presto node

• Create CDH Impala node

You can combine DataWorks general-purpose nodes and compute engine nodes for complex logic processing.

The following are the main nodes:

• Virtual nodes to control business processes.

• HTTP trigger, OSS Object Check, and FTP Check nodes to trigger scheduling from external systems.

• Assignment and parameter nodes to pass parameters between upstream and downstream nodes.

• Do-while nodes to loop code, for-each nodes to iterate and evaluate the assignment results of upstream nodes, and branch nodes.

• Other nodes, such as general-purpose Shell nodes and MySQL database nodes.

• Virtual nodes

• HTTP trigger nodes

• OSS Object Check nodes

• FTP Check nodes

• Assignment nodes

• Parameter nodes

• do-while node logic

• for-each node logic

• Branch nodes

After you finish developing a node task, you can perform the following operations as needed:

• Scheduling configuration

  If the node task needs to run periodically, you can define its scheduling properties, such as scheduling dependencies and parameters.

• Node debugging

  To ensure that production tasks run efficiently and to prevent waste of computing resources, debug a task before you deploy it.

• Node deployment

  A task must be deployed to the production environment to be scheduled periodically. After deployment, you can go to Operation and Maintenance Center > Auto Triggered Node to view and manage the scheduled task.

• Node management

  You can deploy or undeploy node tasks, and batch-modify scheduling properties.

• Development process control

  DataWorks provides development process control for task development and deployment, such as code review, mandatory smoke testing, and custom check logic, to ensure the correctness and security of task-related operations.

• Overview of task scheduling configuration

• Task debugging

• Deploy tasks

• Batch operations

• Development process control

Operation Center

Operation Center is a one-stop big data O&M and monitoring platform that lets you view task running status in real time and provides intelligent diagnostics, reruns, and other O&M operations for abnormal tasks. Its smart baseline feature helps you address issues such as unpredictable completion times for critical tasks and the difficulty of monitoring massive volumes of tasks, ensuring timely task delivery.

Basic O&M for auto-triggered nodes

Data Quality

Data Quality ensures data availability throughout the entire data development lifecycle. Through efficient validation of data quality rules tightly integrated with the task scheduling workflow, it helps you identify quality issues at the earliest stage, effectively prevents them from spreading, and delivers efficient, reliable, and trustworthy data for your business. 

Data quality

Module

Description

References

Data Map

DataWorks Data Map provides an enterprise-level data management platform. Based on unified metadata, it offers data object management and inventory capabilities, lineage viewing, and the ability to quickly find and deeply understand data objects.

Overview

Security Center

Data Security Guard

Approval center

Security Center, Data Security Guard, and the approval center provide an integrated data security governance interface that covers data asset classification, sensitive data identification, data access management, sensitive data masking, sensitive data access auditing, and risk identification and response, helping you implement data security governance.

• Security Center overview

• Overview

• Overview

Data Asset Governance

Data Asset Governance provides proactive and systematic data governance across multiple governance domains by consolidating governance rules, automatically identifying asset optimization issues, and applying pre-event and post-event governance optimization policies.

Overview

Module

Description

Related documentation

DataAnalysis

Lets you run online SQL queries, gain business insights, and edit and share data. You can also save query results as chart cards to quickly build visual data reports.

DataAnalysis

DataService Studio

DataService Studio provides comprehensive data service and sharing capabilities, helping enterprises centrally manage API services for both internal and external use.

DataService Studio

Category

Description

Related documentation

OpenAPI

The OpenAPI feature of Open Platform provides APIs that enable interaction between on-premises services and DataWorks services. This improves the efficiency of big data processing, reduces manual operations and O&M, and lowers data risks and costs.

OpenAPI

OpenEvent

The OpenEvent feature of Open Platform provides a message subscription service. By subscribing to DataWorks event statuses, your application systems can receive real-time status changes, helping you respond to events in a timely manner and meet personalized decision-making requirements.

Overview of OpenEvent

Extensions

DataWorks provides a message push subscription feature based on OpenEvent. You can register a service program as a DataWorks extension to intercept and respond to subscribed event messages. This allows you to implement message notifications and process control for specific events through the extension.

Overview of extensions