DataWorks:Usage notes for development of EMR nodes in DataWorks

Fee

Description

Fees for the DataWorks edition that you use

You must activate DataWorks before you can develop nodes in DataWorks. If you activate an advanced edition such as DataWorks Enterprise Edition, you are charged the related fees when you purchase the edition.

Fees for the scheduling resources that you use to schedule nodes

After nodes are developed, scheduling resources are required to schedule the nodes. You must purchase subscription exclusive resource groups for scheduling based on your business requirements and pay for the resource groups.

Fees for the resources that you use to synchronize data

A data synchronization node consumes scheduling resources and synchronization resources. You must purchase subscription exclusive resource groups for Data Integration based on your business requirements and pay for the resource groups.

Fee

Description

Database fees

When you run data synchronization nodes to read data from and write data to databases, database fees may be generated.

Computing and storage fees

When you run nodes of a specific compute engine type, computing and storage fees of this type of compute engine may be generated.

Network service fees

When you establish network connections between DataWorks and other related services, network service fees may be generated. For example, if you use services, such as Express Connect, Elastic IP Address (EIP), and Internet Shared Bandwidth, to establish network connections between DataWorks and other related services, you may be charged network service fees.

Item

Description

References

Select a DataWorks edition

DataWorks Basic Edition allows you to perform the following basic operations during the development of EMR data: migrate data to the cloud, develop data, schedule EMR nodes, and govern data. If you want to use more advanced data governance and data security solutions, you can purchase DataWorks of an advanced edition.

Differences among DataWorks editions

Select a resource group

You can use only exclusive resource groups for scheduling to run EMR nodes.

Create and use an exclusive resource group for scheduling

Item

Description

References

Register an EMR cluster

Before you enable DataWorks to periodically schedule EMR nodes, you must add an EMR cluster to DataWorks as a data source.

Register an EMR cluster to DataWorks

Prepare a collaborative development environment

To ensure that RAM users can collaborate with each other to develop data in a workspace, you must perform the following operations:

• Add the RAM users to the current workspace as members and assign the Development role to the RAM users in the workspace.

• Add workspace members to the desired EMR cluster.

• Manage permissions on workspace-level services

• Manage users

Module

Description

References

Data Modeling

Data Modeling is the first step for end-to-end data governance. Data Modeling uses the modeling methodology of the Alibaba data mid-end, interprets the business data of an enterprise from a business perspective by using the data warehouse planning, data standard, dimensional modeling, and data metric modules, and allows personnel inside the enterprise to quickly understand and share the idea of measuring and interpreting business data in compliance with data warehousing specifications.

Data Modeling overview

DataStudio

DataStudio is an end-to-end big data development platform that you can use to develop data processing tasks of an EMR compute engine online. DataStudio provides powerful node scheduling capabilities and can support centralized orchestration and scheduling for tens of millions of instances that are generated for nodes. DataStudio also provides a control process for node deployment, which can ensure the stability of node output.

• Development of EMR nodes:

  • Create an EMR Presto node

  • Create an EMR Hive node

  • Create an EMR MR node

  • Create an EMR Spark SQL node

  • Create an EMR Spark node

  • Create an EMR Shell node

  • Create an EMR Spark Streaming node

  • Create an EMR table

  • Create and use an EMR JAR resource

  • Create an EMR function

• Development of other types of nodes:

  • Create and use a zero load node

  • Create an HTTP Trigger node

  • OSS object inspection node

  • Create an FTP Check node

  • Configure an assignment node

  • Create a parameter node

  • Logic of do-while nodes

  • Logic of for-each nodes

  • Configure a branch node

• Scheduling property configuration of nodes: Overview

• Node debugging: Debugging procedure

• Node deployment: Deploy nodes

• Node management: Perform operations on multiple DataWorks objects at a time

• Process control: Process management

Operation Center

Operation Center is an end-to-end big data O&M and monitoring platform. Operation Center allows you to view the status of nodes and perform O&M operations on nodes on which exceptions occur. For example, you can perform intelligent diagnostics and rerun nodes in Operation Center. Operation Center provides the intelligent baseline feature that you can use to resolve issues such as uncontrollable output time of important nodes and difficulties in monitoring of massive nodes. This feature helps you ensure the timeliness of node output.

Perform basic O&M operations on auto triggered nodes

Data Quality

Data Quality ensures data availability for the end-to-end data R&D process and provides reliable data for your business in an efficient manner. Data Quality can help you identify data quality issues at the earliest opportunity and prevent data quality issues from escalating by virtue of effective monitoring rule-based quality checks and the combination of monitoring rules and node scheduling processes.

Data Quality overview

Module

Description

References

Data Map

Data Map is an enterprise-grade data management platform that provides management, sorting, quick search, and in-depth understanding capabilities for data objects based on the underlying unified metadata services.

Data Map overview

Security Center

Data Security Guard

Approval Center

Security Center is an end-to-end data security governance platform that covers classification of data assets, sensitive data identification, management on data-related authorization, masking of sensitive data, audit of access to sensitive data, and risk identification and response. Security Center helps you determine data security governance issues.

• Security Center overview

• Data Security Guard overview

• Approval Center overview

Data Governance Center

Data Governance Center automatically identifies items to be governed for multiple governance fields based on rules that come from experience in data-related fields, and provides governance and optimization solutions covering pre-event issue prevention and post-event issue resolution. Data Governance Center can help you actively and systematically complete data governance.

Data Governance Center overview

Item

Description

References

OpenAPI

The OpenAPI module allows you to call DataWorks API operations so that you can integrate your applications with DataWorks. This can help facilitate big data processing, decrease manual operations and O&M operations, minimize data risks, and reduce costs for enterprises.

OpenAPI

OpenEvent

The OpenEvent module allows you to subscribe to DataWorks change events related to your applications so that you can detect and respond to the changes at the earliest opportunity.

OpenEvent overview

Extensions

You can use the OpenEvent module to subscribe to event messages that are generated in your DataWorks workspace. You can use the Extensions module to register your local program as an extension to manage extension point events and processes.

Extensions overview