DataWorks:Overview of the Data Studio (new version)

Step

Phase name

Core operation and purpose

Key path and reference

1

Associate a compute engine

Associate one or more core compute engines, such as MaxCompute, with the workspace to serve as the execution environment for all SQL tasks.

Console > Workspace Configuration

For more information, see Associate a computing resource.

2

Manage the data catalog

Create or explore the table schemas required for each layer of the data warehouse (such as ODS, DWD, and ADS) in the data catalog. This defines the input and output for data processing.

We recommend that you use the data modeling module to build your data warehouse system.

Data Studio > Data Catalog

For more information, see Data Catalog.

3

Create a scheduled workflow

Create a scheduled workflow in the workspace directory to serve as a container for organizing and managing related ETL tasks.

Data Studio > Workspace Directory > Periodic Scheduling

For more information, see Orchestrate a recurring workflow.

4

Develop and debug nodes

Create nodes such as ODPS SQL nodes. Write the core ETL logic (data cleaning, transformation, and aggregation) in the editor and debug the nodes.

• Data Studio > Node Development > Node Editor

• Data Studio > Node Development > Debugging Configuration

For more information, see Node development.

5

Develop with Copilot assistance

Use DataWorks Copilot to generate, correct, rewrite, and convert SQL and Python code.

• Data Studio > Node Development > Copilot

• Data Studio > Copilot > Agent

  For more information, see DataWorks Copilot.

6

Orchestrate and schedule nodes

On the DAG canvas of the workflow, define the upstream and downstream dependencies between nodes by dragging and connecting them. Various flow control nodes are supported to enable complex flow orchestration.

Configure scheduling properties for the workflow or nodes in the production environment, such as the scheduling cycle, time, and dependencies. Supports large-scale scheduling of tens of millions of tasks per day.

• Data Studio > Workflow > Workflow Canvas

• Data Studio > Node Development > Scheduling Configuration

For more information, see General flow control nodes and Node scheduling configuration.

7

Deploy and O&M

• Deploy: Deploy the debugged node or workflow to the production environment through the deployment.

• O&M: In the Operation Center, monitor tasks, configure alerts, backfill data, and perform recurring validation. Use intelligent baselines to ensure tasks are completed on time and use monitoring alerts to promptly handle abnormal tasks.

• Data Studio > Node/Workflow Details > Deploy Node/Workflow

• Operation Center > Auto Triggered Node O&M > Auto Triggered Nodes

  For more information, see Deploy a node or workflow and Basic O&M operations for auto triggered nodes.

Steps

Stage name

Core operation and purpose

Key path and reference documents

1

Create a personal development environment

Create an isolated and customizable cloud container instance to serve as the environment for installing complex Python dependencies and performing professional AI development.

Data Studio > Personal Development Environment

For more information, see Personal development environment.

2

Create an event-triggered workflow

Create a workflow in the workspace directory that is driven by external events. This provides an orchestration container for real-time AI applications.

Data Studio > Workspace Directory > Event-triggered Workflow

For more information, see Event-triggered workflow.

3

Create and set a trigger

Configure a trigger in the Operation Center to define which external event, such as an OSS event or a Kafka message event, will start the workflow.

• Create: Operation Center > Trigger Management

• Use: Data Studio > Event-triggered Workflow > Scheduling Configuration

For more information, see Manage triggers and Design an event-triggered workflow.

4

Create a Notebook node

Create the core development unit for writing AI/Python code. Typically, exploration is first done in a Notebook in the personal folder.

Project Folder > Event-triggered Workflow > Notebook Node

For more information, see Create a node.

5

Create and use a dataset

Register unstructured data (such as images and documents) stored in OSS/NAS as a dataset. Then, mount it to the development environment or task for code access.

• Create: Data Map > Data Catalog > Dataset

• Use: Data Studio > Personal Development Environment > Dataset Configuration

For more information, see Manage Datasets and Use datasets.

6

Develop and debug the Notebook/node

Write algorithm logic, explore data, validate models, and iterate quickly in the interactive environment provided by the personal development environment.

Data Studio > Notebook Editor

For more information, see Basic Notebook development.

7

Install custom dependency packages

In the terminal of the personal development environment or in a Notebook cell, use tools such as pip to install all third-party Python libraries required for the model.

Data Studio > Personal Development Environment > Terminal

For more information, see Appendix: Complete your personal development environment.

8

Create a custom image

Solidify the personal development environment with all dependencies configured into a standardized image. This ensures that the production environment is identical to the development environment.

If you have not installed custom dependency packages, skip this step.

• Data Studio > Personal Development Environment > Manage Environment

• Console > Custom Image

For more information, see Create a DataWorks image from a personal development environment.

9

Configure node scheduling

In the scheduling configuration of the production node, you must specify the custom image created in the previous step as the runtime environment and mount the required datasets.

Data Studio > Notebook Node > Scheduling

For more information, see Node scheduling configuration.

10

Deploy and O&M

• Deploy: Deploy the configured event-triggered workflow to the production environment.

• O&M: Trigger a real event, such as uploading a file, to verify that the end-to-end process is smooth and perform trigger validation.

• Data Studio > Node/Workflow Details > Deploy Node/Workflow

• Operation Center > Manually Triggered Node O&M > Manually Triggered Node

  For more information, see Deploy node and workflow and Run and manage one-time tasks.