DataWorks:Data Studio overview

Step

Phase

Core operations and objectives

Key path and references

1

Associate a compute engine

Associate one or more core compute engines (such as MaxCompute) with the workspace as the execution environment for all SQL tasks.

Console > Workspace Settings

For more information, see Associate compute engines.

2

Data Catalog management

Create or explore the table structures required for each data warehouse layer (ODS, DWD, ADS, etc.) in the Data Catalog to define the inputs and outputs for data processing.

We recommend that you use the Data Modeling module to build the data warehouse system.

Data Studio > Data Catalog

For more information, see Data Catalog.

3

Create a scheduled workflow

Create a scheduled workflow in the project directory as a container for organizing and managing related ETL tasks.

Data Studio > Project Directory > Scheduled Workflow

For more information, see Scheduled workflows.

4

Node development and debugging

Create ODPS SQL nodes or other nodes, write core ETL logic (data cleansing, transformation, and aggregation) in the editor, and debug the nodes.

• Data Studio > Node Development > Node Editor

• Data Studio > Node Development > Run Configuration

For more information, see Node development.

5

Copilot-assisted development

Use DataWorks Copilot capabilities to generate, correct, rewrite, and convert SQL and Python code.

• Data Studio > Node Development > Copilot

• Data Studio > Copilot > Agent

  For more information, see Copilot.

6

Node orchestration and scheduling

In the DAG canvas of the workflow, define upstream and downstream dependencies between nodes by dragging and connecting them. Various flow control nodes are supported for complex orchestration.

Configure the schedule settings for the production environment at the workflow or node level, including the schedule, time, and dependencies. The platform supports ultra-large-scale scheduling of tens of millions of instances per day.

• Data Studio > Workflow > Workflow Canvas

• Data Studio > Node Development > Schedule Settings

For more information, see General flow control nodes and Schedule settings.

7

Workflow/node deployment and O&M

• Deployment: Deploy the debugged nodes or workflows to the production environment through the deployment process.

• O&M: Monitor, configure alerts for, backfill data for, and perform periodic validation on production tasks in Operation Center. Use intelligent baselines to ensure that tasks are completed on time, and use monitoring alerts to promptly handle task exceptions.

• Data Studio > Node/Workflow Details > Node/Workflow Deployment

• Operation and Maintenance Center > Auto Triggered Task O&M > Auto Triggered Task

  For more information, see Deploy nodes and workflows and Operation Center.

Step

Phase

Core operations and objectives

Key path and references

1

Create a personal development environment

Create an isolated, customizable cloud container instance as the environment for installing complex Python dependencies and performing professional AI development.

Data Studio > Personal Development Environment

For more information, see Personal development environment.

2

Create a trigger-based workflow

Create an event-driven workflow in the project directory as an orchestration container for real-time AI applications.

Data Studio > Project Directory > Trigger-based Workflow

For more information, see Trigger-based workflows.

3

Create and configure a trigger

Configure a trigger in Operation Center to define which external events (such as OSS events or Kafka message events) start the workflow.

• Create: Operation Center > Trigger Management

• Use: Data Studio > Trigger-based Workflow > Schedule Settings

For more information, see Trigger management and Trigger-based workflow schedule settings.

4

Create a Notebook node

Create the core development unit for writing AI/Python code. We recommend that you first explore in a Notebook in the personal directory.

Project Directory > Trigger-based Workflow > Notebook Node

For more information, see Notebook node development.

5

Create and use datasets

Register unstructured data (images, documents, etc.) stored on OSS or NAS as datasets, and mount them to the development environment or tasks for code access.

• Create: Data Map > Data Catalog > Dataset

• Use: Data Studio > Personal Development Environment > Dataset Configuration

For more information, see Create a dataset and Use datasets.

6

Develop and debug Notebooks/nodes

Write algorithm logic in the interactive environment provided by the personal development environment. Perform data exploration, model validation, and rapid iteration.

Data Studio > Notebook Editor

For more information, see Notebook.

7

Install custom dependency packages

In the terminal of the personal development environment or in a Notebook cell, use tools such as pip to install all the third-party Python libraries required by the model.

Data Studio > Personal Development Environment > Terminal

For more information, see Install custom dependency packages.

8

Build a custom image

Freeze the personal development environment with all dependencies configured into a standardized image to ensure that the production environment is fully consistent with the development environment.

If you have not installed custom dependency packages, skip this step.

• Data Studio > Personal Development Environment > Manage Environment

• Console > Custom Image

For more information, see Custom images.

9

Node schedule settings

In the schedule settings of the production node, you must specify the custom image created in the previous step as the runtime environment and mount the required datasets.

Data Studio > Notebook Node > Schedule Settings

For more information, see Schedule settings.

10

Node/workflow deployment and O&M

• Deployment: Deploy the configured trigger-based workflow to the production environment.

• O&M: Trigger a real event (such as uploading a file) to verify that the end-to-end process works correctly, and perform trigger validation.

• Data Studio > Node/Workflow Details > Node/Workflow Deployment

• Operation and Maintenance Center > Manually Triggered Task O&M > Manually Triggered Task

  For more information, see Deploy nodes and workflows and Manually triggered tasks.