DataWorks:Notebook development

Prerequisites

Before you begin, ensure that you have:

• The new Data Studio enabled in your workspace

• A Serverless resource group created — required to run Notebooks in the production environment

• A personal development environment instance created — required to run and debug Notebooks in the development environment. If you haven't created one yet, see Create a personal development environment instance

Steps

1. Create a Notebook node In Data Studio, create a Notebook node under Workspace Directories. Enter a name such as hello_notebook, then submit it.

2. Select a personal development environment In the top navigation bar, click Personal Development Environment and select your personal development environment instance from the drop-down list.

3. Define a Python variable Add a Python cell and enter the following code. This defines the city variable for the SQL query in the next step.

   # Define a variable for the subsequent SQL query
   city = 'Beijing'
   print(f"Defined city variable city = {city}")

4. Write an SQL cell that references the variable Below the Python cell, add an SQL cell. In the lower-right corner of the cell, switch the SQL type to MaxCompute SQL, then enter:

   -- Query using the variable defined in Python
   SELECT '${city}' AS city;

   The ${city} syntax automatically picks up the value from the Python cell above.

5. Run all cells and verify the output Click Run All in the Notebook toolbar. Two things happen:

   • The Python cell prints Defined city variable city = Beijing.

   • The SQL cell displays the query result in a table below it.

You've created and run a Notebook that passes data between Python and SQL.

Development environment vs. production environment

Compute resources and kernels

• Compute resource: The backend compute engine connected to the Notebook — for example, MaxCompute or EMR Serverless Spark. An SQL cell must be bound to a compute resource to run.

• Python kernel: The backend environment that executes Python code — typically your personal development environment instance. Use Magic Commands such as %odps in Python cells to connect to compute resources, submit tasks, or manipulate data.

Folder types

Where you create a Notebook determines its collaboration model, permissions, and deployment path.

Manage cells

Pass parameters between cells

# Python cell
table_name = "dwd_user_info_d"
limit_num = 10

-- SQL cell
SELECT * FROM ${table_name} LIMIT ${limit_num};

SQL to Python

When an SQL cell executes a SELECT query, the result is automatically saved as a DataFrame variable. Access it in any following Python cell.

• Variable name: Starts with df_ by default. Click the name in the lower-left corner of the SQL cell to rename it.

• Variable type: Depends on the SQL engine. Click the DataFrame label in the lower-left corner to switch types if multiple are supported.

SQL engine	Supported types
MaxCompute SQL	Pandas DataFrame, MaxCompute MaxFrame
AnalyticDB for Spark SQL	Pandas DataFrame, PySpark MaxFrame
Other SQL types	Pandas DataFrame

Important

If a cell contains multiple SQL statements, only the result of the last statement is saved to the DataFrame variable.

Example:

Use DataWorks Copilot

DataWorks Copilot is a built-in AI programming assistant that generates and explains code. To open it:

• Click the Copilot icon in the upper-left corner of the selected cell.

• Right-click inside an SQL cell and select Copilot.

• Press Cmd+I (Mac) or Ctrl+I (Windows).

1. Set up parameterized scheduling

Parameterized scheduling lets the Notebook receive different values on each run — useful for processing data from different daily partitions without changing the code.

When to use this: Configure parameterized scheduling when your Notebook needs to operate on different data slices per run, such as reading yesterday's partition each day.

1. In the Python cell that contains your parameter definitions, click ... in the upper-right corner of the cell and select Mark Cell as Parameters. A parameters tag appears, marking it as the parameter entry point.

   

2. In the right pane, click Scheduling, then expand Scheduling Parameters. Assign values to the variables (such as var) defined in your code.

   

At runtime, the scheduling system replaces the variable in your code with the configured value.

2. Configure the runtime environment

1. Select an image: In Scheduling, select an image that includes all Python dependencies the Notebook needs. If you installed packages in your personal development environment instance using pip install, create a DataWorks image from that environment and select it here.

2. Select a resource group: Choose the resource group for executing the task. For a Serverless resource group, configure no more than 16 CU to avoid startup failures. A single task supports a maximum of 64 CU.

3. Associate a RAM role (optional): To apply fine-grained permission control, associate a Resource Access Management (RAM) role with the node. The node runs with that role's identity. For details, see Configure an associated role for a node.

3. Deploy the node

Only nodes in Workspace Directories can be deployed and scheduled periodically.

• Notebook in a Workspace Directory: Click Deploy in the toolbar.

• Notebook in a Personal Directory: Click Save, then click Commit to Workspace Directory, then deploy.

After deployment, monitor and manage the Notebook task on the Auto Triggered Nodes page in Operation Center.