DataWorks:Notebook development

Before you begin, ensure that you have met the following requirements:

• The new Data Studio is enabled in your workspace.

• A serverless resource group has been created. Running Notebooks in the production environment requires a serverless resource group.

• A personal development environment instance has been created. If your Notebook contains Python cells, a personal development environment instance is required to run and debug them in the development environment.

  If you have not created one yet, see Create a personal development environment instance.

Steps:

1. Create a Notebook node

   1. Go to Data Studio. Under Data Development in Workspace Directories, create a Notebook node.

   2. Name the Notebook (for example, hello_notebook) and submit it.

2. Select a personal development environment

   In the top navigation bar, click Personal Development Environment and select your personal development environment instance from the drop-down list.

3. Write a Python cell to define a variable

   Add a Python cell and enter the following code. This step defines a city variable for the SQL query in the next step.

   # Define a variable for subsequent SQL queries
   city = 'Beijing'
   print(f"Defined city variable city = {city}")

4. Write an SQL cell to query data

   1. Below the first cell, add an SQL cell.

   2. In the lower-right corner of the cell, switch the SQL type to MaxCompute SQL.

   3. Enter the following SQL code. The code references the city variable defined in the previous Python cell by using the ${city} syntax.

      -- Query using the variable defined in Python  
      SELECT '${city}' AS city;

5. Run all cells and verify the output

   1. Click Run All in the Notebook toolbar.

   2. Observe the output of each cell:

      • The Python cell prints Defined city variable city = Beijing.

      • The SQL cell displays the query result in a table below it.

You have created and run a Notebook that passes data between Python and SQL.

Notebook modes

DataWorks Notebook supports two modes:

• SQL and Markdown mode: Contains only SQL and Markdown cells. This mode is suitable for pure data querying and documentation. In this mode, only a serverless resource group is required to run in both the development and production environments — no personal development environment instance is needed. This is the default mode, and the Notebook displays SQL Kernel in the upper-right corner.

• Full mode (Python + SQL + Markdown): Supports mixed use of Python, SQL, and Markdown cells. This mode is suitable for complex scenarios that require data processing, analysis, and visualization. In the development environment, a personal development environment instance is required to execute Python code. After you select a personal development environment, the upper-right corner of the Notebook displays (the Python version is for reference only).

Development environment vs. production environment

Compute resources and kernels

In DataWorks Notebook, code execution is determined by two core concepts: compute resources and kernels.

• Compute resource: A compute resource is the backend compute engine that ultimately executes and processes data tasks. It defines the runtime environment, computing power, and execution logic for tasks.

  • Definition: An independent, schedulable compute service instance, such as a MaxCompute project or an EMR Serverless Spark cluster.

  • Purpose: Provides the actual computing power for SQL queries, Spark jobs, and other tasks.

  • Selection: You must explicitly associate a task with a specific compute resource before execution.

• Kernel: A kernel is the component in the Notebook environment that parses and executes the code you write in a cell. It determines the programming language accepted by the cell.

  • Python kernel:

    • Functionality: Executes code written in Python. Supports complex logic such as data processing, algorithm implementation, and task scheduling.

    • Interaction mode: In the Python kernel, you can use Magic Command (such as %sql) or SDK libraries to submit packaged compute tasks (such as SQL statements) to a specified compute resource for execution and retrieve the results for further analysis.

  • SQL kernel:

    • Functionality: Directly interprets and submits SQL queries.

    • Interaction mode: The SQL kernel forwards SQL statements directly to the backend compute resource (such as EMR Spark SQL or MaxCompute SQL Session) that you specify for the cell.

  • Markdown kernel:

    • Functionality: Renders rich-text documents in Markdown format. Does not execute any computation logic.

Summary:

• A kernel is the frontend language interpreter that determines what you write (Python or SQL).

• A compute resource is the backend execution engine that determines where code runs (on MaxCompute or Spark).

Directory types and use cases

The location where you create a Notebook determines its collaboration model, permissions, and deployment process.

Cell management

• Add a cell: Hover over the top or bottom edge of an existing cell and click the + SQL button that appears. You can also use the buttons in the top toolbar.

• Switch the cell type: Click the type label in the lower-right corner of a cell (for example, Python) and select a new type from the menu, such as SQL or Markdown. The code in the cell is preserved when you switch types. You must manually update it to match the new type.

• Move a cell: Hover over the blue vertical bar on the left side of a cell, then hold and drag to reorder.

• Run a cell:

  • Run a single cell: Click the Run button on the left side of the cell.

  • Run all cells: Click the Run All button in the Notebook toolbar.

Variable passing

View lineage (beta)

When Notebook code becomes complex and involves reading from and writing to multiple tables or files, understanding data flow can be difficult. The lineage analysis feature statically parses Python code in Notebooks (with support for the MaxFrame framework) to help you quickly trace data origins and destinations.

Copilot-assisted programming

DataWorks Copilot is a built-in intelligent programming assistant that helps you generate and explain code.

How to invoke Copilot:

• Click the Copilot icon in the upper-left corner of the selected cell.

• Right-click inside an SQL cell and select Copilot.

• Use the keyboard shortcut Cmd+I (Mac) or Ctrl+I (Windows).

Step 1: Configure scheduling parameters (parameterized scheduling)

If you want the parameters in a Notebook to change dynamically for each scheduled run (for example, processing data from different partitions on a daily basis), configure parameterized scheduling.

1. Mark a parameter cell: In the Python cell that contains key parameter definitions, click the ... menu in the upper-right corner and select Mark Cell as Parameters. A parameters tag is added to the cell, indicating that it is the parameter entry point for the scheduled task.

   

2. Configure scheduling parameters:

   1. In the right-side panel of the Notebook, click Schedule Settings.

   2. In the Scheduling Parameters section, assign values to the variables you defined in the code (such as var).

      

When the task is automatically executed by the scheduling system, the actual value of the var parameter in the code is dynamically replaced with the value configured in the scheduling parameters.

Step 2: Configure the runtime environment and resources

1. Configure an image: In the schedule settings, select an image that contains all dependencies required by the Notebook. This is critical to ensuring successful execution in the production environment.

   Important

   If you installed Python packages by using pip install or similar methods in your personal development environment instance, you need to build a DataWorks image from your personal development environment and select that custom image in the schedule settings to ensure the production environment has the same dependencies as the development environment.

2. Configure a resource group: Select the resource group for task execution. For serverless resource groups, we recommend that you configure no more than 16CU to avoid task startup failures caused by insufficient resources. The maximum supported for a single task is 64CU.

3. Configure an associated role: For fine-grained access control, you can associate a specific RAM role with the node so that it runs under that role identity. For more information, see Configure an associated role for a node.

Step 3: Deploy the node

Only nodes in workspace directories can be deployed and periodically scheduled.

• For Notebooks in workspace directories: After you complete the configuration, click Deploy in the top toolbar.

• For Notebooks in personal directories: Click Save first to Submit to Workspace Directory, and then deploy the node.

After successful deployment, you can monitor and manage your Notebook tasks on the Scheduled Tasks page in Operation Center.