DataWorks:Basic notebook development

Prerequisites

• Data Studio is enabled for your workspace.

• You have a serverless resource group.

• You have a personal development environment instance (required for notebooks with Python cells).

  If you have not created one: Create a personal development environment instance.

Procedure:

1. Create a notebook node

   1. Go to Data Studio. In the DataStudio section of Project Directory, create a new notebook node.

   2. Enter a name for the notebook, such as hello_notebook, and submit it.

2. Select a personal development environment

   In the top navigation bar, click Personal development environment and select your personal development environment instance from the drop-down list.

3. Write a Python cell to define a parameter

   Define a city variable for the SQL query.

   # Define a variable for the subsequent SQL query
   city = 'Beijing'
   print(f"Defined city variable city = {city}")

4. Write an SQL cell to query data

   1. Below the first cell, add a new SQL cell.

   2. In the lower-right corner of the cell, switch the SQL type to MaxCompute SQL.

   3. Enter the following SQL. It references the city variable from the Python cell using the ${city} syntax.

      -- Query data using the variable defined in Python  
      SELECT '${city}' AS city;

5. Run the cells and view the results

   1. Click the Run All button on the notebook toolbar.

   2. Observe the execution of each cell:

      • The Python cell outputs Defined city variable city = Beijing.

      • A table with the query result is displayed below the SQL cell.

You have created and run a notebook with Python-SQL interaction.

Notebook modes

DataWorks Notebook operates in two modes:

• SQL and Markdown Mode (default): Supports only SQL and Markdown cells for data querying and documentation. Requires only a serverless resource group — no personal development environment instance is needed. SQL Kernel appears in the upper-right corner.

• Full Mode (Python + SQL + Markdown): Supports Python, SQL, and Markdown cells for data processing, analysis, and visualization. Running Python code requires a personal development environment instance. After you select one, the kernel version appears in the upper-right corner, such as (version is for reference only).

Development environment vs. production environment

Compute resources and kernels

Two concepts determine how code is executed: compute resource and kernel.

• Compute resource: The backend compute engine that executes and processes data tasks, defining the runtime environment and computational power.

  • Definition: An independent, schedulable compute service instance, such as a MaxCompute project or an EMR Serverless Spark cluster.

  • Function: Provides the actual computing power for SQL queries, Spark jobs, and other tasks.

  • Selection: Bind each task to a specific compute resource.

• Kernel: Parses and executes code in a cell, determining which programming language is used.

  • Python kernel:

    • Function: Executes Python code, supporting complex logic for data processing, algorithm implementation, and task orchestration.

    • Interaction model: Within the Python kernel, you can use Magic Commands (like %sql) or SDKs to submit a computational task (such as an SQL query) to a specified compute resource. You can then retrieve the results for further analysis.

  • SQL kernel:

    • Function: Directly interprets and submits queries written in SQL.

    • Interaction model: The SQL kernel forwards SQL statements directly to the designated backend compute resource (e.g., an EMR Spark SQL or MaxCompute SQL session) for execution.

  • Markdown kernel:

    • Function: Renders rich text formatted in Markdown. It does not execute any computational logic.

Relationship summary:

• The kernel is the frontend language interpreter that determines "what you write" (Python or SQL).

• The compute resource is the backend execution engine that determines "where the code runs" (on MaxCompute or Spark).

Directory types and use cases

Where you create a notebook determines its collaboration scope, permissions, and deployment options.

Cell management

• Add a cell: Hover over the top or bottom edge of a cell and click a button such as + SQL, or use the toolbar buttons.

• Switch cell type: Click the type identifier (e.g., Python) in the lower-right corner and select a new type, such as SQL or Markdown. Code is preserved but may need modification.

• Move a cell: Hover over the blue vertical line to the left of a cell, then click and drag to change its order.

• Run cells:

  • Run a single cell: Click the Run icon to the left of the cell.

  • Run all cells: Click the Run All button in the notebook's top toolbar.

Parameter passing

View data lineage (Beta)

The data lineage feature clarifies data flow by statically analyzing Python code (with MaxFrame framework support) in the notebook.

Copilot-assisted programming

DataWorks Copilot is a built-in AI programming assistant that helps you generate and explain code.

How to invoke:

• Click the Copilot icon in the upper-left corner of the selected cell.

• Right-click inside an SQL cell and select Copilot.

• Use the keyboard shortcut Cmd+I (macOS) or Ctrl+I (Windows).

1. Configure scheduling parameters (parameterized scheduling)

Configure parameterized scheduling so that notebook parameters change dynamically per run. For example, you can process a different partition each day.

1. Mark the parameter cell: In the Python cell with parameter definitions, click ... in the upper-right corner and select Mark Cell as Parameters. A parameters tag marks the cell as the parameter entry point.

2. Configure scheduling parameters:

   1. In the right-side panel of the notebook, click Scheduling Settings.

   2. In the Scheduling Parameters area, assign a value to the variables defined in your code, such as var.

      For example, you can set the parameter value to $[yyyymmdd]. At scheduling runtime, it is dynamically replaced with the actual date.

When a task is automatically executed by the scheduling system, the actual value of the var parameter in the code is dynamically replaced by the value configured in the scheduling parameters.

2. Configure runtime environment and resources

1. Configure an image: In Scheduling configuration, select an image with all dependencies required by the notebook. This ensures successful execution in production.

   Important

   If you installed Python packages using pip install, you must create a DataWorks image from the personal development environment to ensure production has the same dependencies. Select this image in scheduling configuration.

2. Resource Group: Select the resource group for the task. For serverless resource groups, configure no more than 16CU to avoid startup failures. Maximum per task: 64CU.

3. Configure an associated role: Associate a RAM role with the node for fine-grained permission control. Associate a role: Securely access other cloud resources.

3. Deploy the node

Only nodes in Workspace Directories can be deployed and scheduled.

• For notebooks in Workspace Directories: After completing the configuration, click the Publish button on the top toolbar.

• For notebooks in your Personal Directory: You must first click the Save button and then submit to the project before deploying it.

After deployment, monitor your notebook task on the Auto Triggered Nodes page in Operation Center.