DataWorks:Notebook development

Before you begin, make sure that you meet the following requirements:

• The current workspace has the new Data Studio enabled.

• A serverless resource group is created. Notebooks require a Serverless resource group to run in the production environment.

• A personal development environment instance is created. Notebooks require a personal development environment instance to run and debug in the development environment.

  If you have not created one, see Create a personal development environment instance.

Procedure:

1. Create a Notebook node

   1. In Data Studio, create a Notebook node under Workspace Directories.

   2. Enter a name for the Notebook, such as hello_notebook, and then submit it.

2. Select a personal development environment

   In the top navigation bar, click Personal Development Environment, and then select your personal development environment instance from the drop-down list.

3. Write a Python cell to define parameters

   Add a Python cell and enter the following code. This step defines a city variable for the subsequent SQL query.

   # Define a variable for the subsequent SQL query
   city = 'Beijing'
   print(f"Defined city variable city = {city}")

4. Write an SQL cell to query data

   1. Below the first cell, add an SQL cell.

   2. In the lower-right corner of the cell, switch the SQL type to MaxCompute SQL.

   3. Enter the following SQL code. The code references the city variable defined in the previous Python cell using the ${city} syntax.

      -- Query using the variable defined in Python  
      SELECT '${city}' AS city;

5. Run the cells and view the results

   1. Click the Run All button on the toolbar at the top of the Notebook.

   2. Observe the execution of each cell:

      • The output below the Python cell shows Defined city variable city = Beijing.

      • The query results are displayed in a table below the SQL cell.

You have now successfully created and run a Notebook that includes Python and SQL interaction.

Development environment vs. production environment

Compute resources and kernels

• Compute resource: The backend compute engine connected to the Notebook, such as MaxCompute or EMR Serverless Spark. An SQL cell must be bound to a compute resource to execute.

• Python kernel: The backend environment that executes Python code. In a DataWorks Notebook, this is typically your personal development environment instance. You can use Magic Commands, such as %odps, in your Python code to connect to compute resources to submit tasks or manipulate data.

Folder types and scenarios

The location where you create a Notebook determines its collaboration model, permissions, and deployment flow.

Manage cells

• Add cell: Hover over the top or bottom edge of an existing cell and click one of the buttons that appear, such as + SQL. You can also use the buttons on the top toolbar.

• Switch cell type: Click the type identifier in the lower-right corner of a cell, such as Python, and select a new type from the menu, such as SQL or Markdown. When you switch the type, the code in the cell is retained. You must manually modify it to fit the new type.

• Move a cell: Hover over the blue vertical line on the left of a cell, and then click and drag to change its order.

• Run a cell:

  • Run a single cell: Click Run.

  • Run all cells: Click Run All in the Notebook toolbar.

Pass parameters

Copilot-assisted programming

DataWorks Copilot is a built-in intelligent programming assistant that helps you generate and explain code.

How to use:

• Click the Copilot icon in the upper-left corner of the selected cell.

• Right-click inside an SQL cell and select Copilot.

• Use the keyboard shortcut Cmd+I (Mac) or Ctrl+I (Windows).

1. Configure scheduling parameters

If you need the parameters in a Notebook to change dynamically for each scheduled run, such as processing data from different partitions daily, you can set up parameterized scheduling.

1. Mark the parameter cell: In the Python cell that contains the core parameter definitions, click the ... menu in the upper-right corner and select Mark Cell as Parameters. A parameters tag is then added to the cell, marking it as the parameter entry point for the scheduling task.

   

2. Configure scheduling parameters:

   1. In the right pane of the Notebook, click Scheduling.

   2. In the Scheduling Parameters area, assign values to the variables (such as var) defined in your code.

      

When the task is automatically executed by the scheduling system, the actual value of the var parameter in the code is dynamically replaced by the value configured in the scheduling parameters.

2. Configure the runtime environment and resources

1. Configure the image: In the Scheduling, select an image that contains all the dependencies required for the Notebook to run. This is key to ensuring successful execution in the production environment.

   Important

   If you install Python packages in your personal development environment instance using methods such as pip install, you must create a DataWorks image from your personal development environment to ensure the production environment has the same dependencies. Then, select this custom image in the Scheduling. For more information, see Create a DataWorks image from a personal development environment.

2. Configure the resource group: Select the resource group to use for executing the task. For a Serverless resource group, we recommend configuring no more than 16 CU to prevent task startup failures due to insufficient resources. A single task supports a maximum of 64 CU.

3. Configure an associated role: For fine-grained permission control, you can associate a specific RAM role with the node. The node then runs with the identity of that role. For more information, see Configure an associated role for a node.

3. Deploy the node

Only nodes in the workspace directories can be deployed and scheduled periodically.

• For Notebooks in the workspace directories: After completing the configuration, click the Deploy button on the top toolbar.

• For Notebooks in the personal directory: you must click the Save button and then click Commit to Workspace Directory before deployment.

After the Notebook is deployed, you can monitor and manage your Notebook task on the Auto Triggered Nodes page in the Operation Center.