DataWorks:Advanced notebook development

Reference project resources (.py files)

Encapsulate common functions or classes into .py files and reference them as MaxCompute resources using ##@resource_reference{"custom_name.py"}.

1. Create and publish a Python resource

   1. In the left-side navigation pane of DataWorks DataStudio, click the icon to go to Resource Management.

   2. In the Resource Management tree, right-click the target directory or click + in the upper-right corner. Select New Resource > MaxCompute Python, and name the resource my_utils.py.

   3. In the Document Content section, click Online Editing, paste your debugged utility function code into the editor, and click Save.

      # my_utils.py
      def greet(name):
          return f"Hello, {name} from resource file!"

   4. In the toolbar, click Save and then Publish. This makes the resource visible to both development and production tasks.

2. Reference the resource in a notebook

   In the first line of a Python cell in your notebook, use the ##@resource_reference syntax to reference the published resource.

   ##@resource_reference{"my_utils.py"}
   # If the resource is in a directory, such as my_folder/my_utils.py, you still use ##@resource_reference{"my_utils.py"} without the directory name.
   from my_utils import greet
   message = greet('DataWorks')
   print(message)

3. Debug and run in the development environment

   Run the Python cell. The following result is printed:

   Hello, DataWorks from resource file!

   Important

   When you debug and run in the development environment, the system detects the ##@resource_reference declaration and downloads the target file from Resource Management to workspace/_dataworks/resource_references in your personal directory.

   If a ModuleNotFoundError error occurs, click the Restart button in the editor toolbar to reload the resource, and then try again.

4. Publish to the production environment and verify

   After you Save and Publish this Notebook node, go to Operation and Maintenance Center > Auto Triggered Task and click Test to run it. After the task runs successfully, you will see the output Hello, DataWorks from resource file! in the logs.

   Important

   If a There is no file with id ... error occurs, ensure you have published the Python resource to the production environment first.

For more operations, see MaxCompute resources and functions.

Read and write datasets (OSS/NAS)

Read and write large files stored on OSS or NAS during notebook task execution.

For more operations, see Use datasets in a personal development environment.

Reference workspace parameters

In addition to scheduling parameters, DataWorks provides Workspace Parameters to reuse global configurations and isolate environments across tasks and nodes. Reference them in SQL and Python cells using ${workspace.param}, where param is the parameter name.

1. Create Workspace Parameters: In DataWorks, go to Operation and Maintenance Center > Scheduling Settings > Workspace Parameters to create the parameters.

2. Reference Workspace Parameters:

• Reference a Workspace Parameter in an SQL cell.

  SELECT '${workspace.param}';

  When the query runs successfully, it prints the resolved value of the parameter.

• Reference a Workspace Parameter in a Python cell.

  print('${workspace.param}')

  When the code runs successfully, it prints the resolved value of the parameter.

For more details, see Use workspace parameters.

Connect to MaxCompute

• %odps: Get a PyODPS entry object

  This eliminates the need to hard-code an AccessKey in your code.

  1. Use a magic command to create a MaxCompute connection. Enter %odps. A MaxCompute compute resource selector appears in the lower-right corner (and automatically selects a compute resource). You can click the MaxCompute project name in the lower-right corner to switch MaxCompute projects.

     o=%odps 

  2. Use the returned MaxCompute compute resource to run a PyODPS script.

     For example, to get all tables in the current project:

     with o.execute_sql('show tables').open_reader() as reader:
         print(reader.raw)

• %maxframe: Establish a MaxFrame connection

  Creates a MaxFrame session for pandas-like distributed processing on MaxCompute.

  # Connect to and access the MaxCompute MaxFrame session.
  mf_session = %maxframe
  df = mf_session.read_odps_table('your_mc_table')
  print(df.head())
  # After development and debugging, destroy the session to release resources.
  mf_session.destroy()

Connect to Spark compute resources

DataWorks Notebook supports multiple Spark engines. Each engine differs in connection method, execution context, and resource lifecycle.

Connect to a Lindorm Ray compute resource

The Lindorm compute engine's Ray resource group provides distributed computing for end-to-end AI workloads. Connect to Lindorm Ray in a notebook for interactive development, then publish the notebook as a scheduled production task.

1. Establish a connection: Run the %lindorm_ray magic command in a Python cell. After you run the command, a compute resource selector appears in the lower-right corner of the cell. Select your Lindorm compute resource and the Ray resource group you created.

   # Connect to the specified Lindorm Ray resource group.
   %lindorm_ray

   Important

   • After you connect to a Lindorm Ray compute resource, you can no longer run SQL cells in the same notebook. The Lindorm Ray engine focuses on running Python and Ray code.

   • If you run the same code cell multiple times, the system automatically terminates the previous Ray job and starts a new one. This prevents resource waste and task conflicts.

2. Execute Ray code: After a successful connection, you can directly write and run Ray code in a new Python cell. Logs are streamed to the output area of the cell in real time for interactive debugging.

   The following example defines a simple remote task using the @ray.remote decorator. The task runs on the Ray cluster, and the logs and final result are returned to the output area of the cell.

   import ray
   import time
   @ray.remote
   def hello_world():
     print("Hello from Lindorm Ray!")
     time.sleep(5)
     return "Task finished."
   # Submit the remote task.
   result_ref = hello_world.remote()
   print(ray.get(result_ref))

3. (Optional) Specify custom startup parameters: To configure additional settings for the Ray environment, such as installing third-party Python packages or uploading local code files, use %%lindorm_ray to establish the connection.

   • Example 1: Install dependencies

     Use the pip parameter to install the jieba package in the Ray environment.

     %%lindorm_ray
     {
       "runtime_env": {
         "pip": ["jieba"]
       }
     }

     After the environment is ready, import and use the package in subsequent Ray tasks. The following example shows how to call jieba in a remote function for Chinese word segmentation:

     import ray 
     @ray.remote
     def do_work(x):
         import jieba
         return "/".join(jieba.cut(x))
     print(ray.get(do_work.remote("Welcome to the DataWorks LindormRay solution")))

   • Example 2: Upload and use DataWorks resources

     Use the working_dir parameter to upload resources from Resource Management to the Ray cluster for import in your tasks.

     Important

     • When you use working_dir to upload resources, the files are uploaded directly from your development environment to the Ray cluster. A size limit of 100 MB applies. If the resource package is too large, the upload may fail or the Ray node may become unstable.

     • For large resources or dependencies (over 100 MB), upload them to OSS and download from OSS in your code, or package them into a custom image for better stability and performance.

     # Reference and declare the path for a resource from Resource Management.
     %%lindorm_ray
     {
         "runtime_env": {
             "working_dir": "/mnt/workspace/_dataworks/resource_references"
         }
     }

     Assume that you have uploaded a file named ray_resource.py to Resource Management in DataWorks. When you run the following cell, the system parses the ##@resource_reference declaration and downloads the corresponding resource to /mnt/workspace/_dataworks/resource_references.

     Example code in ray_resource.py:

     def fun():
         print("This is a test function in ray_resource.py")

     Important

     In the development environment, after you run the cell that contains ##@resource_reference, you must rerun the preceding %%lindorm_ray cell. Rerunning this cell includes the downloaded resources in the working_dir and uploads them to the Ray cluster. You do not need to rerun the cell in the production environment.

     import ray 
     ##@resource_reference{"ray_resource.py"}
     @ray.remote
     def do_work(x):
         print('Ray says:', x)
         from ray_resource import fun
         fun()
         return x
     worker = do_work.remote("Welcome to the DataWorks LindormRay solution")
     print(ray.get(worker))

4. Production scheduling and O&M: After development and debugging, commit and publish this notebook node. It will be scheduled periodically as a Lindorm Ray node in a DAG.

   • Parameterization: Your code can use standard DataWorks scheduling parameters, such as ${bizdate}.

   • Log viewing: In production, the system loads only the first 1 MB of logs by default to prevent performance issues. If logs are truncated, the output provides a link to view full task logs in the Lindorm console.

   • Resource release: After a scheduled production task ends, the Lindorm Ray task enters a terminal state and releases resources. During interactive development, terminate the task by restarting the kernel or closing the notebook.

Appendix: Magic command quick reference