DataWorks:Advanced notebook development

Reference project resources (.py files)

To make your code more modular, reusable, and maintainable, group common functions or classes in standalone .py files. You can then reference these files as MaxCompute resources using the ##@resource_reference{"custom_name.py"} syntax.

1. Create and publish a Python resource

   1. In the left navigation pane of DataWorks DataStudio, click and navigate to Resource Management.

   2. In the Resource Management directory tree, right-click the target directory or click + in the upper-right corner. Select New Resource > MaxCompute Python and name the file my_utils.py.

   3. In the Document Content section, click Online Editing, paste your utility function code into the code editor, and click Save.

      # my_utils.py
      def greet(name):
          return f"Hello, {name} from resource file!"

   4. In the toolbar, click Save and then Publish the resource. This makes the resource available to tasks in both the development and production environments.

2. Reference the resource in a notebook

   In the first line of a Python cell in your notebook, use the ##@resource_reference syntax to reference the published resource.

   ##@resource_reference{"my_utils.py"}
   # If the resource is in a subdirectory, such as my_folder/my_utils.py, reference it by filename only, without the directory path: ##@resource_reference{"my_utils.py"}
   from my_utils import greet
   
   message = greet('DataWorks')
   print(message)

3. Debug in the development environment

   Run the Python cell. The output is:

   Hello, DataWorks from resource file!

   Important

   During debugging in the development environment, the system detects the ##@resource_reference declaration and automatically downloads the corresponding file from Resource Management to the workspace/_dataworks/resource_references path in your personal directory, making it accessible to your code.

   If a ModuleNotFoundError occurs, click the Restart button in the editor toolbar to reload the resource, and then try again.

4. Publish to the production environment and verify

   After you Save and Publish the notebook node, navigate to Operation and Maintenance Center > Auto Triggered Task and click Test to run the task. After the task succeeds, the output Hello, DataWorks from resource file! appears in the logs.

   Important

   If a There is no file with id ... error occurs, ensure that you have published the Python resource to the production environment.

For more information, see MaxCompute resources and functions.

Read and write datasets (OSS/NAS)

When notebook tasks run, you can easily read from and write to large-scale files stored on OSS or NAS.

For more information, see Use datasets in a personal development environment.

Use workspace parameters

DataWorks provides workspace parameters, which extend existing scheduling parameters. These parameters enable the reuse of global configurations and environment isolation across tasks and nodes. You can reference a workspace parameter in a SQL cell or Python cell using the ${workspace.param} format, where param is the name of your workspace parameter.

1. Create a workspace parameter: Before you start, navigate to Operation and Maintenance Center > Scheduling Settings > Workspace Parameters in DataWorks to create the required parameters.

2. Reference a workspace parameter:

• Reference a workspace parameter in a SQL cell.

  SELECT '${workspace.param}';

  When the cell runs, it prints the resolved value of the workspace parameter.

• Reference a workspace parameter in a Python cell.

  print('${workspace.param}')

  When the cell runs, it prints the resolved value of the workspace parameter.

For more details, see Use workspace parameters.

Connect to MaxCompute

• %odps: Get a PyODPS entry object

  This command returns an authenticated PyODPS object bound to the current MaxCompute project. This method is recommended for interacting with MaxCompute because it avoids hard-coding AccessKeys in your code.

  1. Use a magic command to create a MaxCompute connection. Enter %odps. A MaxCompute compute resource selector appears in the lower-right corner and automatically selects a compute resource. Click the MaxCompute project name in the lower-right corner to switch projects.

     o=%odps 

  2. Use the retrieved MaxCompute compute resource to run a PyODPS script.

     For example, to get all tables in the current project:

     with o.execute_sql('show tables').open_reader() as reader:
         print(reader.raw)

• %maxframe: Establish a MaxFrame connection

  This command creates a MaxFrame session, which provides distributed, pandas-like data processing capabilities for MaxCompute.

  # Connect to and access a MaxCompute MaxFrame Session
  mf_session = %maxframe
  
  df = mf_session.read_odps_table('your_mc_table')
  print(df.head())
  
  # After development and debugging, manually destroy the session to release resources
  mf_session.destroy()

Connect to Spark compute resources

DataWorks Notebook supports connections to multiple Spark engines. These engines differ in connection method, execution context, and resource management.

Connect to Lindorm Ray

The RAY resource group of the Lindorm compute engine provides distributed computing services and supports end-to-end AI workloads. You can use a magic command to seamlessly connect to Lindorm Ray resources in a notebook for interactive development and debugging, and then publish the notebook as a scheduled production task.

1. Establish a connection: Run the %lindorm_ray command in a Python cell. A compute resource selector appears in the lower-right corner of the cell. Select your Lindorm compute resource and the created RAY resource group.

   # Connect to the specified Lindorm Ray resource group.
   %lindorm_ray

   Important

   • After you connect to a Lindorm Ray compute resource, you can no longer run SQL cells in the same notebook. The Lindorm Ray engine exclusively executes Python and Ray code.

   • If you run the same code cell multiple times, the system automatically terminates the previous Ray job and starts a new one. This helps prevent resource waste and task conflicts.

2. Execute Ray code: After the connection is established, you can write and execute Ray code directly in a new Python cell. Logs stream back to the cell's output area in real time, which facilitates interactive debugging.

   The following example defines a simple remote task (using the @ray.remote decorator) that executes on a Ray cluster and returns the logs and final result to the output area of the cell.

   import ray
   import time
   
   @ray.remote
   def hello_world():
     print("Hello from Lindorm Ray!")
     time.sleep(5)
     return "Task finished."
   # Submit the remote task
   result_ref = hello_world.remote()
   print(ray.get(result_ref))

3. (Optional) Specify custom startup parameters: If you need to specify additional configurations for the Ray environment, such as installing third-party Python packages or uploading local code files, use the %%lindorm_ray command to establish the connection.

   • Example 1: Install dependencies

     Use the pip parameter to install the jieba package in the Ray environment.

     %%lindorm_ray
     {
       "runtime_env": {
         "pip": ["jieba"]
       }
     }

     Once the environment is ready, import and use the package in subsequent Ray jobs. The following example shows how to call jieba in a remote function to perform Chinese word segmentation:

     import ray 
     
     @ray.remote
     def do_work(x):
         import jieba
     
         return "/".join(jieba.cut(x))
     
     print(ray.get(do_work.remote("Welcome to the DataWorks+LindormRay solution")))

   • Example 2: Upload and use DataWorks resources

     The working_dir parameter is used to upload resources from DataWorks Resource Management to a Ray cluster so that they can be imported and called in tasks.

     Important

     • When you use working_dir to upload resources, files are uploaded directly from your development environment to the Ray cluster and are subject to a 100 MB size limit. If a resource package is too large, the upload may fail or the Ray nodes may become unstable.

     • For large files or dependencies (>100 MB), upload them to OSS and access them from your code, or package them into a custom image. This approach provides better stability and performance.

     # Reference a resource uploaded in DataStudio Resource Management and declare its path.
     %%lindorm_ray
     {
         "runtime_env": {
             "working_dir": "/mnt/workspace/_dataworks/resource_references"
         }
     }

     Suppose you upload a ray_resource.py file to Resource Management in DataStudio. When you write and execute the following cell, the system automatically parses the ##@resource_reference declaration in the subsequent code and downloads the corresponding resource to the /mnt/workspace/_dataworks/resource_references path.

     Sample code for ray_resource.py:

     def fun():
         print("This is a test function in ray_resource.py")

     Important

     In a development environment, after you execute the cell that contains ##@resource_reference, you must rerun the %%lindorm_ray cell above to upload the downloaded resources in the working_dir to the Ray cluster. In a production environment, you do not need to rerun.

     import ray 
     
     ##@resource_reference{"ray_resource.py"}
     
     @ray.remote
     def do_work(x):
         print('Ray says:', x)
     
         from ray_resource import fun
         fun()
         return x
     
     worker = do_work.remote("Welcome to the DataWorks+LindormRay solution")
     print(ray.get(worker))

4. Production scheduling and O&M: After development and debugging, you can commit and publish the notebook node. It will then be periodically scheduled as a Lindorm Ray node in a DAG.

   • Parameterization: Your code can use standard DataWorks scheduling parameters, such as ${bizdate}.

   • Log viewing: In the production environment, to prevent excessive logs from affecting performance, the system loads only the first 1 MB of logs by default. If logs are truncated, the output includes a link that directs you to the Lindorm console to view the complete task logs.

   • Resource release: After a scheduled production task ends, the Lindorm Ray task enters a terminal state and no longer occupies resources. During interactive development, you can terminate the Lindorm Ray task by restarting the kernel or closing the notebook.

Appendix: Magic command quick reference