Use MaxFrame in a DataWorks notebook

Prerequisites

Step 1: Activate MaxCompute

MaxCompute is a large-scale data warehousing and computing service from Alibaba Cloud that supports the MaxFrame distributed Python computing framework.

1. Log on to the MaxCompute console, and select a region in the upper-left corner.

2. In the navigation pane on the left, choose Manage Configurations > Projects.

3. On the Projects page, click Create Project to open the Create Project dialog box. Then, configure the project information as prompted and click OK.

   • Region: Must be the same region that you select for DataWorks, for example, Region.

   • Project Name (Globally Unique): Enter a custom name. We recommend including a business identifier.

   • Billing Method:

     • Subscription: Suitable for high-load production environments with long-term, stable workloads. You pre-purchase a fixed number of Compute Units (CUs) to guarantee resource availability and avoid resource contention.

     • Pay-as-you-go: (Recommended) Suitable for short-term or testing phases. This method offers flexible billing based on the actual CUs consumed.

New users who have completed real-name verification are eligible for a free trial quota that includes computing and storage. For more information, see Free trial for new users.

Step 2: Activate DataWorks

DataWorks is a unified big data development and governance platform that provides capabilities such as notebook development, job scheduling, data integration, and data governance.

1. Go to the DataWorks product page to activate a DataWorks edition.

   Parameter	Description	Recommended value
   DataWorks edition	Features and pricing vary by edition.	Basic Edition (Free, meets requirements for notebook development).
   Region and availability zone	Must be the same as the MaxCompute project region.	Select based on your business needs.

2. Go to the Resource Group product page to purchase compute resources. Configure the following parameters:

   Parameter	Description
   Region and availability zone	Must be the same as the MaxCompute project region. Select based on your business needs.
   Billing method	Pay-as-you-go (Recommended) Subscription
   Resource group name	Enter a custom name. The name must start with a letter and can contain letters, digits, and underscores (_). The name can be up to 128 characters long.
   VPC and vSwitch	If you do not have a Virtual Private Cloud (VPC) and vSwitch, DataWorks creates them automatically. If you have an existing VPC and vSwitch, select them. We recommend using the same ones as the cloud services or storage you intend to access.
   Service-linked role	Follow the on-screen instructions and click Create Service-linked Role.

3. Click Confirm Order and Pay.

If you choose the DataWorks Basic Edition and a pay-as-you-go resource group, no payment is required at this step.

Step 3: Create a DataWorks workspace

A workspace is the basic management unit in DataWorks, used to isolate code, resources, and team members.

1. Log on to the DataWorks console and select a region in the upper-left corner.

2. On the Workspaces page, click Create Workspace.

3. Configure the basic information.

   Parameter	Description
   Workspace name	The name must start with a letter and can contain only letters, digits, and underscores (_).
   Display name	A user-friendly name. You can customize this name.
   Isolate production and development environments	Select whether to enable Standard Mode. For single-user development, you can disable this option (Simple Mode).
   Workspace template	A template is a predefined set of features. We recommend selecting "All Capabilities".

4. Configure the advanced information.

   Parameter	Description
   Workspace administrator	If you are developing with a RAM user, add the RAM user as a workspace administrator.
   Create PAI workspace with the same name	Specifies whether to simultaneously create a PAI workspace. We recommend disabling this option.
   Default resource group	Used for scheduling tasks. Select the general-purpose resource group created in Step 2.
   Alibaba Cloud resource group	An identifier for classifying Alibaba Cloud resources. This parameter is optional.

5. Click Create Workspace.

Step 4: Associate a MaxCompute compute engine

Associate the MaxCompute project with your DataWorks workspace to use it as a compute engine for development.

1. Log on to the DataWorks console and select a region in the upper-left corner.

2. On the Workspaces page, find the target workspace and click Manage in the Actions column.

3. On the Workspace Details page, click Computing Resource in the left navigation pane.

   On the Computing Resource page, click Associate Computing Resource, and select MaxCompute.

   Configure the following parameters:

   Parameter	Description
   MaxCompute project	Select the MaxCompute project created in Step 1.
   Default access identity	The identity used to execute jobs. We recommend selecting your Alibaba Cloud account.
   Endpoint	The service address. This is automatically configured.
   Compute engine instance name	Enter a custom name.
   Resource group name	Connects and enables communication between the compute engine and DataWorks. Select the general-purpose DataWorks resource group created in Step 2 and click Test Connectivity to confirm that the connection is successful.

4. Click Confirm.

Step 5: Create a personal development instance

To execute code, DataWorks notebook nodes must run on a personal development environment instance.

1. On the Workspaces page, find the target workspace and in the Actions column, choose Shortcuts > DataStudio.

2. From the top menu, select Personal development environment and click New Instance.

3. Configure the following parameters:

   Parameter	Description
   Instance name	Enter a custom instance name.
   Instance owner	Your Alibaba Cloud account or RAM user.
   Resource group	The group that contains the compute resources. Select the general-purpose resource group you created.
   Resource type	Select CPU or GPU. For most scenarios, CPU is sufficient.
   Resource quota	Select an instance specification, such as ecs.g8i.xlarge (4 vCPUs, 16 GiB), which supports a 4 CU deduction.
   Select image	The runtime environment. Select an image version that includes MaxFrame, such as dataworks-maxcompute:py3.11-ubuntu20.04:py3.11-ubuntu20.04-202504-1.
   VPC	If you need to access services within a VPC, select the same VPC as your MaxCompute or OSS resources.
   vSwitch	A subnet. Select a vSwitch in the target availability zone under your VPC.
   Security group	Network access control. Select a configured security group that allows outbound traffic.
   Instance RAM role	The authorization role. We recommend selecting the "DataWorks Default Role".

4. Click OK.

5. Wait 2 to 3 minutes until the instance status changes to Running.

Step 6: Create a notebook and run code

1. On the Workspaces page, find the target workspace and in the Actions column, choose Shortcuts > DataStudio.

2. From the top menu, select the personal development environment instance that you started.

3. In Data Studio, click the icon in the left-side navigation pane to go to the Data Studio page.

4. Right-click the target directory and select New Node > Notebook.

5. In the notebook editor, confirm that the kernel in the upper-right corner is Python 3, and then run the following code to verify the MaxFrame version:

   import maxframe
   print("MaxFrame version:", maxframe.__version__)
   # If the version is earlier than 2.2.0, run the following command to upgrade:
   # !pip install maxframe -U

6. In a Python cell in your DataWorks notebook, you can use the built-in Magic Command to establish a connection to MaxCompute compute resources. This avoids repeatedly defining connection information and exposing plaintext AccessKeys in your Python code.

   import maxframe.dataframe as md
   import numpy as np
   from maxframe import new_session
   from maxframe.config import options
   from maxframe.udf import with_running_options
   from odps import ODPS
   import pandas as pd
   
   
   # -------------------------------
   # Initialize the MaxFrame Session
   # -------------------------------
   mf_session = %maxframe
   
   print("LogView Address:", mf_session.get_logview_address())
   
   
   d = {'col1': [1, 2], 'col2': [3, 4]}
   df = md.DataFrame(data=d)
   print(df.execute().fetch())
   
   df = md.DataFrame(data=d, dtype=np.int8)
   print(df.execute().fetch())
   
   d = {'col1': [0, 1, 2, 3], 'col2': pd.Series([2, 3], index=[2, 3])}
   df = md.DataFrame(data=d, index=[0, 1, 2, 3])
   print(df.execute().fetch())
   
   data = np.array([(1, 2, 3), (4, 5, 6), (7, 8, 9)],
                   dtype=[("a", "i4"), ("b", "i4"), ("c", "i4")])
   df = md.DataFrame(data, columns=['c', 'a'])
   df.execute().fetch()
   
   mf_session.destroy()

Related practices

from maxframe import options

# Use a pay-as-you-go quota to avoid insufficient subscription resources.
options.session.quota_name = "Default pay-as-you-go quota"

# Control write concurrency to avoid overwhelming OSS.
options.sql.settings = {
    "odps.sql.split.dop": {"target_table": 5}
}

FAQ