All Products
Search
Document Center

MaxCompute:Overview

Last Updated:Dec 25, 2023

Python on MaxCompute (PyODPS) is MaxCompute SDK for Python, which helps you use PyODPS to interact with MaxCompute and process data. You can use PyODPS to develop MaxCompute jobs, analyze data, and manage MaxCompute resources. This topic describes how to use PyODPS.

Introduction to PyODPS

PyODPS supports the DataFrame framework and basic operations on MaxCompute objects. You can use PyODPS to develop MaxCompute jobs and process data in MaxCompute. PyODPS supports Python 2 and Python 3. Python 2 can be Python 2.6 or later.

For more information about PyODPs, see the following documentation:

If you want to help build the PyODPS ecosystem, you can perform the following operations:

Initialization

Before you can use PyODPS, you must initialize a connection to MaxCompute by using your Alibaba Cloud account. To initialize a connection, run the following command:

import os
from odps import ODPS
# Set the environment variable ALIBABA_CLOUD_ACCESS_KEY_ID to the AccessKey ID of your Alibaba Cloud account. 
# Set the environment variable ALIBABA_CLOUD_ACCESS_KEY_SECRET to the AccessKey secret of your Alibaba Cloud account. 
o = ODPS(
    os.getenv('ALIBABA_CLOUD_ACCESS_KEY_ID'),
    os.getenv('ALIBABA_CLOUD_ACCESS_KEY_SECRET'),
    project='your-default-project',
    endpoint='your-end-point',
)

Parameters:

  • ALIBABA_CLOUD_ACCESS_KEY_ID: the AccessKey ID of your Alibaba Cloud account. Make sure that the account has the permissions to manage objects in a MaxCompute project. For more information about the permissions, see Operation permissions. In the upper-right corner of the MaxCompute console, click the profile picture and select AccessKey Management to obtain the AccessKey ID.

  • ALIBABA_CLOUD_ACCESS_KEY_SECRET: the AccessKey secret that corresponds to the AccessKey ID. In the upper-right corner of the MaxCompute console, click the profile picture and select AccessKey Management to obtain the AccessKey secret.

  • your-default-project: the name of your MaxCompute project. You can log on to the MaxCompute console. In the top navigation bar, select a region. In the left-side navigation pane, choose Workspace > Projects to view the name of the MaxCompute project.

  • your-end-point: the endpoint of the region where your MaxCompute project resides. For more information, see Endpoints.

Description

The following table describes the methods that you can use to perform basic operations on MaxCompute objects.

Item

Operation

Description

Projects

get_project(project_name)

Obtains the name of a MaxCompute project.

exist_project(project_name)

Checks whether a MaxCompute project exists.

Tables

list_tables()

Lists all tables in a MaxCompute project.

exist_table(table_name)

Checks whether a table exists.

get_table(table_name, project=project_name)

Obtains a specified table. You can obtain a table from another MaxCompute project.

create_table()

Creates a table.

read_table()

Reads data from a table.

write_table()

Writes data to a table.

delete_table()

Deletes an existing table.

Table partitions

exist_partition()

Checks whether a partition exists.

get_partition()

Obtains information about a partition.

create_partition()

Creates a partition.

delete_partition()

Deletes an existing partition.

SQL

execute_sql()/run_sql()

Executes SQL statements.

open_reader()

Reads execution results of SQL statements.

Instances

list_instances()

Lists all instances in a MaxCompute project.

exist_instance()

Checks whether an instance exists.

get_instance()

Obtains information about an instance.

stop_instance()

Terminates an instance.

Resources

create_resource()

Creates a resource.

open_resource()

Opens a resource.

get_resource()

Obtains information about a resource.

list_resources()

Lists all existing resources.

exist_resource()

Checks whether a resource exists.

delete_resource()

Deletes an existing resource.

Functions

create_function()

Creates a function.

delete_function()

Deletes an existing function.

Uploads and downloads tunnels

create_upload_session()

Creates a session that is used to upload data.

create_download_session()

Creates a session that is used to download data.

Note

You must specify parameters when you use create_table(), read_table(), write_table(), and delete_table() methods. For more information, see Examples of using the SDK for Python: tables.