Document Center
MaxCompute
MaxCompute
Release Note
Announcements
Document updates
Updates in 2021
Updates in year 2020
Updates in year 2019
Product Introduction
What is MaxCompute?
Usage notes
Definitions
Terms
Terms
Project
Quota
Table
Partition
Lifecycle
Resource
Function
Task
Task instance
ACID semantics
Limits
Computing models enabled in each region
Customer stories
MaxCompute development history
Pricing
Billing method
Storage pricing (pay-as-you-go)
Computing pricing
Download pricing (pay-as-you-go)
View billing details
Switch billing methods
Upgrade or downgrade configurations
Overdue payment warning and service suspension policies
Renewal management
Consumption control
Type
Subscription
Pay-as-you-go
Prepare
Overview
Create an Alibaba Cloud account
Create a RAM user
Activate MaxCompute and DataWorks
Create a MaxCompute project
Add a workspace member and configure roles
Prepare an environment and install required development tools
Endpoints
Users and roles
Quick Start
Use the MaxCompute client
Start the MaxCompute client
Create tables
Import data to tables
Execute SQL statements and export the result data
Delete a table or a MaxCompute project
Additional information
Connect to Maxcompute
Query editor
MaxCompute client
MaxCompute Studio
What is MaxCompute Studio?
Tools Installation and version history
Install IntelliJ IDEA
Install MaxCompute Studio
View and update the version of MaxCompute Studio
Configure MaxCompute Studio
Manage project connections
Manage data and resources
Manage project data
Import and export table data
Manage tables in a visualized manner
Develop SQL procedure
Create a MaxCompute script module
Develop and submit an SQL script
Developing Java
Overview
Create a MaxCompute Java module
Develop a UDF
Develop a MapReduce program
Develop Graph
Query unstructured data
Package, upload, and register a Java program
Develop a Python program
Configure a Python development environment
Develop a Python UDF
Develop a PyODPS script
Manage MaxCompute jobs
View jobs
Job instance details
Tool integration
Integrate with MaxCompute client
FAQ
How to develop UDF using Studio
How to manage MaxCompute metadata using Studio
Connections in the MaxCompute ecosystem architecture
MaxCompute ecosystem
BI tools
Compatibility overview
Connect Tableau to MaxCompute
Connect Superset to MaxCompute
Connect Yonghong BI to MaxCompute
Connect Quick BI to MaxCompute
Database management tools
Compatibility overview
Connect DBeaver to MaxCompute
Connect DataGrip to MaxCompute
Connect SQL Workbench/J to MaxCompute
ETL tools
Use Kettle to schedule MaxCompute jobs
Use Apache Airflow to schedule MaxCompute jobs
Use Azkaban to schedule MaxCompute jobs
Java SDK
Version updates
SDK for Java
Run security-related commands
Instance Logview
Return error logs
Set the SQL flag
Use SQLTask and Tunnel to export a large amount of data
Python SDK
Overview
SDK for Python
JDBC Reference
Version updates
Overview
Usage notes
Downloads
Tutorials
Build an online operation analysis platform
Business scenarios and development process
Activate services
Prepare data
Build data models and analyze data
Create tables
Design workflows
Configure nodes
Commit and test nodes
Display data on dashboards
Data Lake Integration
Lakehouse of MaxCompute
Data Migration
Data upload and download
Overview
Connection to the Tunnel or DataHub service
Data migration scenarios
Data migration tools
Run Tunnel commands to upload and download data
Tunnel commands
Usage notes
Tunnel SDK
Overview
TableTunnel
InstanceTunnel
UploadSession
DownloadSession
TunnelBufferedWriter
Bulk data channel SDK example
Overview
Simple upload
Simple download example
Data upload in multi-threaded mode
Multi-threaded download
Data upload by using BufferedWriter
Data upload by using BufferedWriter in multi-threaded mode
Examples of uploading and downloading complex data
Import or export data by using Data Integration
Export data by using Data Integration
Import data from Message Queue for Apache Kafka to MaxCompute
Migration example
Development
Data types
Data type editions
MaxCompute V1.0 data type edition
MaxCompute V2.0 data type edition
Hive-compatible data type edition
Compatibility between components and data types
Common commands
Common SQL statements
Project operations
SET operations
Security operations
User and role operations
Authorization operations
Table operations
Partition and column operations
Instance operations
Resource operations
Function operations
Tunnel operations
Time zone configuration operations
Other operations
SQL
Overview of MaxCompute SQL
Differences in the support for SQL statements
MaxCompute SQL limits
DDL SQL
Table operations
Partition and column operations
Lifecycle management operations
View-related operations
Insert Operation
Insert or update data into a table or static partitions (INSERT INTO and INSERT OVERWRITE)
Insert or overwrite data into dynamic partitions (DYNAMIC PARTITION)
UPDATE and DELETE
MERGE INTO
MULTI INSERT
VALUES
Select Operation
SELECT syntax
Sequence for executing clauses in a SELECT statement
Subqueries
INTERSECT, UNION, EXCEPT, and MINUS
JOIN
SEMI JOIN
MAPJOIN hints
SKEWJOIN HINT
LATERAL VIEW
HAVING clause
GROUPING SETS
SELECT TRANSFORM
SQL enhancement operations
LOAD
UNLOAD
EXPLAIN
Common table expressions
CLONE TABLE
Parameterized view
Builtin functions
Comparison of functions built in MaxCompute, MySQL, and Oracle
Date functions
Mathematical functions
Window functions
Aggregate functions
String functions
Other functions
UDF
Overview
UDF
Java UDF
Python UDF
Python UDF examples
Code-embedded UDFs
SQL functions
Open source geospatial UDFs
UDT
Overview
Usage examples
UDJ
Run MaxCompute SQL in script mode
Appendix
Operators
Escape character
LIKE usage
Regular expressions
Reserved words and keywords
Data type mappings
Type conversions
Dynamic parameters
MaxCompute SQLML
Overview
Quick start
Billing
PyODPS
Quick start
Installation guide and limits
Platform instructions
Overview
Migrate PyODPS nodes from a data development platform to a local PyODPS environment
Use PyODPS in DataWorks
Basic operations
Overview
Projects
Tables
SQL
Task instances
Resources
Functions
Configurations
DataFrame
Overview
Quick start
Create a DataFrame object
Sequence
Collection
Execution
Column operations
Aggregation
Sort, deduplicate, sample, and transform data
Use UDFs and the third-party Python libraries
MapReduce API
Data merging
Window functions
Plotting
Debugging
User experience enhancement
Command line
IPython
Jupyter Notebook
API overview
Examples
Reference a third-party package in a PyODPS node
Use a PyODPS node to query data based on specific criteria
Use a PyODPS node to pass parameters
Use a PyODPS node to read data from a partitioned table
Use a PyODPS node to read data from the level-1 partition of the specified table
Use a PyODPS node to perform sequence operations
Use a PyODPS node to aggregate data
Use a PyODPS node to perform column operations
Use a PyODPS node to sort data
Use a PyODPS node to de-duplicate data
Use a PyODPS node to sample data
Use a PyODPS node to scale data
Use a PyODPS node to process NULL values
MapReduce
Summary
Overview
Extended MapReduce model
Open source MapReduce
Limits
Quick start
Function Introduction
Terms
Submit a MapReduce job
Input and output
Resource usage
Job running in local mode
Program Example
WordCount
Map-only jobs
MultipleInOut
MultiJobs samples
SecondarySort
Resources
UserDefinedCounters
Grep
JOIN
Sleep samples
Unique samples
Sort examples
Partition samples
Pipeline examples
Java SDK
Overview
Compatibility with Hadoop MapReduce
Java sandbox
Mars
Version updates
Overview
Preparations
Usage notes
Spark
Overview
Set up a Spark on MaxCompute development environment
Running modes
Java and Scala development examples
Overview
Spark 1.x examples
Spark 2.x examples
Develop a Spark on MaxCompute application by using PySpark
Access instances in a VPC from Spark on MaxCompute
Configure Spark on MaxCompute to access OSS resources
Check jobs
MaxCompute Query Acceleration
Overview
Usage notes
External table
Overview
STS authorization
Access external tables
Access OSS data by using a built-in extractor
Use a custom extractor to access OSS
Process OSS data stored in open source formats
Partition paths of an OSS external table
Export unstructured data to OSS
Access Tablestore data
Hologres external tables
Interactive SQL (Lightning)
The Lightning service of MaxCompute will Stop Serving
Upgrade MaxCompute Lightning to Hologres
Overview
Endpoints
Limits
Quick start
Usage notes
Access services using JDBC interfaces
JDBC driver
Configure a JDBC connection
Use common tools to connect to MaxCompute Lightning
Query syntax and built-in functions
View or cancel query jobs
FAQ
Graph
Overview
Aggregator overview
Limits
Graph jobs
Submit a Graph job
SDK configuration
Development and debugging
Examples
SSSP
PageRank
K-means clustering
Bipartite matching
Strongly connected component
Connected component
Topological sorting
Linear regression
Triangle count
Vertex table import
Edge table import
View Job Running Information
Use Logview to view job information
Use Logview V2.0 to view job execution information
Use errors and alerts in the MaxCompute compiler for troubleshooting
Develop scheduling nodes through DataWorks
Create an ODPS SQL node
Create an SQL component node
Create an ODPS Spark node
Create a PyODPS 2 node
Create a PyODPS 3 node
Create an ODPS Script node
Create an ODPS MR node
Create a MaxCompute table
Create MaxCompute resources
Create a MaxCompute function
Collect information for the optimizer of MaxCompute
Open source features of MaxCompute
Security Management
Security management
Security models
Manage roles
Management role permissions
Assign management roles to a RAM user
Use DataWorks to manage permissions of a role on a project
Permission relationships between MaxCompute and DataWorks
Management of users and permissions
Enable security features
Configure security features
Overview of security features
Quick Start
Add a user and grant permissions to the user
Add a role and assign the role to users
Enable project data protection
Manage users and permissions
User authentication
Manage users
Authorize users
Role management
Check permissions
Column-level access control
Policy-based access control and download control
Resource share across project space
Package-based resource sharing across projects
Use a package
Project security configurations
Project data protection
Configure IP address whitelists
Security command list
Statements for project security configurations
Statements for project permission management
Statements for package-based resource sharing
Security management use cases
Create a project
Package-based authorization
Data security self-check
Manage permissions by row
Manage permissions by using a RAM user
Security white paper
MaxCompute security white paper
Backup and restoration
Data encryption
Maintenance Management
Information schema
Overview of Information Schema
Metadata views
Resource and job management
MaxCompute Management
Audit logs
Best Practices
SQL
Write MaxCompute SQL statements
Rewrite incompatible SQL statements
Export SQL execution results
Check whether partition pruning is effective
Query the first N data records of each group
Merge multiple rows of data into one row
JOIN operations in MaxCompute SQL
Data migration
Overview
Migrate data across DataWorks workspaces
Migrate data from Hadoop to MaxCompute
Best practice to migrate data from Oracle to MaxCompute
Migrate data from Kafka to MaxCompute
Migrate data from Elasticsearch to MaxCompute
Migrate data from ApsaraDB RDS to MaxCompute based on dynamic partitioning
Migrate JSON-formatted data from MongoDB to MaxCompute
Migrate JSON data from OSS to MaxCompute
Migrate data from MaxCompute to Tablestore
Migrate data from MaxCompute to OSS
Migrate data from a user-created MySQL database on an ECS instance to MaxCompute
Migrate data from Amazon Redshift to MaxCompute
Migrate data from BigQuery to MaxCompute
Migrate log data to MaxCompute
Overview
Use Tunnel to upload log data to MaxCompute
Use DataHub to migrate log data to MaxCompute
Use DataWorks Data Integration to migrate log data to MaxCompute
Data development
Use IntelliJ IDEA to develop a Java UDF
Use MaxCompute to query geolocations of IP addresses
Resolve the issue that you cannot upload files that exceed 10 MB to DataWorks
Grant access to a specific UDF to a specified user
Use a PyODPS node to segment Chinese text based on Jieba
Use a PyODPS node to download data to a local directory for processing or to process data online
Compute optimization
Optimize SQL statements
Optimize JOIN long tails
Long-tail computing optimization
Optimize the calculation for long-period metrics
Job diagnostics
Use Logview to diagnose jobs that run slowly
Cost optimization
Overview
Select a billing method
Optimize computing costs
Optimize storage costs
Optimize the costs of data uploads and downloads
Manage costs
Command reference
Analyze MaxCompute bills
Security management
Set a RAM user as the super administrator for a MaxCompute project
Error Code Appendix
Error code overview
Common errors (ODPS-00CCCCX)
SQL errors (ODPS-01CCCCX)
PL errors (ODPS-02CCCCX)
API errors (ODPS-04CCCCX)
Xlib errors (ODPS-05CCCCX)
MapReduce errors (ODPS-07CCCCX)
ODPS SDK or CLT errors (ODPS-08CCCCX)
AlgoTask errors (ODPS-12CCCCX)
Common Tunnel errors
Public Datasets
Overview
Service Agreements
MaxCompute SLA
all-products-head
This Product
This Product
All Products
All Products
Show more
Show less