MaxCompute:Connect using the local client (odpscmd)

Billing

You are not charged for connecting to a project using the MaxCompute client. However, operations that you perform through the client may incur MaxCompute fees. For example, if you use the client to submit an SQL query and a write command, the SQL command consumes computing resources when it runs on MaxCompute, and the written data occupies storage space. This generates computing and storage fees. For more information about MaxCompute billing, see Billable items and billing methods.

Install and configure the MaxCompute client

The client supports the MaxCompute V2.0 data type edition starting from V0.27.0. We recommend that you use the new data types. For a list of supported data types, see Data type editions.

Follow these steps:

1. Download the MaxCompute client installation package (GitHub).

   Note

   • Click the link to go to the client release page and download the latest version of the MaxCompute client installation package (odpscmd_public.zip).

   • If you cannot download the package from the preceding link, try to click MaxCompute client installation package (OSS) to download it. If the GitHub link fails, we recommend that you search for a solution on a search engine.

2. Decompress the downloaded installation package to get the bin, conf, lib, and plugins folders.

3. Go to the conf folder and configure the odps_config.ini file.

   In the odps_config.ini file, lines that start with a number sign (#) are comments. The following table describes the parameters.

   Parameter	Required	Description	Example
   project_name	Yes	The name of the target MaxCompute project that you want to access. If you create a workspace in standard mode, note the difference between the project names for the production environment and the development environment (_dev) when you configure the project_name parameter. For more information, see Differences between workspace modes. Log on to the MaxCompute console and select a region in the top-left corner. In the navigation pane on the left, choose Manage Configurations > Projects. On the Projects page, you can find the name of your MaxCompute project.	doc_test_dev
   access_id	Yes	The AccessKey ID of your Alibaba Cloud account or RAM user. You can go to the AccessKey Management page to obtain the AccessKey ID.	None
   access_key	Yes	The AccessKey secret that corresponds to the AccessKey ID.	None
   end_point	Yes	The endpoint of the MaxCompute service. Configure the endpoint based on the region and network connection type that you selected when you created the MaxCompute project. For information about the endpoints for different regions and networks, see Endpoints. Important The endpoint is for the MaxCompute service, and the Tunnel endpoint is for the MaxCompute Tunnel service. Enter the endpoint for the MaxCompute service here. If the endpoint is configured incorrectly, an access error occurs. Make sure that the endpoint is correct.	http://service.cn-hangzhou.maxcompute.aliyun.com/api
   log_view_host	No	The Logview address. We recommend that you configure this parameter. If you do not configure this parameter, you cannot quickly locate the problem when a job fails. You can use this address to view the detailed runtime information of a job, which provides a basis for troubleshooting. The value is fixed at http://logview.odps.aliyun.com.	http://logview.odps.aliyun.com
   https_check	No	Specifies whether to enable HTTPS access to encrypt requests to access MaxCompute projects. Valid values: True: HTTPS is used. False: HTTP is used. Default value: False.	True
   data_size_confirm	No	The maximum size of input data in GB. This parameter has no value range limit. We recommend that you set this parameter to 100 GB.	100
   update_url	No	A reserved parameter that you can ignore.	None
   use_instance_tunnel	No	Specifies whether to use InstanceTunnel to download the execution results of SQL statements. Valid values: True: Use InstanceTunnel to download the execution results of SQL statements. False: Do not use InstanceTunnel to download the execution results of SQL statements. Default value: False.	True
   instance_tunnel_max_record	No	The maximum number of records that the client can return for an SQL execution result. If you set use_instance_tunnel to True, you must configure this parameter. The maximum value is 10000.	10000
   tunnel_endpoint	No	The public endpoint of the Tunnel service. If you do not configure a Tunnel endpoint, Tunnel automatically routes traffic to the Tunnel endpoint that corresponds to the network where the MaxCompute service resides. If you configure a Tunnel endpoint, the configured endpoint is used, and automatic routing is not performed. For information about the Tunnel endpoints for different regions and networks, see Endpoints.	http://dt.cn-hangzhou.maxcompute.aliyun.com
   set.<key>	No	Sets the properties of the MaxCompute project. For more information about properties, see Properties.	set.odps.sql.decimal.odps2=true

   Make sure that the preceding information is correctly configured. Incorrect configuration will cause the project connection to fail.

Run the MaxCompute client

1. Create a MaxCompute project.

2. If you use a RAM user to run the MaxCompute client, use your Alibaba Cloud account to add the RAM user to the target MaxCompute project. For more information about adding workspace members, see Grant permissions to other users.

3. Start the MaxCompute client using one of the following methods:

   Method 1: Use the script file in the installation package

   In the bin folder of the MaxCompute client installation directory, double-click the odpscmd.bat file for Windows or the odpscmd file for macOS to start the MaxCompute client. If the following information is returned, you have successfully connected to the MaxCompute project.

   Method 2: Use the command-line window of your operating system

   In the command-line window of your operating system, go to the bin directory of the MaxCompute client installation path. Run the odpscmd command for Windows or sh odpscmd for Linux or macOS to start the MaxCompute client. If the following information is returned, you have successfully connected to the MaxCompute project.

   Note

   If you run sh odpscmd in Ubuntu, an error is reported. Try to run the ./odpscmd command to start the client.

   

   If you start the MaxCompute client from the command-line window of your operating system, you can specify parameters to run commands. For more information about the parameters, see Startup parameters.

Related operations on the MaxCompute client

...\odpscmd\bin>odpscmd -h

odps@project_name>whoami;

odps@project_name>quit;
-- This command is equivalent to the following command.
odps@project_name>q;

References

After you log on to the MaxCompute client, you can run SQL commands in the MaxCompute project. For more information, see Use the MaxCompute client.

For more information about the command syntax that the MaxCompute client supports, see Common commands or SQL commands and functions.

FAQ

After you configure the odps_config.ini file, the following common errors may occur when you start the MaxCompute client:

Startup parameters

Usage: odpscmd [OPTION]...
where options include the following:
    --help                                  (-h)for help
    --config=<config_file>                  specify another config file
    --project=<prj_name>                    use project
    --endpoint=<http://host:port>           set endpoint
    -k <n>                                  will skip begining queries and start from specified position
    -r <n>                                  set retry times
    -f <"file_path;">                       execute command in file
    -e <"command;[command;]...">            execute command, include sql command

In a Shell window or Windows Command Prompt, you may need to use the dynamic return value from an odpscmd -e <SQL_statement> command. A Shell variable gets this dynamic return value and then uses it in subsequent jobs in the Shell. In this scenario, the return value must not contain extra information such as runtime information or table headers. To facilitate Shell calls, set the use_instance_tunnel parameter in the odps_config.ini file to false to disable the Instance Tunnel service. Then, run the set odps.sql.select.output.format={"needHeader":false,"fieldDelim":" "}; command to hide the table header.

--Run the following command in the Windows Command Prompt.
...\odpscmd\bin>odpscmd -e "set odps.sql.select.output.format={""needHeader"":false,""fieldDelim"":"" ""};select * from noheader;" >D:\test.txt
--The result is saved in D:\test.txt.

--Run the following command in a Shell window.
/Users/.../odpscmd/bin/odpscmd -e "set odps.sql.select.output.format={\"needHeader\":false,\"fieldDelim\":\"\"};select * from noheader;" >/Users/A/temp/test.txt 
--The result is saved in /Users/A/temp/test.txt.
--The following result is returned.
1
2
3

Version update history

The following table describes the recent version updates of the MaxCompute client. For more information, click the link for the corresponding version.

For more information about MaxCompute client versions and related feature updates, see GitHub.