You can use the MaxCompute client to access MaxCompute projects and run commands. This topic describes how to install, configure, and run the MaxCompute client and provides other related instructions.

Prerequisites

Before you use the MaxCompute client, make sure that the following conditions are met:
  • Java 8 or later is installed on the device on which you want to install the MaxCompute client.
  • A MaxCompute project is created.

    For more information about how to create a MaxCompute project, see Create a MaxCompute project.

  • A RAM user as which you want to use the MaxCompute client is added to the DataWorks workspace. The MaxCompute project belongs to the DataWorks workspace.

    For more information about how to add members to a workspace, see Add a workspace member and configure roles.

Background information

After the MaxCompute client is installed and configured, you can start the MaxCompute client by using the script file under the installation path or by running commands in the command-line interface (CLI) of the system. The MaxCompute client allows you to perform the following operations:

Limits

MaxCompute client V0.28.0 and later support Java Development Kit (JDK) 1.9. The MaxCompute client of a version earlier than V0.28.0 supports only JDK 1.8. You can view the client version in the CLI after you start the MaxCompute client. For more information about how to start the MaxCompute client, see Run the MaxCompute client.

Usage notes

  • The output format of the MaxCompute client may not be forward compatible. The command syntax and execution rules of the client vary based on the client version. We recommend that you do not rely on the output format of the client to parse data. For more information about client versions, see aliyun-odps-console.
  • Two consecutive minus signs (--) are used to comment out a command line on the MaxCompute client.

Install and configure the MaxCompute client

Note MaxCompute client V0.27.0 and later support the MaxCompute V2.0 data type edition. We recommend that you use the MaxCompute V2.0 data type edition. For more information about the supported data types, see MaxCompute V2.0 data type edition.

To install and configure the MaxCompute client, perform the following steps:

  1. Download the MaxCompute client installation package (Github).
    Note If you cannot download the package after you click the preceding link, click here to download the MaxCompute client installation package (OSS). For more information about how to solve the failure to download, we recommend that you search for related solutions in search engines.
  2. Decompress the downloaded package to obtain the bin, conf, lib, and plugins folders.
  3. Open the conf folder and configure the odps_config.ini file.
    The following example shows the content in the odps_config.ini file. 
    project_name=
access_id=
access_key=
end_point=
log_view_host=
https_check=
# confirm threshold for query input size(unit: GB)
data_size_confirm=
# this url is for odpscmd update
update_url=
# download sql results by instance tunnel
use_instance_tunnel=
# the max records when download sql results by instance tunnel
instance_tunnel_max_record=
# IMPORTANT:
#   If leaving tunnel_endpoint untouched, console will try to automatically get one from odps service, which might charge networking fees in some cases.
#   Please refer to Configure endpoints
# tunnel_endpoint=

# use set.<key>=
# e.g. set.odps.sql.select.output.format=

    In the odps_config.ini file, lines that start with a number sign (#) are comments. The following table describes the parameters in the file.

    Parameter Required Description Example
    project_name Yes The name of the MaxCompute project that you want to access.

    If you create a workspace in standard mode, pay attention to the differences of the project names between the production environment and development environment when you specify this parameter. The names of the projects in the development environment end with _dev. For more information, see Basic mode and standard mode.

    You can log on to the MaxCompute console and view the MaxCompute project names on the Project Management tab.

    		doc_test_dev
    access_id Yes The AccessKey ID of your Alibaba Cloud account or a RAM user within the Alibaba Cloud account.

    You can obtain the AccessKey ID from the Security Management page.

    		None
    access_key Yes The AccessKey secret that corresponds to the AccessKey ID.

    You can obtain the AccessKey secret from the Security Management page.

    		None
    end_point Yes The endpoint of MaxCompute.

    You must set this parameter based on the region and network connection method you selected when you create the MaxCompute project. For more information about the endpoints that correspond to each region and network, see Endpoints.

    Notice If the endpoint that you configured is invalid, an error occurs when you access MaxCompute.
    		http://service.cn-hangzhou.maxcompute.aliyun.com/api
    log_view_host No The Logview Uniform Resource Locator (URL). You can view the detailed runtime information of a job by using this URL. This information helps you locate job errors. Set the value to http://logview.odps.aliyun.com.
    Note We recommend that you set this parameter. If you do not set this parameter, you cannot locate the cause of job errors.
    		http://logview.odps.aliyun.com
    https_check No Specifies whether to enable HTTPS access. If HTTPS access is enabled, requests to access MaxCompute projects are encrypted. Valid values:
    • True: HTTPS access is used.
    • False: HTTP access is used.

    Default value: False.

    		True
    data_size_confirm No The maximum size of input data, in GB. The value range is unlimited. We recommend that you set this parameter to 100. 100
    update_url No A reserved parameter. None
    use_instance_tunnel No Specifies whether to use InstanceTunnel to download the results of SQL statements. Valid values:
    • True: InstanceTunnel is used to download the results of SQL statements.
    • False: InstanceTunnel is not used to download the results of SQL statements.

    Default value: False.

    		True
    instance_tunnel_max_record No The maximum number of SQL execution results that can be returned by the client. You must specify this parameter if the use_instance_tunnel parameter is set to True. Maximum value: 10000. 10000
    tunnel_endpoint No The public endpoint of MaxCompute Tunnel. If you do not specify this parameter, traffic is automatically routed to the Tunnel endpoint that corresponds to the network where MaxCompute resides. If you specify this parameter, traffic is routed to the specified endpoint and automatic routing is not performed.

    For more information about the Tunnel endpoints that correspond to each region and network, see Endpoints.

    		http://dt.cn-hangzhou.maxcompute.aliyun.com
    set.<key> No The properties of the MaxCompute project.

    For more information about the properties of MaxCompute projects, see Properties.

    		set.odps.sql.decimal.odps2=true

Run the MaxCompute client

You can start the MaxCompute client by using one of the following methods:
  • Method 1: In the bin folder under the installation path of the MaxCompute client, double-click the odpscmd.bat file to start the MaxCompute client. If the information shown in the following figure is returned, the MaxCompute project is connected. Connection established
  • Method 2: In the CLI of the system, go to the bin folder under the installation path of the MaxCompute client and run the odpscmd command to start the MaxCompute client. If the information shown in the following figure is returned, the MaxCompute project is connected. Connection established
You can enter a command at the cursor shown in the preceding figure and press Enter to run the command. The following figure shows the sample command and returned results. Returned results
Note For more information about the command syntax supported by the MaxCompute client, see Common commands or SQL commands and functions.

Obtain information about the current logon user

You can run the following command in the CLI to obtain the information of the current logon user: 
odps@project_name>whoami;
The following figure shows the returned result. Current logon user
  • Name: the account of the current logon user.
  • Source IP: the IP address of the device where the MaxCompute client is located.
  • End_Point: the endpoint of MaxCompute.
  • Project: the name of the MaxCompute project.

Exit the MaxCompute client

You can run the following command in the CLI to exit the MaxCompute client: 
odps@project_name>quit;
-- The preceding command is equivalent to the following command. 
odps@project_name>q;

Obtain help information about commands

You can obtain help information for the commands of the MaxCompute client by using one of the following methods:
  • Method 1: View the help information for commands on the MaxCompute client.
    • View the help information for all commands. 
      odps@project_name>help;
-- The preceding command is equivalent to the following command. 
odps@project_name>h;
    • Specify a keyword to view the help information for the related commands.
      Example: Obtain the help information for the commands related to table operations. 
      odps@project_name>help table;
-- Returned results: 
Usage: alter table <tablename> merge smallfiles
Usage: export table <tablename>
Usage: show tables [in <project_name>] [like '<prefix>']
       list|ls tables [-p,-project <project_name>]
Usage: describe|desc [<projectname>.]<tablename> [partition(<spec>)]
Usage: read [<project_name>.]<table_name> [(<col_name>[,..])] [PARTITION (<partition_spec>)] [line_num]
  • Method 2: In the CLI of the system, go to the bin folder under the installation path of the MaxCompute client, and run the following command to view the help information about all commands. If you start the MaxCompute client by running commands in the CLI, you can specify a series of parameters in the commands. For more information about these parameters, see Specify startup parameters. 
    ..\odpscmd\bin>odpscmd -h

Specify startup parameters

In the CLI of the system, you can specify a series of parameters to run a command. The following code shows the usage of these parameters. 
Usage: odpscmd [OPTION]...
where options include:
    --help                                  (-h)for help
    --config=<config_file>                  specify another config file
    --project=<prj_name>                    use project
    --endpoint=<http://host:port>           set endpoint
    -k <n>                                  will skip begining queries and start from specified position
    -r <n>                                  set retry times
    -f <"file_path;">                       execute command in file
    -e <"command;[command;]...">            execute command, include sql command
The following table describes the startup parameters.
Parameter Description Sample command
--help or -h Obtains the help information for all commands of the MaxCompute client. odpscmd --help
--config Specifies the directory where the configuration file odps_config.ini is saved. The default directory is odpscmd_public/conf/odps_config.ini. odpscmd --config=D:/odpscmd/conf/odps_config.ini
--project Specifies the name of the MaxCompute project that you want to access. odpscmd --project=doc_test
--endpoint Specifies the endpoint of MaxCompute. For more information about endpoints, see Configure endpoints. odpscmd --endpoint=http://service.cn-shanghai.maxcompute.aliyun.com/api
-k Executes the statement from the specified location. If n is set to a value that is less than or equal to 0, the execution starts from the first statement. Multiple statements are separated by semicolons (;). Run the following command to ignore the first two statements and start to execute from the third statement: odpscmd -k 3 -e "drop table table_name;create table table_name (dummy string);insert overwrite table table_name select count(*) from table_name;"
-r Specifies the number of retries allowed after the job fails to run. odpscmd -r 2 -e "select * from sale_detail;select * from table_test;"
-f Specifies the file to read.
  1. Prepare a script file named script.txt. In this example, the file is stored in drive D and contains the following data: 
    drop table if exists test_table_mj;
create table test_table_mj (id string, name string);
drop table test_table_mj;
  2. In the CLI, go to the bin folder under the installation path of the MaxCompute client and run the following command: 
    ..\odpscmd\bin>odpscmd -f D:/script.txt;
-e Specifies the command that you want to run. odpscmd -e "select * from sale_detail;"

The dynamic return value of an odpscmd -e command may be called by a shell script that is run in a shell window or the Command Prompt in Windows. A shell variable may obtain the return value and use it in subsequent jobs. In this scenario, only field values are required. Other information, such as runtime information and headers, cannot be returned. To facilitate shell calls, you can run the set odps.sql.select.output.format={"needHeader":false,"fieldDelim":""}; command to disable header display.

For example, a table named noheader has one column and three rows of data. The field values are 1, 2, and 3. After you run the following command to redirect the standard output of the query result to the destination handle, the output contains only field values.
odpscmd -e "set odps.sql.select.output.format={"needHeader":false,"fieldDelim":""};select * from noheader;" >/temp/test.txt
-- Returned results: 
1
2
3

Version updates

The following table describes the latest version updates of the MaxCompute client. For more information, click the URL of a specific version.

Version: Change type Description
v0.37.4-public Enhanced feature
  • The help information for commands is optimized.
  • The desc extended partition command returns more information about properties of partitions.
v0.36.0-public New feature An external project can be created to connect to Data Lake Formation (DLF). This implements the Alibaba Cloud LakeHouse feature.
Fixed issue The issue that caused the nanosecond part of data of the TIMESTAMP type to be incorrectly processed is fixed.