Notes and examples for using odpscmd in Cloud Shell - MaxCompute

Prerequisites

If you log in to odpscmd in Cloud Shell as a RAM user, the RAM user must have the AliyunCloudShellFullAccess permission. For more information, see Resource Access Management (RAM).

Limitations

Note

Cloud Shell is a web-based command-line tool provided by Alibaba Cloud. MaxCompute provides a cloud-based client environment based on Cloud Shell, allowing you to use odpscmd without manual installation. For more information about Cloud Shell, see What is Cloud Shell?.

Cloud Shell limitations

Limit	Description
Virtual machines	Only one virtual machine runs at a time. All session windows connect to it automatically.
Session windows	You can open up to five session windows.
Inactivity timeout	After 30 minutes of inactivity or when all session windows close, the session stops. The virtual machine is destroyed 15 minutes later. Restarting Cloud Shell creates a new virtual machine.
Expiration	Each virtual machine has a 1-hour lifetime. After expiration, Cloud Shell destroys it immediately. Restarting Cloud Shell creates a new virtual machine.
File storage	Cloud Shell provides 10 GB of temporary storage in the /home/shell folder. This space resets when the virtual machine is destroyed. For persistent storage, mount a bucket. The bucket loads automatically each time Cloud Shell starts.
Prohibited use	Compute-intensive, network-intensive, long-running tasks, and malicious processes are prohibited. Violations may result in your session being stopped or disabled without notice.

Region limitations

Currently, you can use Cloud Shell to connect to MaxCompute only in the following regions: China (Hangzhou), China (Shanghai), China (Beijing), and China (Shenzhen).

Usage notes

Regardless of the region where you start odpscmd for MaxCompute, the Cloud Shell service and its temporary storage reside in the China (Shanghai) region by default. If your project is in a different region, using this feature results in cross-region data transfer, which may pose a data security risk. Use with caution.

Enter the odpscmd environment

Log in to the MaxCompute console and select a region in the upper-left corner.
On the Workspace > Projects page, click Command Line in the top-right corner.

In the Choose Project dialog box, select the project that you want to manage and click OK.

Note

If a prompt about mounting a NAS file system appears, you can mount one as needed. For more information, see (Optional) Mount a NAS file system.

After odpscmd loads in Cloud Shell, the output is similar to the following:

inflating: odpscmd_pub/plugins/cupid/lib/jackson-core-2.9.5.jar
inflating: odpscmd_pub/plugins/cupid/lib/odps-console-cupid-public-0.43.1-public.jar
inflating: odpscmd_pub/plugins/cupid/plugin.ini
inflating: odpscmd_pub/plugins/volume2/lib/odps-console-volume2-0.43.1-public.jar
inflating: odpscmd_pub/plugins/volume2/lib/odps-sdk-volumefs-0.43.1-public.jar
inflating: odpscmd_pub/plugins/volume2/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar
inflating: odpscmd_pub/plugins/volume2/lib/hadoop-annotations-3.3.3.jar
inflating: odpscmd_pub/plugins/volume2/lib/hadoop-auth-3.3.3.jar
inflating: odpscmd_pub/plugins/volume2/lib/hadoop-common-3.3.3.jar
/home/shell
Aliyun ODPS Command Line Tool
Version 0.43.1-public
@Copyright 2020 Alibaba Cloud Computing Co., Ltd. All rights reserved.
Connecting to http://service.odps.aliyun.com/api, project: xxx
Endpoint: http://service.odps.aliyun.com/api
Project: xxx
Schema: default
Quota: default in region N/A
Timezone: Asia/Shanghai
Connected!
xxx (default)>

(Optional) Mount NAS

Cloud Shell provides 10 GB of temporary storage that resets when the virtual machine is destroyed. For persistent storage, you can mount a NAS file system, which Cloud Shell automatically loads each time you start a session. The following steps describe how to mount a NAS file system.

In the MaxCompute Cloud CMD window, click the icon and select Mount Storage Space to bind a NAS file system.

Note

Mounting a NAS file system incurs NAS storage fees. Unbind the storage space when you no longer need it. For more information about NAS storage fees, see Billing overview.

Use odpscmd in Cloud Shell

odpscmd in Cloud Shell supports all SQL commands that the local client can run. The following sections demonstrate how to use odpscmd by running SQL commands and transferring data.

Run SQL commands

Create the table result_table1. The following is a sample command.

CREATE TABLE IF NOT EXISTS result_table1 (
 education STRING comment 'Education level',
 num BIGINT comment 'Number of people');

After the table is created, run the following command to view it:

SHOW TABLES;

Data transfer

Upload a file

Click the icon and select Upload to transfer a local file. This topic uses the result.txt file as an example.

After the upload is complete, press Ctrl+c to exit the Cloud Shell (odpscmd) environment, and then run the ll command to check if the upload was successful and view the file path.

shell@Alicloud:~$ ll
total 24
drwxr-xr-x 3 shell shell 4096 Jul 19 08:21 ./
drwxr-xr-x 1 root  root  4096 Jul 19 08:00 ../
-rw------- 1 shell shell  292 Jul 19 08:22 .bash_history
drwx------ 3 shell shell 4096 Jul 19 08:00 .config/
-rw-r--r-- 1 shell shell  325 Jul 19 08:15 .odpscmd.history
-rw-r--r-- 1 shell shell   30 Jul 19 08:21 result.txt
-rw-r--r-- 1 shell shell    0 Jul 19 08:09 whoami
shell@Alicloud:~$

After confirming the file is present, run the following command to return to the odpscmd environment:

cloudshell-odpscmd -p <project.name>

Upload the result.txt file to MaxCompute. The following is a sample command.
```
tunnel upload result.txt result_table1;
```

Download a file
1. The following is a sample command to download the MaxCompute table result_table1 to Cloud Shell (odpscmd).
```
Tunnel download result_table1 result_table1.txt;
```
2. Click the icon and select Download to save the result_table1.txt file to your local computer.