Use Tunnel commands to upload and download data - MaxCompute

Subcommands

tunnel <subcommand> [options] [args]

Subcommand	Alias	Description
`upload`	`u`	Upload local data to a MaxCompute table
`download`	`d`	Download data from a MaxCompute table to a local file
`upsert`	`us`	Write data using UPDATE and INSERT semantics (Transaction Table 2.0 only)
`resume`	`r`	Resume an interrupted upload session
`show`	`s`	Display transfer history and logs
`purge`	`p`	Clear the session directory
`help`	`h`	Show help information

How the commands relate: Use upload, download, or upsert to transfer data. If an upload is interrupted, use resume to continue from where it stopped. Use show to inspect transfer history, and purge to clean up old session logs.

Upload

Uploads local data to a MaxCompute table in append mode. Existing data in the table is not overwritten — both existing and newly uploaded data coexist in the table.

Each upload operation targets one table or one partition. For partitioned tables, specify the lowest-level partition.

Syntax

tunnel upload [options] <path> <[project.]table[/partition]>

Examples:

# Upload log.txt to a two-level partitioned table (file is in the MaxCompute client bin directory)
tunnel upload log.txt test_project.test_table/p1="b1",p2="b2"

# Upload from an absolute path (Windows)
tunnel upload D:\test\log.txt test_project.test_table/p1="b1",p2="b2"

# Scan the file before uploading to validate against the table schema
tunnel upload log.txt test_table --scan=true

# Overwrite a partition instead of appending
tunnel upload -ow true data.txt sale_detail/sale_date=201312,region=hangzhou

On macOS, path must be an absolute path, such as /Users/username/MaxCompute/bin/file.txt.

Upload supports TXT files by default. To upload a CSV file, set -cf true and use the latest MaxCompute client. File splitting is not supported in CSV format.

Parameters

Required

Parameter	Description
`path`	Path to the local data file. Files in the MaxCompute client's `bin` directory can be referenced by filename only; files in other directories require the full path.
`[project.]table[/partition]`	Target table. Include the project name if the table is in a different project. Specify the lowest-level partition for partitioned tables.

Optional — Format

Option	Default	Description
`-c`, `-charset`	(not set)	File character encoding. When not set, raw data is uploaded without conversion. Default encoding is UTF-8.
`-cf`, `-csv-format`	`false`	Set to `true` to upload a CSV file. When enabled, file splitting is not supported.
`-fd`, `-field-delimiter`	`,`	Column delimiter. Supports multiple characters and Unicode escapes such as `\u0001`. Cannot contain the row delimiter.
`-rd`, `-record-delimiter`	`\r\n`	Row delimiter. Supports multiple characters, Unicode escapes, and the escape sequences `\r`, `\n`, `\t`.
`-h`, `-header`	`false`	Set to `true` if the file has a header row. The header row is skipped during upload.
`-ni`, `-null-indicator`	`""` (empty string)	String that represents NULL in the source file. When not set, empty strings are treated as NULL.
`-dfp`, `-date-format-pattern`	`yyyy-MM-dd HH:mm:ss`	DATETIME format pattern. Use `yyyy-MM-dd HH:mm:ss.SSS` for millisecond precision. See Data type editions.

Optional — Error handling

Option	Default	Description
`-s`, `-scan`	`true`	Controls pre-upload file scanning. `true`: scan first, then upload if valid. `false`: skip scanning and upload directly. `only`: scan without uploading.
`-ss`, `-strict-schema`	`true`	Set to `false` to allow lenient schema matching: extra columns are discarded, and missing columns are filled with NULL.
`-dbr`, `-discard-bad-records`	`false`	Set to `true` to skip bad records (extra columns, missing columns, or type mismatches) instead of failing. When `false`, the upload stops and returns an error on the first bad record detected. Use with `-mbr` to set the maximum number of bad records to skip.
`-mbr`, `-max-bad-records`	`1000`	Maximum number of bad records to skip. Effective only when `-dbr true` is set. If the number of bad records exceeds this limit, the upload stops.

Optional — Performance

Option	Default	Description
`-t`, `-threads`	`1`	Number of concurrent upload threads. Increase this value to improve throughput for large files.
`-bs`, `-block-size`	`100` MiB	Size of each uploaded block. 1 MiB = 1,048,576 bytes.
`-cp`, `-compress`	`true`	Compress data before uploading to reduce network usage.

Optional — Session

Option	Default	Description
`-acp`, `-auto-create-partition`	`false`	Automatically create the target partition if it does not exist.
`-ow`, `-overwrite`	`false`	Set to `true` to overwrite the target table or partition instead of appending.
`-sd`, `-session-dir`	Client plugin directory	Directory used to store session state.
`-te`, `-tunnel_endpoint`	(not set)	Tunnel endpoint. See Endpoints.
`-tz`, `-time-zone`	Local time zone	Time zone for DATETIME data. Default is the local time zone (for example, `Asia/Shanghai`). See Time zones.
`-time`	`false`	Track and display elapsed time after the upload completes.

Example: overwrite a partition

Create a partitioned table and add an initial partition:

CREATE TABLE IF NOT EXISTS sale_detail(
      shop_name     STRING,
      customer_id   STRING,
      total_price   DOUBLE)
PARTITIONED BY (sale_date STRING, region STRING);

ALTER TABLE sale_detail ADD PARTITION (sale_date='201312', region='hangzhou');

Upload initial data (data.txt contains shopx,x_id,100 and shopy,y_id,200):

tunnel upload d:\data.txt sale_detail/sale_date=201312,region=hangzhou

Query result:

+------------+-------------+-------------+------------+------------+
| shop_name  | customer_id | total_price | sale_date  | region     |
+------------+-------------+-------------+------------+------------+
| shopx      | x_id        | 100.0       | 201312     | hangzhou   |
| shopy      | y_id        | 200.0       | 201312     | hangzhou   |
+------------+-------------+-------------+------------+------------+

Update data.txt to contain shopx,x_id,300 and shopy,y_id,400, then overwrite the partition:

tunnel upload -ow true data.txt sale_detail/sale_date=201312,region=hangzhou

Query result after overwrite:

+------------+-------------+-------------+------------+------------+
| shop_name  | customer_id | total_price | sale_date  | region     |
+------------+-------------+-------------+------------+------------+
| shopx      | x_id        | 300.0       | 201312     | hangzhou   |
| shopy      | y_id        | 400.0       | 201312     | hangzhou   |
+------------+-------------+-------------+------------+------------+

Download

Downloads data from a MaxCompute table or a specific instance's execution result to a local file.

Each download operation targets one table or one partition. For partitioned tables, specify the lowest-level partition.

Prerequisites

Before downloading, make sure you have the Download permission on the target table. If not, contact the project owner or a user with the Super_Administrator role to grant access. See Policy-based access control.

Syntax

# Download from a table or partition
tunnel download [options] <[project.]table[/partition]> <path>

# Download an instance's execution result
tunnel download [options] instance://<[project/]instance_id> <path>

Examples:

# Download a two-level partitioned table to a local file
tunnel download test_project.test_table/p1="b1",p2="b2" test_table.txt

# Download to a specific directory (Windows)
tunnel download test_project.test_table/p1="b1",p2="b2" D:\test\test_table.txt

# Download the execution result of a specific instance
tunnel download instance://test_project/test_instance log.txt

Download supports TXT files by default. To download a CSV file, set -cf true and use the latest MaxCompute client. When -cf true is set, the delimiter is fixed to comma (,); the -fd setting is ignored.

Parameters

Required

Parameter	Description
`path`	Path to save the downloaded file. Files in the MaxCompute client's `bin` directory can be referenced by filename only; files in other directories require the full path.
`[project.]table[/partition]`	Source table. Include the project name if the table is in a different project. Specify the lowest-level partition for partitioned tables.
`[project/]instance_id`	Instance ID for downloading an instance's execution result.

Optional — Format

Option	Default	Description
`-c`, `-charset`	(not set)	File character encoding. When not set, raw data is downloaded without conversion.
`-cf`, `-csv-format`	`false`	Set to `true` to download as a CSV file. When enabled, the delimiter is fixed to `,`; the `-fd` setting is ignored.
`-ci`, `-columns-index`	(all columns)	Comma-separated list of column indexes to download (0-based).
`-cn`, `-columns-name`	(all columns)	Comma-separated list of column names to download.
`-fd`, `-field-delimiter`	`,`	Column delimiter. Supports multiple characters and Unicode escapes such as `\u0001`. Ignored when `-cf true` is set.
`-rd`, `-record-delimiter`	`\r\n`	Row delimiter. Supports multiple characters, Unicode escapes, and the escape sequences `\r`, `\n`, `\t`.
`-h`, `-header`	`false`	Set to `true` to include a header row in the downloaded file. Cannot be used with `-t` greater than `1`.
`-ni`, `-null-indicator`	`""` (empty string)	String used to represent NULL values in the output file. When not set, NULL values appear as empty strings.
`-dfp`, `-date-format-pattern`	`yyyy-MM-dd HH:mm:ss`	DATETIME format pattern.
`-e`, `-exponential`	`false`	Set to `true` to use exponential notation for DOUBLE values. When `false`, up to 20 digits are retained.
`-limit`	(not set)	Maximum number of records to download. When not set, all records are downloaded.

Optional — Performance

Option	Default	Description
`-t`, `-threads`	`1`	Number of concurrent download threads. Cannot be greater than `1` when `-h true` is set.
`-cp`, `-compress`	`true`	Compress data during download to reduce network usage.

Optional — Session

Option	Default	Description
`-sd`, `-session-dir`	Client plugin directory	Directory used to store session state.
`-te`, `-tunnel_endpoint`	(not set)	Tunnel endpoint. See Endpoints.
`-tz`, `-time-zone`	Local time zone	Time zone for DATETIME data (for example, `Asia/Shanghai`). See Time zones.
`-time`	`false`	Track and display elapsed time after the download completes.

Upsert

Writes data to a Transaction Table 2.0 table using UPDATE and INSERT semantics: inserts a new record if no matching record exists, or updates the existing record if one is found.

Prerequisites

MaxCompute client (odpscmd) version V0.47 or later
The target table must be a Transaction Table 2.0 table

Syntax

tunnel upsert [options] <path> <[project.]table[/partition]>

Example:

tunnel upsert log.txt test_project.test_table/p1="b1",p2="b2"

Upsert supports TXT files by default. To upsert from a CSV file, set -cf true and use the latest MaxCompute client.

Parameters

Required

Parameter	Description
`path`	Path to the local data file. Files in the MaxCompute client's `bin` directory can be referenced by filename only; files in other directories require the full path. On macOS, this must be an absolute path.
`[project.]table[/partition]`	Target table. Include the project name if the table is in a different project. Specify the lowest-level partition for partitioned tables.

Optional — Format

Option	Default	Description
`-c`, `-charset`	(not set)	File character encoding. When not set, raw data is uploaded without conversion.
`-cf`, `-csv-format`	`false`	Set to `true` to upsert from a CSV file.
`-fd`, `-field-delimiter`	`,`	Column delimiter. Supports multiple characters and Unicode escapes. Cannot contain the row delimiter.
`-rd`, `-record-delimiter`	`\n`	Row delimiter. Default for upsert is `\n`, which differs from upload (`\r\n`).
`-h`, `-header`	`false`	Set to `true` if the file has a header row. The header row is skipped.
`-ni`, `-null-indicator`	`""` (empty string)	String that represents NULL in the source file. When not set, empty strings are treated as NULL.
`-dfp`, `-date-format-pattern`	`yyyy-MM-dd HH:mm:ss`	DATETIME format pattern. Use `yyyy-MM-dd HH:mm:ss.SSS` for millisecond precision. See Data type editions.

Optional — Error handling

Option	Default	Description
`-ss`, `-strict-schema`	`true`	Set to `false` to discard extra columns and fill missing columns with NULL.
`-dbr`, `-discard-bad-records`	`false`	Set to `true` to skip bad records instead of failing. Use with `-mbr` to limit the number of skipped records.
`-mbr`, `-max-bad-records`	`1000`	Maximum number of bad records to skip. Effective only when `-dbr true` is set. If the number of bad records exceeds this limit, the operation stops.

Optional — Performance

Option	Default	Description
`-bs`, `-block-size`	`100` MiB	Size of each uploaded block. 1 MiB = 1,048,576 bytes.
`-cp`, `-compress`	`true`	Compress data before uploading to reduce network usage.

Optional — Session

Option	Default	Description
`-acp`, `-auto-create-partition`	`false`	Automatically create the target partition if it does not exist.
`-qn`, `-quota_name`	(not set)	Tunnel quota to use for accessing MaxCompute. View quota names in the MaxCompute console under Workspace > Quotas. See Manage quotas in the new MaxCompute console.
`-sd`, `-session-dir`	Client plugin directory	Directory used to store session state.
`-te`, `-tunnel_endpoint`	(not set)	Tunnel endpoint. See Endpoints.
`-tz`, `-time-zone`	Local time zone	Time zone for DATETIME data (for example, `Asia/Shanghai`). See Time zones.
`-time`	`false`	Track and display elapsed time after the operation completes.

Resume

Resumes an interrupted upload session. Only uploads can be resumed; downloads cannot.

Syntax

tunnel resume [session_id] [-force]

Parameter	Description
`session_id`	Required. The ID of the failed upload session. Retrieve it from `tunnel show history`.
`-f`, `-force`	Force-resume the session without confirmation.

Example

odps@ project_name>tunnel resume 20150610xxxxxxxxxxx70a002ec60c -force;
start resume
20150610xxxxxxxxxxx70a002ec60c
Upload session: 20150610xxxxxxxxxxx70a002ec60c
Start upload:d:\data.txt
Resume 1 blocks
2015-06-10 16:46:42     upload block: '1'
2015-06-10 16:46:42     upload block complete, blockid=1
upload complete, average speed is 0 KB/s
OK

Show

Displays transfer history and logs.

Syntax

# Show transfer history
tunnel show history [-n <number>]

# Show the log of the most recent transfer
tunnel show log

-n <number>: Number of history entries to display. Up to 500 entries are stored by default.

Session status values

Each history entry shows a session ID, status, and the command that was run. The status values are:

Status	Description
`success`	The transfer completed without errors.
`bad`	The transfer completed, but some records were skipped (requires `-dbr true`).
`failed`	The transfer failed and no data was written.

Examples

Show all stored history:

tunnel show history

Output:

20230505xxxxxxxxxxxxxx0b0d5b3c  bad     'upload d:\data.txt sale_detail/sale_date=201312,region=hangzhou -dbr true -time true'
20230505xxxxxxxxxxxxxx0ad720a3  failed  'upload d:\data.txt sale_detail/sale_date=201312,region=hangzhou -time true'
20230505xxxxxxxxxxxxxx0ad5ca68  bad     'upload d:\data.txt sale_detail/sale_date=201312,region=hangzhou -dbr true'
......

Show the last 5 entries:

tunnel show history -n 5

Output:

20230505xxxxxxxxxxxxxx0aa48c4b  success 'download sale_detail/sale_date=201312,region=hangzhou result.txt'
20230505xxxxxxxxxxxxxx0aa6165c  success 'download sale_detail/sale_date=201312,region=hangzhou result.txt'
20230505xxxxxxxxxxxxxx0af11472  failed  'upload d:\data.txt sale_detail/sale_date=201312,region=hangzhou -s false'
20230505xxxxxxxxxxxxxx0b464374  success 'upload d:\data.txt sale_detail/sale_date=201312,region=hangzhou -s false'
20230505xxxxxxxxxxxxxx02dbb6bd  failed  'upload d:\data.txt sale_detail/sale_date="201312",region="hangzhou" -s false'

Purge

Clears session history from the session directory.

Syntax

tunnel purge [n]

n: Number of days of history to clear. Default: 3 (clears logs from the last 3 days).

Example: Clear the last 5 days of logs:

tunnel purge 5

Usage notes

Supported data types

Data type	Upload behavior	Download behavior
STRING	Max length: 8 MB	Max length: 8 MB
BOOLEAN	Accepts `True`, `False`, `0`, or `1` (case-insensitive)	Returns `True` or `False`
BIGINT	Range: -9,223,372,036,854,775,807 to 9,223,372,036,854,775,807	Same range
DOUBLE	A 16-digit number. Expressed in scientific notation.	Expressed as a number (up to 20 digits). Positive infinity: `Infinity`. Negative infinity: `-Infinity`. Max: `1.7976931348623157E308`. Min: `4.9E-324`.
DATETIME	Default time zone: GMT+8. Use `-dfp` to set the format. See Data type editions.	—

Supported DATETIME format patterns:

Pattern	Example
`yyyyMMddHHmmss`	`20140209101000`
`yyyy-MM-dd HH:mm:ss` (default)	`2014-02-09 10:10:00`
`MM/dd/yyyy`	`09/01/2014`

NULL values

By default, empty strings are treated as NULL. To use a custom null indicator:

tunnel upload log.txt test_table -ni "NULL"

File encoding

Default encoding is UTF-8. To specify a different encoding:

tunnel upload log.txt test_table -c "gbk"

Delimiters

Both row and column delimiters support multiple characters. The column delimiter cannot contain the row delimiter. Supported escape sequences: \r, \n, \t.

tunnel upload log.txt test_table -fd "||" -rd "\r\n"

Subcommands

Upload

Syntax

Parameters

Required

Optional — Format

Optional — Error handling

Optional — Performance

Optional — Session

Example: overwrite a partition

Download

Prerequisites

Syntax

Parameters

Required

Optional — Format

Optional — Performance

Optional — Session

Upsert

Prerequisites

Syntax

Parameters

Required

Optional — Format

Optional — Error handling

Optional — Performance

Optional — Session

Resume

Syntax

Example

Show

Syntax

Session status values

Examples

Purge

Syntax

Usage notes

Supported data types

NULL values

File encoding

Delimiters

What's next