Tunnel is a command-line tool for loading large volumes of data into MaxCompute tables in bulk. It is designed for offline computing scenarios. This tutorial walks you through creating a destination table and uploading a local CSV log file using the odpscmd client.
Prerequisites
Before you begin, ensure that you have:
-
The odpscmd client installed and configured. For more information, see Install and configure the MaxCompute client.
-
A local log file ready to upload. This tutorial uses loghub.csv as the example data file.
Step 1: Create the destination table
In the odpscmd client, run the following commands to create a table named loghub.
The first command enables MaxCompute V2.0 data types. Submit it together with the CREATE TABLE statement in the same session.
set odps.sql.type.system.odps2=true;
-- Create a table named loghub.
CREATE TABLE loghub
(
client_ip STRING,
receive_time STRING,
topic STRING,
id STRING,
name VARCHAR(32),
salenum STRING
);Step 2: Upload the log data
Run the following command to upload the log file to the loghub table. Adjust the file path to match your operating system.
Windows
Tunnel u D:\loghub.csv loghub;Linux or macOS
Tunnel u /path/to/loghub.csv loghub;| Parameter | Description |
|---|---|
D:\loghub.csv or /path/to/loghub.csv |
Path to the local log data file |
loghub |
Name of the destination MaxCompute table |
Wildcards and regular expressions are not supported for Tunnel-based uploads. Specify the exact file path.
Step 3: Verify the upload
Run the following query to check whether the data was loaded into the table:
SELECT * FROM loghub;If the upload succeeded, the query returns the uploaded rows as shown below.
xxx >SELECT * FROM loghub;
ID = 20191224054915??erwinv21
Job Queueing...
Summary:
+--------------+--------------+-------+------+---------+----------+
| client_ip | receive_time | topic | id | name | salenum |
+--------------+--------------+-------+------+---------+----------+
| 47.95.63.148 | 1572925369 | NULL | 62 | ????13 | 2000 |
| 47.95.63.148 | 1572925369 | NULL | 61 | ????12 | 606 |
| 47.95.63.148 | 1572925369 | NULL | 60 | ????11 | 909 |
| 47.95.63.148 | 1572925369 | NULL | 32 | ????3 | 2000 |
| 47.95.63.148 | 1572925369 | NULL | 31 | ????2 | 606 |
| 47.95.63.148 | 1572925369 | NULL | 30 | ????1 | 909 |
| 47.95.63.148 | 1572925369 | NULL | 31 | ????2 | 3000 |
| 47.95.63.148 | 1572925369 | NULL | 30 | ????1 | 1000 |
| 47.95.63.148 | 1572925369 | NULL | 20 | ??????? | 21 |
| 47.95.63.148 | 1572925369 | NULL | 6 | ???? | 40 |
| 47.95.63.148 | 1572925369 | NULL | 5 | ???? | 20 |
| 47.95.63.148 | 1572925369 | NULL | 2 | ??? | 260 |
| 47.95.63.148 | 1572925369 | NULL | 2 | ??? | 200 |
| 47.95.63.148 | 1572925369 | NULL | 1 | C?? | 100 |
| 47.95.63.148 | 1572925369 | NULL | 9 | vivo | 400 |
| 47.95.63.148 | 1572925369 | NULL | 8 | oppo | 200 |
| 47.95.63.148 | 1572925369 | NULL | 7 | ??? | 30 |
| 47.95.63.148 | 1572925369 | NULL | 12 | ?????? | 0 |
| 47.95.63.148 | 1572925369 | NULL | 11 | ????? | 600 |
| 47.95.63.148 | 1572925369 | NULL | 10 | ??? | 90 |
+--------------+--------------+-------+------+---------+----------+
20 records (at most 10000 supported) fetched by instance tunnel.If the query returns no rows or an error, check the following:
-
The file path in Step 2 matches the actual location of your CSV file.
-
The table schema matches the columns in your CSV file.
-
The odpscmd client is connected to the correct MaxCompute project.
Next steps
For a full reference of Tunnel commands, including features and how to use Tunnel commands to upload or download data, see Tunnel usage notes.