Data upload

The Data Upload feature in DataWorks lets you upload data from various sources, such as local files, spreadsheets from Data Analysis, OSS files, and HTTP files, to engines like MaxCompute, EMR Hive, Hologres, and StarRocks for analysis and management. This convenient data transfer service helps you quickly drive your business with data. This topic describes how to use the Data Upload feature to upload data.

Notes

If your operations involve cross-border data transfers (for example, transferring data from the Chinese mainland to a location outside the Chinese mainland, or between different countries or regions), you must first read and understand the related compliance statement. Otherwise, the upload may fail, and you could face legal liability.
Before you upload data, we recommend using English headers in your source file. Chinese headers can cause parsing failures and may cause the upload to fail.

Limitations

Resource group limitations: The Data Upload feature requires you to specify a resource group for scheduling and a resource group for Data Integration.
- This feature supports only a serverless resource group (recommended), an exclusive resource group for scheduling, and an exclusive resource group for Data Integration. You must configure a resource group for scheduling and a resource group for Data Integration for the respective engine in Data Analysis > More > System Management.
- The selected resource group must be associated with the DataWorks workspace that contains the destination table. You must also ensure network connectivity between the selected resource group and the data source used for the upload task.
  Note
  - To configure the resource group for an engine in Data Analysis, see System management.
  - To establish network connectivity between a data source and a resource group, see Network connectivity solutions.
  - To associate an exclusive resource group with its designated workspace, see Use exclusive resource groups for scheduling and Use an exclusive resource group for Data Integration.
Permissions required for data upload:
- MaxCompute tables (ODPS tables): You only need to be the table owner. You do not need the developer, analyst, or operator role in the DataWorks workspace.
- Non-MaxCompute tables (such as Hologres, EMR Hive, and StarRocks): In addition to being the table owner, you must have a role assigned in the DataWorks workspace.
  - Tables in the development environment: You must have the developer role.
  - Tables in the production environment: You must have the operator role.
Table type limitations: You can only upload data to internal tables or tables under the default catalog (applicable to StarRocks).

Billing

Data upload incurs the following fees:

Data transfer fees.
If new tables are created, compute and storage fees are charged.

All fees are charged by the engine. For more information, see MaxCompute billing, Hologres billing, E-MapReduce billing, and EMR Serverless StarRocks billing.

Go to the Data Upload page

Go to the Upload and Download page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Integration > Data Upload and Download. On the page that appears, click Go to Data Upload and Download.
Click the icon in the left-side navigation pane to go to the Upload Data page.
Click Upload Data and follow the on-screen instructions to upload the data.

Select data to upload

You can upload data from local files, spreadsheets, OSS, and HTTP files. Select a data source based on your business requirements.

Note

When you upload a file, specify whether to skip dirty data as needed.

Yes: If dirty data is detected, the system automatically skips it and continues the upload.
No: If dirty data is detected, the system does not skip it, and the upload is interrupted.

Local file

If the data to upload is stored in a local file, select this method.

Set Data Source to Local File.
Specify Data to Be Uploaded: Follow the on-screen instructions to drag the local file to the Select File area.
Note
- The following formats are supported: CSV, XLS, XLSX, and JSON. The maximum data size for a CSV file is 5GB. The maximum data size for other file types is 100MB.
- By default, only the first sheet of a file is uploaded. To upload data from multiple sheets in a file, you must create a separate file for each sheet and place it as the first sheet.
- Uploading files in SQL format is not supported.

Spreadsheet

If the data to upload is stored in a DataWorks Data Analysis spreadsheet, select this method.

Set Data Source to Workbook.
Specify Data to Be Uploaded:
1. Select the spreadsheet to upload from the drop-down list next to Select File.
2. If the spreadsheet does not exist, click the Create button to create one. You can also Create a workbook through the Data Analysis module and import data.

OSS

If the data to upload is stored in OSS, select this method.

Prerequisites:

You have created an OSS bucket and stored the data to upload in the OSS bucket. You can then upload OSS data to the corresponding data source.
To avoid permission issues, grant the Alibaba Cloud account that performs the upload operation the permission to access the target bucket through Overview of permissions and access control before you upload data.

Upload procedure:

Set Data Source to OSS.
Specify Data to Be Uploaded:
1. From the Select Bucket drop-down list, select the target OSS bucket that stores the data to upload.
  
  Note
  You can only upload data from buckets in the same region as the current DataWorks workspace.
2. In the Select File area, select the file to upload.
  
  Note
  Only files in CSV, XLS, XLSX, and JSON formats are supported.

HTTP file

If the data to upload is stored in an HTTP file, select this method.

Set Data Source to HTTP File.

Specify Data to Be Uploaded:

Parameter	Description
File URL	Select the URL of the file. Note HTTP and HTTPS file URLs are supported.
File Type	The file type is automatically detected based on the file you upload. The following file formats are supported: `CSV`, `XLS`, and `XLSX`. The maximum data size for a `CSV` file is 5 GB. The maximum data size for other file types is 50 MB.
Request Method	Three methods are supported: GET, POST, and PUT. We recommend using GET to retrieve data, but you can select a method based on the allowed request methods that you defined.
Advanced Parameters	You can also configure Request Header and Request Body in Advanced Parameters based on your business requirements.

Configure the destination table

In the Configure Destination Table section, select a Compute Engine for the data upload and configure the relevant parameters based on the selected engine.

Important

When you configure the destination table, make sure that you distinguish between PROD (production environment) and DEV (development environment) when selecting a data source. If you select the wrong environment, the data will be uploaded to an unintended environment.

MaxCompute

To upload data to an internal table in MaxCompute, configure the parameters as described in the following table.

Parameter		Description
MaxCompute project name		Select a MaxCompute data source associated with the current region. If the data source is not found, you can Associate a MaxCompute compute resource to the current workspace to generate a data source with the same name.
Destination table		You can select Existing Table or Create Table.
Destination Table > Existing Table	Select destination table	The table to which you want to upload data. You can search for a table by keyword. Note You can only upload data to a table that you own (you are the table owner). For more information, see Usage limits.
Destination Table > Existing Table	Upload method	Select a method for adding data to the destination table. Clear Table Data First: Clears all existing data in the destination table and then imports data into the corresponding mapped columns. Append: Appends the uploaded data to the corresponding mapped columns in the destination table.
Destination Table > Create Table	Table name	Specify a custom name for the new table. Note When creating a table in MaxCompute, the system uses the MaxCompute account information configured in the DataWorks compute resource to create the table in the corresponding MaxCompute project.
	Table type	Select Non-partitioned Table or Partitioned Table as needed. If you select a partitioned table, you must specify the partition column and its values.
	Lifecycle	Specify the lifecycle of the table. After the lifecycle expires, the table may become unavailable. For more information about table lifecycle, see Lifecycle and Lifecycle management operations.

EMR Hive

To upload data to an internal table in EMR Hive, configure the parameters as described in the following table.

Parameter	Description
Data source	Select an EMR Hive data source (Alibaba Cloud instance mode) associated with the current workspace in the current region.
Destination table	You can only upload data to an Existing Table.
Select destination table	The table to which you want to upload data. You can search for a table by keyword. Note If the destination table does not exist, follow the on-screen instructions to go to the Table management page in Data Studio to create a table. You can only upload data to a table that you own (you are the table owner). For more information, see Usage limits.
Upload method	Select a method for adding data to the destination table. Clear Table Data First: Clears all existing data in the destination table and then imports data into the corresponding mapped columns. Append: Appends the uploaded data to the corresponding mapped columns in the destination table.

Hologres

To upload data to an internal table in Hologres, configure the parameters as described in the following table.

Parameter	Description
Data source	Select a Hologres data source associated with the current workspace in the current region. If the data source is not found, you can Associate a Hologres computing resource to the current workspace to generate a data source with the same name.
Destination table	You can only upload data to an Existing Table.
Select destination table	The table to which you want to upload data. You can search for a table by keyword. Note If the destination table does not exist, follow the on-screen instructions to go to the Hologres console to create a table. You can only upload data to a table that you own (you are the table owner). For more information, see Usage limits.
Upload method	Select a method for adding data to the destination table. Clear Table Data First: Clears all existing data in the destination table and then imports data into the corresponding mapped columns. Append: Appends the uploaded data to the corresponding mapped columns in the destination table.
Primary key conflict policy	If uploading data causes a primary key conflict in the destination table, you can use the following policies. Ignore: Ignores the uploaded data. The data in the destination table is not updated. Update (replace): The uploaded data fully overwrites the existing data in the destination table. Columns without column mapping are forcibly set to NULL. update: The uploaded data overwrites the existing data in the destination table, but only the columns with column mapping are updated.

StarRocks

To upload data to a table under the StarRocks default catalog, configure the parameters as described in the following table.

Parameter	Description
Data source	Select a StarRocks data source associated with the current workspace in the current region.
Destination table	You can only upload data to an Existing Table.
Select destination table	The table to which you want to upload data. You can search for a table by keyword. Note If the destination table does not exist, follow the on-screen instructions to go to the EMR Serverless StarRocks instance page to create a table. You can only upload data to a table that you own (you are the table owner). For more information, see Usage limits.
Upload method	Select a method for adding data to the destination table. Clear Table Data First: Clears all existing data in the destination table and then imports data into the corresponding mapped columns. Append: Appends the uploaded data to the corresponding mapped columns in the destination table.
Advanced parameters	You can configure Stream Load request parameters.

Preview data to upload

After you configure the destination table, you can adjust the file encoding and column mapping based on the data preview.

Note

Currently, only the first 20 rows of data can be previewed.

File Encoding Format: If the data contains garbled characters, you can switch the encoding format. The following encoding formats are supported: UTF-8, GB18030, Big5, UTF-16LE, and UTF-16BE.
Preview data and configure destination table columns:
- Upload data to an existing table: You must configure the mapping between the columns in the source file and the columns in the destination table. Data can be uploaded only after the mapping is configured. The mapping methods include Mapping by Column Name and Mapping by Order. After the mapping is configured, you can also customize the column names in the destination table.
  Note
  - If the data to upload has no mapping to a destination table column, the data is grayed out and will not be uploaded.
  - Duplicate mappings between the data to upload and destination table columns are not allowed.
  - Column names and column types cannot be empty. Otherwise, data cannot be uploaded.
- Upload data to a new table: You can use Intelligent Field Generation to automatically populate column information, or manually modify the column information.
  Note
  - Column names and column types cannot be empty. Otherwise, data cannot be uploaded.
  - EMR Hive, Hologres, and StarRocks engines do not support creating new tables during data upload.
Ignore First Row: Specifies whether to upload the first row of the file data (usually column names) to the destination table.
- Selected: If the first row contains column names, it is not uploaded to the destination table.
- Cleared: If the first row contains data, it is uploaded to the destination table.

Upload data

After you preview the data, click the Upload Data button in the lower-left corner to upload the data.

Next steps

After the data is uploaded, click the icon in the left-side navigation pane to go to the Upload Data page. Find the created data upload task and perform the following operations as needed:

Continue uploading: Click Resume in the Operation column to upload the data again.
Query data: Click Query Data in the Operation column to query and analyze the data.
View upload details: Click the target Table Name to go to Data Map and view detailed information about the destination table. For more information, see Metadata retrieval.

Appendix: Compliance statement for cross-border data upload

Important

If your operations involve cross-border data transfers (for example, transferring data from the Chinese mainland to a location outside the Chinese mainland, or between different countries or regions), you must first read and understand the related compliance statement. Otherwise, the upload may fail, and you could face legal liability.

Cross-border data operations will transfer your cloud business data to the region you select or the region where the product is deployed. You must ensure that the operations comply with the following requirements:

You have the permission to process the relevant cloud business data.
You have adopted sufficient data security protection technologies and policies.
The data transfer complies with the requirements of applicable laws and regulations. For example, the data being transferred does not contain any content that is restricted, prohibited from being transferred, or prohibited from being disclosed under applicable laws.

Alibaba Cloud reminds you that if your data upload operations may result in cross-border data transfers, consult professional legal or compliance personnel before performing such operations to ensure that the cross-border data transfers comply with applicable laws, regulations, and regulatory policies (for example, obtaining valid authorization from data subjects, signing and filing relevant contractual clauses, and completing relevant security assessments and other statutory obligations).

If you perform cross-border data operations without complying with this compliance statement, you will bear the corresponding legal consequences. You are also liable for any losses incurred by Alibaba Cloud and its affiliates.

References

Data Studio also supports uploading local CSV or text files to MaxCompute tables. For more information, see Upload data.
For more operations on MaxCompute tables, see Manage MaxCompute tables.
For more operations on Hologres tables, see Create a Hologres table.
For more operations on EMR tables, see Create an EMR table.

FAQ

Resource group configuration issue.

Error message: The current file source or destination engine requires a resource group for data upload. Contact the workspace administrator to configure a resource group.

Solution: To configure the resource group for an engine in Data Analysis, see System management.
Resource group association issue.

Error message: The resource group configured for global data upload in your workspace is not associated with the workspace to which the target table belongs. Contact the workspace administrator to associate them.

Solution: You can associate the resource group as a workspace resource group that is configured in system management.