All Products
Search

This Product

    DataWorks:Sync a single OSS table to MaxCompute

    Document Center

    DataWorks:Sync a single OSS table to MaxCompute

    Last Updated:Jun 03, 2026

    Walks through data source configuration, network connectivity, and batch sync task setup to load data from a single OSS table into MaxCompute.

    OSS overview

    Alibaba Cloud OSS provides 99.9999999999% (twelve 9s) data durability and 99.995% availability with multiple storage classes for cost optimization. Data Integration supports syncing data between OSS and other sources or destinations.

    Get OSS bucket information

    In the OSS console, locate the target bucket and note its public and internal endpoints from the bucket overview page.

    • The public endpoint allows internet access. Inbound traffic (writes) is free; outbound traffic (reads) incurs charges per OSS Pricing and Billing Items.

    • The internal endpoint connects Alibaba Cloud products within the same region with no traffic charges. Use the internal endpoint when the OSS bucket and your Data Integration resource group are in the same region; otherwise, use the public endpoint.

    • Regions and Endpoints.

    Prerequisites

    Limitations

    Syncing source data to MaxCompute external tables is not supported.

    Procedure

    Note

    This tutorial uses the Data Studio (new version) UI.

    Step 1: Create a node and task

    Create a batch sync node following Codeless UI configuration.

    Step 2: Configure the source and destination

    Configure the source (OSS)

    Configure the following OSS source parameters.

    Parameter

    Description

    File Type

    Select the file type to sync. The Codeless UI supports reading files in csv, text, orc, and parquet formats.

    File Path

    Enter the path to the source file.

    • A single object path uses one thread for extraction.

    • Multiple object paths use multiple threads. Adjust concurrency as needed.

    • Wildcards match multiple objects. For example, abc*[0-9] matches abc0, abc1, abc2, and abc3. abc?.txt matches files starting with abc, ending with .txt, with one character in between.

    Field Delimiter

    Column delimiter used in the file.

    Encoding

    Character encoding for reading the source file.

    Null String

    • Do not process: source values remain unchanged.

    • Visible characters: enter a string representing null. An empty field is treated as an empty string.

    • Invisible Characters: enter a Unicode code (\u001b, \u007c) or escape character (\t). Required.

    Compression Format

    Compression format. Supported: Gzip, Bzip2, Zip, or uncompressed.

    Skip Header

    Whether to skip the header row in CSV-like files. Default: header included.

    Note

    Skipping the header is not supported for compressed files.

    Table Data Structure

    After you configure the data source parameters, click Confirm Data Structure to verify the data format.

    Configure the destination (MaxCompute)

    Configure the following MaxCompute destination parameters.

    Note

    Parameters not listed below can use default values.

    Parameter

    Description

    Tunnel Resource Group

    MaxCompute data transfer resource (Tunnel Quota). Defaults to the free public quota. If an exclusive Tunnel Quota becomes unavailable due to overdue payments or expiration, the task falls back to the public quota.

    Table

    Select the destination MaxCompute table. In a standard workspace, ensure a table with the same name and schema exists in both development and production environments.

    Alternatively, click Generate Destination Table Schema to auto-create a destination table and adjust the DDL as needed.

    Note

    Consider the following:

    • If the table does not exist in the development environment, it does not appear in the destination table list.

    • If the table does not exist in the production environment, the published task fails.

    • Inconsistent schemas between development and production can cause column mapping mismatches and incorrect data writes at runtime.

    Partition Information

    Partition column values for partitioned destination tables.

    • Fixed value, such as ds=20220101.

    • Scheduling parameter, such as ds=${bizdate}, replaced with actual values at runtime.

    Write Method

    Clear existing data or append new data to the target table.

    Step 3: Configure field mapping

    Map columns between source and destination using Map Fields with the Same Name, Map Fields in the Same Line, Delete All Mappings, or Edit Field Mappings.

    Step 4: Configure advanced settings

    Configure advanced settings such as Expected Maximum Concurrency and Policy for Dirty Data Records. For this tutorial, set the Policy for Dirty Data Records to Disallow Dirty Data Records and keep defaults for other settings. Codeless UI configuration.

    Step 5: Configure and run the debug task

    1. Click Run Configuration, set the Resource Group and Script Parameters for the debug run, and then click Run to test the task.

    2. In the left-side navigation pane, click image, then click the new icon next to Personal Directory to create an SQL file. Run the following query to verify the destination table data.

      Note 
      SELECT * FROM <your_maxcompute_table_name> WHERE pt=<your_partition> LIMIT 20;

    Step 6: Configure scheduling and publish the task

    In the right pane, click Scheduling Settings to configure scheduling parameters. Then click Publish to publish the task.