OSS Tables is a structured data lake storage service provided by Alibaba Cloud Object Storage Service (OSS). Based on the Apache Iceberg open table format, OSS Tables provides low-cost, high-performance structured data storage for AI training data management and OLAP analysis scenarios. OSS Tables has a built-in automated data maintenance mechanism (compaction, snapshot management, and unreferenced file cleanup) that delivers a zero-ops serverless data lake experience. DataWorks Data Integration allows you to write data to OSS Tables data sources. This topic describes how to use the OSS Tables data source.
Limitation
Item | Description |
Resource group | Only Serverless resource groups are supported. |
Read/write direction | Only write operations as a destination are supported. |
Synchronization mode | Only Hive is supported as the source, and data can be written only in batch offline mode. |
Create a data source
Before you begin, make sure the following conditions are met:
The Alibaba Cloud OSS Tables service is activated, and an OSS Tables bucket and namespace are created.
A DataWorks workspace is available, and you have access to the workspace Management Center.
An AccessKey pair or RAM role is available for access authentication.
Perform the following steps:
Log on to the DataWorks console and select the target region in the top navigation bar.
In the left-side navigation pane, choose . In the drop-down list, select the target workspace, and then click Enter Management Center.
In the left-side navigation pane of the Management Center page, click Data Sources.
On the Data Sources page, click Add Data Source.
In the Add Data Source dialog box, search for and select OSS Tables.
On the OSS Tables data source configuration page, configure the parameters described in the following table.
Parameter
Description
Data Source Name
The name of the data source. The name must be unique within the workspace and can contain only letters, digits, and underscores (_). The name cannot start with a digit or underscore.
Configuration Mode
Only Connection String Mode is supported.
Endpoint
Select the endpoint of the OSS Tables engine instance from the drop-down list.
Access Identity
Select the authentication method: AccessKey Pair (authenticate by using the AccessKey ID and AccessKey Secret) or Alibaba Cloud RAM Role (authenticate by using a RAM role).
Bucket Name
Enter the name of the OSS Tables bucket.
Namespace
Enter the OSS Tables namespace.
After you configure the parameters, click Test Connectivity in the connection configuration section to verify the network connectivity between the data source and the Serverless resource group.
After the connectivity test is passed, click Complete to create the data source.
If the connectivity test fails, see Network connectivity to troubleshoot network issues.
Develop data synchronization tasks
Batch offline synchronization for entire databases
Supported source: Hive.
Configuration guide: Synchronize data from an entire database to OSS Tables in batch offline mode.