Create an OSS data source to enable Dataphin to read business data from or write data to OSS.
Overview
Alibaba Cloud Object Storage Service (OSS) is a secure, cost-effective, and highly reliable cloud storage service for large-scale data. Create an OSS data source before using Dataphin for data development or writing data to OSS. What is OSS?
Permissions
Only custom global roles with the Create Data Source permission and system roles such as Super Administrator, Data Source Administrator, Division Architect, and Project Administrator can create data sources.
Procedure
-
On the Dataphin homepage, click Management Center > Datasource Management in the top navigation bar.
-
On the Datasource page, click +Create Data Source.
-
On the Create Data Source page, select OSS in the File section.
Alternatively, select OSS from the Recently Used section or search by keyword.
-
On the Create OSS Data Source page, configure the connection parameters.
-
Configure basic information.
Parameter
Description
Datasource Name
The name must meet the following requirements:
-
It can contain only Chinese characters, letters, digits, underscores (_), and hyphens (-).
-
It cannot exceed 64 characters in length.
Datasource Code
Reference data source tables in Flink_SQL tasks using the
data source code.table nameordata source code.schema.table nameformat. To auto-match the current environment, use the variable format${data source code}.tableor${data source code}.schema.table. For more information, see Dataphin data source table development method.ImportantThe data source code cannot be modified after it is configured.
You can preview data on the object details page in the asset directory and asset checklist only after the data source code is configured.
In Flink SQL, only MySQL, Hologres, MaxCompute, Oracle, StarRocks, Hive, SelectDB, and GaussDB data warehouse service (DWS) data sources are currently supported.
Data Source Description
Brief description of the data source. Maximum 128 characters.
Data Source Configuration
Select the configuration scope:
-
Production + Development Data Source: separate production and development environments.
-
Production Data Source: single shared environment.
Tag
Categorize data sources with tags. Manage data source tags.
-
-
Configure connection parameters.
If you select Production + Development Data Source, configure connection parameters for both environments. If you select Production Data Source, configure only the production connection. Production data source Production data source
NoteBest practice: use separate data sources for production and development to isolate environments. However, Dataphin supports using the same data source with identical parameters for both.
Parameter
Description
Endpoint
Regional endpoint in the format
http://{oss-Region}.aliyuncs.com. Example:https://oss-cn-hangzhou.aliyuncs.comfor China (Hangzhou).Each region has a unique endpoint domain.
Bucket
The bucket in the target region. A bucket is a container for storing objects. Find your bucket on the Bucket List page.
You can create multiple buckets and add objects to each. During synchronization, only objects in the specified bucket are accessible.
Directory
If you only have permissions for a specific directory, you can specify the directory path here. For example,
/dataphin/.CNAME
Optional. Custom domain name bound to the OSS bucket. Use custom domain names to access OSS resources.
Access ID, Access Key
The AccessKey ID and AccessKey Secret of the account to which the OSS data source belongs.
-
-
Select a Default Resource Group for running data source tasks (SQL queries, offline migration, and data preview).
-
Click Test Connection to verify connectivity, or click OK to save the data source directly.
Test Connection validates connectivity to Dataphin. If you click OK directly, the system tests all selected clusters automatically. The data source is saved regardless of test results.