Dataphin:Create an Amazon S3 data source

Permissions

Only users with a custom global role that includes the New Data Source permission, or users with the super administrator, data source administrator, domain architect, or project administrator system role can create data sources.

Procedure

1. On the Dataphin homepage, choose Admin Center > Data Source Management from the top navigation bar.

2. On the Data Source Management page, click + New Data Source.

3. On the New Data Source page, in the File section, select Amazon S3.

   If you have recently used Amazon S3, you can also select it from the Recently Used section. You can also search for Amazon S3 in the search box.

4. On the New Amazon S3 Data Source page, configure the connection parameters.

   1. Basic information

      Parameter	Description
      Data source name	Enter a name for the data source. The name must meet the following requirements: It can contain Chinese characters, uppercase and lowercase letters, digits, underscores (_), and hyphens (-). It must be 64 characters or fewer.
      Data source code	After you configure the data source code, you can reference a table from this data source in a Flink SQL task by using the data_source_code.table_name or data_source_code.schema.table_name format. To automatically access the data source for the current environment, use the ${data_source_code}.table or ${data_source_code}.schema.table variable format. For more information, see Dataphin data source table development and . Important The data source code cannot be modified after it is configured. You can preview data on the object details page in the asset directory and asset checklist only after the data source code is configured. In Flink SQL, only MySQL, Hologres, MaxCompute, Oracle, StarRocks, Hive, SelectDB, and GaussDB data warehouse service (DWS) data sources are currently supported.
      Data source description	Enter a brief description of the data source, 128 characters or fewer.
      Data source type	Select the type of data source to configure: If your business data source distinguishes between production and development environments, select Production and development data sources. If your business data source does not distinguish between environments, select Production data source.
      Tag	You can add a tag to classify the data source. For more information about how to create tags, see Manage data source tags.

   2. Connection parameters

      If you selected Production and development data sources, you must configure connection information for both. Otherwise, you only need to configure it for the production data source.

      Note

      Typically, you should use different data sources for production and development to ensure environment isolation and prevent development activities from affecting the production system. However, Dataphin allows you to use the same connection parameters for both.

      Parameter	Description
      Endpoint	The endpoint for the Amazon S3 region is in the format http://s3-{Region}.amazonaws.com , where Region is the region where the bucket is located. The endpoint for the Amazon S3 service depends on the region. You must use a different domain name to access different regions. For more information, see Amazon S3 Endpoints.
      Region	The region where the bucket is located. This field is optional. If the endpoint does not specify the region, you must enter it here.
      Bucket	The name of the Amazon S3 bucket. A bucket is a container for storing objects. For more information, see Amazon S3 bucket overview to find the correct bucket for your Amazon S3 region.
      Directory	If you only have permissions for a specific directory, you can specify the directory path here. For example, /dataphin/.
      Access mode	Determines how the S3 SDK client constructs the request URL. Virtual-hosted-style: This is the default value and the default addressing mode for the S3 SDK. In this mode, the bucket name is prepended as a subdomain to the endpoint. You must ensure that the DNS for this subdomain is correctly configured, otherwise the connection will fail. Path-style: In this mode, the bucket name becomes the first part of the URL path, and the host is fixed to the endpoint. This mode is suitable for S3-compatible storage services like MinIO and Ceph, as well as custom gateway domains.
      Authentication method	The authentication method that Dataphin uses to access Amazon S3. The default is AccessKey pair. The IAM role proxy option is displayed only when IAM role authentication is enabled for the metastore tenant. AccessKey pair: Access Amazon S3 by providing an Access Key ID and an Access Key Secret. IAM role proxy: Access Amazon S3 by using the IAM role attached to the Amazon EC2 instance hosting Dataphin. The system automatically retrieves the IAM role attached to the current instance. Make sure this role has permissions on the target bucket.
      Access Key ID and Access Key Secret	The Access Key ID and Access Key Secret for the Amazon S3 account. This is required only when the authentication method is set to AccessKey pair. To obtain them, see Amazon access keys. Note This refers to the Access Key ID and Access Key Secret of a non-Alibaba Cloud account.

5. Select the default resource group. This resource group runs tasks related to the data source, such as database SQL queries, offline full database migration, and data preview.

6. Click Test Connection, or click OK to save the data source.

   When you click Test Connection, the system verifies that Dataphin can connect to the data source. If you click OK directly, the system automatically tests the connection. You can create the data source even if the connection tests fail.