Dataphin:Create an Amazon S3 data source

Permissions

Only custom global roles with the Create Data Source permission, and the system roles super administrator, data source administrator, business segment architect, and project administrator can create data sources.

Procedure

1. On the Dataphin homepage, click Management Center > Data Source Management in the top menu bar.

2. On the Data Sources page, click + Create Data Source.

3. On the Create Data Source page, select Amazon S3 in the File section.

   If you have recently used Amazon S3, you can also find and select it in the Recently Used section. Alternatively, you can enter "Amazon S3" in the search box to find it quickly.

4. On the Create Amazon S3 Data Source page, configure the connection parameters.

   1. Configure the basic information for the data source.

      Parameter	Description
      Data source name	Enter a name for the data source. The name must meet the following requirements: Can contain only Chinese characters, uppercase and lowercase letters, digits, underscores (_), and hyphens (-). The length cannot exceed 64 characters.
      Data source code	After you configure the data source code, you can reference tables from this data source in a Flink SQL job by using the format data_source_code.table_name or data_source_code.schema.table_name. To automatically access the data source corresponding to the current environment, use the variable format ${data_source_code}.table or ${data_source_code}.schema.table. For more information, see Development with Dataphin data source tables. Important The data source code cannot be modified after it is configured. You can preview data on the object details page in the asset directory and asset checklist only after the data source code is configured. In Flink SQL, only MySQL, Hologres, MaxCompute, Oracle, StarRocks, Hive, SelectDB, and GaussDB data warehouse service (DWS) data sources are currently supported.
      Data source description	Enter a brief description of the data source. The description cannot exceed 128 characters.
      Data source configuration	Select the type of data source to configure: Production and development data source: Select this option if your business data source is separated into production and development environments. Production data source: Select this option if your business data source does not have separate environments.
      Tag	You can add tags to categorize and manage the data source. To learn how to create tags, see Manage data source tags.

   2. Configure the connection parameters.

      If you selected Production and development data source, you must configure connection information for both environments. If you selected Production data source, you only need to configure the connection information for the Production data source.

      Note

      Typically, production and development data sources should be configured separately to isolate the environments and prevent development activities from affecting production. However, Dataphin allows you to configure them as the same data source by using identical parameter values.

      Parameter	Description
      Endpoint	The endpoint for the region where your Amazon S3 bucket is located. The format is http://s3-{Region}.amazonaws.com, where {Region} is the bucket's region. The endpoint for the Amazon S3 service varies by region. You must enter the correct endpoint for the region you want to access. For more information, see Amazon S3 endpoints.
      Region	The region where the bucket is located. This parameter is optional. You must specify a region if it is not included in the endpoint.
      Bucket	The name of the bucket in the specified Amazon S3 region. A bucket is a container for storing objects. To find your bucket name, see Amazon S3 bucket overview.
      Directory	If you only have permissions for a specific directory, enter its path here. For example, /dataphin/.
      Authentication method	The authentication method for Dataphin to access Amazon S3. The default is AccessKey-based authentication. The IAM role authentication option is displayed only if IAM role authentication is enabled for the metadata warehouse tenant. AccessKey-based authentication: Access Amazon S3 by using the specified Access Key ID and Secret Access Key. IAM role authentication : Access Amazon S3 by using the IAM role attached to the Amazon EC2 instance where Dataphin is deployed. Dataphin automatically obtains the IAM role attached to the current instance. Ensure this role has the necessary permissions to access the target bucket.
      Access Key ID and Secret Access Key	The Access Key ID and Secret Access Key for the AWS account that owns the Amazon S3 data source. This is required only when authentication method is set to AccessKey-based authentication . For information on how to obtain these credentials, see Amazon access keys. Note These are the credentials for your AWS account, not an Alibaba Cloud account.

5. Select the default resource group. This resource group is used to run tasks related to the data source, including database SQL queries, offline full-database migration, and data preview.

6. Click Test Connection or click OK to create the Amazon S3 data source.

   When you click Test Connection, Dataphin verifies the connection to the data source. If you click OK, Dataphin automatically tests the connection for all selected clusters. You can create the data source even if these tests fail.