Realtime Compute for Apache Flink:Overview

Limits

A Realtime Compute for Apache Flink cluster in exclusive mode can access only storage resources in the same virtual private cloud (VPC), region, and security group as the cluster.

Data storage in Realtime Compute for Apache Flink

Use a plaintext AccessKey pair

To reference upstream and downstream storage resources by using a plaintext AccessKey pair, you must configure the accessId and accessKey parameters in the WITH clause of the related DDL statement. For more information, see Overview. This way, you can authorize an Alibaba Cloud account and its RAM users to access the resources of the current or another Alibaba Cloud account. If User A or a RAM user created within the Alibaba Cloud account of User A wants to use the storage resources of User B, User A can set the AccessKey pair of User B in the following DDL statement in plaintext mode:

CREATE TABLE in_stream(
  a varchar,
  b varchar,
  c timestamp
) with (
  type='datahub',
  endPoint='http://dh-cn-hangzhou.aliyuncs.com',
  project='<dataHubProjectName>',
  topic='<dataHubTopicName>',
  accessId='<accessIdOfUserB>',
  accessKey='<accessKeyOfUserB>'
);

Register a storage resource

Realtime Compute for Apache Flink allows you to manage and reference both upstream and downstream storage resources that have been registered with Realtime Compute for Apache Flink. After storage resources are registered, you can preview or sample the relevant data, or obtain the DDL statements that are automatically generated to reference the resources. This helps you manage your cloud storage resources in end-to-end mode.

• Register storage resources
  To register upstream and downstream storage resources with Realtime Compute for Apache Flink before you reference them, perform the following steps:

  1. Log on to the Realtime Compute development platform.
  2. In the top navigation bar, click Development.
  3. In the left-side navigation pane of the Development page, click Storage.
  4. In the upper-right corner of the Storage tab, click +Registration and Connection.
  5. In the Register Data Store and Test Connection dialog box, configure the parameters for storage resources.

     Realtime Compute for Apache Flink allows you to register the following types of storage resources. For more information about how to register storage resources of a specific type, click the following links:

     • Register a Tablestore instance
     • Register an ApsaraDB for RDS instance
     • Register a Log Service project
• Preview data from a registered storage resource
  To preview data from a registered storage resource, perform the following steps:

  1. In the left-side navigation pane of the Development page, click Storage.
  2. On the Storage tab, double-click the folder of a registered storage resource and its subfolder to find the table that you want to view, and double-click the name of the table.
  3. In the Table Details pane, view data of the storage resource in the Data Preview section.
  
• Obtain the DDL statements that are automatically generated to reference a storage resource
  To obtain the DDL statements that are automatically generated to reference a storage resource, perform the following steps:

  1. In the left-side navigation pane of the Development page, click Storage.
  2. On the Storage tab, double-click the folder of a registered storage resource and its subfolder to find the table that you want to view, and double-click the name of the table.
  3. In the Table Details pane, click Reference as Source Table, Reference as Result Table, or Reference as Dimension Table. Then, you can obtain the DDL statements that are automatically generated to reference the table.

  Note The automatically generated DDL statements contain only the basic parameters in the WITH clause to ensure connectivity between Realtime Compute for Apache Flink and storage resources. You can add other parameters to the WITH clause in addition to the basic parameters.

  
• Test network connectivity by using the network detection feature
  Note The network detection feature is not supported in the China (Hangzhou) region of Finance Cloud because Cloud Assistant is not installed in the region.
  Realtime Compute for Apache Flink provides the network detection feature for data storage. This feature allows you to test network connectivity between Realtime Compute for Apache Flink and storage resources. To enable the network detection feature, perform the following steps:

  1. In the left-side navigation pane of the Development page, click Storage.
  2. In the upper-right corner of the Storage tab, click +Registration and Connection.
  3. In the Register Data Store and Test Connection dialog box, turn on Test Connection.