Manage variables and keys - Realtime Compute for Apache Flink

Realtime Compute for Apache Flink allows you to configure variables and keys to prevent security risks that are caused by the use of information such as plaintext AccessKey pairs and passwords. You can reuse variables and keys to avoid repeatedly writing the same code or values and simplify configuration management. You can reference a variable or a key in various scenarios, such as the development of SQL, JAR, or Python drafts, log output configuration, and UI-based parameter configuration. This topic describes how to create and reference a variable or a key.

Background information

Each variable or key has a name and a value. The name reflects the meaning of the value and can be customized. The value is the actual data.

Variable: A variable can be referenced in only one deployment. You can reference variables in the DDL statements of SQL drafts and the input parameters of a Main function of JAR and Python drafts. Variables must be in the ${Variable name} format. For more information about how to create and reference a variable, see Variables.
Key: A key can be referenced in only one namespace. You can reference keys when you write DDL statements of SQL drafts, configure log output, and create catalogs on the UI. Keys must be in the ${secret_values.Key name} format. For more information about how to create and reference a key, see Key hosting.

Precautions

Only accounts that have the permissions of the editor or owner role in the current namespace can configure and reference a variable or key. For more information about how to grant permissions to an account, see Authorize an account to perform operations in a namespace.
You cannot specify multiple values for a variable or key.
In SQL drafts, you can reference a variable or key only when you create a temporary table by using the CREATE TEMPORARY TABLE statement. If you reference a variable or key when you create a persistent table by using the CREATE TABLE statement, the table that you created cannot be used.

Variables

Note

Variable names must be unique within a single draft.

Reference a variable in an SQL draft

You can reference a variable when you develop an SQL draft and change the variable value when you start the deployment.

In the left-side navigation pane of the development console of Realtime Compute for Apache Flink, click SQL Editor.
On the Drafts tab, click the desired draft, and enter a variable in the ${Variable name} format in the SQL editor. The following sample code provides an example.
- If you reference a variable in the WITH clause, you must enclose the variable with a pair of single quotation marks ('). The variable must be in the '${Variable name}' format.
- If you reference a variable as a table name or field name in an SQL statement, you must enclose the variable with a pair of grave accents (`). The variable must be in the `${Variable name}` format.
```
create temporary table `${source_name}`(
    id varchar,
    name varchar
) with (
    'connector' = 'datagen'
);


create temporary table blackhole(
    id varchar,
    `${test_name}` varchar
) with (
    'connector' = '${blackhole}'
);

insert into blackhole select * from `${source_name}`;
```
Enter the variable value by using one of the following methods. Realtime Compute for Apache Flink automatically identifies variables in the required format in drafts.
- Method 1: Enter the variable value in the Variables panel of the SQL editor.
- Method 2: Click the Configurations tab on the right side of the SQL editor. In the Configurations panel, enter the variable value in the Variables section.
After the draft is deployed, view the variable that is referenced in the deployment on the Configuration tab of the Deployments page.
On the Deployments page, find the desired deployment and click Start in the Actions column. You can change the variable value in the Start Job panel.
Important
The new variable value takes effect only for the deployment to be started. The variable value in the statement of the SQL draft remains unchanged.

Reference a variable in a JAR or Python draft

You can reference a variable in the input parameters of a Main function of a JAR or Python draft.

In the development console of Realtime Compute for Apache Flink, click Deployments. On the Deployments page, click Create Deployment.
Enter a variable name based on the format that is shown in the following figure in the Entry Point Main Arguments field and then enter the variable value in the Variables section below the field.
For more information about other deployment parameters, see Create a deployment.
In the Create Deployment dialog box, click Deploy.
On the Deployments page, find the desired deployment and click Start in the Actions column. You can change the variable value in the Start Job panel.

Key hosting

Note

The key names must be unique within a single namespace.
You can only add and delete a key. You cannot view or change the value of a key.
If you delete an existing key, the draft that uses variables may fail to be deployed or you may need to redeploy the draft. Proceed with caution. If you delete an existing key, the deployments that are running remain unaffected.

Add a key

Log on to the Realtime Compute for Apache Flink console. Find the workspace that you want to manage and click Console in the Actions column.
In the left-side navigation pane, click Security. On the Security page, click the Secret Values tab.
On the Secret Values tab, click Add Secret Value. In the Add Secret Value dialog box, configure the Secret Name and Secret Value parameters.
Important
The key values are case-sensitive. Make sure that the letters in the key values are in the correct cases.
Click OK.

Reference a key

You can reference a key in the ${secret_values.Key name} format. When you reference a key, you need to only specify the actual key name. The following examples show how to reference a key in different scenarios.

Reference a key in an SQL draft

You can enter a key in the ${secret_values.Key name} format in the DDL statements of SQL drafts. The following sample code provides an example.

If you reference a key in the WITH clause, you must enclose the key with a pair of single quotation marks ('). The key must be in the '${secret_values.Key name}' format.
If you reference a key as a table name or field name in an SQL statement, you must enclose the key with a pair of grave accents (`). The key must be in the `${secret_values.Key name}` format.

create temporary table `${secret_values.source_name}`(
    id varchar,
    name varchar
) with (
    'connector' = 'datagen'
);


create temporary table blackhole(
    id varchar,
    `${secret_values.test_name}` varchar
) with (
    'connector' = '${secret_values.blackhole}'
);

insert into blackhole select * from `${secret_values.source_name}`;

Reference a key in the Logging section

Go to the Logging section.
- For more information about how to configure parameters to export the logs of a deployment, see Configure parameters to export the logs of a deployment.
- For more information about how to configure parameters to export the logs of all deployments in a namespace, see Configure parameters to export the logs of all deployments in a namespace.

Reference a key in the Logging section.

The following sample code provides an example on how to reference a key named accessKeyId and a key named accessKeySecret when you configure parameters to export the logs of a deployment to Simple Log Service. For more information about the complete log output configuration, see Configure parameters to export logs of a deployment.

<Appender name="SLS" type="SLS">
  <Layout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS}{GMT+8} %-5p %-60c %x - %m%n" type="PatternLayout" charset="UTF-8"/>  

  <!-- The final effective log path is: ${baseUri}/logs/${namespace}/${deploymentId}/{jobId}/ -->
  <Property name="namespace">{{ namespace }}</Property> <!-- Do not modify this line -->
  <Property name="project">YOUR-SLS-PROJECT</Property>  
  <Property name="logStore">YOUR-SLS-LOGSTORE</Property> 
  <Property name="endpoint">YOUR-SLS-ENDPOINT</Property> 
  <Property name="accessKeyId">${secret_values.accessKeyId}</Property> 
  <Property name="accessKeySecret">${secret_values.accessKeySecret}</Property> 
  <Property name="topic">{{ namespace }}:{{ deploymentId }}:{{ jobId }}</Property>
  <Property name="deploymentName">{{ deploymentName }}</Property>
  <Property name="flushIntervalSeconds">10</Property>
  <Property name="flushIntervalEventCount">100</Property>
</Appender>

Reference a key on the UI

When you create a catalog in the Realtime Compute for Apache Flink console, you can reference a key.

In the left-side navigation pane of the development console of Realtime Compute for Apache Flink, click Catalogs.
In the Create Catalog dialog box, reference a key. The following example shows how to reference a key named mysqlpassword in the password parameter when you create a MySQL catalog.

References

For more information about how to develop an SQL draft, see Develop an SQL draft.
For more information about deployment parameters of a JAR or Python draft, see Create a deployment.
For more information about how to create and use a catalog, see Manage catalogs.
For more information about the connectors that are supported by Realtime Compute for Apache Flink, see Supported connectors.