All Products
Search
Document Center

Realtime Compute for Apache Flink:Manage Simple Log Service catalogs

Last Updated:Jan 26, 2024

You can configure a Simple Log Service catalog to automatically parse logs in a Logstore of Simple Log Service to infer the table schema. After you configure a Simple Log Service catalog, you can directly access the Logstore of Simple Log Service on the SQL Editor page of the console of fully managed Flink. This way, you can obtain field information without the need to declare the schema of the Simple Log Service table in Flink SQL. This topic describes how to create, view, use, and delete a Simple Log Service catalog in the console of fully managed Flink.

Context

This topic describes the following operations that you can perform to manage Simple Log Service catalogs:

Limits

  • All fields in a table are parsed as strings. If you want to use other data types, you must use Flink SQL to perform data type conversion.

  • Only Realtime Compute for Apache Flink that uses Ververica Runtime (VVR) 6.0.7 or later supports Simple Log Service catalogs.

  • You cannot use DDL statements to modify existing Simple Log Service catalogs.

  • You can only query data tables by using Simple Log Service catalogs. You are not allowed to create, modify, or delete databases and tables by using Simple Log Service catalogs.

  • The tables provided by Simple Log Service catalogs can be directly used as source tables and result tables in Flink SQL deployments but cannot be used as lookup dimension tables.

Create a Simple Log Service catalog

  1. On the script editing page, enter the following statement to create a Simple Log Service catalog:

    CREATE CATALOG <catalogName> WITH(
      'type'='sls',
      'endpoint'='<brokers>',
      'project'='project',
      'accessId'='${secret_values.ak_id}',
      'accessKey'='${secret_values.ak_secret}'
    )

    Parameter

    Data type

    Description

    Required

    Remarks

    catalogName

    STRING

    The name of the Simple Log Service catalog.

    Yes

    Enter a custom name.

    type

    STRING

    The type of the catalog.

    Yes

    Set the value to sls.

    endpoint

    STRING

    The endpoint of Simple Log Service.

    Yes

    For more information, see Endpoints.

    project

    STRING

    The name of the Simple Log Service project.

    Yes

    N/A.

    accessId

    STRING

    The AccessKey ID of the Alibaba Cloud account.

    Yes

    For more information, see How do I view information about the AccessKey ID and AccessKey secret of the account?

    Important

    To protect your AccessKey pair, we recommend that you specify the AccessKey ID by using the key management method. For more information, see Manage keys.

    accessKey

    STRING

    The AccessKey secret of your Alibaba Cloud account.

    Yes

    For more information, see How do I view information about the AccessKey ID and AccessKey secret of the account?

    Important

    To protect your AccessKey pair, we recommend that you specify the AccessKey secret by using the key management method. For more information, see Manage keys.

  2. Select the code that is used to create a catalog and click Run that appears on the left side of the code. You can also right-click the code that is used to create the catalog and select Run.

  3. In the Catalogs pane on the left side of the Catalog List page, view the catalog that you create.

View a Simple Log Service catalog

After a Simple Log Service catalog is configured, you can perform the following steps to view the metadata of Simple Log Service.

  1. Go to the Catalogs page.

    1. Log on to the Realtime Compute for Apache Flink console.

    2. On the Fully Managed Flink tab, find the workspace that you want to manage and click Console in the Actions column.

    3. In the left-side navigation pane, click Catalogs.

  2. On the Catalog List page, view Name and Type of the desired catalog.

    Note

    If you want to view information about a Logstore in a catalog, click View in the Actions column of the catalog.

View information about a Logstore of Simple Log Service

  1. On the script editing page, enter the following statement:

    DESCRIBE `${catalog_name}`.`${project_name}`.`${logstore_name}`;

    Parameter

    Description

    ${catalog_name}

    The name of the Simple Log Service catalog.

    ${project_name}

    The name of the Simple Log Service project.

    ${logstore_name}

    The name of the Logstore of Simple Log Service.

  2. Select the code that is used to view a catalog and click Run that appears on the left side of the code.

    After the code is run, you can view the information about the table in the result.

    表信息

Use a Simple Log Service catalog

  • Use a table of a Simple Log Service catalog as a source table and read data from the Logstore of Simple Log Service.

    INSERT INTO ${other_sink_table}
    SELECT...
    FROM `${catalog_name}`.`${project_name}`.`${logstore_name}`/*+OPTIONS('startTime'='2023-06-01 00:00:00')*/;
    Note

    If you want to specify other parameters in the WITH clause when you use a table of a Simple Log Service catalog, we recommend that you use SQL hints to add the parameters. For example, the SQL hints in the preceding SQL statement specify that data consumption starts from '2023-06-01 00:00:00'. For more information about other parameters, see Simple Log Service connector.

  • Use a table of a Simple Log Service catalog as a result table and write data to the Logstore of Simple Log Service.

    INSERT INTO `${catalog_name}`.`${project_name}`.`${logstore_name}`
    SELECT ... 
    FROM ${other_source_table}
Important

Before you use a Simple Log Service catalog to read data from the source Logstore, you must enable indexing. Similarly, you must enable indexing before you use a Simple Log Service catalog to write data to the destination Logstore. This is because a catalog needs to read specific data from the result table first to determine whether the schema of the data that needs to be written is the same as the schema of the destination Logstore of Simple Log Service. For more information about how to enable indexing, see Create indexes.

Delete a Simple Log Service catalog

Warning

After you delete a Simple Log Service catalog, the deployments that are running are not affected. However, the deployments that use a table of the catalog can no longer find the table if the deployments are published or restarted. Proceed with caution when you delete a Simple Log Service catalog.

  1. On the script editing page, enter the following statement:

    DROP CATALOG ${catalog_name};

    In the preceding statement, ${catalog_name} indicates the name of the Simple Log Service catalog that you want to delete.

  2. Right-click the statement that is used to delete the catalog and choose Run from the shortcut menu.

  3. View the Catalogs pane on the left side of the Catalog List page to check whether the catalog is deleted.

Information about a table that is obtained by using a Simple Log Service catalog

When you use a Simple Log Service catalog to obtain data from a Simple Log Service table, the default parameters and metadata are added to the derived table. This helps you use the table in an efficient manner. A table that is obtained by using a Simple Log Service catalog contains the following information:

  • Schema inference of the related Simple Log Service table

    When a Simple Log Service catalog parses logs in Log Simple Service to obtain the schema of a topic, the catalog attempts to consume a message and parse the message to obtain the schema of the data. The Simple Log Service catalog parses logs in Simple Log Service to obtain the field names and data types. The logs in Simple Log Service are stored as data of the STRING type. Therefore, all fields are of the STRING type.

  • Table parameters that need to be added by default

    Parameter

    Description

    Remarks

    connector

    The type of the connector.

    Set the value to sls.

    endpoint

    The endpoint of Simple Log Service.

    For more information, see Endpoints.

    project

    The name of the Simple Log Service project.

    N/A.

    logstore

    The name of a Logstore or Metricstore of Simple Log Service.

    N/A.

    accessId

    The AccessKey ID of the Alibaba Cloud account.

    For more information, see How do I view information about the AccessKey ID and AccessKey secret of the account?

    Important

    To protect your AccessKey pair, we recommend that you specify the AccessKey ID by using the key management method. For more information, see Manage keys.

    accessKey

    The AccessKey secret of your Alibaba Cloud account.

    For more information, see How do I view information about the AccessKey ID and AccessKey secret of the account?

    Important

    To protect your AccessKey pair, we recommend that you specify the AccessKey secret by using the key management method. For more information, see Manage keys.