This topic describes how to use Log Service to transfer log data to AnalyticDB for MySQL.

Background information

Logs are a special type of data that plays an important role in processing historical data, diagnosing problems, and tracing system activities. Logs are essential data sources for data analysts, developers, and O&M personnel.

Alibaba Cloud allows you to use Log Service to synchronize logs such as NGINX access logs, Log4j logs, Apache access logs, and structured text to AnalyticDB for MySQL in real time. This allows you to use AnalyticDB for MySQL to perform real-time log analytics.

Prerequisites

  • When you use Log Service for the first time, you must use your Alibaba Cloud account to log on to the Log Service product page and click Get it Free. The system automatically redirects you to the buy page. Read and select I agree with Log Service Agreement of Service and click Enable Now. The Log Service console appears.
    Note If you have activated Log Service, you can start to create a project.
  • Before you use AnalyticDB for MySQL, make sure that the following requirements are met:
    1. An AnalyticDB for MySQL cluster is created. For more information, see Create a cluster.
    2. A database account is created. For more information, see Create a database account.
    3. A database is created. For more information, see Create a database.
    4. A public endpoint is applied for if you need to use a public endpoint to connect to AnalyticDB for MySQL clusters. For more information, see Apply for or release a public endpoint.
    5. A table is created through the CREATE TABLE statement to store the log data transferred to AnalyticDB for MySQL. For more information, see CREATE TABLE.

Create a LogShipper task

You can use the Export function in the Data Transformation module of Log Service to transfer the collected logs in Logstores to AnalyticDB for MySQL.

  1. Create a project and a Logstore.

    In this example, a project named adb-test and a Logstore named adb-source are created. For more information, see Create a project and Create a Logstore.

    Note Log Service projects and AnalyticDB for MySQL clusters must be located in the same region because logs can be transferred only within one region.
    Create a project and a Logstore
  2. On the Logstores tab, find the target Logstore and choose > > Data Transformation > Export > AnalyticDB.

    If you use Log Service to transfer log data to AnalyticDB for MySQL for the first time, you must grant access permissions on Log Service to AnalyticDB for MySQL.

    1. Click + to the right of AnalyticDB. The system prompts you to complete authorization. Click permissions.
    2. On the Cloud Resource Access Authorization page, click Confirm Authorization Policy and assign the RAM role AliyunAnalyticDBAccessingLogRole to AnalyticDB for MySQL.Grant permissions on Log Service
  3. After you complete authorization, click + to the right of AnalyticDB. You can click Ship to directly transfer log data. You can also click Go to Data Transformation to process log data before the data is transferred.
    Note The log data is transferred directly in this example.
  4. On the LogHub - Data Shipper page, configure the parameters.
    Table 1. LogHub - Data Shipper parameters
    Parameter Description
    Shipper Name The name of the LogShipper task. We recommend that you specify an identifiable name to facilitate subsequent management.
    Shipper Description The description of the LogShipper task. We recommend that you specify a meaningful description to facilitate subsequent management.
    Cluster Version The version of the AnalyticDB for MySQL cluster. Select 3.0.
    Cluster Name The name of the destination AnalyticDB for MySQL cluster.
    Database Name The name of the database in the destination AnalyticDB for MySQL cluster.
    Table Name The name of the table in the destination AnalyticDB for MySQL cluster.
    Account Name The name of the Create a database used to connect to the destination AnalyticDB for MySQL cluster. You can use one of the following accounts:
    • Privileged account
    • Standard account
    Account Password The password of the database account used to connect to the destination AnalyticDB for MySQL cluster.
    Field Mapping Log Service automatically extracts all log fields of the last ten minutes and maps these fields to the destination fields in AnalyticDB for MySQL.
    Delivery Start Time The start time of the LogShipper task.

    After log data is written into Log Service, Log Service can transfer the data to AnalyticDB for MySQL in real time.

    Filter Dirty Data Specifies whether to filter dirty data.
    • Disable: automatically interrupts the LogShipper task if dirty data is found.
    • Enable: filters dirty data.

      Dirty data refers to data whose data type fails to be converted or whose required fields are empty.

    LogShipper task 1LogShipper task 2LogShipper task 3
  5. After you configure the preceding parameters, click OK. Log Service will transfer log data to AnalyticDB for MySQL tables in real time at the specified delivery time.
    Log data transferred

View log data

After logs are transferred to AnalyticDB for MySQL, you can execute the SELECT statement to analyze log data. For more information, see Syntax.

Manage a LogShipper task

  • You can click the task name to view the execution status of the LogShipper task. For example, you can view the amount of data that is transferred and fails to be transferred and delivery latency.Monitor a LogShipper task
  • To modify a LogShipper task, click Modify LogShipper.Modify LogShipper
  • To restart a LogShipper task, first click Stop. After the LogShipper task is stopped, click Start to restart the LogShipper task.