After you collect logs by using Simple Log Service, you can ship the logs to AnalyticDB for MySQL for data storage and analysis. This topic describes how to ship logs from Simple Log Service to AnalyticDB for MySQL.
Data is shipped to AnalyticDB for MySQL based on the REPLACE INTO semantics. You can execute the REPLACE INTO statement to insert data to a table by overwriting the existing data in real time. When you write data, the REPLACE INTO statement checks whether the data to be written exists in the table based on the primary key and then writes the data based on the check result.
If the data to be written already exists, the statement overwrites the existing data.
If the data to be written does not exist, the statement inserts the data.
For more information, see REPLACE INTO.
Logs are collected to a destination Logstore. For more information, see Data collection overview.
The following operations are performed in the AnalyticDB for MySQL console:
An AnalyticDB for MySQL cluster is created in the region where the Simple Log Service project resides. For more information, see Create a cluster.Note
Simple Log Service can ship logs only in the same region.
A database account is created. For more information, see Create a database account.
A database is created. For more information, see Create a database.
If you need to access AnalyticDB for MySQL clusters over the Internet, you must first apply for a public endpoint. For more information, see Apply for or release a public endpoint.
A table is created in the database. For more information, see CREATE TABLE.
Create a data shipping job
- Log on to the Log Service console.
In the Projects section, click the project that you want to manage.
On the tab, click the > icon next to the destination Logstore. Then, choose , and click + next to AnalyticDB.
In the Shipping Notes dialog box, click Ship.
The first time you create a data shipping job to ship logs from Simple Log Service to AnalyticDB for MySQL, you must create the AliyunServiceRoleForAnalyticDBForMySQL service-linked role.
In the Create Service Linked Role dialog box, click AliyunServiceRoleForAnalyticDBForMySQL.
In the Create Service Linked Role message, click OK.
On the LogHub - Data Shipper page, configure the parameters and click OK.
The name of the data shipping job.
The description of the data shipping job.
The version of the AnalyticDB for MySQL cluster. In this example, 3.0 is selected.
The AnalyticDB for MySQL cluster that you created.
The database that you created in the AnalyticDB for MySQL cluster.
The table that you created in the AnalyticDB for MySQL cluster.
The account of the database that you created in the AnalyticDB for MySQL cluster.
The account password of the database that you created in the AnalyticDB for MySQL cluster.
Simple Log Service extracts all log fields of the last 10 minutes and maps the fields to the destination fields in the AnalyticDB for MySQL table. Enter the names of log fields in the left text box. Enter the names of fields in the AnalyticDB for MySQL table in the right text box.Note
The field mapping of fields of the timestamp type is accurate to seconds. For example, the timestamp of a log in Simple Log Service is 2022-01-05 10:12:13 13.145. After the log is shipped to AnalyticDB for MySQL, the timestamp is recorded as 2022-01-05 10:12:13.
Delivery Start Time
The start time of the data shipping job.
After logs are collected to Simple Log Service, the logs are shipped to AnalyticDB for MySQL in real time.
Filter Dirty Data
Dirty data includes data whose data type fails to be converted and whose required fields are empty.
If you turn on Filter Dirty Data, the data shipping job filters out dirty data.
If you turn off Filter Dirty Data and dirty data exists, the data shipping job is interrupted. Proceed with caution.
After you create a data shipping job, you can manage the job in the Simple Log Service console. You can view the job details, modify the shipping rule, and start, stop, or delete the job. For more information, see Manage a data shipping job.
View log data
After logs are shipped to AnalyticDB for MySQL, you can execute a SELECT statement to query log data.