Logstash is an open source data collection engine with real-time pipelining capabilities. It is first used to write log data into Elastic Stack. As the open source community develops, Logstash can dynamically unify data from disparate sources and normalize the data into destinations of your choice.
For example, Java Database Connectivity (JDBC) is allowed to access AnalyticDB for MySQL. You can use a Logstash plug-in named logstash-output-jdbc to import log data to AnalyticDB for MySQL for subsequent analysis. However, JDBC writes single records. If you use JDBC to write large amounts of log data to AnalyticDB for MySQL, the write performance is low and the CPU utilization is high. To manage this issue, AnalyticDB for MySQL has optimized a Logstash plug-in named logstash-ouput-analyticdb based on JDBC. This plug-in is dedicated to write log data to AnalyticDB for MySQL in aggregate.
The logstash-output-analyticdb plug-in provides five times the write speed of the logstash-output-jdbc plug-in at a lower CPU utilization.
Install Logstash
The following section describes how to install the logstash-output-analyticdb plug-in. For information about how to install Logstash, visit Installing Logstash.
- Run the following command to go to the root directory of Logstash:
cd logstash
. - Run the following command to install the logstash-output-analyticdb plug-in:
bin/logstash-plugin install logstash-output-analyticdb
. - Run the following command to create a directory named
vendor/jar/jdbc
under the root directory of Logstash:mkdir -p vendor/jar/jdbc
. - Run the following command to copy the installation package of JDBC to
vendor/jar/jdbc
:cd vendor/jar/jdbc; wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.36/mysql-connector-java-5.1.36.jar
.
After you complete the preceding steps, the logstash-output-analyticdb plug-in is installed.
Configure Logstash
Create a configuration file named logstash-analyticdb.conf
under the config directory. You can also customize the file name. logstash-analyticdb.conf
contains the following content:
input
{
stdin { }
}
output {
analyticdb {
driver_class => "com.mysql.jdbc.Driver"
connection_string => "jdbc:mysql://HOSTNAME:PORT/DATABASE? user=USER&password=PASSWORD"
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
commit_size => 4194304
}
}
connection_string
: the JDBC connection string used to connect to AnalyticDB for MySQL.statement
: the declared arrays in the INSERT statement.
Other parameters:
max_flush_exceptions
: specifies the maximum number of retries that can be attempted if an exception occurs during data write. Default value: 100.skip_exception
: specifies whether to skip exceptions. The default value is FALSE, indicating that the maximum number of retries specified by themax_flush_exceptions
parameter are attempted if an exception occurs. If all of these retries fail, an exception will be thrown to terminate the synchronization task. If this parameter is set to TRUE and the maximum number of retries fail, the exception is skipped and simply written to a log.flush_size
: the maximum number of data records that can buffer simultaneously. This parameter is used in combination with thecommit_size
parameter.commit_size
: the maximum amount of data that can buffer simultaneously. This parameter is used in combination with theflush_size
parameter. Data write tasks are submitted when upper limits are reached.
The preceding configuration file is provided for reference. You must configure the
logstash-analyticdb.conf
file. For more information about configurations related to AnalyticDB for MySQL,
visit README. For more information about configurations and rules of Logstash, see the Logstash
documentation.
After you configure the preceding parameters, the configuration is complete.
Start a task
Run the following command in the installation directory of Logstash to start a task:
bin/logstash -f config/logstash-analyticdb.conf
.
Precautions
Before you write data into AnalyticDB for MySQL, we recommend that you run the following command to upgrade Logstash to the latest version:
bin/logstash-plugin update logstash-output-analyticdb