Log storage and analytics

Logstash

1. Download and install the Logstash Doris Output plugin. You can use one of the following methods:

   • Direct download: Click here to download.

   • Build from source and run the following command to install the plugin:

     ./bin/logstash-plugin install logstash-output-doris-1.2.0.gem

2. Configure Logstash. You need to configure the following files:

   • logstash.yml: Configure the batch size and delay to improve ingest performance.

     pipeline.batch.size: 1000000  
     pipeline.batch.delay: 10000

   • logstash_demo.conf: Configure the input path for the logs to be collected and the output settings for SelectDB.

     input {  
         file {  
         path => "/path/to/your/log"  
       }  
     }  
     
     output {  
       doris {  
         http_hosts => [ "<http://fehost1:http_port>", "<http://fehost2:http_port>", "<http://fehost3:http_port">]  
         user => "your_username"  
         password => "your_password"  
         db => "your_db"  
         table => "your_table"  
         
         # doris stream load http headers  
         headers => {  
         "format" => "json"  
         "read_json_by_line" => "true"  
         "load_to_single_tablet" => "true"  
         }  
         
         # field mapping: doris fileld name => logstash field name  
         # %{} to get a logstash field, [] for nested field such as [host][name] for host.name  
         mapping => {  
         "ts" => "%{@timestamp}"  
         "host" => "%{[host][name]}"  
         "path" => "%{[log][file][path]}"  
         "message" => "%{message}"  
         }  
         log_request => true  
         log_speed_interval => 10  
       }  
     }

3. Run the following command to start Logstash, collect logs, and output them to SelectDB.

   ./bin/logstash -f logstash_demo.conf

For more information about Logstash configuration and usage, see Logstash Doris Output Plugin.

Filebeat

1. Download the Filebeat binary that supports output to SelectDB: Download link.

2. Configure Filebeat. You need to configure the following file:

   • filebeat_demo.yml: Configure the input log path and the output settings for SelectDB.

     # input
     filebeat.inputs:
     - type: log
       enabled: true
       paths:
         - /path/to/your/log
       # The multiline setting merges multi-line logs (such as Java stack traces) into a single event.
       multiline:
         type: pattern
         # Behavior: Treat lines that start with yyyy-mm-dd HH:MM:SS as a new log entry; append other lines to the previous entry.
         pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
         negate: true
         match: after
         skip_newline: true
     
     processors:
     # Use the js script processor to replace \t with spaces to avoid JSON parsing errors.
     - script:
         lang: javascript
         source: >
             function process(event) {
                 var msg = event.Get("message");
                 msg = msg.replace(/\t/g, "  ");
                 event.Put("message", msg);
             }
     # Use the dissect processor for simple log parsing.
     - dissect:
         # 2024-06-08 18:26:25,481 INFO (report-thread|199) [ReportHandler.cpuReport():617] begin to handle
         tokenizer: "%{day} %{time} %{log_level} (%{thread}) [%{position}] %{content}"
         target_prefix: ""
         ignore_failure: true
         overwrite_keys: true
     
     # queue and batch
     queue.mem:
       events: 1000000
       flush.min_events: 100000
       flush.timeout: 10s
     
     # output
     output.doris:
       fenodes: [ "http://fehost1:http_port", "http://fehost2:http_port", "http://fehost3:http_port" ]
       user: "your_username"
       password: "your_password"
       database: "your_db"
       table: "your_table"
       # output string format
       ## %{[agent][hostname]} and %{[log][file][path]} are built-in Filebeat metadata fields.
       ## A commonly used Filebeat metadata field is the collection timestamp %{[@timestamp]}.
       ## %{[day]} and %{[time]} are fields parsed by the dissect processor above.
       codec_format_string: '{"ts": "%{[day]} %{[time]}", "host": "%{[agent][hostname]}", "path": "%{[log][file][path]}", "message": "%{[message]}"}'
       headers:
         format: "json"
         read_json_by_line: "true"
         load_to_single_tablet: "true"

3. Run the following commands to start Filebeat, collect logs, and output them to SelectDB.

   chmod +x filebeat-doris-2.1.1  
   ./filebeat-doris-2.1.1 -c filebeat_demo.yml

For more information about Filebeat configuration and usage, see Beats Doris Output Plugin.

Kafka

Write JSON-formatted logs to a Kafka message queue and create a Kafka Routine Load job. SelectDB pulls data from Kafka automatically.

Use the following example as a reference. The property.* settings are librdkafka client configurations. Set them based on your Kafka cluster setup.

-- Prepare a Kafka cluster and a topic named log__topic_.
-- Create a routine load job to import data from the kafka topic log__topic_ into the log_table table.
CREATE ROUTINE LOAD load_log_kafka ON log_db.log_table  
COLUMNS(ts, clientip, request, status, size)  
PROPERTIES (
"max_batch_interval" = "60",
"max_batch_rows" = "20000000",
"max_batch_size" = "1073741824", 
"load_to_single_tablet" = "true",
"format" = "json"
)
FROM KAFKA (  
"kafka_broker_list" = "host:port",  
"kafka_topic" = "log__topic_",  
"property.group.id" = "your_group_id",  
"property.security.protocol"="SASL_PLAINTEXT",  
"property.sasl.mechanism"="GSSAPI",  
"property.sasl.kerberos.service.name"="kafka",  
"property.sasl.kerberos.keytab"="/path/to/xxx.keytab",  
"property.sasl.kerberos.principal"="<xxx@yyy.com>"  
);  
-- View the status of the routine load job.
SHOW ROUTINE LOAD;

For more information about Kafka configuration and usage, see Routine Load.

Custom program

You can use the Stream Load HTTP API to import log data into SelectDB with a custom program:

curl   
--location-trusted   
-u username:password   
-H "format:json"   
-H "read_json_by_line:true"   
-H "load_to_single_tablet:true"   
-H "timeout:600"   
-T logfile.json   
http://fe_host:fe_http_port/api/log_db/log_table/_stream_load

When using a custom program, note the following:

• Use Basic Auth for HTTP authentication and perform the calculation with the command echo -n 'username:password' | base64.

• Set the HTTP header "format:json" to specify the data format as JSON.

• Set the HTTP header "read_json_by_line:true" to specify one JSON object per line.

• Set the HTTP header "load_to_single_tablet:true" so each import writes to a single bucket, which reduces the generation of small files.

• Use a client-side batch size of 100 MB to 1 GB. For Apache Doris 2.1 and later, use the server-side Group Commit feature to reduce the client-side batch size.