This topic describes how to use LogHub Log4j Appenders or Logtail to collect Log4j logs.

Background information

Log4j is an open-source project of Apache. Log4j allows you to set the output destination and format of logs. The severity levels of logs are classified into ERROR, WARN, INFO, and DEBUG in descending order. The output destination specifies whether logs are sent to the console or files. The output format specifies the format of logs. The following configurations are the default configurations of Log4j:
<Configuration status="WARN">
  <Appenders>
    <Console name="Console" target="SYSTEM_OUT">
      <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss:SSS zzz} [%t] %-5level %logger{36} - %msg%n"/>
    </Console>
  </Appenders>
  <Loggers>
    <Logger name="com.foo.Bar" level="trace">
      <AppenderRef ref="Console"/>
    </Logger>
    <Root level="error">
      <AppenderRef ref="Console"/>
    </Root>
  </Loggers>
</Configuration>
The following example shows a sample log entry:
2013-12-25 19:57:06,954 [10.10.10.10] WARN impl.PermanentTairDaoImpl - Fail to Read Permanent Tair,key:e:470217319319741_1,result:com.example.tair.Result@172e3ebc[rc=code=-1, msg=connection error or timeout,value=,flag=0]

Collect Log4j logs by using LogHub Log4j Appenders

For information about how to collect Log4j logs by using LogHub Log4j Appenders, see Log4j Appender.

Use Logtail to collect Log4j logs

  1. Log on to the Log Service console.
  2. On the page that appears, click RegEx - Text Log in the Import Data section.
  3. In the Specify Logstore step, select the project and Logstore, and then click Next.
    You can also click Create Now to create a project and a Logstore.
  4. In the Create Machine Group step, create a machine group.
    • If a machine group is available, click Using Existing Machine Groups.
    • This section uses ECS instances as an example to describe how to create a machine group. To create a machine group, perform the following steps:
      1. Install Logtail on ECS instances. For more information, see Install Logtail on ECS instances.

        If Logtail is installed on the ECS instances, click Complete Installation.

        Note If you need to collect logs from user-created clusters or servers of third-party cloud service providers, you must install Logtail on these servers. For more information, see Install Logtail in Linux or Install Logtail in Windows.
      2. After the installation is complete, click Complete Installation.
      3. On the page that appears, specify the parameters for the machine group. For more information, see Create an IP address-based machine group or Create a custom ID-based machine group.
  5. In the Machine Group Settings step, apply the configurations to the machine group.
    Select the created machine group and move the group from Source Server Groups to Applied Server Groups.
  6. In the Logtail Config step, create a Logtail configuration file.
    Parameter Description
    Config Name The name of the Logtail configuration file. The name cannot be modified after the Logtail configuration file is created.

    You can also click Import Other Configuration to import Logtail configurations from another project.

    Log Path The directories and files from which log data is collected.
    The file names can be complete names or names that contain wildcards. For more information, visit Wildcard matching. The log files in all levels of subdirectories under a specified directory are monitored if the log files match the specified pattern. Examples:
    • /apsara/nuwa/ … /*.log indicates that the files whose extension is .log in the /apsara/nuwa directory and its subdirectories are monitored.
    • /var/logs/app_* … /*.log* indicates that each file that meets the following conditions is monitored: The file name contains .log. The file is stored in a subdirectory (at all levels) of the /var/logs directory. The name of the subdirectory matches the app_* pattern.
    Note
    • Each log file can be collected by using only one Logtail configuration file.
    • You can include only asterisks (*) and question marks (?) as wildcard characters in the log path.
    Blacklist If you turn on this switch, you can configure a blacklist in the Add Blacklist field. You can configure a blacklist to skip the specified directories or files during log data collection. You can use exact match or wildcard match to specify directories and files. Example:
    • If you select Filter by Directory from the Filter Type drop-down list and enter /tmp/mydir in the Content column, all files in the directory are skipped.
    • If you select Filter by File from the Filter Type drop-down list and enter /tmp/mydir/file in the Content column, only the specified file is skipped.
    Docker File If you collect logs from Docker containers, you can configure the paths and tags of the containers. Logtail monitors the creation and destruction of the containers, filters the logs of the containers by tag, and collects the filtered logs. For more information, see Use the console to collect Kubernetes text logs in the DaemonSet mode.
    Mode Set the value to Full Regex Mode.
    Singleline Turn off the Singleline switch.
    Log Sample Enter the following sample log entry in the Log Sample field:
    2013-12-25 19:57:06,954 [10.10.10.10] WARN impl.PermanentTairDaoImpl - Fail to Read Permanent Tair,key:e:470217319319741_1,result:com.example.tair.Result@172e3ebc[rc=code=-1, msg=connection error or timeout,value=,flag=0]
    Regex to Match First Line After you enter the sample log entry, click Auto Generate. A regular expression is generated to match the first line of the log entry. The sample log entry starts with a timestamp. Therefore, the generated regular expression is \d+-\d+-\d+\s. *.
    Extract Field If you turn on the Extract Field switch, you can use a regular expression to extract field values from logs.
    RegEx Set the value to (\d+-\d+-\d+\s\d+:\d+:\d+,\d+)\s\[([^\]]*)\]\s(\S+)\s+(\S+)\s-\s(. *). You can configure a regular expression based on one of the following methods:
    • Automatically generate a regular expression

      In the Log Sample field, select the field values to be extracted, and click Generate Regular Expression. A regular expression is automatically generated.

    • Manually enter a regular expression

      Click Manual. In the RegEx field, enter a regular expression. After you enter a regular expression in the field, click Validate to check whether the regular expression can parse the log content. For more information, see How do I modify a regular expression?.

    Extracted Content After you use a regular expression to extract field values, you must specify a key for each value.
    Use System Time Turn off the Use System Time switch. Configure the time field in the %Y-%m-%dT%H:%M:%S format. You can use one of the following methods to configure the time field:
    • Specifies whether to the use system time. If you enable the Use System Time feature, the timestamp of a log entry is the system time of the server when the log entry is collected.
    • If you disable the Use System Time feature, you must find the value that indicates time information in the Extracted Content and configure a key named time for the value. Specify the value and then click Auto Generate in the Time Conversion Format field to automatically parse the time. For more information, see Time formats.
    Drop Failed to Parse Logs
    • Specifies whether to drop failed-to-parse logs. If you enable the Drop Failed to Parse Logs feature, logs that fail to be parsed are not uploaded to Log Service.
    • If you disable the Drop Failed to Parse Logs feature, raw logs are uploaded to Log Service when the raw logs fail to be parsed.
    Maximum Directory Monitoring Depth The maximum depth at which the specified log directory is monitored. Valid values: 0 to 1000. The value 0 indicates that only the directory that is specified in the log path is monitored.
    You can configure advanced options based on your business requirements. We recommend that you do not modify the settings. The following table describes the parameters in the advanced options.
    Parameter Description
    Enable Plug-in Processing Specifies whether to enable the plug-in processing feature. If you enable this feature, plug-ins are used to process logs. For more information, see Process data.
    Upload Raw Log Specifies whether to upload raw logs. If you enable this feature, raw logs are written to the __raw__ field and uploaded together with the parsed logs.
    Topic Generation Mode
    • Null - Do not generate topic: This mode is selected by default. In this mode, the topic field is set to an empty string. You can query logs without the need to enter a topic.
    • Machine Group Topic Attributes: This mode is used to differentiate logs that are generated by different servers.
    • File Path Regex: In this mode, you must configure a regular expression in the Custom RegEx field. The part of a log path that matches the regular expression is used as the topic name. This mode is used to differentiate logs that are generated by different users or instances.
    Log File Encoding
    • utf8: indicates that UTF-8 encoding is used.
    • gbk: indicates that GBK encoding is used.
    Timezone The time zone where logs are collected. Valid values:
    • System Timezone: This option is selected by default. It indicates that the time zone where logs are collected is the same as the time zone to which the server belongs.
    • Custom: Select a time zone.
    Timeout The timeout period of log files. If a log file is not updated within the specified period, Logtail considers the file to be timed out. Valid values:
    • Never: All log files are continuously monitored and never time out.
    • 30 Minute Timeout: If a log file is not updated within 30 minutes, Logtail considers the file to be timed out and no longer monitors the file.

      If you select 30 Minute Timeout, you must specify the Maximum Timeout Directory Depth parameter. Valid values: 1 to 3.

    Filter Configuration The filter conditions that are used to collect logs. Only logs that match the specified filter conditions are collected. Examples:
    • Collect logs that meet a condition: Specify the filter condition to Key:level Regex:WARNING|ERROR if you need to collect only logs of only the WARNING or ERROR severity level.
    • Filter out logs that do not meet a condition:
      • Specify the filter condition to Key:level Regex:^(?!. *(INFO|DEBUG)). * if you need to filter out logs of the INFO or DEBUG severity level.
      • Specify the filter condition to Key:url Regex:. *^(?!.*(healthcheck)). * if you need to filter out logs whose URL contains the keyword healthcheck. For example, logs in which the value of the url key is /inner/healthcheck/jiankong.html are not collected.

    For more examples, visit regex-exclude-word and regex-exclude-pattern.

    After you complete the Logtail configurations, Log Service starts to collect Log4j logs.

  7. In the Configure Query and Analysis step, configure the indexes.
    Indexes are configured by default. You can re-configure the indexes based on your business requirements. For more information, see Enable and configure the index feature for a Logstore.
    Note
    • You must configure Full Text Index or Field Search. If you configure both of them, the settings of Field Search are applied.
    • If the data type of index is long or double, the Case Sensitive and Delimiter settings are unavailable.