Python logs

The Python logging module provides different log levels and logging methods such as file-based, HTTP GET, HTTP POST, SMTP, and Socket logging. You can also customize a logging method as needed. The Python logging module functions the same as the Log4j logging module except for their implementation details. The Python logging module provides logger, handler, filter, and formatter components.

Python log format

Formatter specifies the output format of log records. It uses two parameters to specify the log format: message format string and message date string. Both of the parameters are optional.

Log format:

import logging  
import logging.handlers  
LOG_FILE = 'tst.log'  
handler = logging.handlers.RotatingFileHandler(LOG_FILE, maxBytes = 1024*1024, backupCount = 5) # Instantiate the handler.   
fmt = '%(asctime)s - %(filename)s:%(lineno)s - %(name)s - %(message)s'  
formatter = logging.Formatter(fmt)   # Instantiate the formatter.  
handler.setFormatter(formatter)      # Add the formatter to the handler.  
logger = logging.getLogger('tst')    # Obtain the logger named tst.  
logger.addHandler(handler)           # Add the handler to the logger.  
logger.setLevel(logging.DEBUG)  
logger.info('first info message')  
logger.debug('first debug message')

Field description

Sample logs

Sample output logs:

2015-03-04 23:21:59,682 - log_test.py:16 - tst - first info message   
2015-03-04 23:21:59,682 - log_test.py:17 - tst - first debug message

Common Python logs and related regular expressions:

• Sample log:

  2016-02-19 11:03:13,410 - test.py:19 - tst - first debug message

  Regular expression:

  (\d+-\d+-\d+\s\S+)\s+-\s+([^:]+):(\d+)\s+-\s+(\w+)\s+-\s+(. *)

• Log format:

  %(asctime)s - %(filename)s:%(lineno)s - %(levelno)s %(levelname)s %(pathname)s %(module)s %(funcName)s %(created)f %(thread)d %(threadName)s %(process)d %(name)s - %(message)s

  Sample log:

  2016-02-19 11:06:52,514 - test.py:19 - 10 DEBUG test.py test <module> 1455851212.514271 139865996687072 MainThread 20193 tst - first debug message

  Regular expression:

  (\d+-\d+-\d+\s\S+)\s-\s([^:]+):(\d+)\s+-\s+(\d+)\s+(\w+)\s+(\S+)\s+(\w+)\s+(\S+)\s+(\S+)\s+(\d+)\s+(\w+)\s+(\d+)\s+(\w+)\s+-\s+(. *)

Configure Logtail to collect Python logs

Select configurations based on the network deployment and your actual situation. For more information, see Get started.

1. Create a project and a Logstore. For more information, see Preparations.
2. Click Import Data. On the Import Data page that appears, click RegEx - Text Log.
3. Select a Logstore.

   You can select an existing Logstore or create a project or Logstore.

4. Create and configure a server group.

   Make sure that you have installed Logtail on the specified server before you create a server group. After you install Logtail, click Complete Installation to create a server group. If you have created a server group, click Use Existing Server Groups.

   Select a server group and move the server group from Source Server Groups to Applied Server Groups.

5. Configure a data source.
   1. Specify Config Name and Log Path. Then, set Mode to Full Regex Mode.
   2. Turn on Singleline.
   3. Specify Log Sample.
   4. Turn on Extract Field.
   5. Set RegEx.
      • Select fields in the log sample to generate a regular expression.

        If the regular expression that is automatically generated does not match your log sample, you can select fields in the log sample to generate a regular expression. Log Service automatically parses the fields you selected in the log sample to generate a regular expression. You can select log fields in Log Sample and click Generate Regular Expression. A regular expression of each selected field is displayed in RegEx. To generate a full regular expression for the log sample, you can select log fields in the log sample for multiple times.

        
      • Modify a regular expression.

        Actual data formats may vary. In this case, you can click Manual under the text box of RegEx to adjust the regular expression that is automatically generated based on the actual situation. This ensures that the regular expression is suitable for all formats of the collected logs.

      • Validate a regular expression.

        After you modify a regular expression, click Validate next to the text box of RegEx. If a regular expression is correct, the extraction results are displayed. If a regular expression is incorrect, rectify it.

   6. Confirm the extraction results of log fields.

      View the extraction results of log fields and specify keys for the extracted fields.

      Specify a descriptive field name for each log field in the extraction results. For example, specify a time for the time field. If you do not use the system time, you must specify a field where both the value and key are time.
   7. Turn on Use System Time.

      If you use the system time, the time of each log is the time when Logtail parses the log.

   8. Optional. Configure Advanced Options.

   After you complete the Logtail configuration, apply the configuration to the server group to collect Python logs.

6. Configure log query and analysis.

   An index is configured by default. You can reconfigure the index as needed.