The Python logging module provides a general logging system, which can be used for third-party modules or applications.

The Python logging module provides different log levels and logging methods such as file-based, HTTP GET, HTTP POST, SMTP, and Socket logging. You can also customize a logging method as needed. The Python logging module functions the same as the Log4j logging module except for their implementation details. The Python logging module provides logger, handler, filter, and formatter components.

Python log format

Formatter specifies the output format of log records. It uses two parameters to specify the log format: message format string and message date string. Both of the parameters are optional.

Log format:

import logging  
import logging.handlers  
LOG_FILE = 'tst.log'  
handler = logging.handlers.RotatingFileHandler(LOG_FILE, maxBytes = 1024*1024, backupCount = 5) # Instantiate the handler.   
fmt = '%(asctime)s - %(filename)s:%(lineno)s - %(name)s - %(message)s'  
formatter = logging.Formatter(fmt)   # Instantiate the formatter.  
handler.setFormatter(formatter)      # Add the formatter to the handler.  
logger = logging.getLogger('tst')    # Obtain the logger named tst.  
logger.addHandler(handler)           # Add the handler to the logger.  
logger.setLevel(logging.DEBUG)  
logger.info('first info message')  
logger.debug('first debug message')

Field description

The fields of the formatter are in the %(key)s format. The following table describes the fields of the formatter.
Field Description
%(name)s The name of the logger that generates a log.
%(levelno)s The log level in numeric format. Valid values: DEBUG, INFO, WARNING, ERROR, and CRITICAL.
%(levelname)s The log level in text format. Valid values: DEBUG, INFO, WARNING, ERROR, and CRITICAL.
%(pathname)s The full path of the source file that contains the statement used to generate the log.
%(filename)s The name of the source file.
%(module)s The name of the module where the statement used to generate the log resides.
%(funcName)s The name of the function that calls the log output function.
%(lineno)d The line of code that contains the statement used to call the log output function.
%(created)f The time when the log is generated, in the UNIX time format. It indicates the number of seconds that have elapsed since the epoch time January 1, 1970 00:00:00 (UTC).
%(relativeCreated)d The interval between the time when a log is created and the time when the logging module is loaded, in milliseconds.
%(asctime)s The time when the log is created. The value of 2003-07-08 16:49:45,896 is an example of the default format. The number after the comma (,) indicates the number of milliseconds.
%(msecs)d The time when the log is created, in milliseconds.
%(thread)d The ID of a thread.
%(threadName)s The name of a thread.
%(process)d The ID of a process.
%(message)s The log message.

Sample logs

Sample output logs:

2015-03-04 23:21:59,682 - log_test.py:16 - tst - first info message   
2015-03-04 23:21:59,682 - log_test.py:17 - tst - first debug message

Common Python logs and related regular expressions:

  • Sample log:
    2016-02-19 11:03:13,410 - test.py:19 - tst - first debug message
    Regular expression:
    (\d+-\d+-\d+\s\S+)\s+-\s+([^:]+):(\d+)\s+-\s+(\w+)\s+-\s+(. *)
  • Log format:
    %(asctime)s - %(filename)s:%(lineno)s - %(levelno)s %(levelname)s %(pathname)s %(module)s %(funcName)s %(created)f %(thread)d %(threadName)s %(process)d %(name)s - %(message)s
    Sample log:
    2016-02-19 11:06:52,514 - test.py:19 - 10 DEBUG test.py test <module> 1455851212.514271 139865996687072 MainThread 20193 tst - first debug message

    Regular expression:

    (\d+-\d+-\d+\s\S+)\s-\s([^:]+):(\d+)\s+-\s+(\d+)\s+(\w+)\s+(\S+)\s+(\w+)\s+(\S+)\s+(\S+)\s+(\d+)\s+(\w+)\s+(\d+)\s+(\w+)\s+-\s+(. *)

Configure Logtail to collect Python logs

Select configurations based on the network deployment and your actual situation. For more information, see Get started.

  1. Create a project and a Logstore. For more information, see Preparations.
  2. Click Import Data. On the Import Data page that appears, click RegEx - Text Log.
  3. Select a Logstore.

    You can select an existing Logstore or create a project or Logstore.

  4. Create and configure a server group.

    Make sure that you have installed Logtail on the specified server before you create a server group. After you install Logtail, click Complete Installation to create a server group. If you have created a server group, click Use Existing Server Groups.

    Select a server group and move the server group from Source Server Groups to Applied Server Groups.

  5. Configure a data source.
    1. Specify Config Name and Log Path. Then, set Mode to Full Regex Mode.
    2. Turn on Singleline.
    3. Specify Log Sample.
    4. Turn on Extract Field.
    5. Set RegEx.
      • Select fields in the log sample to generate a regular expression.

        If the regular expression that is automatically generated does not match your log sample, you can select fields in the log sample to generate a regular expression. Log Service automatically parses the fields you selected in the log sample to generate a regular expression. You can select log fields in Log Sample and click Generate Regular Expression. A regular expression of each selected field is displayed in RegEx. To generate a full regular expression for the log sample, you can select log fields in the log sample for multiple times.

        Regular expression
      • Modify a regular expression.

        Actual data formats may vary. In this case, you can click Manual under the text box of RegEx to adjust the regular expression that is automatically generated based on the actual situation. This ensures that the regular expression is suitable for all formats of the collected logs.

      • Validate a regular expression.

        After you modify a regular expression, click Validate next to the text box of RegEx. If a regular expression is correct, the extraction results are displayed. If a regular expression is incorrect, rectify it.

    6. Confirm the extraction results of log fields.

      View the extraction results of log fields and specify keys for the extracted fields.

      Specify a descriptive field name for each log field in the extraction results. For example, specify a time for the time field. If you do not use the system time, you must specify a field where both the value and key are time.Extraction results of log fields
    7. Turn on Use System Time.

      If you use the system time, the time of each log is the time when Logtail parses the log.

    8. Optional. Configure Advanced Options.

    After you complete the Logtail configuration, apply the configuration to the server group to collect Python logs.

  6. Configure log query and analysis.

    An index is configured by default. You can reconfigure the index as needed.