edit-icon download-icon

Python logs

Last Updated: Mar 05, 2018

The logging module of Python provides a general logging system, which can be used by third-party modules or applications. The logging module provides different log levels and records logs in many methods, such as file, HTTP GET/POST, SMTP, and Socket. You can customize a log recording method as needed. The logging module has the same mechanism as Log4j except for the different implementation details. The logging module provides the logger, handler, filter, and formatter features.

Python log format

Log format

The log format specifies the specific output format of log recording in formatter. The construction method of formatter needs two parameters: message format string and message date string. Both of the parameters are optional.

Python log format:

  1. import logging
  2. import logging.handlers
  3. LOG_FILE = 'tst.log'
  4. handler = logging.handlers.RotatingFileHandler(LOG_FILE, maxBytes = 1024*1024, backupCount = 5) # Instantiate the handler
  5. fmt = '%(asctime)s - %(filename)s:%(lineno)s - %(name)s - %(message)s'
  6. formatter = logging.Formatter(fmt) # Instantiate the formatter
  7. handler.setFormatter(formatter) # Add the formatter to the handler
  8. logger = logging.getLogger('tst') # Obtain the logger named tst
  9. logger.addHandler(handler) # Add the handler to the logger
  10. logger.setLevel(logging.DEBUG)
  11. logger.info('first info message')
  12. logger.debug('first debug message')

Field description

The formatter is configured in the %(key)s format, that is, replacing the dictionary keywords. The following keywords are provided.

Format Description
%(name)s The logger name of the generated log.
%(levelno)s The log level in numeric format, including DEBUG, INFO, WARNING, ERROR, and CRITICAL.
%(levelname)s The log level in text format, including DEBUG, INFO, WARNING, ERROR, and CRITICAL.
%(pathname)s The full path of the source file where the statement that outputs the log resides (if available).
%(filename)s The file name.
%(module)s The name of the module where the statement that outputs the log resides.
%(funcName)s The name of the function that calls the log output.
%(lineno)d The code line where the function statement that calls the log output resides (if available).
%(created)f The time (in the UNIX standard time format) when the log is created, which indicates the number of seconds since 1970-1-1 00:00:00 UTC.
%(relativeCreated)d The interval (in milliseconds) between the log created time and the time that the logging module is loaded.
%(asctime)s The log created time, which is in the format of “2003-07-08 16:49:45,896” by default (the number after the comma (,) is the number of milliseconds).
%(msecs)d The log created time in the millisecond level.
%(thread)d The thread ID (if available).
%(threadName)s The thread name (if available).
%(process)d The process ID (if available).
%(message)s The log message.

Log sample

  1. 2015-03-04 23:21:59,682 - log_test.py:16 - tst - first info message
  2. 2015-03-04 23:21:59,682 - log_test.py:17 - tst - first debug message

Configure Logtail to collect Python logs

For the detailed procedure of collecting Python logs by using Logtail, see Quick start and Apache logs. Select the corresponding configuration based on your network deployment and actual situation.

The automatically generated regular expression is only based on the log sample and does not cover all the situations of logs. Therefore, you must adjust the regular expression slightly after it is automatically generated.

See the following common Python logs and the corresponding regular expressions:

  • Log sample:

    1. 2016-02-19 11:03:13,410 - test.py:19 - tst - first debug message

    Regular expression:

    1. (\d+-\d+-\d+\s\S+)\s+-\s+([^:]+):(\d+)\s+-\s+(\w+)\s+-\s+(.*)
  • Log format:

    1. %(asctime)s - %(filename)s:%(lineno)s - %(levelno)s %(levelname)s %(pathname)s %(module)s %(funcName)s %(created)f %(thread)d %(threadName)s %(process)d %(name)s - %(message)s

    Log sample:

    1. 2016-02-19 11:06:52,514 - test.py:19 - 10 DEBUG test.py test <module> 1455851212.514271 139865996687072 MainThread 20193 tst - first debug message

    Regular expression:

    1. (\d+-\d+-\d+\s\S+)\s-\s([^:]+):(\d+)\s+-\s+(\d+)\s+(\w+)\s+(\S+)\s+(\w+)\s+(\S+)\s+(\S+)\s+(\d+)\s+(\w+)\s+(\d+)\s+(\w+)\s+-\s+(.*)
Thank you! We've received your feedback.