Python logs

Last Updated: Jul 05, 2017

Python logging module provides a general logging system for use by third-party modules or applications. The logging module provides different log levels and records logs by different methods including file, HTTP GET/POST, SMTP, and Socket. You can customize a log recording method as needed. The logging module is the same as Log4j except that they have different implementation details. The logging module provides the logger, handler, filter, and formatter features.

Python log format

The formatter specifies the log record output format. The formatter construction mode are defined by two parameters: message format string and message date string. The parameters are optional.

Python sample log:

  1. import logging
  2. import logging.handlers
  3. LOG_FILE = 'tst.log'
  4. handler = logging.handlers.RotatingFileHandler(LOG_FILE, maxBytes = 1024*1024, backupCount = 5) # Instantiate the handler
  5. fmt = '%(asctime)s - %(filename)s:%(lineno)s - %(name)s - %(message)s'
  6. formatter = logging.Formatter(fmt) # Instantiate the formatter
  7. handler.setFormatter(formatter) # Add the formatter to the handler
  8. logger = logging.getLogger('tst') # Obtain the logger named tst
  9. logger.addHandler(handler) # Add the handler to the logger
  10. logger.setLevel(logging.DEBUG)
  11. logger.info('first info message')
  12. logger.debug('first debug message')

Output sample log:

  1. 2015-03-04 23:21:59,682 - log_test.py:16 - tst - first info message
  2. 2015-03-04 23:21:59,682 - log_test.py:17 - tst - first debug message

The formatter is configured in the %(key)s format, that is, replacing the dictionary keywords. The following keywords are provided.

Format Description
%(name)s Name of the logger (logging channel).
%(levelno)s Numeric logging level for the message (DEBUG, INFO, WARNING, ERROR, CRITICAL).
%(levelname)s Text logging level for the message (‘DEBUG’, ‘INFO’, ‘WARNING’, ‘ERROR’, ‘CRITICAL’).
%(pathname)s Full path of the source file where the logging call was issued (if available).
%(filename)s File name portion of the path name.
%(module)s Module (name portion of the file name).
%(funcName)s Name of function containing the logging call.
%(lineno)d Source line number where the logging call was issued (if available).
%(created)f Time when the LogRecord was created, in UNIX standard time format and seconds calculated from 1970-1-1 00:00:00 UTC.
%(relativeCreated)d Time in milliseconds when the LogRecord was created, relative to the time the logging module was loaded.
%(asctime)s Human-readable time when the LogRecord was created. By default this is of the form “2003-07-08 16:49:45,896” (the numbers after the comma are millisecond portion of the time).
%(msecs)d Millisecond portion of the time when the LogRecord was created.
%(thread)d Thread ID (if available).
%(threadName)s Thread name (if available).
%(process)d Process ID (if available).
%(message)s The logged message.

Python log collection by Logtail

For details about the standard process, refer to Quick start.

The automatically generated regular expression is based on the sample log and does not cover every log type. Therefore, you need to tune the regular expression after it is generated.

A regular expression and other information for a common Python log is shown below.

  • Log sample:

    1. 2016-02-19 11:03:13,410 - test.py:19 - tst - first debug message

    Regular expression:

    1. (\d+-\d+-\d+\s\S+)\s+-\s+([^:]+):(\d+)\s+-\s+(\w+)\s+-\s+(.*)
  • Log format:

    1. %(asctime)s - %(filename)s:%(lineno)s - %(levelno)s %(levelname)s %(pathname)s %(module)s %(funcName)s %(created)f %(thread)d %(threadName)s %(process)d %(name)s - %(message)s

    Log output:

    1. 2016-02-19 11:06:52,514 - test.py:19 - 10 DEBUG test.py test <module> 1455851212.514271 139865996687072 MainThread 20193 tst - first debug message

    Regular expression:

    1. (\d+-\d+-\d+\s\S+)\s-\s([^:]+):(\d+)\s+-\s+(\d+)\s+(\w+)\s+(\S+)\s+(\w+)\s+(\S+)\s+(\S+)\s+(\d+)\s+(\w+)\s+(\d+)\s+(\w+)\s+-\s+(.*)
Thank you! We've received your feedback.