This topic describes how to use logging handlers to automatically upload Python logs.

Benefits

Logging handlers provided by Python SDK can automatically upload logs of Python applications to Log Service without storing the logs on a local disk. Compared with other tools that store logs in files before uploading them, logging handlers have the following benefits:
  • Logs are uploaded in real time without being locally stored.
  • Massive volumes of logs can be uploaded in an asynchronous manner.
  • Logging handlers are easy to configure, requiring no modification of code. The configurations immediately take effect after you modify the configuration file. The physical location of the host is not required for uploading logs.
  • JSON formatted log information and key-value formatted log information are automatically parsed.

Configurations

Logging handlers are compatible with the Python logging module. For more information about the Python logging module, see Python Logging.

The Python logging module allows you to configure logging by using a script or using a configuration file. The following example describes how to configure logging by using the logging.conf file.
[loggers]
keys=root,sls

[handlers]
keys=consoleHandler, slsHandler

[formatters]
keys=simpleFormatter, rawFormatter

[logger_root]
level=DEBUG
handlers=consoleHandler

[logger_sls]
level=INFO
handlers=consoleHandler, slsHandler
qualname=sls
propagate=0

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[handler_slsHandler]
class=aliyun.log.QueuedLogHandler
level=INFO
formatter=rawFormatter
args=(os.environ.get('ALIYUN_LOG_SAMPLE_ENDPOINT', ''), os.environ.get('ALIYUN_LOG_SAMPLE_ACCESSID', ''), os.environ.get('ALIYUN_LOG_SAMPLE_ACCESSKEY', ''), os.environ.get('ALIYUN_LOG_SAMPLE_TMP_PROJECT', ''), "logstore")

[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s


[formatter_rawFormatter]
format=%(message)s
A root logging handler and a sls logging handler are configured. The sls logging handler is an object that is created from the aliyun.log.QueuedLogHandler class. The following shows the parameters that are passed for the configurations of the sls logging handler. For more information about the parameters, see the parameter list.
args=(os.environ.get('ALIYUN_LOG_SAMPLE_ENDPOINT', ''), os.environ.get('ALIYUN_LOG_SAMPLE_ACCESSID', ''), os.environ.get('ALIYUN_LOG_SAMPLE_ACCESSKEY', ''), os.environ.get('ALIYUN_LOG_SAMPLE_TMP_PROJECT', ''), "logstore")
Note os.environ is used to obtain the values of environment variables. You can also enter actual parameter values.

Upload logs

You can use the logging configuration file to print logs to Log Service.
import logging
import logging.config

# Configurations
logging.config.fileConfig('logging.conf')
logger = logging.getLogger('sls')

# Use logger
logger.info("test1")

try:
    1/0
except ZeroDivisionError as ex:
    logger.exception(ex)

Then, logs are automatically uploaded to Log Service. To use the search and analytics feature, you must enable the index of the corresponding Logstore.

Configure the index for a Logstore

Enable the index of the Logstore that receives logs and configure the index for specific fields. We recommend that you use Command Line Interface - CLI to configure the index as follows:
aliyunlog log update_index --project_name="project1" --logstore_name="logstore1" --index_detail="file:///Users/user1/loghandler_index.json"

For more information about the configuration file, see python_logging_handler_index.json.

Change the log fields to be collected

The following table shows the log fields that Log Service support. By default, all of these fields are collected.
Field Description
message The content of the log.
record_name The name of the logging handler. In the preceding example, the record_name field is sls.
level The level of the log, such as INFO and ERROR.
file_path The full path of the configuration file.
func_name The name of the function that is used to print the log.
line_no The number of the code line where the function is called.
module The module where the function resides.
thread_id The ID of the current thread.
thread_name The name of the current thread.
process_id The ID of the current process.
process_name The name of the current process.

You can choose the fields to be collected based on the fields parameter of QueuedLogHandler. For more information, see aliyun.log.LogFields.

In the following example, the preceding configuration file is modified to only collect the level, func_name, module, and line_no fields.
[handler_slsHandler]
class=aliyun.log.QueuedLogHandler
level=INFO
formatter=rawFormatter
args=('cn-beijing.log.aliyuncs.com', 'ak_id', 'ak_key', 'project1', "logstore1", 'mytopic', ['level', 'func_name', 'module', 'line_no']  )
Note
  • The message field is collected regardless of your configurations.
  • You can also add a prefix and suffix to the names of these fields by using the buildin_fields_prefix and buildin_fields_suffix parameters. An example of a prefix or suffix is __level__.

Configure logging by using a JSON scrip

You can use a JSON scrip to achieve a more flexible logging configuration.
#encoding: utf8
import logging, logging.config, os

# Configurations
conf = {'version': 1,
        'formatters': {'rawformatter': {'class': 'logging.Formatter',
                                        'format': '%(message)s'}
                       },
        'handlers': {'sls_handler': {'()':
                                     'aliyun.log.QueuedLogHandler',
                                     'level': 'INFO',
                                     'formatter': 'rawformatter',

                                     # custom args:
                                     'end_point': os.environ.get('ALIYUN_LOG_SAMPLE_ENDPOINT', ''),
                                     'access_key_id': os.environ.get('ALIYUN_LOG_SAMPLE_ACCESSID', ''),
                                     'access_key': os.environ.get('ALIYUN_LOG_SAMPLE_ACCESSKEY', ''),
                                     'project': 'project1',
                                     'log_store': "logstore1"
                                     }
                     },
        'loggers': {'sls': {'handlers': ['sls_handler', ],
                                   'level': 'INFO',
                                   'propagate': False}
                    }
        }
logging.config.dictConfig(conf)

# Use the logger
logger = logging.getLogger('sls')
logger.info("Hello world")
Note QueuedLogHandler is initialized by passing named parameters. For more information about parameters, see the parameters.

For more information about how to use dicConfig in Python, see dictConfig.

Use Python logging handlers for uWSGI logs

QueuedLogHandler cannot perform as expected due to the scheduling of uWSGI processes. The following two logging handlers are provided as alternatives to upload uWSGI logs.
  • UwsgiQueuedLogHandler. It has the same functions and configurations as the QueuedLogHandler class. To use this class, you must install the third-party library uwsgidecorators.
  • SimpleLogHandler. It can upload logs in real time and has the same configurations as the QueuedLogHandler class. We recommend that you only use this class for testing purposes.