Mastering Logging in Python

Logging is an indispensable tool in a programmer’s kit. It aids in understanding the flow of a program and uncovering unforeseen scenarios during development.

Logs give developers an extra set of eyes to monitor the flow of an application. They can capture details such as which user accessed the application and, in the event of an error, provide insights beyond a simple stack trace, revealing the state of the program leading up to the error.

By strategically logging useful data, you can not only debug errors with ease but also analyze application performance to plan for scaling or track usage patterns for marketing strategies.

Python’s standard library includes a logging system, streamlining the addition of logging to your app. This article explores why adopting this module is optimal for logging and provides a quick start guide, along with an introduction to its advanced features.

The Logging Module

The logging module in Python caters to both novices and enterprise teams with its robust design. It's widely used by third-party libraries, enabling seamless integration of log messages into a cohesive log for your app.

Quick Start

Adding logging is straightforward:

import logging

Using a "logger," you can log messages you want to monitor. There are five standard levels of severity:

• DEBUG
• INFO
• WARNING
• ERROR
• CRITICAL

Here's an example of logging messages at each severity level:

import logging

logging.debug('This is a debug message')
logging.info('This is an info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')

Output:

WARNING:root:This is a warning message
ERROR:root:This is an error message
CRITICAL:root:This is a critical message

Notice root in the output, which is the default logger’s name. The debug() and info() messages aren’t logged by default because the logging module logs messages with severity WARNING or higher. This can be adjusted by configuring the module to include all levels.

Basic Configurations

Utilize basicConfig() to set up logging:

Python’s logging module uses camelCase naming from Java’s Log4j, impacting PEP8 style compliance. Nevertheless, it ensures backwards compatibility.

Common Parameters:

• level: Sets the root logger’s severity level.
• filename: Specifies the output file.
• filemode: Determines the file’s opening mode (a by default for append).
• format: Dictates the log message format.

To log all levels, set:

import logging

logging.basicConfig(level=logging.DEBUG)
logging.debug('This will get logged')

Output:

DEBUG:root:This will get logged

For logging to a file and defining format, use:

import logging

logging.basicConfig(filename='app.log', filemode='w', format='%(name)s - %(levelname)s - %(message)s')
logging.warning('This will get logged to a file')

The log appears in app.log like:

root - ERROR - This will get logged to a file

Formatting the Output

Include dynamic data in logs easily with LogRecord attributes. For example, to log a process ID:

import logging

logging.basicConfig(format='%(process)d-%(levelname)s-%(message)s')
logging.warning('This is a Warning')

Example:

import logging

logging.basicConfig(format='%(asctime)s - %(message)s', level=logging.INFO)
logging.info('Admin logged in')

Output:

2018-07-11 20:12:06,288 - Admin logged in

Logging Variable Data

Use format strings to include variables directly:

import logging

name = 'John'
logging.error('%s raised an error', name)

F-strings:

import logging

name = 'John'
logging.error(f'{name} raised an error')

Capturing Stack Traces

Capture stack traces with exc_info:

import logging

a = 5
b = 0

try:
  c = a / b
except Exception as e:
  logging.error("Exception occurred", exc_info=True)

For succinct handling, use logging.exception():

import logging

a = 5
b = 0
try:
  c = a / b
except Exception as e:
  logging.exception("Exception occurred")

Classes and Functions

Create custom loggers with Logger objects for multi-module apps.

Classes include:

• Logger: Direct usage in applications.
• LogRecord: Automatically created, holds event info.
• Handler: Sends logs to destinations.
• Formatter: Specifies output format.

Using Handlers

Handlers manage log messages' destinations, allowing different severities for each. Example:

import logging

logger = logging.getLogger(__name__)

c_handler = logging.StreamHandler()
f_handler = logging.FileHandler('file.log')
c_handler.setLevel(logging.WARNING)
f_handler.setLevel(logging.ERROR)

c_format = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
f_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
c_handler.setFormatter(c_format)
f_handler.setFormatter(f_format)

logger.addHandler(c_handler)
logger.addHandler(f_handler)

logger.warning('This is a warning')
logger.error('This is an error')

Other Configuration Methods

Configurations can be loaded from files or dictionaries:

File Configuration:

[loggers]
keys=root,sampleLogger

... (rest of the file config)

Load with: fileConfig()

Dictionary Approach (YAML):

version: 1
... (YAML content)

Load with:

import logging.config
yaml

with open('config.yaml', 'r') as f:
    config = yaml.safe_load(f.read())
    logging.config.dictConfig(config)

Keep Calm and Read the Logs

Python’s logging module is versatile and practical, fitting most use cases. Whether for a small project or a large-scale application, logging can significantly ease development challenges and open up further insights for advancement.