Handling your Logs

After fellow DES7 member, Joe O'Connor's introductory blog on logging, I thought it'd be worth adding to his work, and going into one of loggings most useful tools.
Handlers.
While the logger in python creates the records, you can utilise handlers to decide where those logs go.

File Handler

One of the most common handlers to use is the logging.FileHandler('file_name'), which will add the log message into a logging file which you have created.
This is really useful for when you want to be able to retroactively review what happened after your code ran, but there are some instances where you may want to see what's happening while your code runs. In these instance you may want to use a logging.StreamHandler(). It sends logs to a stream such as sys.stdout (standard output) or sys.stderr (standard error).

Stream Handler

A stream handler can be configured to 'print' to your standard error
logging.StreamHandler(stream=sys.stderr)
or your standard output
logging.StreamHandler(stream=sys.stdout)
In your terminal, these will look the same, but there are use cases where the distinction matters
One of which is within an orchestrator such as Kestra. I won't go into too much detail as to what an orchestrator is and how they work, but for now just know they are places your code can be scheduled to run. Within Kestra if you've set up these stream handlers you can have different logging instances tagged as info (standard output) and the others flagged as errors (standard error).
stdout handler → “INFO logs in Kestra UI (non-error)”
stderr handler → “ERROR/WARNING logs in Kestra UI”

Other Handlers

These are just a couple examples of handlers, focused on logging locally. But as I'm sure you can imagine there are situations where you'd like to log to more remote locations. There are email handlers (SMTPHandler), web handlers (HTTPHandler), System log Handlers (SysLogHandler) and many more. So when logging make sure to consider all your handling options.

Defining a Logging Function with Multiple Handlers

import logging
import os
import sys

def log_config(timestamp: str, log_dir: str):
    # Ensure the target logging directory exists
    os.makedirs(log_dir, exist_ok=True)
    log_filename = f"{log_dir}/{timestamp}.log"

    # Grab the root logger and clear any default handlers
    logger = logging.getLogger(__name__)
    logger.setLevel(logging.DEBUG)
    logger.handlers = []

    # Define a consistent format across all outputs
    formatter = logging.Formatter(
        '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
    )

    # 1. Route INFO and above to stdout
    stdout_handler = logging.StreamHandler(stream=sys.stdout)
    stdout_handler.setLevel(logging.INFO)
    stdout_handler.setFormatter(formatter)

    # 2. Route WARNING and above to stderr
    stderr_handler = logging.StreamHandler(stream=sys.stderr)
    stderr_handler.setLevel(logging.WARNING)
    stderr_handler.setFormatter(formatter)

    # 3. Save EVERYTHING (DEBUG and above) to a physical log file
    file_handler = logging.FileHandler(log_filename)
    file_handler.setLevel(logging.DEBUG)
    file_handler.setFormatter(formatter)

    # Attach handlers to our logger
    logger.addHandler(stdout_handler)
    logger.addHandler(stderr_handler)
    logger.addHandler(file_handler)

    return logger