Intermediate

How To Use Python Dataclasses For Clean Data Structures

You’re building a model to represent a user, a product, an order—some structured data. In older Python, you’d write boilerplate code: __init__ to initialize fields, __repr__ to show a nice string representation, __eq__ to compare instances. A simple 5-field data class would require 30+ lines of code. Dataclasses, added in Python 3.7, solve this problem entirely. One decorator (@dataclass) gives you automatic __init__, __repr__, __eq__, and more. Your class definition becomes clean, readable, and type-safe. This is one of the best quality-of-life improvements in modern Python.

In this article, we’ll explore dataclasses from basics to advanced patterns. You’ll learn when to use them, how to configure them, and how to build practical systems with them. By the end, you’ll write less boilerplate and more focused business logic.

Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Dataclasses: Quick Example

Here’s what would normally require 20+ lines of boilerplate code, now expressed as pure declarations:

#simple_dataclass.py
from dataclasses import dataclass

@dataclass
class Person:
    name: str
    age: int
    email: str

person = Person("Alice", 30, "alice@example.com")
print(person)
# Person(name='Alice', age=30, email='alice@example.com')

person2 = Person("Alice", 30, "alice@example.com")
print(person == person2)
# True

Output:

Person(name='Alice', age=30, email='alice@example.com')
True

The @dataclass decorator gave us an __init__ that accepts all three fields, a __repr__ that shows the class and values, and __eq__ that compares instances by their fields. All with zero boilerplate code. This is why dataclasses are a game-changer for Python development.

Why Dataclasses Beat Dictionaries and Simple Classes

You might think “Can’t I just use dictionaries?” or “Why not write a normal class?” Let’s compare the approaches side by side. Each approach has tradeoffs, and dataclasses hit the sweet spot for most scenarios:

#comparison.py
from dataclasses import dataclass

# Dictionary approach (old way)
person_dict = {
    'name': 'Bob',
    'age': 28,
    'email': 'bob@example.com'
}
print(person_dict['name'])  # Must use quotes, typos aren't caught
print(person_dict.get('phone'))  # Missing keys return None silently

# Regular class (lots of boilerplate)
class PersonClass:
    def __init__(self, name, age, email):
        self.name = name
        self.age = age
        self.email = email

    def __repr__(self):
        return f'PersonClass(name={self.name}, age={self.age}, email={self.email})'

person_class = PersonClass('Bob', 28, 'bob@example.com')
print(person_class.name)  # Attribute access is nicer

# Dataclass approach (best of both worlds)
@dataclass
class Person:
    name: str
    age: int
    email: str

person = Person('Bob', 28, 'bob@example.com')
print(person.name)  # Attribute access like classes
print(person)  # Nice repr for free
# Person(name='Bob', age=28, email='bob@example.com')

Output:

Bob
None
Bob
Person(name='Bob', age=28, email='bob@example.com')

Dataclasses give you attribute access (cleaner than dict[‘key’]), automatic __repr__ (debug-friendly output), type hints (IDE support, safety), and zero boilerplate. Compared to dictionaries, you get type safety and better tooling. Compared to regular classes, you lose nothing but lines of code. They’re the right choice for modeling data.

Basic Dataclass Syntax With Methods

Dataclasses aren’t just data containers—they can have methods too. This makes them useful for modeling entities with both state and behavior:

#dataclass_with_methods.py
from dataclasses import dataclass

@dataclass
class Point:
    x: float
    y: float

p = Point(3.5, 4.2)
print(f"Point coordinates: ({p.x}, {p.y})")

@dataclass
class Rectangle:
    width: float
    height: float

    def area(self):
        # Calculate area using width and height
        return self.width * self.height

    def perimeter(self):
        # Calculate perimeter
        return 2 * (self.width + self.height)

rect = Rectangle(5, 3)
print(f"Area: {rect.area()}")
print(f"Perimeter: {rect.perimeter()}")

Output:

Point coordinates: (3.5, 4.2)
Area: 15
Perimeter: 16

Dataclasses combine data (the fields) with behavior (the methods). This is object-oriented programming the right way—encapsulation of related data and operations.

Default Values and Field Configuration

Often your fields have default values, or they need special handling (like lists that start empty). The field() function from dataclasses gives you fine-grained control:

#dataclass_defaults.py
from dataclasses import dataclass, field
from datetime import datetime
from typing import List

@dataclass
class Article:
    title: str
    content: str
    author: str = "Anonymous"  # Simple default
    created_at: datetime = field(default_factory=datetime.now)  # Factory for mutable defaults
    tags: List[str] = field(default_factory=list)  # Always use field for lists
    views: int = 0
    is_published: bool = False

article1 = Article("Python Tips", "Learn about dataclasses...")
print(article1)

article2 = Article(
    "Flask Tutorial",
    "Build APIs with Flask...",
    author="John Doe",
    is_published=True
)
print(article2)

Output:

Article(title='Python Tips', content='Learn...', author='Anonymous', created_at=datetime.datetime(2026, 3, 12, 10, 15, 30), tags=[], views=0, is_published=False)
Article(title='Flask Tutorial', content='Build...', author='John Doe', created_at=datetime.datetime(2026, 3, 12, 10, 16, 45), tags=[], views=0, is_published=True)

Notice that created_at uses field(default_factory=…). This is important: mutable defaults (lists, dicts, datetime.now()) must use default_factory, otherwise all instances share the same list. Without field(), if you created two articles, both would share the same tags list (a common bug). The field() function prevents this.

Frozen Dataclasses: Immutability When You Need It

Sometimes you want data that can’t be changed after creation. Colors, coordinates, configuration objects—these are often immutable for good reason. The frozen=True parameter makes dataclass instances immutable:

#frozen_dataclass.py
from dataclasses import dataclass

@dataclass(frozen=True)
class Color:
    red: int
    green: int
    blue: int

sky_blue = Color(135, 206, 235)
print(f"Sky blue: {sky_blue}")

try:
    sky_blue.red = 100  # Try to modify
except Exception as e:
    print(f"Error: {e}")

darker_blue = Color(115, 186, 215)
print(f"Darker blue: {darker_blue}")

Output:

Sky blue: Color(red=135, green=206, blue=235)
Error: cannot assign to field 'red'
Darker blue: Color(red=115, green=186, blue=215)

Immutable objects are safer—they can’t be modified accidentally, they’re thread-safe, and they work as dictionary keys. Use frozen=True for configuration objects, constants, and anything that should never change.

Inheritance: Building Class Hierarchies

Dataclasses work beautifully with inheritance. Child classes inherit parent fields and can add their own:

#dataclass_inheritance.py
from dataclasses import dataclass

@dataclass
class Employee:
    name: str
    employee_id: int
    salary: float

@dataclass
class Manager(Employee):
    department: str
    team_size: int

emp = Employee("Alice", 101, 50000)
mgr = Manager("Bob", 102, 80000, "Engineering", 5)

print(emp)
print(mgr)
print(f"Manager {mgr.name} manages {mgr.team_size} people")

Output:

Employee(name='Alice', employee_id=101, salary=50000)
Manager(name='Bob', employee_id=102, salary=80000, department='Engineering', team_size=5)
Manager Bob manages 5 people

Manager inherits name, employee_id, and salary from Employee, then adds department and team_size. The __init__ signature includes all fields in the right order. This is how you model real-world hierarchies—employee types, vehicle types, etc.

Dataclasses vs NamedTuple: Understanding the Differences

NamedTuple is similar to dataclasses but makes different tradeoffs. Both are useful, but for different scenarios. Let’s compare them so you can choose wisely:

#dataclass_vs_namedtuple.py
from dataclasses import dataclass
from typing import NamedTuple

@dataclass
class PersonDataclass:
    name: str
    age: int

class PersonNamedTuple(NamedTuple):
    name: str
    age: int

p1 = PersonDataclass("Charlie", 35)
p2 = PersonNamedTuple("Charlie", 35)

print(p1)
print(p2)

p1.age = 36  # Dataclass is mutable
print(f"Updated dataclass: {p1}")

try:
    p2.age = 36  # NamedTuple is immutable
except Exception as e:
    print(f"NamedTuple error: {e}")

name, age = p2  # NamedTuple supports unpacking
print(f"Unpacked: {name}, {age}")

Output:

PersonDataclass(name='Charlie', age=35)
PersonNamedTuple(name='Charlie', age=35)
Updated dataclass: PersonDataclass(name='Charlie', age=36)
NamedTuple error: can't set attribute
Unpacked: Charlie, 35

Key differences:

• Dataclasses: Mutable (can change fields), more flexible, better for modeling evolving objects
• NamedTuple: Immutable, lighter weight, can be unpacked like tuples, good for fixed data

Use dataclasses when you might modify objects or want methods. Use NamedTuple when you want immutable, lightweight data containers. For most modern Python code, dataclasses are the default choice.

Real-Life Example: Product Inventory System

Let’s build a practical inventory management system using dataclasses. This demonstrates real-world patterns: enums for categories, computed properties, and business logic:

#inventory_system.py
from dataclasses import dataclass, field
from datetime import datetime
from typing import List
from enum import Enum

class Category(Enum):
    ELECTRONICS = "Electronics"
    CLOTHING = "Clothing"
    BOOKS = "Books"

@dataclass
class Product:
    sku: str
    name: str
    category: Category
    price: float
    quantity_in_stock: int
    reorder_level: int = 10
    last_restocked: datetime = field(default_factory=datetime.now)

    def needs_restock(self) -> bool:
        return self.quantity_in_stock <= self.reorder_level

    def total_value(self) -> float:
        return self.price * self.quantity_in_stock

    def sell(self, quantity: int) -> bool:
        if quantity > self.quantity_in_stock:
            return False
        self.quantity_in_stock -= quantity
        return True

    def restock(self, quantity: int) -> None:
        self.quantity_in_stock += quantity
        self.last_restocked = datetime.now()

@dataclass
class Inventory:
    products: List[Product] = field(default_factory=list)

    def add_product(self, product: Product) -> None:
        self.products.append(product)

    def get_product(self, sku: str) -> Product:
        for product in self.products:
            if product.sku == sku:
                return product
        return None

    def low_stock_items(self) -> List[Product]:
        return [p for p in self.products if p.needs_restock()]

    def total_value(self) -> float:
        return sum(p.total_value() for p in self.products)

    def inventory_report(self) -> None:
        print("=== INVENTORY REPORT ===")
        for product in self.products:
            status = "LOW STOCK" if product.needs_restock() else "OK"
            print(f"{product.sku}: {product.name} - {product.quantity_in_stock} units ({status})")
        print(f"Total Inventory Value: ${self.total_value():.2f}")
        print(f"Items needing restock: {len(self.low_stock_items())}")

inventory = Inventory()
laptop = Product(sku="LAPTOP001",name="MacBook Pro 14 inch",category=Category.ELECTRONICS,price=1999.99,quantity_in_stock=5,reorder_level=3)
book = Product(sku="BOOK001",name="Python Mastery",category=Category.BOOKS,price=29.99,quantity_in_stock=2,reorder_level=10)
inventory.add_product(laptop)
inventory.add_product(book)
laptop.sell(2)
book.sell(1)
inventory.inventory_report()
book.restock(15)
print(f"\nAfter restocking: {book}")

Output:

=== INVENTORY REPORT ===
LAPTOP001: MacBook Pro 14 inch - 3 units (OK)
BOOK001: Python Mastery - 1 units (LOW STOCK)
Total Inventory Value: $5999.97
Items needing restock: 1

After restocking: Product(sku='BOOK001', name='Python Mastery', category=, price=29.99, quantity_in_stock=16, reorder_level=10, last_restocked=datetime.datetime(2026, 3, 12, 10, 30, 45))

This inventory system uses dataclasses effectively: Product holds item data with methods for business logic (sell, restock, total_value). Inventory aggregates products and provides reports. Enums ensure category values are consistent. Timestamps track when restocking happened. This is maintainable, testable code that scales from a small store to enterprise inventory systems.

Frequently Asked Questions

Q: When should I use dataclasses vs regular classes?

Use dataclasses when your class is primarily a data container. Use regular classes when you have complex initialization logic, property decorators, or inheritance patterns that don’t fit the dataclass model. When in doubt, start with a dataclass and upgrade to a regular class if needed.

Q: Can I use dataclasses with type hints?

Type hints are required for dataclasses—they tell the decorator which fields to create. This is a feature, not a limitation. Type hints make your code clearer and enable better IDE support, type checking, and documentation. Python’s type system is optional, but use it with dataclasses.

Q: How do I convert a dataclass to a dictionary?

Use the asdict() function from dataclasses module: from dataclasses import asdict; my_dict = asdict(my_dataclass). This is useful for JSON serialization, logging, or comparing to dictionaries.

Q: Can I add validators to dataclasses?

Use the __post_init__ method to validate after initialization: def __post_init__(self): if self.age < 0: raise ValueError("Age must be positive"). You can also use field with validate_before for pre-validation, or external libraries like Pydantic for more complex validation.

Q: Do dataclasses work with JSON serialization?

Convert to dict first with asdict(), then to JSON: json.dumps(asdict(my_dataclass)). For automatic JSON support with validation, consider Pydantic instead. Pydantic is built on dataclasses but adds JSON schema support and validation.

Conclusion

Dataclasses are a game-changer for Python developers. They eliminate boilerplate code, add type safety, and make your code more readable. Whether you're building simple data models, complex business objects, or inventory systems, dataclasses scale from simple to sophisticated use cases. Start using them in your next project and you'll never want to go back to writing __init__ methods by hand.

Key takeaways:

• Dataclasses eliminate __init__, __repr__, __eq__ boilerplate with one decorator
• Use field() for mutable defaults (lists, dicts) and special handling
• Use frozen=True for immutable objects
• Dataclasses support inheritance naturally
• Add methods to dataclasses for business logic—they're not just data containers
• Dataclasses work with type hints, IDE tooling, and modern Python patterns
• Use __post_init__ for validation after initialization
• Choose dataclasses over dictionaries for type safety and over regular classes for less boilerplate

References

• Python Dataclasses Documentation
• Python typing Module
• Python Enum Module
• Pydantic for Advanced Validation

Related Articles

• How To Use Python attrs Library for Data Classes Without Boilerplate
• How To Create Data Visualizations with Seaborn in Python
• How To Use Pydantic V2 for Data Validation in Python
• How To Store Data Separately For Testing and Production in Python 3
• How To Use Python dynaconf for Configuration Management
• How To Use Python snoop for Function Tracing and Debugging
• 7 Python Programming Myths

Intermediate

You have written a function that works perfectly. Then one day it starts running slowly in production, and you have no idea which function is the bottleneck. Or maybe you need to log every time a critical function gets called, but you do not want to litter every function body with print() statements. These are the exact problems that Python decorators solve — they let you wrap extra behavior around your functions without touching the function code itself.

The good news is that decorators are built into Python’s core syntax. You do not need to install anything — just standard Python 3. The concept uses closures and first-class functions, which sounds intimidating, but the pattern is surprisingly simple once you see it in action. By the end of this article you will be writing your own decorators from scratch with confidence.

In this article we will start with a quick working example so you can see the payoff immediately. Then we will break down what decorators actually are and how they work under the hood. From there we will build a logging decorator, a timing decorator, a retry decorator, and learn how to stack multiple decorators together. We will also cover functools.wraps — a small but critical detail that most tutorials skip. Finally, we will tie everything together with a real-life performance monitoring toolkit you can drop into any project.

Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Python Decorators for Logging: Quick Example

Let us jump straight into a working example. Here is a decorator that automatically logs every time a function is called, including the arguments it received and the value it returned. You can copy this code, run it, and see the result immediately.

# quick_example.py
import functools

def log_calls(func):
    """Decorator that logs function calls with arguments and return values."""
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        print(f"CALL: {func.__name__}({args}, {kwargs})")
        result = func(*args, **kwargs)
        print(f"RETURN: {func.__name__} -> {result}")
        return result
    return wrapper

@log_calls
def add(a, b):
    return a + b

print(add(3, 5))
print(add(10, 20))

Output:

CALL: add((3, 5), {})
RETURN: add -> 8
8
CALL: add((10, 20), {})
RETURN: add -> 30
30

With just the @log_calls line above the function definition, every call to add() now prints a log message automatically. The function itself has no idea it is being logged — it just does its job and returns the sum. That separation of concerns is the entire point of decorators. The functools.wraps call inside the decorator preserves the original function’s name and docstring, which we will explain in detail later.

Want to go deeper? Below we cover exactly how this pattern works, how to build timing and retry decorators, and how to combine multiple decorators on a single function.

What Are Python Decorators and Why Use Them?

A decorator is a function that takes another function as input, adds some behavior to it, and returns a new function. Think of it like gift wrapping — the present inside (your original function) stays exactly the same, but the wrapping paper (the decorator) adds something extra on the outside. When someone opens the gift, they get both the wrapping experience and the present itself.

In Python, functions are first-class objects. That means you can pass a function as an argument to another function, return a function from a function, and assign a function to a variable. Decorators take advantage of all three of these capabilities. The @decorator_name syntax above a function definition is just shorthand — writing @log_calls above def add() is exactly the same as writing add = log_calls(add) after the function definition.

Here is when decorators are the right tool for the job versus when you should use something else:

The key principle is that decorators work best when you want to add the same cross-cutting behavior to multiple functions. If you find yourself copying the same three lines of logging code into every function, that is a strong signal that a decorator would clean things up. Let us start building decorators from scratch.

Basic Decorator Syntax in Python

Before we build useful decorators, let us understand the basic pattern. A decorator is a function that accepts a function as its argument, defines an inner function (called a wrapper) that adds behavior, and returns the wrapper. Here is the simplest possible decorator that does nothing except call the original function.

# basic_decorator.py
def my_decorator(func):
    """A minimal decorator that wraps a function without changing behavior."""
    def wrapper(*args, **kwargs):
        # You could add behavior here (before the call)
        result = func(*args, **kwargs)
        # You could add behavior here (after the call)
        return result
    return wrapper

@my_decorator
def greet(name):
    return f"Hello, {name}!"

print(greet("Alice"))
print(greet("Bob"))

Output:

Hello, Alice!
Hello, Bob!

The wrapper function uses *args and **kwargs to accept any combination of positional and keyword arguments. This is important because the decorator needs to work with functions that have different signatures — a function with two parameters, a function with five parameters, or a function with keyword-only parameters should all work the same way. The result = func(*args, **kwargs) line calls the original function with whatever arguments were passed in, and return result passes the return value back to the caller.

Now let us replace those placeholder comments with real, useful behavior. We will start with a logging decorator.

Building a Logging Decorator

A logging decorator captures information about function calls as they happen. This is incredibly useful for debugging — instead of sprinkling print() statements throughout your code and then removing them later, you simply add or remove the @log_calls decorator. Here is a robust version that formats the output nicely and handles both positional and keyword arguments.

# logging_decorator.py
import functools
from datetime import datetime

def log_calls(func):
    """Log every call to the decorated function with timestamp, args, and result."""
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
        # Format arguments for readable output
        arg_parts = [repr(a) for a in args]
        kwarg_parts = [f"{k}={v!r}" for k, v in kwargs.items()]
        all_args = ", ".join(arg_parts + kwarg_parts)

        print(f"[{timestamp}] CALL: {func.__name__}({all_args})")
        result = func(*args, **kwargs)
        print(f"[{timestamp}] RETURN: {func.__name__} -> {result!r}")
        return result
    return wrapper

@log_calls
def calculate_discount(price, discount_percent=10):
    """Calculate the discounted price."""
    discount_amount = price * (discount_percent / 100)
    return round(price - discount_amount, 2)

# Test with positional and keyword arguments
print(calculate_discount(99.99))
print()
print(calculate_discount(49.99, discount_percent=25))

Output:

[2026-03-13 10:30:00] CALL: calculate_discount(99.99)
[2026-03-13 10:30:00] RETURN: calculate_discount -> 89.99
89.99

[2026-03-13 10:30:00] CALL: calculate_discount(49.99, discount_percent=25)
[2026-03-13 10:30:00] RETURN: calculate_discount -> 37.49
37.49

Notice how the decorator captures the function name, the exact arguments passed (including keyword arguments like discount_percent=25), and the return value. The !r format specifier uses repr() so strings show up with quotes, making it easy to distinguish "hello" from hello in your logs. The timestamp tells you exactly when each call happened, which is essential for debugging timing-related issues in production. You can easily swap print() for Python’s built-in logging module to write these messages to a file instead of the console.

Building a Timing Decorator

Performance matters, and the first step to improving performance is measuring it. A timing decorator wraps a function and records how long it takes to execute. This is far more convenient than manually adding time.time() calls at the start and end of every function you want to profile.

# timing_decorator.py
import functools
import time

def timer(func):
    """Measure and print the execution time of the decorated function."""
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        start_time = time.perf_counter()  # High-precision timer
        result = func(*args, **kwargs)
        end_time = time.perf_counter()
        elapsed = end_time - start_time
        print(f"⏱ {func.__name__} took {elapsed:.4f} seconds")
        return result
    return wrapper

@timer
def slow_operation():
    """Simulate a slow operation."""
    total = 0
    for i in range(1_000_000):
        total += i * i
    return total

@timer
def fast_operation():
    """Simulate a fast operation."""
    return sum(range(1000))

print(f"Result: {slow_operation()}")
print(f"Result: {fast_operation()}")

Output:

⏱ slow_operation took 0.0892 seconds
Result: 333332833333500000
⏱ fast_operation took 0.0000 seconds
Result: 499500

We use time.perf_counter() instead of time.time() because it provides the highest resolution timer available on your operating system. This matters when measuring fast functions — time.time() might show 0.0 seconds for something that actually takes 0.3 milliseconds. The :.4f format gives us four decimal places, which is enough precision for most profiling needs. If you need sub-millisecond accuracy, change it to :.6f for microsecond resolution.

Using functools.wraps for Metadata Preservation

You may have noticed that every decorator we have written includes @functools.wraps(func) on the wrapper function. This is not optional — without it, the decorated function loses its identity. Let us see exactly what goes wrong when you skip it.

# without_wraps.py
def bad_decorator(func):
    """Decorator WITHOUT functools.wraps — demonstrates the problem."""
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

@bad_decorator
def calculate_tax(amount, rate=0.08):
    """Calculate sales tax on a purchase amount."""
    return round(amount * rate, 2)

# Check the function's identity
print(f"Function name: {calculate_tax.__name__}")
print(f"Docstring: {calculate_tax.__doc__}")
print(f"Help output:")
help(calculate_tax)

Output:

Function name: wrapper
Docstring: None
Help output:
Help on function wrapper in module __main__:

wrapper(*args, **kwargs)

The function thinks its name is wrapper and its docstring is None. This breaks debugging tools, documentation generators, and any code that inspects function metadata. Now let us see the same decorator with functools.wraps applied.

# with_wraps.py
import functools

def good_decorator(func):
    """Decorator WITH functools.wraps — preserves the original function's metadata."""
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

@good_decorator
def calculate_tax(amount, rate=0.08):
    """Calculate sales tax on a purchase amount."""
    return round(amount * rate, 2)

# Check the function's identity
print(f"Function name: {calculate_tax.__name__}")
print(f"Docstring: {calculate_tax.__doc__}")
print(f"Help output:")
help(calculate_tax)

Output:

Function name: calculate_tax
Docstring: Calculate sales tax on a purchase amount.
Help output:
Help on function calculate_tax in module __main__:

calculate_tax(amount, rate=0.08)
    Calculate sales tax on a purchase amount.

With @functools.wraps(func), the decorated function retains its original name, docstring, and even its parameter signature in the help output. This is a one-line addition that prevents hours of confusion. Every decorator you write should include it — no exceptions. The rule is simple: if your decorator defines a wrapper function, put @functools.wraps(func) on the line directly above it.

Decorators That Accept Arguments

Sometimes you want your decorator to be configurable. For example, a logging decorator where you can set the log level, or a retry decorator where you can specify the number of retries. This requires an extra layer of nesting — a function that returns a decorator, which returns a wrapper. It sounds complicated, but the pattern is consistent once you see it.

# decorator_with_args.py
import functools

def log_with_level(level="INFO"):
    """Decorator factory that accepts a log level argument."""
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            arg_parts = [repr(a) for a in args]
            kwarg_parts = [f"{k}={v!r}" for k, v in kwargs.items()]
            all_args = ", ".join(arg_parts + kwarg_parts)
            print(f"[{level}] Calling {func.__name__}({all_args})")
            result = func(*args, **kwargs)
            print(f"[{level}] {func.__name__} returned {result!r}")
            return result
        return wrapper
    return decorator

@log_with_level("DEBUG")
def fetch_user(user_id):
    """Simulate fetching a user from a database."""
    return {"id": user_id, "name": "Alice", "email": "alice@example.com"}

@log_with_level("WARNING")
def delete_user(user_id):
    """Simulate deleting a user — dangerous operation."""
    return f"User {user_id} deleted"

print(fetch_user(42))
print()
print(delete_user(7))

Output:

[DEBUG] Calling fetch_user(42)
[DEBUG] fetch_user returned {'id': 42, 'name': 'Alice', 'email': 'alice@example.com'}
{'id': 42, 'name': 'Alice', 'email': 'alice@example.com'}

[WARNING] Calling delete_user(7)
[WARNING] delete_user returned 'User 7 deleted'
User 7 deleted

The key difference is that log_with_level("DEBUG") is not the decorator itself — it is a decorator factory that returns the actual decorator. When Python sees @log_with_level("DEBUG"), it first calls log_with_level("DEBUG"), which returns the decorator function. Then Python applies that decorator function to fetch_user. This three-layer structure (factory → decorator → wrapper) is the standard pattern for any decorator that needs configuration.

Building a Retry Decorator

Network calls fail. APIs time out. Databases hiccup. A retry decorator automatically re-runs a function when it raises an exception, with a configurable number of attempts and delay between retries. This is one of the most practical decorators you will ever write.

# retry_decorator.py
import functools
import time
import random

def retry(max_attempts=3, delay=1.0):
    """Retry a function up to max_attempts times with a delay between retries."""
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            last_exception = None
            for attempt in range(1, max_attempts + 1):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    last_exception = e
                    if attempt < max_attempts:
                        print(f"⚠ {func.__name__} failed (attempt {attempt}/{max_attempts}): {e}")
                        print(f"  Retrying in {delay}s...")
                        time.sleep(delay)
                    else:
                        print(f"✗ {func.__name__} failed after {max_attempts} attempts: {e}")
            raise last_exception
        return wrapper
    return decorator

@retry(max_attempts=3, delay=0.5)
def unreliable_api_call():
    """Simulate an API call that fails randomly."""
    if random.random() < 0.7:  # 70% chance of failure
        raise ConnectionError("Server unavailable")
    return {"status": "ok", "data": [1, 2, 3]}

# Set seed for reproducible output
random.seed(42)
try:
    result = unreliable_api_call()
    print(f"Success: {result}")
except ConnectionError as e:
    print(f"Final failure: {e}")

Output:

⚠ unreliable_api_call failed (attempt 1/3): Server unavailable
  Retrying in 0.5s...
⚠ unreliable_api_call failed (attempt 2/3): Server unavailable
  Retrying in 0.5s...
✗ unreliable_api_call failed after 3 attempts: Server unavailable
Final failure: Server unavailable

The retry decorator stores the last exception in last_exception so it can re-raise it if all attempts fail. This way the caller still gets the original exception type (ConnectionError) rather than a generic error. The time.sleep(delay) adds a pause between retries, which is important for network operations — hammering a server that just failed with immediate retries usually makes things worse. In production, you would typically use exponential backoff (doubling the delay each time) instead of a fixed delay.

Stacking Multiple Decorators

One of the most powerful features of decorators is that you can stack them. When you put multiple @decorator lines above a function, they are applied from bottom to top — the decorator closest to the function definition runs first, and the outermost decorator wraps everything. This lets you combine logging, timing, and retry logic on a single function.

# stacking_decorators.py
import functools
import time

def log_calls(func):
    """Log function calls."""
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        print(f"LOG: Calling {func.__name__}")
        result = func(*args, **kwargs)
        print(f"LOG: {func.__name__} returned {result!r}")
        return result
    return wrapper

def timer(func):
    """Time function execution."""
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        elapsed = time.perf_counter() - start
        print(f"⏱ {func.__name__} took {elapsed:.4f}s")
        return result
    return wrapper

@log_calls   # Applied second (outermost)
@timer       # Applied first (innermost)
def process_data(items):
    """Process a list of items with a simulated delay."""
    time.sleep(0.1)  # Simulate work
    return [item.upper() for item in items]

result = process_data(["hello", "world", "python"])

Output:

LOG: Calling process_data
⏱ process_data took 0.1003s
LOG: process_data returned ['HELLO', 'WORLD', 'PYTHON']

The execution order matters. Because @timer is closest to the function, it wraps process_data first. Then @log_calls wraps the already-timed version. When you call process_data(), the log decorator runs first (printing the "Calling" message), then the timer starts, then the actual function runs, then the timer stops (printing the elapsed time), and finally the log decorator prints the return value. If you reversed the order, the timer would measure the time including the logging overhead, which is usually not what you want.

Real-Life Example: Performance Monitoring Toolkit

Let us tie everything together into a practical project. This performance monitoring toolkit gives you three decorators that you can drop into any Python project: @monitor for combined logging and timing, @retry for fault tolerance, and @validate_types for runtime type checking. Together they form a lightweight monitoring layer for any application.

# performance_toolkit.py
import functools
import time
from datetime import datetime

def monitor(func):
    """Combined logging and timing decorator for production monitoring."""
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        timestamp = datetime.now().strftime("%H:%M:%S")
        arg_parts = [repr(a) for a in args]
        kwarg_parts = [f"{k}={v!r}" for k, v in kwargs.items()]
        signature = ", ".join(arg_parts + kwarg_parts)

        print(f"[{timestamp}] → {func.__name__}({signature})")
        start = time.perf_counter()

        try:
            result = func(*args, **kwargs)
            elapsed = time.perf_counter() - start
            print(f"[{timestamp}] ← {func.__name__} returned {result!r} ({elapsed:.4f}s)")
            return result
        except Exception as e:
            elapsed = time.perf_counter() - start
            print(f"[{timestamp}] ✗ {func.__name__} raised {type(e).__name__}: {e} ({elapsed:.4f}s)")
            raise
    return wrapper

def validate_types(**expected_types):
    """Validate argument types at runtime before the function executes."""
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            # Check keyword arguments against expected types
            for param_name, expected_type in expected_types.items():
                if param_name in kwargs:
                    value = kwargs[param_name]
                    if not isinstance(value, expected_type):
                        raise TypeError(
                            f"{param_name} must be {expected_type.__name__}, "
                            f"got {type(value).__name__}"
                        )
            return func(*args, **kwargs)
        return wrapper
    return decorator

# --- Use the toolkit ---

@monitor
def fetch_user_profile(user_id):
    """Simulate fetching a user profile from a database."""
    time.sleep(0.05)  # Simulate database query
    users = {1: "Alice", 2: "Bob", 3: "Charlie"}
    if user_id not in users:
        raise ValueError(f"User {user_id} not found")
    return {"id": user_id, "name": users[user_id], "active": True}

@monitor
@validate_types(amount=float, description=str)
def process_payment(amount=0.0, description=""):
    """Process a payment transaction."""
    time.sleep(0.02)  # Simulate payment processing
    return {"status": "completed", "amount": amount, "ref": "TXN-001"}

# Run the toolkit
print("=== Performance Monitoring Toolkit Demo ===\n")

# Successful call
profile = fetch_user_profile(1)
print(f"Got profile: {profile}\n")

# Successful payment
payment = process_payment(amount=29.99, description="Monthly subscription")
print(f"Payment result: {payment}\n")

# Failed call (user not found)
try:
    fetch_user_profile(999)
except ValueError:
    print("(Error handled gracefully)\n")

# Type validation failure
try:
    process_payment(amount="not a number", description="Bad payment")
except TypeError as e:
    print(f"Type error caught: {e}")

Output:

=== Performance Monitoring Toolkit Demo ===

[10:30:00] → fetch_user_profile(1)
[10:30:00] ← fetch_user_profile returned {'id': 1, 'name': 'Alice', 'active': True} (0.0503s)
Got profile: {'id': 1, 'name': 'Alice', 'active': True}

[10:30:00] → process_payment(amount=29.99, description='Monthly subscription')
[10:30:00] ← process_payment returned {'status': 'completed', 'amount': 29.99, 'ref': 'TXN-001'} (0.0201s)
Payment result: {'status': 'completed', 'amount': 29.99, 'ref': 'TXN-001'}

[10:30:00] → fetch_user_profile(999)
[10:30:00] ✗ fetch_user_profile raised ValueError: User 999 not found (0.0501s)
(Error handled gracefully)

Type error caught: amount must be float, got str

This toolkit demonstrates how decorators keep your business logic clean. The fetch_user_profile function only cares about looking up users — it knows nothing about logging, timing, or error formatting. All of that cross-cutting behavior lives in the decorators. You could add @monitor to every function in your application with a single line per function, giving you instant visibility into how your code performs. To extend this project, try adding a decorator that caches results (memoization) or one that limits how many times a function can be called per minute (rate limiting).

Frequently Asked Questions

What is the difference between a decorator and a regular function call?

A decorator wraps a function at definition time, not at call time. When you put @timer above a function, the wrapping happens once when Python loads the file. After that, every call to the function automatically goes through the wrapper. A regular function call like timer(my_func)(args) does the same thing but only for that one call. Decorators give you permanent, reusable wrapping with clean syntax.

Can you use a class as a decorator instead of a function?

Yes. Any callable object can be a decorator, and classes are callable (calling a class creates an instance). You define __init__ to accept the function and __call__ to act as the wrapper. Class-based decorators are useful when you need to maintain state between calls, such as counting how many times a function has been called. For stateless decorators like logging and timing, function-based decorators are simpler and more common.

How do you debug a decorated function?

Always use @functools.wraps(func) in your decorators — this is the single most important step. Without it, tracebacks show the wrapper function name instead of your actual function name, making debugging nearly impossible. If you need to temporarily remove a decorator for debugging, just comment out the @decorator line. You can also access the original unwrapped function via decorated_func.__wrapped__, which functools.wraps sets automatically.

Do decorators add performance overhead?

Yes, but it is typically negligible. Each decorated call adds the overhead of one extra function call plus whatever your wrapper does. For a simple logging decorator, this is microseconds. The overhead only matters if you are decorating a function that gets called millions of times in a tight loop. In that case, measure with time.perf_counter() and decide whether the convenience is worth the cost. For most applications — web servers, CLI tools, data processing scripts — the overhead is invisible.

Can decorators work with async functions?

Yes, but you need to make the wrapper function async too. Use async def wrapper(*args, **kwargs) and result = await func(*args, **kwargs) inside the decorator. If you want a single decorator that works with both sync and async functions, use import asyncio and check asyncio.iscoroutinefunction(func) to decide whether to use await. The functools.wraps pattern works the same way for async decorators.

Does the order of stacked decorators matter?

Absolutely. Decorators are applied bottom-to-top (closest to the function first), but they execute top-to-bottom when the function is called. If you stack @log_calls on top of @timer, the logging will record the total time including timer overhead. If you reverse them, the timer will only measure the function itself. The general rule is: put the decorator whose behavior you want to see first (outermost) on top, and put the decorator that should run closest to the actual function on the bottom.

Conclusion

In this article we covered everything you need to start using Python decorators effectively in your own projects. We started with the basic decorator pattern — a function that takes a function and returns a wrapper function. We built a logging decorator that captures function calls with timestamps and arguments, a timing decorator that measures execution time with time.perf_counter(), and a retry decorator that gracefully handles transient failures. We also covered functools.wraps (always use it), decorator factories for configurable decorators, and how stacking multiple decorators affects execution order.

The performance monitoring toolkit in the real-life example gives you a foundation you can build on. Try extending it with a caching decorator using functools.lru_cache, a rate limiter that tracks calls per time window, or an authentication decorator for a web application. Decorators are one of Python's most elegant features — once you get comfortable with the pattern, you will find uses for them everywhere.

For a deeper dive into decorators, closures, and first-class functions, check out the official Python documentation on decorators and the functools module documentation.

Related Articles

• How To Use Python structlog for Structured Logging
• How To Use Python Loguru for Modern Application Logging
• How To Use Python Decorators: A Complete Guide
• How To Use the Python logging Module for Application Logging
• How To Use Python Closures and Nested Functions
• How To Use Python Property Decorators and Descriptors
• Comparing Python to other Web Development Languages

Intermediate

Every Python developer starts with print() for debugging. It works fine when you are learning, but the moment your code runs in production — on a server, in a scheduled task, or as a background service — print statements become useless. They disappear when the terminal closes, they have no timestamps, no severity levels, and no way to separate important messages from noise. That is where Python’s built-in logging module comes in, and when you combine it with rotating file handlers, you get a production-grade logging system that manages itself.

The good news is that Python ships with everything you need. The logging module is part of the standard library, so there is nothing to install. It supports multiple log levels (DEBUG through CRITICAL), custom formatting, and a variety of handlers that control where your logs go — console, files, network sockets, email, and more. The RotatingFileHandler and TimedRotatingFileHandler are especially useful because they automatically manage log file sizes and rotation, preventing your disk from filling up.

In this article we will set up a complete logging system from scratch. We will start with a quick working example, then explain why logging beats print(), walk through log levels and formatting, set up size-based rotation with RotatingFileHandler, time-based rotation with TimedRotatingFileHandler, combine multiple handlers for simultaneous console and file logging, add structured JSON logging, and finish with a real-life project — a Production Application Logger class you can drop into any project.

Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Python Logging With Rotation: Quick Example

Here is a minimal setup that logs messages to both the console and a rotating file. The file automatically rolls over when it hits 1 MB, keeping the last 3 backups.

# quick_logging.py
import logging
from logging.handlers import RotatingFileHandler

# Create logger
logger = logging.getLogger("myapp")
logger.setLevel(logging.DEBUG)

# Console handler
console = logging.StreamHandler()
console.setLevel(logging.INFO)

# Rotating file handler (1 MB max, keep 3 backups)
file_handler = RotatingFileHandler("app.log", maxBytes=1_000_000, backupCount=3)
file_handler.setLevel(logging.DEBUG)

# Formatter
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
console.setFormatter(formatter)
file_handler.setFormatter(formatter)

# Add handlers
logger.addHandler(console)
logger.addHandler(file_handler)

# Test it
logger.debug("This goes to the file only")
logger.info("This goes to both console and file")
logger.warning("Something might be wrong")
logger.error("Something is definitely wrong")

Output (console):

2026-03-13 14:30:00,123 - myapp - INFO - This goes to both console and file
2026-03-13 14:30:00,124 - myapp - WARNING - Something might be wrong
2026-03-13 14:30:00,124 - myapp - ERROR - Something is definitely wrong

Notice that the DEBUG message only appears in the file, not the console — because we set the console handler to INFO level. The file handler captures everything from DEBUG up. When app.log reaches 1 MB, it automatically renames to app.log.1, creates a fresh app.log, and deletes the oldest backup beyond 3 files. Zero maintenance required.

Want to go deeper? Below we cover why logging beats print, all five log levels, custom formatting, both types of rotation handlers, and a production-ready logger class you can reuse in any project.

Why Not Just Use print()?

The print() function is great for quick debugging, but it falls apart in any serious application. Understanding its limitations helps explain why the logging module exists and why every production codebase uses it.

Here is a side-by-side comparison of what you get with each approach:

The most important difference is control. With logging, you can set your production server to WARNING level (ignoring all DEBUG and INFO messages) without changing a single line of code. You can send errors to a file while keeping info messages in the console. You can add email alerts for CRITICAL failures. None of this is possible with print().

Basic Logging Setup

The simplest way to start logging is with logging.basicConfig(), which configures the root logger with a single function call. This is fine for scripts and small programs, though for larger applications you will want the more flexible approach we show later.

# basic_setup.py
import logging

# Configure the root logger
logging.basicConfig(
    level=logging.DEBUG,
    format="%(asctime)s - %(levelname)s - %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S"
)

# These all use the root logger
logging.debug("Detailed information for diagnosing problems")
logging.info("Confirmation that things are working as expected")
logging.warning("Something unexpected happened, but the program still works")
logging.error("A more serious problem - something failed")
logging.critical("The program may not be able to continue")

Output:

2026-03-13 14:30:00 - DEBUG - Detailed information for diagnosing problems
2026-03-13 14:30:00 - INFO - Confirmation that things are working as expected
2026-03-13 14:30:00 - WARNING - Something unexpected happened, but the program still works
2026-03-13 14:30:00 - ERROR - A more serious problem - something failed
2026-03-13 14:30:00 - CRITICAL - The program may not be able to continue

The basicConfig() function is a convenience wrapper. The level parameter sets the minimum severity to capture — anything below this level is silently ignored. The format string controls what each log line looks like, using placeholder variables like %(asctime)s for the timestamp and %(levelname)s for the severity. The datefmt parameter controls the timestamp format.

Understanding Log Levels

Log levels are the filtering mechanism that makes logging so powerful. Each level has a numeric value, and the logger only processes messages at or above the configured threshold. Understanding when to use each level is critical for writing logs that are actually useful when you need them.

# log_levels.py
import logging

logging.basicConfig(level=logging.DEBUG, format="%(levelname)-8s %(message)s")

# DEBUG (10) - Detailed diagnostic info, only useful during development
logging.debug(f"Processing user_id=42, payload_size=1024 bytes")

# INFO (20) - Routine operational messages
logging.info("Server started on port 8080")
logging.info("User 'alice' logged in successfully")

# WARNING (30) - Something unexpected but not broken
logging.warning("Disk usage at 85% - consider cleanup")
logging.warning("API response took 4.2s (threshold: 3.0s)")

# ERROR (40) - Something failed, but the app continues
logging.error("Failed to connect to database: Connection refused")
logging.error("Payment processing failed for order #1234")

# CRITICAL (50) - The app may crash or is in an unrecoverable state
logging.critical("Out of memory - shutting down worker process")
logging.critical("Security breach detected: unauthorized admin access")

Output:

DEBUG    Processing user_id=42, payload_size=1024 bytes
INFO     Server started on port 8080
INFO     User 'alice' logged in successfully
WARNING  Disk usage at 85% - consider cleanup
WARNING  API response took 4.2s (threshold: 3.0s)
ERROR    Failed to connect to database: Connection refused
ERROR    Payment processing failed for order #1234
CRITICAL Out of memory - shutting down worker process
CRITICAL Security breach detected: unauthorized admin access

A common production strategy is to log DEBUG and INFO to files (for post-mortem analysis) while only showing WARNING and above in the console (to avoid drowning operators in noise). The %-8s in the format string left-aligns the level name in an 8-character field, making the output easier to scan visually.

RotatingFileHandler for Size-Based Rotation

The RotatingFileHandler automatically creates a new log file when the current one reaches a specified size. Old log files are renamed with numeric suffixes (.1, .2, etc.) and the oldest files beyond your backup count are deleted automatically. This prevents log files from growing unbounded and filling up your disk.

# rotating_handler.py
import logging
from logging.handlers import RotatingFileHandler

logger = logging.getLogger("rotating_demo")
logger.setLevel(logging.DEBUG)

# Create rotating handler: 500 KB max, keep 5 backups
handler = RotatingFileHandler(
    filename="demo.log",
    maxBytes=500_000,       # 500 KB per file
    backupCount=5,          # Keep demo.log.1 through demo.log.5
    encoding="utf-8"        # Always specify encoding
)

formatter = logging.Formatter(
    "%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S"
)
handler.setFormatter(formatter)
logger.addHandler(handler)

# Simulate logging activity
for i in range(1000):
    logger.info(f"Processing record {i}: status=OK, duration=0.{i % 100:02d}s")
    if i % 100 == 0:
        logger.warning(f"Batch {i // 100} checkpoint reached")

logger.info("Processing complete")

Output (in demo.log):

2026-03-13 14:30:00 | INFO     | rotating_demo | Processing record 0: status=OK, duration=0.00s
2026-03-13 14:30:00 | WARNING  | rotating_demo | Batch 0 checkpoint reached
2026-03-13 14:30:00 | INFO     | rotating_demo | Processing record 1: status=OK, duration=0.01s
...

After running this, you will see files like demo.log, demo.log.1, demo.log.2, and so on in your directory. The demo.log file is always the current, active log. When it hits 500 KB, the handler renames it to demo.log.1 (pushing the previous .1 to .2, etc.) and starts writing to a fresh demo.log. Files beyond demo.log.5 are automatically deleted. The total maximum disk usage is maxBytes x (backupCount + 1) — in this case, about 3 MB.

TimedRotatingFileHandler for Time-Based Rotation

Sometimes you want logs rotated by time rather than size — a new file every day, every hour, or every week. The TimedRotatingFileHandler handles this automatically. This is especially useful for daily log files that you can easily search by date.

# timed_handler.py
import logging
from logging.handlers import TimedRotatingFileHandler

logger = logging.getLogger("timed_demo")
logger.setLevel(logging.DEBUG)

# Create timed rotating handler: rotate at midnight, keep 30 days
handler = TimedRotatingFileHandler(
    filename="daily.log",
    when="midnight",        # Rotate at midnight
    interval=1,             # Every 1 day
    backupCount=30,         # Keep 30 days of logs
    encoding="utf-8",
    utc=False               # Use local time, not UTC
)

# Customize the backup file suffix to include the date
handler.suffix = "%Y-%m-%d"

formatter = logging.Formatter(
    "%(asctime)s | %(levelname)-8s | %(funcName)s | %(message)s"
)
handler.setFormatter(formatter)
logger.addHandler(handler)

# Example usage
def process_order(order_id, amount):
    logger.info(f"Processing order #{order_id} for ${amount:.2f}")
    if amount > 1000:
        logger.warning(f"High-value order #{order_id}: ${amount:.2f}")
    logger.debug(f"Order #{order_id} details sent to payment gateway")

process_order(1001, 49.99)
process_order(1002, 1500.00)
process_order(1003, 25.50)

Output (in daily.log):

2026-03-13 14:30:00,123 | INFO     | process_order | Processing order #1001 for $49.99
2026-03-13 14:30:00,123 | DEBUG    | process_order | Order #1001 details sent to payment gateway
2026-03-13 14:30:00,124 | INFO     | process_order | Processing order #1002 for $1500.00
2026-03-13 14:30:00,124 | WARNING  | process_order | High-value order #1002: $1500.00
2026-03-13 14:30:00,124 | DEBUG    | process_order | Order #1002 details sent to payment gateway
2026-03-13 14:30:00,125 | INFO     | process_order | Processing order #1003 for $25.50
2026-03-13 14:30:00,125 | DEBUG    | process_order | Order #1003 details sent to payment gateway

At midnight, the handler renames daily.log to daily.log.2026-03-13 and creates a fresh daily.log. The when parameter accepts several values: "S" for seconds, "M" for minutes, "H" for hours, "D" for days, "midnight" for midnight rotation, and "W0" through "W6" for specific weekdays (Monday through Sunday). The %(funcName)s formatter variable automatically includes the function name where the log call was made — extremely useful for tracing issues across a large codebase.

Multiple Handlers: Console AND File Logging

In practice, you almost always want both console output (for real-time monitoring) and file output (for historical records). Python’s logging module makes this easy — a single logger can have multiple handlers, each with its own level and format.

# multi_handler.py
import logging
from logging.handlers import RotatingFileHandler

def setup_logger(name, log_file="app.log", console_level=logging.INFO, file_level=logging.DEBUG):
    """Create a logger with both console and rotating file handlers."""
    logger = logging.getLogger(name)
    logger.setLevel(logging.DEBUG)  # Capture everything; handlers filter

    # Console handler - concise format, higher threshold
    console_handler = logging.StreamHandler()
    console_handler.setLevel(console_level)
    console_fmt = logging.Formatter("%(levelname)-8s %(message)s")
    console_handler.setFormatter(console_fmt)

    # File handler - detailed format, captures everything
    file_handler = RotatingFileHandler(
        log_file, maxBytes=5_000_000, backupCount=5, encoding="utf-8"
    )
    file_handler.setLevel(file_level)
    file_fmt = logging.Formatter(
        "%(asctime)s | %(levelname)-8s | %(name)s:%(funcName)s:%(lineno)d | %(message)s"
    )
    file_handler.setFormatter(file_fmt)

    logger.addHandler(console_handler)
    logger.addHandler(file_handler)
    return logger

# Usage
logger = setup_logger("myapp")

def connect_to_database(host, port):
    logger.debug(f"Attempting connection to {host}:{port}")
    logger.info(f"Connected to database at {host}:{port}")
    return True

def fetch_users():
    logger.debug("Executing SELECT * FROM users")
    logger.info("Fetched 42 users from database")
    logger.warning("Query took 2.3 seconds (threshold: 1.0s)")

connect_to_database("localhost", 5432)
fetch_users()

Output (console — concise):

INFO     Connected to database at localhost:5432
INFO     Fetched 42 users from database
WARNING  Query took 2.3 seconds (threshold: 1.0s)

Output (app.log — detailed):

2026-03-13 14:30:00,123 | DEBUG    | myapp:connect_to_database:30 | Attempting connection to localhost:5432
2026-03-13 14:30:00,124 | INFO     | myapp:connect_to_database:31 | Connected to database at localhost:5432
2026-03-13 14:30:00,125 | DEBUG    | myapp:fetch_users:35 | Executing SELECT * FROM users
2026-03-13 14:30:00,125 | INFO     | myapp:fetch_users:36 | Fetched 42 users from database
2026-03-13 14:30:00,126 | WARNING  | myapp:fetch_users:37 | Query took 2.3 seconds (threshold: 1.0s)

The key insight is that the logger level must be set to the lowest level you want to capture (DEBUG), and each handler then filters independently. The console handler only shows INFO and above, keeping the terminal clean. The file handler captures everything including DEBUG messages, giving you full diagnostic detail when you need to investigate an issue after the fact. The file format includes the function name and line number (%(funcName)s:%(lineno)d), which makes tracing bugs significantly faster.

Real-Life Example: Production Application Logger

Let us build a reusable AppLogger class that you can drop into any project. It combines everything we have covered — console and file handlers, rotating files, structured formatting, and exception logging — into a clean, configurable package.

# app_logger.py
import os
import logging
import traceback
from logging.handlers import RotatingFileHandler, TimedRotatingFileHandler
from datetime import datetime

class AppLogger:
    """Production-ready logger with console and rotating file output."""

    def __init__(self, name, log_dir="logs", console_level="INFO",
                 file_level="DEBUG", max_bytes=10_000_000, backup_count=10):
        # Create log directory if it doesn't exist
        os.makedirs(log_dir, exist_ok=True)

        self.logger = logging.getLogger(name)
        self.logger.setLevel(logging.DEBUG)

        # Prevent duplicate handlers if called multiple times
        if self.logger.handlers:
            return

        # Console handler - human-readable, colored by level
        console = logging.StreamHandler()
        console.setLevel(getattr(logging, console_level.upper()))
        console_fmt = logging.Formatter(
            "%(asctime)s | %(levelname)-8s | %(message)s",
            datefmt="%H:%M:%S"
        )
        console.setFormatter(console_fmt)

        # Rotating file handler - detailed, size-based rotation
        log_path = os.path.join(log_dir, f"{name}.log")
        file_handler = RotatingFileHandler(
            log_path, maxBytes=max_bytes, backupCount=backup_count, encoding="utf-8"
        )
        file_handler.setLevel(getattr(logging, file_level.upper()))
        file_fmt = logging.Formatter(
            "%(asctime)s | %(levelname)-8s | %(name)s:%(funcName)s:%(lineno)d | %(message)s"
        )
        file_handler.setFormatter(file_fmt)

        # Error-only file handler - quick access to errors
        error_path = os.path.join(log_dir, f"{name}_errors.log")
        error_handler = RotatingFileHandler(
            error_path, maxBytes=max_bytes, backupCount=5, encoding="utf-8"
        )
        error_handler.setLevel(logging.ERROR)
        error_handler.setFormatter(file_fmt)

        self.logger.addHandler(console)
        self.logger.addHandler(file_handler)
        self.logger.addHandler(error_handler)

    def debug(self, msg): self.logger.debug(msg)
    def info(self, msg): self.logger.info(msg)
    def warning(self, msg): self.logger.warning(msg)
    def error(self, msg): self.logger.error(msg)
    def critical(self, msg): self.logger.critical(msg)

    def exception(self, msg):
        """Log an error with full traceback."""
        self.logger.error(f"{msg}\n{traceback.format_exc()}")


# Demo usage
if __name__ == "__main__":
    log = AppLogger("myapp")

    log.info("Application started")
    log.debug("Loading configuration from config.json")
    log.info("Database connection established")

    # Simulate processing
    for i in range(5):
        log.info(f"Processing batch {i + 1} of 5")
        if i == 2:
            log.warning("Batch 3 had 12 skipped records")

    # Simulate an error with traceback
    try:
        result = 1 / 0
    except ZeroDivisionError:
        log.exception("Math operation failed")

    log.info("Application shutdown complete")

Output (console):

14:30:00 | INFO     | Application started
14:30:00 | INFO     | Database connection established
14:30:00 | INFO     | Processing batch 1 of 5
14:30:00 | INFO     | Processing batch 2 of 5
14:30:00 | INFO     | Processing batch 3 of 5
14:30:00 | WARNING  | Batch 3 had 12 skipped records
14:30:00 | INFO     | Processing batch 4 of 5
14:30:00 | INFO     | Processing batch 5 of 5
14:30:00 | ERROR    | Math operation failed
                      Traceback (most recent call last):
                        File "app_logger.py", line 74, in <module>
                          result = 1 / 0
                      ZeroDivisionError: division by zero
14:30:00 | INFO     | Application shutdown complete

This logger class gives you three outputs: a clean console for real-time monitoring, a detailed log file for everything, and a separate error-only file for quick problem diagnosis. The exception() method automatically captures the full Python traceback, which is invaluable for debugging production errors. The duplicate handler check (if self.logger.handlers) prevents the common bug where creating multiple instances of the same logger adds duplicate handlers, causing each message to appear multiple times.

You could extend this logger with JSON-formatted output for log aggregation tools like ELK Stack, email alerts for CRITICAL messages using SMTPHandler, or Slack notifications via a custom handler.

Frequently Asked Questions

When should I use basicConfig vs manual handler setup?

Use basicConfig() for quick scripts, one-file programs, and learning. It is a single function call that handles the most common case. Switch to manual handler setup (creating Logger, Handler, and Formatter objects explicitly) when you need multiple handlers with different levels, custom formatting per output, or when building a library or larger application. The manual approach gives you complete control.

How do I silence noisy logs from third-party libraries?

Third-party libraries like requests, urllib3, and boto3 often produce verbose DEBUG logs. Set their logger level to WARNING: logging.getLogger("urllib3").setLevel(logging.WARNING). This silences their DEBUG and INFO messages without affecting your own logging. You can also use logging.getLogger("urllib3").propagate = False to completely stop their messages from reaching your root logger.

What format variables are available in log formatters?

The most useful ones are: %(asctime)s for timestamp, %(levelname)s for level, %(name)s for logger name, %(funcName)s for function name, %(lineno)d for line number, %(filename)s for file name, %(message)s for the actual message, and %(process)d for process ID. You can combine them in any order. For production, include at minimum the timestamp, level, and message.

How do I log in JSON format for tools like ELK Stack?

Install the python-json-logger package (pip install python-json-logger) and use its JsonFormatter. Replace your standard formatter with JsonFormatter("%(asctime)s %(levelname)s %(name)s %(message)s"). This outputs each log line as a JSON object with those fields as keys, which tools like Elasticsearch, Splunk, and CloudWatch can parse automatically without custom regex patterns.

Is Python logging thread-safe?

Yes. The logging module uses locks internally to ensure that log messages from different threads do not interleave or corrupt each other. Each handler has its own lock. This means you can safely use the same logger from multiple threads without any additional synchronization. For multi-process applications, however, you need to be more careful — RotatingFileHandler can have issues when multiple processes write to the same file. Use QueueHandler with a separate logging process in that case.

Does logging slow down my application?

Logging has minimal overhead when configured correctly. The biggest performance tip is to use lazy formatting: write logger.debug("Processing %d items", count) instead of logger.debug(f"Processing {count} items"). With the first form, the string formatting only happens if DEBUG level is enabled. With the f-string, the formatting happens every time regardless of level. For most applications, logging overhead is negligible compared to I/O, network calls, or database queries.

Conclusion

You now have a complete understanding of Python’s logging system. We covered why print() falls short in production, how to use basicConfig() for quick setup, the five log levels and when to use each one, custom formatting with Formatter, size-based rotation with RotatingFileHandler, time-based rotation with TimedRotatingFileHandler, combining multiple handlers for simultaneous console and file output, and a reusable AppLogger class for production applications.

The AppLogger class from the real-life example is ready to use in your own projects. Try extending it with JSON output for log aggregation, email alerts for critical errors, or integration with monitoring tools like Sentry. Proper logging is one of those investments that pays for itself the first time you need to debug a production issue at 2 AM.

For the complete reference on handlers, formatters, filters, and configuration, check out the official Python logging documentation and the Logging Cookbook for advanced patterns.

Related Articles

• How To Set Up Logging in Python 3 (Output to File and Console)
• How To Use Python structlog for Structured Logging
• How To Use Python Loguru for Modern Application Logging
• How To Use the Python logging Module for Application Logging
• How To Use Python Decorators To Add Logging and Timing to Functions
• How To Use Python dynaconf for Configuration Management
• Comparing Python to other Web Development Languages

Intermediate

Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

How To Store Data Separately For Testing and Production in Python 3

Quick Example (TLDR)

Use environment variables to switch between test and production settings:

# config.py
import os

ENVIRONMENT = os.getenv('ENV', 'development')

if ENVIRONMENT == 'production':
    DATABASE_URL = 'postgresql://prod.example.com:5432/maindb'
    DEBUG = False
else:
    DATABASE_URL = 'sqlite:///test.db'
    DEBUG = True

# main.py
from config import DATABASE_URL, DEBUG
print(f"Database: {DATABASE_URL}, Debug: {DEBUG}")

Output (development):

Database: sqlite:///test.db, Debug: True

Output (production):

Database: postgresql://prod.example.com:5432/maindb, Debug: False

Why Separate Configurations Matter

Your test environment should be completely isolated from your production data. Testing on production is a recipe for disaster: you could delete real data, send test emails to real customers, or charge real credit cards. The solution is environment-specific configuration.

Using Environment Variables

Environment variables are the simplest and most secure way to manage different configs:

import os

# Read environment variable, use default if not set
DATABASE_HOST = os.getenv('DB_HOST', 'localhost')
DATABASE_PORT = int(os.getenv('DB_PORT', '5432'))
API_KEY = os.getenv('API_KEY')  # No default - must be set
DEBUG_MODE = os.getenv('DEBUG', 'False').lower() == 'true'

print(f"Host: {DATABASE_HOST}")
print(f"Port: {DATABASE_PORT}")
print(f"Debug: {DEBUG_MODE}")

Output:

Host: localhost
Port: 5432
Debug: False

Using .env Files with python-dotenv

For development, load environment variables from a .env file:

# .env (development)
DATABASE_URL=sqlite:///dev.db
API_KEY=dev-key-12345
DEBUG=True
MAIL_SERVICE=fake

# .env.production
DATABASE_URL=postgresql://prod.example.com/maindb
API_KEY=prod-key-secret
DEBUG=False
MAIL_SERVICE=sendgrid

Load them in Python:

from dotenv import load_dotenv
import os

# Load from .env file
load_dotenv('.env')

DATABASE_URL = os.getenv('DATABASE_URL')
API_KEY = os.getenv('API_KEY')
print(f"Database: {DATABASE_URL}")

Output:

Database: sqlite:///dev.db

ConfigParser for Multi-Environment Files

For complex configurations, use ConfigParser with separate .ini files:

# config_development.ini
[database]
host = localhost
port = 5432
name = testdb
user = testuser
password = testpass

[email]
service = console
debug = True

[api]
key = dev-key-123
timeout = 10

Read it in Python:

import configparser
import os

# Determine which config to load
env = os.getenv('ENV', 'development')
config_file = f'config_{env}.ini'

# Parse the config file
config = configparser.ConfigParser()
config.read(config_file)

# Access values
db_host = config.get('database', 'host')
db_port = config.getint('database', 'port')
email_debug = config.getboolean('email', 'debug')

print(f"Database: {db_host}:{db_port}")
print(f"Email debug: {email_debug}")

Output:

Database: localhost:5432
Email debug: True

The Settings Pattern: Dev, Staging, and Production

Create a flexible settings module that supports multiple environments:

# settings/base.py - Common settings for all environments
import os

class Settings:
    # Common to all environments
    SECRET_KEY = os.getenv('SECRET_KEY', 'dev-key')
    ALLOWED_HOSTS = ['localhost', '127.0.0.1']
    
    # Default values
    DATABASE_URL = 'sqlite:///db.sqlite3'
    API_TIMEOUT = 30
    DEBUG = False
    LOG_LEVEL = 'INFO'

# settings/development.py
from .base import Settings

class DevelopmentSettings(Settings):
    DEBUG = True
    DATABASE_URL = 'sqlite:///dev.db'
    API_TIMEOUT = 60  # Longer timeout for debugging
    LOG_LEVEL = 'DEBUG'

# settings/production.py
from .base import Settings

class ProductionSettings(Settings):
    DEBUG = False
    DATABASE_URL = os.getenv('DATABASE_URL')  # From env vars
    API_TIMEOUT = 10  # Strict timeout in production
    LOG_LEVEL = 'WARNING'

# settings/staging.py
from .base import Settings

class StagingSettings(Settings):
    DEBUG = False  # But with more logging than prod
    DATABASE_URL = os.getenv('STAGING_DATABASE_URL')
    LOG_LEVEL = 'INFO'

Use it in your app:

# main.py
import os
from settings.development import DevelopmentSettings
from settings.production import ProductionSettings
from settings.staging import StagingSettings

# Load appropriate settings
env = os.getenv('ENV', 'development')

if env == 'production':
    settings = ProductionSettings()
elif env == 'staging':
    settings = StagingSettings()
else:
    settings = DevelopmentSettings()

# Use the settings
print(f"Debug mode: {settings.DEBUG}")
print(f"Database: {settings.DATABASE_URL}")
print(f"Log level: {settings.LOG_LEVEL}")

Output (development):

Debug mode: True
Database: sqlite:///dev.db
Log level: DEBUG

Output (production):

Debug mode: False
Database: postgresql://prod.example.com/maindb
Log level: WARNING

Real-Life Example: Flask App With Environment-Based Config

Here’s a complete Flask application setup with separate environments:

# config.py
import os

class Config:
    # Common
    SECRET_KEY = os.getenv('SECRET_KEY', 'dev-secret-key')
    SQLALCHEMY_TRACK_MODIFICATIONS = False

class DevelopmentConfig(Config):
    DEBUG = True
    SQLALCHEMY_DATABASE_URI = 'sqlite:///dev.db'
    SQLALCHEMY_ECHO = True  # Log all SQL
    MAIL_BACKEND = 'console'  # Print emails to console

class TestingConfig(Config):
    TESTING = True
    SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:'  # In-memory DB
    MAIL_BACKEND = 'testing'  # Don't send emails

class ProductionConfig(Config):
    DEBUG = False
    SQLALCHEMY_DATABASE_URI = os.getenv('DATABASE_URL')
    SQLALCHEMY_ECHO = False  # Don't log SQL
    MAIL_BACKEND = 'sendgrid'  # Send real emails

# app.py
from flask import Flask
import os
from config import DevelopmentConfig, TestingConfig, ProductionConfig

def create_app():
    app = Flask(__name__)
    
    # Load config based on environment
    env = os.getenv('FLASK_ENV', 'development')
    
    if env == 'production':
        app.config.from_object(ProductionConfig)
    elif env == 'testing':
        app.config.from_object(TestingConfig)
    else:
        app.config.from_object(DevelopmentConfig)
    
    @app.route('/status')
    def status():
        return {
            'environment': env,
            'debug': app.config['DEBUG'],
            'database': app.config['SQLALCHEMY_DATABASE_URI']
        }
    
    return app

# Run with: FLASK_ENV=production python app.py
if __name__ == '__main__':
    app = create_app()
    app.run()

Output (development):

{
  "environment": "development",
  "debug": true,
  "database": "sqlite:///dev.db"
}

FAQ

Q: Should I commit .env files to git?

A: Never! Add .env to .gitignore. Commit .env.example with dummy values so others know what variables are needed. This keeps secrets out of version control.

Q: Can I use environment variables for all settings?

A: Yes, but config files are often easier for complex setups. Use environment variables for secrets (API keys, passwords) and config files for regular settings.

Q: How do I test with a test database without affecting production?

A: Use an in-memory database or a separate test database for testing. Set ENV=testing to automatically use test configuration with no risk to production.

Q: What if I forget to set an environment variable?

A: Use defaults wisely. For critical values like DATABASE_URL, don’t provide defaults so the app fails loudly. For optional values, provide sensible defaults.

Q: How do I know which environment my code is running in?

A: Always have a way to check: print(os.getenv(‘ENV’)) or check your settings object. In Flask: app.config[‘DEBUG’] or app.config[‘ENV’]

Conclusion

Separating configuration by environment is a fundamental practice in professional software development. It protects your production data, makes testing safer, and allows different teams to work without stepping on each other’s toes. Use environment variables for secrets and configuration files for complex settings, and your code will be more flexible and secure.

References

• Python ConfigParser documentation
• os.getenv documentation
• python-dotenv package
• The Twelve Factor App – Config

The Pattern: Env Var Says Which Config to Load

The most reliable pattern across frameworks: read a single environment variable (APP_ENV or PYTHON_ENV) at startup, branch on it to load the right config file. Everything else flows from that one decision:

# File: config/__init__.py
import os
from importlib import import_module

env = os.environ.get("APP_ENV", "development")
config = import_module(f"config.{env}").Config

# config/development.py
class Config:
    DEBUG = True
    DATABASE_URL = "sqlite:///dev.db"
    CACHE_TYPE = "SimpleCache"
    LOG_LEVEL = "DEBUG"

# config/testing.py
class Config:
    DEBUG = True
    TESTING = True
    DATABASE_URL = "sqlite:///:memory:"
    CACHE_TYPE = "NullCache"

# config/production.py
class Config:
    DEBUG = False
    DATABASE_URL = os.environ["DATABASE_URL"]  # require in prod
    CACHE_TYPE = "RedisCache"
    LOG_LEVEL = "WARNING"

The trick: development and testing have safe defaults inline, but production REQUIRES the env var. os.environ["DATABASE_URL"] raises KeyError at startup if missing — fail-fast is exactly what you want in production.

Pydantic Settings — The Modern Approach

For real applications, hand-rolled config classes get unwieldy fast. pydantic-settings (the modern replacement for pydantic.BaseSettings) gives you type-checked config with automatic env-var loading, validation, and .env file support:

# pip install pydantic-settings

from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8")

    database_url: str
    secret_key: str = Field(min_length=32)
    debug: bool = False
    log_level: str = "INFO"
    redis_url: str | None = None

settings = Settings()
print(settings.database_url)

It loads from .env in dev, from real env vars in production, and the type annotations get validated at startup. secret_key with min_length=32 catches misconfigured staging where someone copy-pasted a short test value.

Secrets vs Config

Treat secrets (API keys, DB passwords, signing keys) differently from configuration (feature flags, timeouts, hostnames). Three rules:

• Never commit secrets. Put them in .env (gitignored), in a secrets manager (AWS Secrets Manager, Vault, GCP Secret Manager), or in your platform’s encrypted-env-var feature.
• Never log secrets. Add a __repr__ override that returns " for any field marked as a secret. Pydantic v2 supports SecretStr for exactly this.
• Rotate periodically. Secrets that never change are secrets that have been leaked. Build rotation into your config-loading code (multiple valid keys at once, with a deprecation window).

Common Pitfalls

• Default-to-production. os.environ.get("APP_ENV", "production") is the opposite of what you want. Default to development so a forgotten env var in CI doesn’t accidentally point at the prod database.
• Config drift between environments. A new feature flag added in dev but not staging means staging tests pass, prod breaks. Use a single Settings class with the same fields across envs; only the values differ.
• Config loaded at module import time. If your settings = Settings() runs at import, you can’t override env vars in tests without re-importing the module. Wrap config in a function (get_settings()) and cache it lazily.
• Reading env vars throughout the codebase. Scattered os.environ.get(...) calls make config hard to audit. Centralize all env-var access in one Settings class; the rest of the code imports from there.
• No validation. A typo in a numeric setting (e.g., PORT="abc") shouldn’t fail when the first request arrives — it should fail at startup. Pydantic Settings gives you this for free.

FAQ

Q: .env file or real environment variables?
A: Both. .env for local development (never committed), real env vars in production via your platform’s secrets management. pydantic-settings reads from both transparently.

Q: How do I share config between Python and other services in my stack?
A: Use a portable format — env vars, JSON, or YAML — not a Python module. Then each service loads from the shared source. Resist the temptation to import config across language boundaries.

Q: What about feature flags?
A: Same Settings class can hold feature flags as booleans. For dynamic flags that change without redeployment, use a flag service like LaunchDarkly or Unleash — they’re worth it once you have more than 5-10 flags.

Q: How do I test config-dependent code?
A: Inject the config object rather than importing it. Tests pass a custom Settings instance with the values they need. pytest fixtures make this easy with autouse=True overrides.

Q: 12-factor app — do I have to do all twelve?
A: Config-in-environment is factor 3 and the one that matters most. The other eleven are great guidelines, but most teams gain 80% of the benefit just by getting config and secrets out of code.

Wrapping Up

Environment-driven config is one of those infrastructure habits that compounds: small now, life-saving when you’re trying to debug a production outage at 2 AM. Start with the APP_ENV pattern, move to pydantic-settings when the codebase grows, and never let secrets touch your git history. The setup cost is an hour; the lifetime cost of getting it wrong is incalculable.

• How To Manage Python Environment Variables With dotenv
• How To Use Python TOML For Configuration Files Instead of INI
• How To Set Up Python Logging With Rotating File Handlers

Related Articles

• How To Use Python attrs Library for Data Classes Without Boilerplate
• How To Use Hypothesis for Property-Based Testing in Python
• How To Create Data Visualizations with Seaborn in Python
• How To Use Pydantic V2 for Data Validation in Python
• How To Use Python Dataclasses For Clean Data Structures
• How To Use Python snoop for Function Tracing and Debugging
• Storing settings data in Config File in Python