Intermediate

Your Downloads folder is a graveyard. PDFs from three jobs ago, screenshots with names like image(47).png, ZIP archives you extracted months ago, and a dozen video files you swore you would watch. Every time you need to find something, you scroll past hundreds of unrelated files. Sound familiar? The good news is you can fix this permanently with about 50 lines of Python.

Python’s standard library gives you everything you need to build a real file organizer: pathlib for readable path manipulation, shutil for safe file moves, os for directory creation, and argparse for a proper command-line interface. No third-party packages required. The whole script runs on any machine with Python 3.6 or later installed.

In this article, we will build a desktop file organizer step by step. First, we will cover the Quick Example to get a feel for the approach. Then we will look at scanning directories with pathlib, moving files safely with shutil, organizing by file extension, and adding a CLI with dry-run mode. By the end you will have a production-ready script you can run on any folder — and set up as a scheduled task if you want it to run automatically.

Organizing a Folder: Quick Example

Here is the core idea in 25 lines. This script scans a target folder and moves every file into a subfolder named after its category (Images, Documents, Videos, and so on).

# quick_organizer.py
import shutil
from pathlib import Path

FOLDER = Path.home() / "Downloads"

EXT_MAP = {
    ".pdf": "Documents", ".docx": "Documents", ".txt": "Documents",
    ".jpg": "Images",    ".jpeg": "Images",    ".png": "Images", ".gif": "Images",
    ".mp4": "Videos",    ".mov": "Videos",     ".avi": "Videos",
    ".mp3": "Audio",     ".wav": "Audio",
    ".zip": "Archives",  ".tar": "Archives",   ".gz":  "Archives",
    ".py":  "Code",      ".js":  "Code",       ".html":"Code",
}

moved = 0
for item in FOLDER.iterdir():
    if not item.is_file():
        continue
    category = EXT_MAP.get(item.suffix.lower(), "Other")
    dest_dir = FOLDER / category
    dest_dir.mkdir(exist_ok=True)
    shutil.move(str(item), str(dest_dir / item.name))
    print(f"Moved: {item.name}  ->  {category}/")
    moved += 1

print(f"\nDone. {moved} file(s) organized.")

Output:

Moved: budget-2025.pdf  ->  Documents/
Moved: screenshot-2026-01.png  ->  Images/
Moved: project.zip  ->  Archives/
Moved: notes.txt  ->  Documents/

Done. 4 file(s) organized.

The key players are Path.iterdir() to list every item in the folder, item.suffix.lower() to get the file extension normalized to lowercase, and shutil.move() to relocate the file. The dest_dir.mkdir(exist_ok=True) call creates the category subfolder only if it does not already exist — so running the script twice is completely safe.

The sections below build this into a proper, reusable tool with a CLI, duplicate protection, dry-run mode, and a configurable extension map.

What Is a File Organizer and Why Build One in Python?

A file organizer is a script that inspects the name or metadata of each file in a directory and moves it to a predefined location based on rules you define. The simplest rule is the one we use here: sort by file extension. An extension like .pdf tells you the file is a document; .mp4 tells you it is a video. More sophisticated organizers can read EXIF metadata from photos to sort by date, or inspect file contents for type detection — but extension-based sorting covers 90% of real-world use cases with zero added complexity.

Why Python? Because the standard library is batteries-included for this task. Compare your options:

Python gives you the most flexibility with the least friction. Once written, you can run the same script on Windows, macOS, and Linux without changing a line.

Scanning Directories with pathlib

The pathlib module, introduced in Python 3.4, treats file paths as objects with useful methods instead of plain strings. For a file organizer, the two most important methods are Path.iterdir() — which lists everything in a single directory — and Path.rglob() — which searches recursively through nested subdirectories.

# scan_demo.py
from pathlib import Path

target = Path("/tmp/messy_folder")

# List only direct children (non-recursive)
print("Direct children:")
for item in target.iterdir():
    kind = "DIR " if item.is_dir() else "FILE"
    print(f"  [{kind}] {item.name}  (suffix: {repr(item.suffix)})")

Output:

Direct children:
  [FILE] report.pdf  (suffix: '.pdf')
  [FILE] Photo_2026.JPG  (suffix: '.JPG')
  [DIR ] old_projects  (suffix: '')
  [FILE] notes.txt  (suffix: '.txt')

Notice that item.suffix preserves the original case (e.g., '.JPG' not '.jpg'). Always call .lower() on it before looking it up in your extension map, otherwise .JPG and .jpg will be treated as different file types. The item.is_dir() check lets us skip subdirectories so we only process actual files.

For deep folder trees, swap iterdir() for rglob("*") and add an if item.is_file() check:

# recursive_scan.py
from pathlib import Path

target = Path("/tmp/messy_folder")

print("All files (recursive):")
for item in target.rglob("*"):
    if item.is_file():
        print(f"  {item.relative_to(target)}  ->  ext: {item.suffix.lower() or 'none'}")

Output:

All files (recursive):
  report.pdf  ->  ext: .pdf
  Photo_2026.JPG  ->  ext: .jpg
  notes.txt  ->  ext: .txt
  old_projects/draft.docx  ->  ext: .docx

item.relative_to(target) shows a clean path relative to the root folder rather than the full absolute path — useful for readable log output.

Moving Files Safely with shutil

Once you know where a file should go, shutil.move(src, dst) relocates it. The function accepts both string paths and Path objects. It handles the cross-device case automatically — if source and destination are on different drives, it copies then deletes rather than doing a raw rename that would fail.

# move_demo.py
import shutil
from pathlib import Path

src = Path("/tmp/messy_folder/report.pdf")
dst_dir = Path("/tmp/messy_folder/Documents")
dst_dir.mkdir(parents=True, exist_ok=True)

dst = dst_dir / src.name
shutil.move(str(src), str(dst))
print(f"Moved: {src.name} -> Documents/")

Output:

Moved: report.pdf -> Documents/

The dst_dir.mkdir(parents=True, exist_ok=True) call is a two-for-one: parents=True creates any missing parent directories, and exist_ok=True suppresses the error if the folder already exists. Always include both flags so the script is idempotent — safe to run multiple times on the same folder.

One edge case to handle: what if a file with the same name already exists in the destination? shutil.move() will silently overwrite it. To prevent data loss, check for a collision and rename the incoming file if needed:

# safe_move.py
import shutil
from pathlib import Path

def safe_move(src, dst_dir, dry_run=False):
    dst_dir.mkdir(parents=True, exist_ok=True)
    dst = dst_dir / src.name
    counter = 1
    while dst.exists():
        dst = dst_dir / f"{src.stem}({counter}){src.suffix}"
        counter += 1
    if not dry_run:
        shutil.move(str(src), str(dst))
    return dst

src = Path("/tmp/messy_folder/report.pdf")
final = safe_move(src, Path("/tmp/messy_folder/Documents"))
print(f"File is now at: {final}")

Output (when a report.pdf already exists in Documents/):

File is now at: /tmp/messy_folder/Documents/report(1).pdf

This pattern — loop until the target path does not exist — is simple and collision-proof. It is the same approach used by macOS Finder and Windows Explorer when you copy a file into a folder that already contains a file with the same name.

Adding a CLI with argparse

Hard-coding the target folder path inside the script is fine for personal use but gets annoying fast. Adding an argparse CLI lets you pass the folder as an argument — and add useful flags like --dry-run to preview what would move without actually moving anything.

# cli_demo.py
import argparse
from pathlib import Path

def parse_args():
    parser = argparse.ArgumentParser(
        description="Organize files in a folder by extension."
    )
    parser.add_argument(
        "folder",
        type=Path,
        help="Path to the folder you want to organize."
    )
    parser.add_argument(
        "--dry-run",
        action="store_true",
        help="Preview what would be moved without making any changes."
    )
    parser.add_argument(
        "--recursive",
        action="store_true",
        help="Also organize files in subdirectories."
    )
    return parser.parse_args()

args = parse_args()
print(f"Target folder : {args.folder}")
print(f"Dry run       : {args.dry_run}")
print(f"Recursive     : {args.recursive}")

Output (run as: python cli_demo.py ~/Downloads –dry-run):

Target folder : /home/user/Downloads
Dry run       : True
Recursive     : False

The --dry-run flag is invaluable when you are running the organizer on an unfamiliar folder. It lets you audit the plan before committing. We will wire it into the main loop in the Real-Life Example so that passing --dry-run prints every planned move but touches nothing on disk.

Real-Life Example: A Production-Ready File Organizer

Here is the complete script combining everything above — scanning, safe moving, a full extension map, CLI arguments, dry-run mode, and a summary report at the end. Save it as organize.py and run it from anywhere.

# organize.py
import argparse
import shutil
from pathlib import Path

EXT_MAP = {
    ".pdf": "Documents",  ".docx": "Documents", ".txt": "Documents",
    ".xls": "Documents",  ".xlsx": "Documents", ".pptx": "Documents",
    ".csv": "Documents",  ".doc": "Documents",
    ".jpg": "Images",  ".jpeg": "Images", ".png": "Images",
    ".gif": "Images",  ".webp": "Images", ".heic": "Images",
    ".svg": "Images",  ".raw": "Images",
    ".mp4": "Videos",  ".mov": "Videos",  ".avi": "Videos",
    ".mkv": "Videos",  ".webm": "Videos",
    ".mp3": "Audio",   ".wav": "Audio",   ".aac": "Audio",
    ".flac": "Audio",  ".m4a": "Audio",
    ".zip": "Archives", ".tar": "Archives", ".gz": "Archives",
    ".rar": "Archives", ".7z": "Archives",
    ".py": "Code",  ".js": "Code",   ".ts": "Code",
    ".html": "Code", ".css": "Code", ".json": "Code",
    ".yaml": "Code", ".yml": "Code", ".ipynb": "Code",
    ".exe": "Installers", ".dmg": "Installers", ".pkg": "Installers",
}

def safe_move(src, dst_dir, dry_run):
    dst_dir.mkdir(parents=True, exist_ok=True)
    dst = dst_dir / src.name
    counter = 1
    while dst.exists():
        dst = dst_dir / f"{src.stem}({counter}){src.suffix}"
        counter += 1
    if not dry_run:
        shutil.move(str(src), str(dst))
    return dst

def organize(folder, recursive, dry_run):
    stats = {}
    iterator = folder.rglob("*") if recursive else folder.iterdir()
    for item in iterator:
        if not item.is_file() or not item.suffix:
            continue
        category = EXT_MAP.get(item.suffix.lower(), "Other")
        dst_dir = folder / category
        dst = safe_move(item, dst_dir, dry_run)
        label = "[DRY RUN] " if dry_run else ""
        print(f"  {label}{item.name}  ->  {category}/")
        stats[category] = stats.get(category, 0) + 1
    return stats

def main():
    parser = argparse.ArgumentParser(
        description="Organize files in a folder by extension."
    )
    parser.add_argument("folder", type=Path, help="Folder to organize.")
    parser.add_argument("--dry-run", action="store_true",
                        help="Preview moves without touching files.")
    parser.add_argument("--recursive", action="store_true",
                        help="Organize subdirectories too.")
    args = parser.parse_args()

    if not args.folder.is_dir():
        print(f"Error: {args.folder} is not a valid directory.")
        return

    print(f"Organizing: {args.folder}")
    if args.dry_run:
        print("(DRY RUN -- no files will be moved)\n")

    stats = organize(args.folder, args.recursive, args.dry_run)

    print("\n--- Summary ---")
    for category, count in sorted(stats.items()):
        print(f"  {category:15s} {count} file(s)")
    print(f"  {'TOTAL':15s} {sum(stats.values())} file(s)")

if __name__ == "__main__":
    main()

Output (dry-run on a cluttered Downloads folder):

Organizing: /home/user/Downloads
(DRY RUN -- no files will be moved)

  [DRY RUN] budget-2025.pdf  ->  Documents/
  [DRY RUN] photo-holiday.jpg  ->  Images/
  [DRY RUN] project.zip  ->  Archives/
  [DRY RUN] meeting-notes.docx  ->  Documents/
  [DRY RUN] setup.exe  ->  Installers/
  [DRY RUN] demo-clip.mp4  ->  Videos/

--- Summary ---
  Archives        1 file(s)
  Documents       2 file(s)
  Images          1 file(s)
  Installers      1 file(s)
  Videos          1 file(s)
  TOTAL           6 file(s)

Run it for real by dropping --dry-run: python organize.py ~/Downloads. The script is structured so you can extend it easily — add more extensions to EXT_MAP, add a --log flag that writes moves to a text file, or drop in a date-based sorting layer for photos.

Frequently Asked Questions

What happens if two files have the same name?

The safe_move() function handles this with a counter loop. If report.pdf already exists in Documents/, the incoming file becomes report(1).pdf. If that also exists, it becomes report(2).pdf, and so on. This protects you from silent data overwrites that shutil.move() would otherwise allow by default.

Can I undo the organization after running it?

Not automatically — shutil.move() does not maintain an undo log. For a safety net, always run with --dry-run first to verify the plan. For a real undo capability, modify the script to write every move to a JSON log file ({"src": "...", "dst": "..."}), then write a companion script that reverses each entry. Alternatively, test on a copy of the folder before touching the original.

How do I organize files by date instead of extension?

Swap the extension lookup for a date lookup using the file’s modification time. item.stat().st_mtime returns a Unix timestamp; pass it to datetime.fromtimestamp() and use strftime("%Y/%m") to get a 2026/08 folder path. For photos with EXIF data, the Pillow library can read the DateTimeOriginal tag for more accurate dates than the filesystem timestamp.

How do I run this automatically every day?

On macOS and Linux, add a cron job: open a terminal, run crontab -e, and add 0 8 * * * python3 /path/to/organize.py ~/Downloads to run it at 8 AM daily. On Windows, use Task Scheduler and point it at a batch file that calls your Python script. Both approaches run the script silently in the background without any manual action required.

Does this script handle hidden files (dotfiles)?

Path.iterdir() returns hidden files (files starting with . on Unix, like .DS_Store or .gitignore). The script currently skips files with no extension (the if not item.suffix guard), which silently ignores many dotfiles. If you want to explicitly skip all hidden files, add if item.name.startswith("."): continue right after the is_file() check.

What if I want different category names than the defaults?

Edit the EXT_MAP dictionary directly. The values are plain strings — change "Documents" to "Docs", or "Images" to "Photos", and the script will create folders with those names. You can also load the map from a JSON config file at runtime if you want to share settings across multiple machines without editing the script itself.

Conclusion

You have built a complete, production-ready file organizer in Python using only the standard library. The key tools are pathlib.Path.iterdir() for scanning, item.suffix.lower() for extension detection, shutil.move() for relocating files, and argparse for a proper CLI with --dry-run and --recursive flags. The safe_move() helper prevents silent data overwrites by appending a counter to duplicate filenames.

From here, consider extending the script with date-based sorting for photos (using datetime.fromtimestamp(item.stat().st_mtime)), a JSON move log for undo support, or a --watch mode using the watchdog library to organize files as they arrive. The structure is already in place — each extension is another entry in the map, and each new mode is a flag away.

Official documentation: pathlib — Object-oriented filesystem paths and shutil — High-level file operations.

Related Articles

• How To Automate Tasks with Python and Schedule
• How To Use argparse for Command-Line Interfaces in Python
• How To Copy, Move, and Delete Files with Python shutil

Intermediate

You are debugging a crash in production. The traceback lands on a generic RuntimeError("Failed to load config") — but you have no idea what actually went wrong. Was it a missing file? A JSON parse error? A permissions problem? The original exception is gone, buried somewhere inside the code that caught and re-raised it without preserving the cause. You are left guessing at the root cause from a vague error message, and that is a bad situation to be in at 2am when a service is down.

Python’s exception chaining mechanism exists to solve exactly this problem. When you catch an exception and need to raise a different one, you can use raise NewException(...) from original_exc to link the two together. The traceback then shows the full chain — both the original cause and the new exception — so whoever reads it can trace the problem all the way back to its source. Everything covered here is built into Python’s standard exception system, with no third-party packages required.

This article covers how Python’s implicit and explicit exception chaining works, how to use raise from to set an explicit cause, the difference between __cause__ and __context__, when to use raise from None to suppress the original exception, how to re-raise a caught exception cleanly with bare raise, and how to inspect exception chains programmatically. By the end you will be able to write error-handling code that gives callers accurate, complete stack traces instead of mystery messages.

Exception Chaining in Python: Quick Example

Here is a minimal example showing the two most common patterns — re-raising with a cause, and suppressing the cause — side by side:

# exception_chain_quick.py

def load_config(path):
    try:
        with open(path) as f:
            import json
            return json.load(f)
    except (FileNotFoundError, json.JSONDecodeError) as exc:
        # Chain: tell the caller WHY loading failed, and preserve the original error
        raise RuntimeError(f"Failed to load config from {path}") from exc

def get_user_id(raw_value):
    try:
        return int(raw_value)
    except ValueError as exc:
        # Suppress: the ValueError is an internal detail; expose a cleaner error
        raise ValueError(f"User ID must be a number, got: {raw_value!r}") from None

# Try chaining
try:
    load_config("/nonexistent/config.json")
except RuntimeError as exc:
    print(type(exc).__name__, "->", exc)
    print("Caused by:", type(exc.__cause__).__name__, exc.__cause__)

# Try suppression
try:
    get_user_id("abc")
except ValueError as exc:
    print(type(exc).__name__, "->", exc)
    print("Caused by:", exc.__cause__)   # None -- suppressed

Output:

RuntimeError -> Failed to load config from /nonexistent/config.json
Caused by: FileNotFoundError [Errno 2] No such file or directory: '/nonexistent/config.json'
ValueError -> User ID must be a number, got: 'abc'
Caused by: None

The first case shows chaining: the RuntimeError carries its cause in __cause__, and a real traceback would print both exceptions. The second case shows suppression: the inner ValueError is discarded, and the caller sees only the clean public error. The sections below explain when to use each pattern and how the underlying machinery works.

What Is Exception Chaining and Why Does It Matter?

Every Python exception object can hold a reference to another exception. This lets you capture the full history of what went wrong, not just the final symptom. Without chaining, catching an exception and raising a new one loses all the context from the original — the stack frame, the message, the type. With chaining, a reader of the traceback sees the complete story from root cause to final failure.

Python has two forms of exception chaining. Implicit chaining happens automatically whenever you raise an exception inside an except block. Python sets exc.__context__ to the active exception automatically, so the “During handling of the above exception, another exception occurred” message in a traceback is implicit chaining at work. Explicit chaining happens when you use raise NewExc from original_exc — this sets exc.__cause__ and also sets exc.__suppress_context__ = True, which tells Python to display “The above exception was the direct cause of the following exception” instead of the implicit message.

The rule of thumb: use explicit chaining (raise from) when the original exception is genuinely useful context for the caller. Use suppression (raise from None) when the original exception is an internal implementation detail that would only confuse the caller. Use bare raise when you want to re-raise the exact same exception without modification.

Using raise from to Set an Explicit Cause

The raise NewException from original syntax explicitly links two exceptions. The new exception is what propagates to the caller; the original is attached as __cause__ and displayed in the traceback above the new exception. Use this pattern when your function catches a low-level exception (like a database error or a network timeout) and raises a higher-level one that makes sense in your abstraction layer.

# raise_from_demo.py
import json

class ConfigError(Exception):
    """Raised when configuration loading or parsing fails."""

def parse_settings(raw_json: str) -> dict:
    """Parse a JSON settings string. Raises ConfigError on failure."""
    try:
        return json.loads(raw_json)
    except json.JSONDecodeError as exc:
        raise ConfigError(
            f"Settings file contains invalid JSON at line {exc.lineno}, col {exc.colno}"
        ) from exc

# Simulate bad JSON input
bad_json = '{"timeout": 30, "host": "db.example.com"'  # missing closing brace

try:
    parse_settings(bad_json)
except ConfigError as exc:
    print(f"ConfigError: {exc}")
    print(f"__cause__ type: {type(exc.__cause__).__name__}")
    print(f"__cause__ msg:  {exc.__cause__}")
    print(f"__suppress_context__: {exc.__suppress_context__}")

Output:

ConfigError: Settings file contains invalid JSON at line 1, col 41
__cause__ type: JSONDecodeError
__cause__ msg:  Expecting property name enclosed in double quotes: line 1 column 41 (char 40)
__suppress_context__: True

The caller catches a clean ConfigError with a human-readable message, but the original JSONDecodeError is preserved in __cause__. In a real traceback (not caught by a try/except), Python would print the JSONDecodeError first with “The above exception was the direct cause of the following exception”, then print the ConfigError. Notice that __suppress_context__ is automatically set to True by raise from — this tells Python to use the “direct cause” wording instead of the implicit “during handling” wording.

Understanding __cause__ vs __context__

Python maintains both __cause__ and __context__ on every exception, and they serve different purposes. __context__ is set automatically whenever an exception is raised while another exception is already being handled — it is the exception that was active at the time. __cause__ is only set when you use raise ... from ..., and it signals an intentional, semantic relationship: “this exception was directly caused by that one.”

# cause_vs_context.py

def implicit_context():
    """Raises inside an except block -- sets __context__ automatically."""
    try:
        int("not-a-number")       # raises ValueError
    except ValueError:
        raise RuntimeError("Something went wrong internally")

def explicit_cause():
    """Uses raise from -- sets __cause__ explicitly."""
    try:
        int("not-a-number")       # raises ValueError
    except ValueError as exc:
        raise RuntimeError("Input conversion failed") from exc

def mixed():
    """Both are set when you use raise from inside an except block."""
    try:
        int("not-a-number")
    except ValueError as exc:
        new_exc = RuntimeError("Conversion error")
        raise new_exc from exc
    # new_exc.__cause__ == the ValueError (explicit)
    # new_exc.__context__ == the ValueError (implicit, same exception here)

# Inspect implicit chaining
try:
    implicit_context()
except RuntimeError as exc:
    print("=== Implicit context ===")
    print(f"__cause__:              {exc.__cause__}")
    print(f"__context__ type:       {type(exc.__context__).__name__}")
    print(f"__suppress_context__:   {exc.__suppress_context__}")

# Inspect explicit cause
try:
    explicit_cause()
except RuntimeError as exc:
    print("\n=== Explicit cause ===")
    print(f"__cause__ type:         {type(exc.__cause__).__name__}")
    print(f"__context__ type:       {type(exc.__context__).__name__}")
    print(f"__suppress_context__:   {exc.__suppress_context__}")

Output:

=== Implicit context ===
__cause__:              None
__context__ type:       ValueError
__suppress_context__:   False

=== Explicit cause ===
__cause__ type:         ValueError
__context__ type:       ValueError
__suppress_context__:   True

In the implicit case, __context__ is set (because we raised inside an except block) but __cause__ is None and __suppress_context__ is False. Python will show the “During handling” message. In the explicit case, both __cause__ and __context__ point to the same ValueError, but __suppress_context__ is True — so Python uses the “direct cause” wording and ignores __context__ for display purposes. The key rule: raise from always sets __suppress_context__ = True, which changes both the traceback wording and which exception Python chooses to display.

Suppressing the Original Exception with raise from None

Sometimes you catch a low-level exception but the implementation detail it represents should not leak to the caller. A classic example: a KeyError inside a dictionary lookup that you expose as a public LookupError. If you let the KeyError surface in the traceback, callers start depending on your internal data structure — they write except KeyError instead of except LookupError, and now you cannot change the implementation. Using raise NewExc from None suppresses both __cause__ and __context__, producing a clean traceback that shows only the new exception.

# raise_from_none.py

class UserNotFoundError(Exception):
    """Public exception for the user lookup API."""

# Internal implementation uses a dict -- an implementation detail
_USER_STORE = {
    "alice": {"id": 1, "role": "admin"},
    "bob":   {"id": 2, "role": "viewer"},
}

def get_user(username: str) -> dict:
    """Look up a user by name. Raises UserNotFoundError if not found."""
    try:
        return _USER_STORE[username]    # raises KeyError internally
    except KeyError:
        # Suppress: callers should not see or depend on KeyError
        raise UserNotFoundError(f"No user named {username!r}") from None

# Caller never sees the KeyError
try:
    get_user("mallory")
except UserNotFoundError as exc:
    print(f"Caught: {exc}")
    print(f"__cause__:            {exc.__cause__}")
    print(f"__context__:          {exc.__context__}")
    print(f"__suppress_context__: {exc.__suppress_context__}")

Output:

Caught: No user named 'mallory'
__cause__:            None
__context__:          None
__suppress_context__: True

Both __cause__ and __context__ are None. This is the special case of raise from None: it sets __cause__ = None, __context__ = None, and __suppress_context__ = True, so the traceback shows only the new exception with no chaining information. Use this sparingly — only when you are certain the original exception would actively mislead callers. In most cases, preserving the cause with raise from exc is more helpful for debugging.

Re-raising Exceptions with bare raise

Sometimes you want to inspect a caught exception, take some action (log it, update a counter, release a resource), and then let it propagate unchanged. Bare raise — with no argument — re-raises the current exception exactly as it was, including all its attributes, original traceback, and any chaining information. It does not create a new exception or modify the existing one.

# bare_raise.py
import logging

logging.basicConfig(level=logging.ERROR, format="%(levelname)s: %(message)s")

_error_count = 0

def risky_parse(value: str) -> int:
    """Parse an integer, log any failure, then re-raise."""
    global _error_count
    try:
        return int(value)
    except ValueError:
        _error_count += 1
        logging.error("Parse failed for value %r (total errors: %d)", value, _error_count)
        raise   # re-raise the original ValueError unchanged

# First call -- fails
try:
    risky_parse("bad")
except ValueError as exc:
    print(f"Caller caught: {type(exc).__name__}: {exc}")
    print(f"Error count so far: {_error_count}")

# Second call -- also fails
try:
    risky_parse("also bad")
except ValueError as exc:
    print(f"Caller caught: {type(exc).__name__}: {exc}")
    print(f"Error count so far: {_error_count}")

# Third call -- succeeds
result = risky_parse("42")
print(f"Parsed successfully: {result}")

Output:

ERROR: Parse failed for value 'bad' (total errors: 1)
Caller caught: ValueError: invalid literal for int() with base 10: 'bad'
Error count so far: 1
ERROR: Parse failed for value 'also bad' (total errors: 2)
Caller caught: ValueError: invalid literal for int() with base 10: 'also bad'
Error count so far: 2
Parsed successfully: 42

The ValueError the caller receives is the original exception object — not a copy, not a new instance. Its traceback points back to the int(value) call inside risky_parse, not to the bare raise statement. This is the critical difference between bare raise and raise exc (re-raising by variable): raise exc resets the traceback start to the line containing raise exc, hiding where the exception actually originated. Always prefer bare raise when re-raising the same exception in an except block.

Inspecting Exception Chains Programmatically

For logging frameworks, error-reporting systems, or debugging utilities, you sometimes need to walk the full exception chain rather than just letting Python print it. You can traverse __cause__ and __context__ yourself to collect every exception in the chain, from the outermost to the innermost root cause.

# inspect_chain.py

def collect_chain(exc):
    """Walk the exception chain and return a list of all exceptions, outermost first."""
    chain = []
    visited = set()
    current = exc
    while current is not None and id(current) not in visited:
        chain.append(current)
        visited.add(id(current))
        # Prefer __cause__ (explicit) over __context__ (implicit)
        if current.__cause__ is not None:
            current = current.__cause__
        elif not current.__suppress_context__ and current.__context__ is not None:
            current = current.__context__
        else:
            break
    return chain

def format_chain(exc):
    """Return a structured summary of the full exception chain."""
    chain = collect_chain(exc)
    parts = []
    for i, e in enumerate(chain):
        relationship = ""
        if i == 0:
            relationship = "[outermost]"
        elif chain[i-1].__cause__ is e:
            relationship = "[direct cause]"
        else:
            relationship = "[context]"
        parts.append(f"  {relationship} {type(e).__name__}: {e}")
    return "\n".join(parts)

# Build a three-level chain
try:
    try:
        try:
            int("xyz")                          # root: ValueError
        except ValueError as e1:
            raise OSError("IO layer failed") from e1   # middle: OSError
    except OSError as e2:
        raise RuntimeError("Top-level failure") from e2  # outer: RuntimeError
except RuntimeError as final:
    print("Exception chain (outermost -> root cause):")
    print(format_chain(final))
    print()
    chain = collect_chain(final)
    print(f"Root cause: {type(chain[-1]).__name__}: {chain[-1]}")

Output:

Exception chain (outermost -> root cause):
  [outermost] RuntimeError: Top-level failure
  [direct cause] OSError: IO layer failed
  [direct cause] ValueError: invalid literal for int() with base 10: 'xyz'

Root cause: ValueError: invalid literal for int() with base 10: 'xyz'

The traversal prefers __cause__ over __context__ because explicit chaining is the intentional relationship. It also guards against cycles using a visited set — a defensive measure since circular exception chains are theoretically possible (though rare in practice). The format_chain function is a lightweight building block you can drop into any logging or error-reporting utility to include the full chain in your structured logs.

Real-Life Example: Resilient Database Query Handler

Here is a practical module that wraps a database layer (simulated with a dictionary) and demonstrates all four patterns — explicit chaining, suppression, bare re-raise, and chain inspection — in a realistic context. The module translates low-level database exceptions into domain-specific ones while preserving enough context for effective debugging.

# db_handler.py
import logging

logging.basicConfig(level=logging.WARNING, format="%(levelname)s %(name)s: %(message)s")
logger = logging.getLogger("db_handler")


# --- Exceptions ---

class DatabaseError(Exception):
    """Base class for all database errors in this module."""

class RecordNotFoundError(DatabaseError):
    """Raised when a queried record does not exist."""

class DataCorruptionError(DatabaseError):
    """Raised when stored data cannot be deserialized."""


# --- Simulated low-level storage ---

import json

_STORE = {
    "user:1": '{"name": "Alice", "role": "admin"}',
    "user:2": '{"name": "Bob", "role": "viewer"}',
    "user:3": '{CORRUPTED DATA}',        # simulates corruption
}

def _raw_fetch(key: str) -> str:
    """Low-level fetch -- raises KeyError if the key does not exist."""
    return _STORE[key]                   # KeyError is an implementation detail


# --- Public API ---

def get_user(user_id: int) -> dict:
    """
    Fetch a user record by ID.

    Raises:
        RecordNotFoundError: if the user does not exist.
        DataCorruptionError: if the stored data cannot be parsed.
        DatabaseError: for unexpected low-level failures.
    """
    key = f"user:{user_id}"

    # Fetch the raw record
    try:
        raw = _raw_fetch(key)
    except KeyError:
        # Suppress: KeyError is internal; the public contract is RecordNotFoundError
        raise RecordNotFoundError(f"User {user_id} not found") from None

    # Parse the record
    try:
        data = json.loads(raw)
    except json.JSONDecodeError as exc:
        # Chain: the JSON error is directly responsible and useful for debugging
        raise DataCorruptionError(
            f"Stored data for user {user_id} is not valid JSON"
        ) from exc

    # Validate expected shape
    if not isinstance(data, dict) or "name" not in data:
        raise DataCorruptionError(
            f"User {user_id} record is missing required fields"
        ) from None

    return data


def safe_get_user(user_id: int) -> dict | None:
    """
    Like get_user, but returns None for missing records instead of raising.
    Re-raises DataCorruptionError -- corruption is always worth knowing about.
    """
    try:
        return get_user(user_id)
    except RecordNotFoundError:
        return None
    except DataCorruptionError:
        logger.warning("Corruption detected for user %d -- propagating", user_id)
        raise   # bare raise: preserve the original exception and its chain


# --- Demo ---

print("=== Normal fetch ===")
user = get_user(1)
print(f"Found: {user}")

print("\n=== Missing user (suppressed) ===")
try:
    get_user(99)
except RecordNotFoundError as exc:
    print(f"RecordNotFoundError: {exc}")
    print(f"  __cause__: {exc.__cause__}")

print("\n=== Corrupted record (chained) ===")
try:
    get_user(3)
except DataCorruptionError as exc:
    print(f"DataCorruptionError: {exc}")
    print(f"  __cause__ type: {type(exc.__cause__).__name__}")
    print(f"  __cause__ msg:  {exc.__cause__}")

print("\n=== safe_get_user: missing ===")
result = safe_get_user(99)
print(f"Result: {result}")

print("\n=== safe_get_user: corrupted -- re-raised ===")
try:
    safe_get_user(3)
except DataCorruptionError as exc:
    print(f"Re-raised: {type(exc).__name__}: {exc}")
    # __cause__ is still intact from the original raise in get_user
    print(f"  __cause__ still attached: {type(exc.__cause__).__name__}")

Output:

=== Normal fetch ===
Found: {'name': 'Alice', 'role': 'admin'}

=== Missing user (suppressed) ===
RecordNotFoundError: User 99 not found
  __cause__: None

=== Corrupted record (chained) ===
DataCorruptionError: Stored data for user 3 is not valid JSON
  __cause__ type: JSONDecodeError
  __cause__ msg:  Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

=== safe_get_user: missing ===
Result: None

=== safe_get_user: corrupted -- re-raised ===
WARNING db_handler: Corruption detected for user 3 -- propagating
Re-raised: DataCorruptionError: Stored data for user 3 is not valid JSON
  __cause__ still attached: JSONDecodeError

The module demonstrates all four patterns in a coherent API. get_user uses suppression for the missing-record case (callers should not see KeyError) and explicit chaining for the corruption case (the JSONDecodeError details are valuable). safe_get_user uses bare raise to let corruption propagate without altering the exception — the __cause__ chain from the original raise in get_user is still fully intact when the caller catches it. Extend this pattern by adding retry logic around _raw_fetch, structured JSON logging of the chain using collect_chain, or a context manager that wraps arbitrary code and translates all Exception subclasses into domain errors.

Frequently Asked Questions

What is the difference between bare raise and raise exc?

Bare raise (with no argument) re-raises the current exception exactly as it was, preserving its original traceback. The traceback will point to the original site where the exception was first raised, not to the bare raise statement. raise exc (raising the caught exception by name) creates a new raise site — the traceback will now start at the line with raise exc, hiding the original origin. In an except block, always prefer bare raise unless you intentionally want to change the traceback origin. The only reason to use raise exc is if you are storing the exception and raising it later, outside of the except block where it was caught.

When should I use raise from None instead of raise from exc?

Use raise from None when the original exception is an internal implementation detail that would actively mislead callers. The classic cases are: catching a KeyError from a private dictionary and raising a public LookupError, catching a StopIteration inside a generator and re-raising a different error, or translating third-party library exceptions into your own domain exceptions when the library’s exception type would expose your dependency choices. If you are unsure, default to raise from exc — preserving context is almost always more helpful for debugging than hiding it.

How do I read a chained exception traceback?

Python prints exception chains from the oldest (innermost) exception at the top to the newest (outermost) at the bottom. Read them bottom-up to understand what failed from a user perspective, and top-down to understand the root cause. The separator line tells you the relationship: “The above exception was the direct cause of the following exception” means explicit chaining (raise from); “During handling of the above exception, another exception occurred” means implicit chaining (raised in an except block without from). The outermost exception (at the very bottom) is the one your except clause catches.

Can exception chaining happen in a finally block?

Yes, and it can be surprising. If an exception is raised inside a finally block while another exception is already propagating, Python sets __context__ on the new exception, creating an implicit chain. The new exception from the finally block replaces the original as the propagating exception, but the original is still accessible via __context__. This means a noisy finally block (one that can itself raise) can swallow or obscure the original error. The fix is to wrap the body of your finally block in its own try/except and handle errors there, so cleanup failures do not mask the original exception.

Do I need to do anything special for custom exception classes?

No — all exception chaining behavior is inherited from BaseException, so any class that inherits from Exception (directly or indirectly) already has __cause__, __context__, and __suppress_context__ as built-in attributes. You do not need to add anything to your custom exception class to use raise from with it. If you want to add extra context fields (like a request ID or a user ID) that travel with the exception, define them as constructor arguments and store them as instance attributes — they will be accessible alongside the chaining attributes.

Does exception chaining work the same way in async code?

Yes, exception chaining works identically in async code. raise from, bare raise, and __cause__/__context__ all behave the same inside async def functions, async with blocks, and async for loops. The only complication is with asyncio.TaskGroup or asyncio.gather: when multiple tasks fail simultaneously, Python wraps them in an ExceptionGroup (Python 3.11+), which is a separate mechanism for collecting multiple simultaneous exceptions. The except* syntax handles those. For single-task async error handling, use the same patterns covered in this article.

Conclusion

In this article we covered how Python’s two exception chaining mechanisms work — implicit chaining via __context__ and explicit chaining via raise from and __cause__. We saw how to use raise NewExc from original to preserve context across abstraction boundaries, how raise from None suppresses the original exception when it would only mislead callers, and how bare raise re-raises the current exception without altering its traceback origin. The collect_chain utility showed how to walk the chain programmatically for logging and reporting, and the database handler example tied all four patterns together in a realistic API.

A good next step is to add collect_chain to your application’s logging configuration so that every logged exception automatically includes its full chain, not just the outermost message. For structured logging (JSON logs), serialize each exception in the chain as a separate entry in a list field — that way log analysis tools can filter by root cause type without parsing traceback strings. The official Python documentation on exception chaining covers the language spec in full detail, and PEP 3134 is the original proposal that introduced the mechanism — it explains the design rationale clearly and is worth reading if you want to understand why the two-chain model exists.

Related Articles

• How To Manage Multiple Context Managers with contextlib.ExitStack
• How To Use Python contextmanager Decorator
• How To Use Python Logging

Intermediate

You are writing a data pipeline that opens five CSV files, acquires a threading lock, and connects to a database — all at once. The classic with statement handles one or two resources cleanly, but by the time you have five nested with blocks, your code has drifted four indentation levels to the right and cleanup order is anyone’s guess. Worse, the number of files is only known at runtime, so you cannot even write five static with statements — you need to open them in a loop.

Python’s contextlib.ExitStack was designed exactly for this situation. It is a context manager that manages other context managers, letting you enter and exit any number of them — determined at runtime — with a single with block. Everything in the standard library that works with with works with ExitStack. No third-party packages required.

This article covers what ExitStack is and when to reach for it, how to add context managers dynamically with enter_context(), how to register arbitrary cleanup functions with callback(), how error handling and suppression work, and patterns for optional or conditional resources. By the end you will have built a real multi-file log merger that opens any number of files cleanly and guarantees cleanup even when exceptions occur.

ExitStack in Python: Quick Example

Here is the minimal case: opening a variable number of files with ExitStack and reading from all of them, with guaranteed cleanup regardless of errors.

# exitstack_quick.py
import contextlib
import tempfile
import os

# Create three temporary files with sample data
paths = []
for i in range(3):
    f = tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False)
    f.write(f"Line from file {i+1}\n")
    f.close()
    paths.append(f.name)

# Open all three files at once using ExitStack
with contextlib.ExitStack() as stack:
    file_handles = [stack.enter_context(open(p)) for p in paths]
    for fh in file_handles:
        print(fh.read().strip())

# Cleanup
for p in paths:
    os.unlink(p)

Output:

Line from file 1
Line from file 2
Line from file 3

The key call is stack.enter_context(open(p)). Each call activates one context manager and registers its __exit__ method with the stack. When the with block ends — normally or via exception — the stack calls every registered __exit__ in reverse order, exactly as nested with statements would. Whether you enter two context managers or twenty, the pattern stays the same.

The sections below explain how this mechanism works, cover more advanced usage including callback() and error suppression, and show patterns for optional and conditional resources.

What Is ExitStack and When Do You Need It?

A context manager in Python is any object that implements __enter__ and __exit__. The with statement calls __enter__ on entry and __exit__ on exit — even if an exception is raised. This is why with open(...) closes the file even when your code crashes halfway through reading it.

Nested with statements handle two or three resources reasonably well. But they have two hard limits: the number must be known at write time, and each level adds one indentation. ExitStack removes both limits. Think of it as a list that accumulates __exit__ callbacks as you enter context managers, then drains the list in reverse order when the block ends. The “last in, first out” cleanup order mirrors what nested with statements would produce.

ExitStack lives in the contextlib module, which ships with Python’s standard library — nothing to install. Import it with from contextlib import ExitStack or use contextlib.ExitStack directly.

Adding Context Managers Dynamically with enter_context()

enter_context(cm) is the main API. It calls cm.__enter__(), registers cm.__exit__ with the stack, and returns whatever __enter__ returned — exactly the same value you would get from as in a plain with statement. You can call it as many times as you need, in any order, including inside loops and conditionals.

# enter_context_demo.py
import contextlib
import threading
import tempfile
import os

lock = threading.Lock()

# Open a dynamic number of files plus a lock in one block
filenames = [tempfile.mktemp(suffix='.log') for _ in range(4)]
for name in filenames:
    open(name, 'w').close()  # create them

with contextlib.ExitStack() as stack:
    # Acquire the lock first -- it is released last (LIFO order)
    stack.enter_context(lock)
    # Open all log files
    handles = [stack.enter_context(open(name, 'r')) for name in filenames]

    print(f"Lock acquired: {lock.locked()}")
    print(f"Files open: {len(handles)}")
    # ... process handles here ...

# After the block: lock released, all files closed
print(f"Lock still held: {lock.locked()}")
print(f"File closed: {handles[0].closed}")

# Cleanup temp files
for name in filenames:
    os.unlink(name)

Output:

Lock acquired: True
Files open: 4
Lock still held: False
File closed: True

Notice the cleanup order: the lock was entered first and released last. ExitStack always calls __exit__ in reverse entry order, which mirrors what you would get from innermost-to-outermost nested with blocks. If cleanup order matters for your resources — and for locks it usually does — enter them in the order you want them released in reverse.

Registering Arbitrary Cleanup with callback()

Not everything you need to clean up is a context manager. Sometimes you have a function call, a database rollback, a temp directory deletion, or a log flush that needs to run on exit. stack.callback(fn, *args, **kwargs) registers any callable as a cleanup step — it runs at the same time as __exit__ calls, in LIFO order, and is guaranteed to run even if an exception is raised.

# callback_demo.py
import contextlib
import tempfile
import shutil
import os

def create_workspace():
    """Create a temp directory and return its path."""
    return tempfile.mkdtemp(prefix='job_')

def log_cleanup(path):
    print(f"Cleaning up workspace: {path}")

with contextlib.ExitStack() as stack:
    workspace = create_workspace()
    # Register cleanup as a callback -- shutil.rmtree is not a context manager
    stack.callback(log_cleanup, workspace)
    stack.callback(shutil.rmtree, workspace, ignore_errors=True)

    # Do work inside the workspace
    output_file = os.path.join(workspace, 'result.txt')
    with open(output_file, 'w') as f:
        f.write('processing complete\n')

    print(f"Workspace exists: {os.path.isdir(workspace)}")
    print(f"Output file: {os.path.exists(output_file)}")

print(f"Workspace removed: {not os.path.isdir(workspace)}")

Output:

Workspace exists: True
Output file: True
Cleaning up workspace: /tmp/job_xyz123
Workspace removed: True

The callback registered last runs first. Here shutil.rmtree is registered before log_cleanup, so the log message prints before the directory is deleted — that is the reverse registration order. Think carefully about the order when callbacks depend on each other. Also note that callback() accepts positional and keyword arguments directly, so you never need a lambda wrapper: stack.callback(shutil.rmtree, workspace, ignore_errors=True) is cleaner and safer than stack.callback(lambda: shutil.rmtree(workspace, ignore_errors=True)).

Optional and Conditional Resources

One of the most practical uses of ExitStack is managing resources that are only opened under certain conditions — for example, opening a second output file only when a --verbose flag is passed, or acquiring a distributed lock only in production. With nested with statements, conditional resources require awkward outer if blocks or dummy no-op context managers. With ExitStack, you simply call enter_context() inside a conditional.

# conditional_resource.py
import contextlib
import sys
import os
import tempfile

verbose = '--verbose' in sys.argv

# Simulate an output file path
output_path = tempfile.mktemp(suffix='.log')

with contextlib.ExitStack() as stack:
    out = open(output_path, 'w')
    stack.enter_context(out)

    # Only open a debug log if verbose mode is on
    if verbose:
        debug_path = tempfile.mktemp(suffix='.debug')
        debug = stack.enter_context(open(debug_path, 'w'))
        debug.write("debug mode active\n")
        print(f"Debug log: {debug_path}")
    else:
        debug = None
        print("Running in normal mode (no debug log)")

    out.write("result: OK\n")
    if debug:
        debug.write("result written to output\n")

print(f"Output file closed: {out.closed}")
os.unlink(output_path)

Output (without –verbose flag):

Running in normal mode (no debug log)
Output file closed: True

The single with contextlib.ExitStack() block handles both cases. When verbose is False, only the output file is registered. When it is True, both files are registered and both are closed on exit. No dummy context managers, no nested if/with combinations, no cleanup logic repeated in two branches.

Error Handling and Exception Suppression

ExitStack mirrors the exception behavior of nested with statements. If an exception occurs inside the block, it is passed to each registered __exit__ in reverse order. Any __exit__ that returns a truthy value suppresses the exception. If the exception survives all cleanup calls, it propagates normally. You can also suppress exceptions explicitly with contextlib.suppress entered onto the stack.

# error_handling.py
import contextlib

class NoisyResource:
    """A context manager that prints on entry and exit."""
    def __init__(self, name):
        self.name = name

    def __enter__(self):
        print(f"  Opened: {self.name}")
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        print(f"  Closed: {self.name} (exc={exc_type is not None})")
        return False  # do not suppress exceptions

print("--- Normal exit ---")
with contextlib.ExitStack() as stack:
    stack.enter_context(NoisyResource("A"))
    stack.enter_context(NoisyResource("B"))
    stack.enter_context(NoisyResource("C"))

print()
print("--- Exception during body ---")
try:
    with contextlib.ExitStack() as stack:
        stack.enter_context(NoisyResource("X"))
        stack.enter_context(NoisyResource("Y"))
        raise ValueError("something went wrong")
except ValueError as e:
    print(f"  Caught: {e}")

print()
print("--- Suppress a specific exception type ---")
with contextlib.ExitStack() as stack:
    stack.enter_context(contextlib.suppress(FileNotFoundError))
    stack.enter_context(NoisyResource("Z"))
    raise FileNotFoundError("missing.txt")  # suppressed
print("  Continued after suppressed exception")

Output:

--- Normal exit ---
  Opened: A
  Opened: B
  Opened: C
  Closed: C (exc=False)
  Closed: B (exc=False)
  Closed: A (exc=False)

--- Exception during body ---
  Opened: X
  Opened: Y
  Closed: Y (exc=True)
  Closed: X (exc=True)
  Caught: something went wrong

--- Suppress a specific exception type ---
  Opened: Z
  Closed: Z (exc=True)
  Continued after suppressed exception

All three resources are closed even when an exception is raised — the exception information is passed to each __exit__ but neither suppresses it, so it propagates after cleanup. The third block shows contextlib.suppress(FileNotFoundError) entered onto the stack: when FileNotFoundError is raised, the suppress context manager’s __exit__ returns True, and the exception is silently discarded.

Detaching the Stack for Deferred Cleanup

Sometimes you want to open resources and pass them elsewhere — to a class, a worker thread, or a long-lived object — and delay cleanup until that object is done. stack.pop_all() transfers all registered cleanup handlers to a new ExitStack and returns it, leaving the original empty. The new stack can then be stored as an instance variable and closed explicitly later.

# deferred_cleanup.py
import contextlib
import tempfile
import os

class FileProcessor:
    """Opens a set of files and keeps them available until close() is called."""

    def __init__(self, paths):
        self._stack = contextlib.ExitStack()
        self.handles = []
        try:
            for path in paths:
                fh = self._stack.enter_context(open(path))
                self.handles.append(fh)
        except Exception:
            # If any open fails, close what we opened so far
            self._stack.close()
            raise

    def process(self):
        for fh in self.handles:
            print(f"  Reading {fh.name}: {fh.read().strip()}")

    def close(self):
        self._stack.close()

# Create temp files
paths = []
for i in range(3):
    f = tempfile.NamedTemporaryFile(mode='w', suffix='.dat', delete=False)
    f.write(f"data from file {i+1}")
    f.close()
    paths.append(f.name)

processor = FileProcessor(paths)
print("Processing:")
processor.process()
processor.close()
print(f"File 0 closed: {processor.handles[0].closed}")

for p in paths:
    os.unlink(p)

Output:

Processing:
  Reading /tmp/tmpXXX.dat: data from file 1
  Reading /tmp/tmpXXX.dat: data from file 2
  Reading /tmp/tmpXXX.dat: data from file 3
File 0 closed: True

The try/except block inside __init__ is important: if the third file fails to open, self._stack.close() ensures the first two are also closed before the exception propagates. Without it, those file handles would leak. This pattern — open resources inside a stack, then store the stack, then close it explicitly — is the standard approach for long-lived objects that manage multiple resources.

Real-Life Example: Multi-File Log Merger

Here is a practical script that merges any number of timestamped log files into a single sorted output file. The number of input files is determined at runtime from a directory scan, making ExitStack the right tool for the job.

# log_merger.py
import contextlib
import tempfile
import os
import heapq

def create_sample_logs(base_dir, num_files=4):
    """Create sample log files with interleaved timestamps."""
    import random
    paths = []
    for i in range(num_files):
        path = os.path.join(base_dir, f"service_{i+1}.log")
        with open(path, 'w') as f:
            lines = sorted(
                f"2026-08-13T0{h:02d}:{m:02d}:00 [service-{i+1}] event-{j}\n"
                for j, (h, m) in enumerate(
                    (random.randint(0, 9), random.randint(0, 59))
                    for _ in range(5)
                )
            )
            f.writelines(lines)
        paths.append(path)
    return paths

def merge_logs(input_paths, output_path):
    """Merge sorted log files into one sorted output file."""
    with contextlib.ExitStack() as stack:
        # Open all input files dynamically
        handles = [stack.enter_context(open(p)) for p in input_paths]
        # Open the output file on the same stack
        out = stack.enter_context(open(output_path, 'w'))

        # Register a callback to print a summary on exit
        line_count = [0]
        def report():
            print(f"  Merged {len(handles)} files -> {line_count[0]} lines total")
        stack.callback(report)

        # Merge-sort across all files using heapq
        for line in heapq.merge(*handles):
            out.write(line)
            line_count[0] += 1

    # Stack has exited: all files closed, callback fired
    print(f"  Output written to: {output_path}")

# Run the demo
with tempfile.TemporaryDirectory() as tmpdir:
    log_paths = create_sample_logs(tmpdir, num_files=4)
    output = os.path.join(tmpdir, 'merged.log')

    print("Merging logs...")
    merge_logs(log_paths, output)

    # Show first 5 lines of result
    print("\nFirst 5 lines of merged output:")
    with open(output) as f:
        for i, line in enumerate(f):
            if i >= 5:
                break
            print(f"  {line.rstrip()}")

Output:

Merging logs...
  Merged 4 files -> 20 lines total
  Output written to: /tmp/tmpXXX/merged.log

First 5 lines of merged output:
  2026-08-13T000:12:00 [service-2] event-3
  2026-08-13T000:17:00 [service-4] event-1
  2026-08-13T001:08:00 [service-1] event-0
  2026-08-13T001:23:00 [service-3] event-2
  2026-08-13T002:45:00 [service-1] event-4

The ExitStack block opens all four input files plus the output file in a single flat with. The callback() for the summary report fires after the output file is flushed and closed — the right order for accurate reporting. You can extend this by adding a threading lock to the stack (if merging concurrently), entering a contextlib.suppress(BrokenPipeError) for piped output, or registering a callback to upload the merged file when done.

Frequently Asked Questions

When should I use ExitStack instead of nested with statements?

Use ExitStack when the number of context managers is not known until runtime (e.g., opening files from a list), when you have more than three or four fixed resources and the indentation becomes unreadable, or when you need to conditionally enter a context manager inside an if block. For two or three fixed, always-opened resources, nested with is simpler and more readable. ExitStack is the upgrade path when nested with starts to hurt.

What happens if an exception is raised inside a cleanup callback?

If a callback registered with stack.callback() raises an exception, ExitStack catches it, continues running the remaining cleanup handlers, and then re-raises the exception (or the original body exception, whichever takes precedence). This means all registered cleanups run even if some of them fail — the same guarantee provided by nested with statements. To suppress errors from specific callbacks, wrap the callable yourself: stack.callback(lambda: quietly_close(resource)) where quietly_close catches and logs its own exceptions.

What does pop_all() do and when do I need it?

stack.pop_all() transfers all registered cleanup handlers from the current stack to a new ExitStack instance and returns it, leaving the original stack empty. This is used when you want to open resources inside a temporary setup block but then keep them alive beyond that block’s scope. The typical pattern is a class that opens resources in __init__ using a temporary ExitStack, then calls pop_all() at the end to store the cleanup handlers as self._stack, which is closed in close() or __exit__.

Is ExitStack thread-safe?

No — ExitStack itself is not thread-safe. If multiple threads could be entering or exiting context managers on the same stack simultaneously, you need to synchronize access with a threading.Lock. In practice, each thread typically has its own ExitStack for its own resources, which avoids the issue entirely. If you need a shared cleanup registry across threads, build one explicitly using a lock.

Does ExitStack work with async context managers?

Standard ExitStack only works with synchronous context managers (those using __enter__/__exit__). For async code using async with and __aenter__/__aexit__, use contextlib.AsyncExitStack instead. It has the same API — enter_async_context(), callback(), push() — but is awaitable and must be used with async with. Both classes are in contextlib and require no extra installation.

Can I use ExitStack without a with statement?

Yes — you can manage it manually. Call stack = contextlib.ExitStack(), use stack.enter_context() to add resources, and call stack.close() when done. Wrap the whole thing in a try/finally to guarantee close() runs even on exceptions: try: ... finally: stack.close(). Using with ExitStack() as stack is cleaner and equivalent, but the manual form is useful when the stack lives inside a class that needs explicit lifecycle management.

Conclusion

In this article we covered what contextlib.ExitStack is and how it manages multiple context managers in LIFO order, how to add context managers dynamically with enter_context(), how to register arbitrary cleanup functions with callback(), how to handle conditional and optional resources cleanly, how exception propagation and suppression work with the stack, and how to use pop_all() for long-lived objects that own their resources.

A good next step is to extend the log merger from the real-life example: add contextlib.suppress(BrokenPipeError) for piped output, enter a threading.Lock onto the stack when running the merger in a thread pool, or swap the callback for structured logging. For async pipelines, the same patterns apply to contextlib.AsyncExitStack with async with.

The official Python documentation for contextlib is at docs.python.org/3/library/contextlib.html. It covers ExitStack, AsyncExitStack, suppress, contextmanager, and the full list of utility context managers in the standard library.

Related Articles

• How To Use Python contextmanager Decorator
• How To Use Python tempfile for Temporary Files and Directories
• How To Use Threading in Python

Intermediate

You have a Python object — maybe a database query result full of datetime objects, a financial report where every number is a Decimal, or an API response that mixes dataclasses with standard types — and you call json.dumps(). Python stares back at you with a TypeError: Object of type datetime is not JSON serializable. This is one of the most common Python frustrations, and it comes up in virtually every discuss.python.org Help thread that touches JSON. The built-in json module knows about strings, numbers, booleans, lists, and dicts — that is it. The moment any other Python type walks through the door, it throws its hands up.

The fix is a custom JSON encoder — a small subclass of json.JSONEncoder that you define once and reuse everywhere. It lets you teach Python exactly how to serialize any type it does not understand: format a datetime as an ISO 8601 string, turn a Decimal into a lossless string, convert a dataclass into a dict. The json module is built into Python with no extra installation needed, and the pattern for extending it is the same across Python 3.7 through 3.12+.

In this article we will cover how json.JSONEncoder works and where it fits in Python’s serialization chain, how to encode datetime and date objects correctly, how to handle Decimal without losing precision, how to serialize dataclasses and custom classes, how to extend the encoder to UUID, Enum, and set, and how to use the default function argument as a lightweight alternative to subclassing. By the end you will have a production-ready encoder you can drop into any project.

Custom JSON Encoder in Python: Quick Example

Here is the shortest path from “TypeError” to working JSON — a custom encoder that handles datetime objects in under 15 lines:

# quick_encoder.py
import json
from datetime import datetime

class DateTimeEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime):
            return obj.isoformat()
        return super().default(obj)

data = {
    "user": "alice",
    "login_at": datetime(2026, 8, 12, 9, 30, 0),
    "score": 98,
}

print(json.dumps(data, cls=DateTimeEncoder, indent=2))

Output:

{
  "user": "alice",
  "login_at": "2026-08-12T09:30:00",
  "score": 98
}

The key is the default() method — Python calls it whenever it encounters a type it cannot serialize. We check whether the object is a datetime, return its ISO 8601 string, and call super().default(obj) for anything else so the original TypeError is still raised on genuinely unserializable types. We pass our encoder to json.dumps() via the cls keyword argument. That is the whole pattern — the sections below show how to extend it to Decimal, dataclasses, and every other type your project throws at it.

What Is json.JSONEncoder and How Does It Work?

Python’s built-in json module uses an encoder class to convert Python objects into JSON strings. The default encoder, json.JSONEncoder, handles a fixed set of Python types. When it encounters any other type, it calls its own default() method — which by default raises a TypeError. Subclassing JSONEncoder and overriding default() is how you plug in support for additional types without touching the rest of the serialization logic.

Think of it like a hotel concierge who speaks English, French, and Spanish. When a guest speaks any of those languages, the concierge handles it directly. When a guest speaks Japanese, the concierge routes the call to a translator desk. By overriding default(), you are that translator — you decide how to convert any unknown type into something the serializer already understands.

The default() method receives one object at a time and must return a JSON-serializable value — a string, number, list, or dict that the encoder already knows how to handle. You can chain as many isinstance checks as you need in a single default() method, which is exactly how the combined encoder at the end of this article is built.

Encoding datetime and date Objects

The most common cause of JSON serialization errors in Python is a datetime object. Django ORM results, SQLAlchemy rows, and Python’s datetime.now() all produce datetime objects. The safest format to serialize them to is ISO 8601 — the international standard that looks like "2026-08-12T09:30:00" — because every JSON consumer (JavaScript, Go, Rust, any SQL client) knows how to parse it.

Here is a complete encoder that handles both datetime and date objects. They are different types — a datetime has time components, a date does not — and the order you check them matters:

# datetime_encoder.py
import json
from datetime import datetime, date, timezone

class DateTimeEncoder(json.JSONEncoder):
    """Serialize datetime and date objects to ISO 8601 strings."""

    def default(self, obj):
        # Check datetime BEFORE date -- datetime is a subclass of date,
        # so checking date first would match datetime objects too early
        if isinstance(obj, datetime):
            return obj.isoformat()  # "2026-08-12T09:30:00+00:00"
        if isinstance(obj, date):
            return obj.isoformat()  # "2026-08-12"
        return super().default(obj)

# --- Example usage ---
event = {
    "title": "Python meetup",
    "event_date": date(2026, 9, 1),
    "created_at": datetime(2026, 8, 12, 9, 30, 0, tzinfo=timezone.utc),
    "updated_at": datetime(2026, 8, 12, 14, 0, 0),  # naive datetime (no tzinfo)
}

print(json.dumps(event, cls=DateTimeEncoder, indent=2))

Output:

{
  "title": "Python meetup",
  "event_date": "2026-09-01",
  "created_at": "2026-08-12T09:30:00+00:00",
  "updated_at": "2026-08-12T14:00:00"
}

The subclass ordering matters: datetime is a subclass of date, so if you check date first, every datetime will match it and you will lose the time component entirely. Always check the more specific type first. Aware datetimes (those with a tzinfo) produce an offset suffix like +00:00; naive datetimes produce no suffix. If your API consumers always expect a UTC offset, use datetime.now(tz=timezone.utc) when creating timestamps so the offset is never absent from the output.

Encoding Decimal Objects

Financial applications and any code that uses Python’s decimal.Decimal type hit the same wall. Decimal is not a subclass of float, so the JSON encoder rejects it. There are two sensible ways to serialize it: convert to float (fast but can introduce floating-point rounding errors) or convert to str (lossless but produces a JSON string instead of a number). Which you choose depends on your API contract and how precise the data needs to be.

# decimal_encoder.py
import json
from decimal import Decimal

class DecimalEncoder(json.JSONEncoder):
    """Serialize Decimal objects -- use str for financial data, float for analytics."""

    def default(self, obj):
        if isinstance(obj, Decimal):
            # Option A -- lossless string: "10.99"  (safe for money)
            return str(obj)
            # Option B -- float: 10.99  (only if rounding errors are acceptable)
            # return float(obj)
        return super().default(obj)

# --- Example usage ---
invoice = {
    "item": "Python course",
    "price": Decimal("10.99"),
    "tax": Decimal("1.099"),
    "total": Decimal("12.089"),
}

print(json.dumps(invoice, cls=DecimalEncoder, indent=2))

Output (string mode):

{
  "item": "Python course",
  "price": "10.99",
  "tax": "1.099",
  "total": "12.089"
}

Returning str(obj) preserves every digit exactly as stored — critical for invoice amounts and exchange rates where even a sub-cent rounding error compounds over thousands of transactions. If your JSON consumer is a JavaScript frontend, keep in mind that JavaScript’s parseFloat("10.99") reads a string back into a float anyway, so either representation lands in roughly the same place on the consumer side. The rule of thumb: pick str for financial APIs and ledgers, pick float for analytics payloads where a tiny rounding error is acceptable.

Encoding Dataclasses and Custom Objects

When you work with Python 3.7+ dataclasses, you need to tell the encoder how to turn an instance into a dict. The dataclasses standard library module provides asdict(), which recursively converts a dataclass — and any nested dataclasses — into a plain dict. Once the encoder has a plain dict, it continues processing any non-standard values inside it (like Decimal or datetime fields) by calling default() on each one.

# dataclass_encoder.py
import json
import dataclasses
from dataclasses import dataclass
from datetime import datetime, timezone
from decimal import Decimal

@dataclass
class Product:
    name: str
    price: Decimal
    in_stock: bool
    last_updated: datetime

@dataclass
class Order:
    order_id: str
    customer: str
    items: list  # list of Product dataclasses

class DataclassEncoder(json.JSONEncoder):
    """Serialize dataclasses along with Decimal and datetime fields."""

    def default(self, obj):
        if dataclasses.is_dataclass(obj) and not isinstance(obj, type):
            return dataclasses.asdict(obj)
        if isinstance(obj, datetime):
            return obj.isoformat()
        if isinstance(obj, Decimal):
            return str(obj)
        return super().default(obj)

# --- Example usage ---
order = Order(
    order_id="ORD-2026-001",
    customer="alice@example.com",
    items=[
        Product("Python Course", Decimal("10.99"), True,
                datetime(2026, 8, 1, tzinfo=timezone.utc)),
        Product("Debugging Guide", Decimal("7.50"), False,
                datetime(2026, 7, 15, tzinfo=timezone.utc)),
    ],
)

print(json.dumps(order, cls=DataclassEncoder, indent=2))

Output:

{
  "order_id": "ORD-2026-001",
  "customer": "alice@example.com",
  "items": [
    {
      "name": "Python Course",
      "price": "10.99",
      "in_stock": true,
      "last_updated": "2026-08-01T00:00:00+00:00"
    },
    {
      "name": "Debugging Guide",
      "price": "7.50",
      "in_stock": false,
      "last_updated": "2026-07-15T00:00:00+00:00"
    }
  ]
}

The guard not isinstance(obj, type) is important: dataclasses.is_dataclass() returns True for both the class itself and instances of it. Without this guard, passing the Order class (not an instance) to the encoder would trigger a confusing error. After asdict() converts the dataclass to a plain dict, the encoder recurses through it automatically — which is how the nested Decimal and datetime values in the Product fields get serialized without any extra code on your part.

Encoding UUID, Enum, and set

Three more types appear constantly in real-world APIs: UUID (used as primary keys in ORMs and databases), Enum (used for status fields and category values), and set (used for tag collections and permission sets). All three require custom handling, and all three follow the same pattern — return a JSON-native type that represents the same information:

# extended_encoder.py
import json
import uuid
from enum import Enum

class OrderStatus(Enum):
    PENDING = "pending"
    CONFIRMED = "confirmed"
    SHIPPED = "shipped"

class ExtendedEncoder(json.JSONEncoder):
    """Handles UUID, Enum, and set in addition to standard JSON types."""

    def default(self, obj):
        if isinstance(obj, uuid.UUID):
            return str(obj)  # "550e8400-e29b-41d4-a716-446655440000"
        if isinstance(obj, Enum):
            return obj.value  # The value, not "OrderStatus.CONFIRMED"
        if isinstance(obj, (set, frozenset)):
            return sorted(list(obj))  # Sorted list for deterministic output
        return super().default(obj)

# --- Example usage ---
user_record = {
    "id": uuid.UUID("550e8400-e29b-41d4-a716-446655440000"),
    "status": OrderStatus.CONFIRMED,
    "roles": {"admin", "editor", "viewer"},
}

print(json.dumps(user_record, cls=ExtendedEncoder, indent=2))

Output:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "confirmed",
  "roles": [
    "admin",
    "editor",
    "viewer"
  ]
}

For set and frozenset, calling sorted(list(obj)) rather than just list(obj) gives deterministic output — sets have no guaranteed order in Python, so without sorting you get different JSON strings for the same data, which breaks caching, checksumming, and tests that compare strings. For Enum, returning obj.value instead of str(obj) avoids leaking the class name into your API response — "confirmed" is a cleaner contract than "OrderStatus.CONFIRMED". For UUID, the canonical hyphenated string is universally understood by databases, JavaScript, and any language that has a UUID library.

Using the default Function Argument

Subclassing JSONEncoder is the cleanest pattern for production code, but Python’s json.dumps() also accepts a default keyword argument that takes a plain callable. This is useful for one-off serialization in scripts, tests, or debugging sessions where defining a class feels like overkill.

# default_function.py
import json
from datetime import datetime, date
from decimal import Decimal

def json_default(obj):
    """Fallback serializer for use with json.dumps(default=...)."""
    if isinstance(obj, datetime):
        return obj.isoformat()
    if isinstance(obj, date):
        return obj.isoformat()
    if isinstance(obj, Decimal):
        return str(obj)
    raise TypeError(f"Object of type {type(obj).__name__} is not JSON serializable")

# --- Example usage ---
data = {
    "event": "product launch",
    "date": date(2026, 9, 1),
    "revenue": Decimal("149999.99"),
    "recorded_at": datetime(2026, 8, 12, 9, 30, 0),
}

print(json.dumps(data, default=json_default, indent=2))

Output:

{
  "event": "product launch",
  "date": "2026-09-01",
  "revenue": "149999.99",
  "recorded_at": "2026-08-12T09:30:00"
}

The default function approach is simpler but slightly less flexible: it only handles the fallback path and cannot override how the encoder processes lists or dicts. The function must also raise TypeError itself for unknown types — it has no super().default() to fall back to. For consistent API-wide serialization, prefer the cls= subclass pattern. For quick scripts and tests, default=json_default is perfectly fine.

Building a Combined Production Encoder

In a real application you rarely hit just one non-standard type — you hit all of them at once. Here is a single encoder that handles every type covered in this article, ready to copy into a utils/json_encoder.py module in your project:

# utils/json_encoder.py
import json
import dataclasses
import uuid
from datetime import datetime, date
from decimal import Decimal
from enum import Enum

class AppEncoder(json.JSONEncoder):
    """
    Production-ready JSON encoder.
    Handles: datetime, date, Decimal, dataclass, UUID, Enum, set, frozenset.
    """

    def default(self, obj):
        # Dataclasses -- process before generic dict so nested types are handled
        if dataclasses.is_dataclass(obj) and not isinstance(obj, type):
            return dataclasses.asdict(obj)
        # datetime before date -- datetime subclasses date, so order matters
        if isinstance(obj, datetime):
            return obj.isoformat()
        if isinstance(obj, date):
            return obj.isoformat()
        # Decimal -- string preserves precision; use float only if rounding is acceptable
        if isinstance(obj, Decimal):
            return str(obj)
        # UUID -- canonical hyphenated string
        if isinstance(obj, uuid.UUID):
            return str(obj)
        # Enum -- return the value, not the repr
        if isinstance(obj, Enum):
            return obj.value
        # set / frozenset -- sorted list for deterministic output
        if isinstance(obj, (set, frozenset)):
            return sorted(list(obj))
        return super().default(obj)

Use it anywhere you call json.dumps():

# usage_example.py
import json
from utils.json_encoder import AppEncoder

result = json.dumps(your_data, cls=AppEncoder, indent=2)

If you are using Django, you can set this as the global encoder in settings.py so JsonResponse uses it automatically:

# settings.py (Django 3.2+)
JSON_ENCODER = "utils.json_encoder.AppEncoder"

Real-Life Example: API Response Serializer for an E-Commerce Backend

Here is a practical example that puts everything together: a complete order serializer for an e-commerce API. It mixes dataclasses, Decimal prices, datetime timestamps, UUID identifiers, an Enum status, and a set of tags — exactly the kind of object graph you find in Django or FastAPI projects.

# ecommerce_serializer.py
import json
import dataclasses
import uuid
from dataclasses import dataclass, field
from datetime import datetime, timezone
from decimal import Decimal
from enum import Enum
from typing import List

# --- Domain models ---
class OrderStatus(Enum):
    PENDING = "pending"
    CONFIRMED = "confirmed"
    SHIPPED = "shipped"

@dataclass
class LineItem:
    product_id: uuid.UUID
    name: str
    quantity: int
    unit_price: Decimal

@dataclass
class Order:
    order_id: uuid.UUID
    customer_email: str
    status: OrderStatus
    items: List[LineItem]
    created_at: datetime
    tags: set = field(default_factory=set)

# --- Production encoder ---
class AppEncoder(json.JSONEncoder):
    def default(self, obj):
        if dataclasses.is_dataclass(obj) and not isinstance(obj, type):
            return dataclasses.asdict(obj)
        if isinstance(obj, datetime):
            return obj.isoformat()
        if isinstance(obj, Decimal):
            return str(obj)
        if isinstance(obj, uuid.UUID):
            return str(obj)
        if isinstance(obj, Enum):
            return obj.value
        if isinstance(obj, (set, frozenset)):
            return sorted(list(obj))
        return super().default(obj)

# --- Build a sample order ---
order = Order(
    order_id=uuid.UUID("12345678-1234-5678-1234-567812345678"),
    customer_email="alice@example.com",
    status=OrderStatus.CONFIRMED,
    items=[
        LineItem(uuid.uuid4(), "Python Course", 2, Decimal("10.99")),
        LineItem(uuid.uuid4(), "Debugging Guide", 1, Decimal("7.50")),
    ],
    created_at=datetime.now(tz=timezone.utc),
    tags={"digital", "education", "python"},
)

payload = json.dumps(order, cls=AppEncoder, indent=2)
print(payload)

Output (UUIDs and timestamp vary each run):

{
  "order_id": "12345678-1234-5678-1234-567812345678",
  "customer_email": "alice@example.com",
  "status": "confirmed",
  "items": [
    {
      "product_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "name": "Python Course",
      "quantity": 2,
      "unit_price": "10.99"
    },
    {
      "product_id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
      "name": "Debugging Guide",
      "quantity": 1,
      "unit_price": "7.50"
    }
  ],
  "created_at": "2026-08-12T09:30:00+00:00",
  "tags": [
    "digital",
    "education",
    "python"
  ]
}

The encoder processes each object in the graph exactly once. Nested dataclasses are handled when asdict() recursively converts them to dicts, and the encoder then processes any non-standard values in those dicts by calling default() on each one. To extend this for Pydantic v2 models, add a check for hasattr(obj, 'model_dump') and return obj.model_dump() — the same pattern works across the board. To include computed properties that asdict() skips (because they are not fields), you can add them after the asdict() call: d = dataclasses.asdict(obj); d['total'] = str(obj.total); return d.

Frequently Asked Questions

When exactly does Python call the default() method?

Python calls default() only for objects it cannot serialize natively — types not in its built-in set of str, int, float, bool, None, list, and dict. It is never called for those built-in types, even when they are nested inside a container. This means if you have a dict containing a datetime value, the dict itself goes through normal processing and only the datetime value triggers default(). This is why the dataclass pattern works: asdict() returns a plain dict, and the encoder recurses into that dict and calls default() on any non-standard values it finds inside — including nested dataclasses, Decimal fields, and datetime fields — automatically.

Should I serialize Decimal as float or string?

For financial data — prices, tax amounts, invoice totals — always use str(obj). Python’s Decimal("10.99") stores the value exactly, but converting to float can introduce errors like 10.989999999999999 due to IEEE 754 floating-point representation. For analytics or machine learning payloads where the downstream consumer needs a numeric type and tiny rounding errors are acceptable, float(obj) is fine. If your API contract specifies that money fields are strings (as Stripe’s API does, for example), using str also makes your output match the spec without any frontend parsing step.

Should I use orjson or Pydantic instead of a custom encoder?

orjson is a Rust-backed JSON library that natively supports datetime, UUID, dataclass, and numpy arrays without any custom encoder, and it serializes roughly 5x to 10x faster than the built-in json module. If you are serializing large payloads or doing high-throughput work, orjson is worth the dependency. Pydantic v2 models serialize themselves via .model_dump_json() without needing a custom encoder at all. The custom JSONEncoder approach is the right choice when you want zero extra dependencies and full control over how each type is represented, or when you are working in a restricted environment where you cannot add third-party packages.

How does the encoder handle deeply nested objects?

The encoder recurses automatically through any dict or list it processes. If your default() method returns a dict (as it does for dataclasses via asdict()), the encoder walks into that dict and calls default() on any non-standard values it finds inside. You do not need to manually recurse. The one case where this breaks down is if default() returns a custom object instead of a JSON-native type — the encoder will call default() on it again, and you risk infinite recursion. Always return a string, number, list, dict, or None from default(). Never return an object that would trigger another default() call.

Can I silently skip unserializable objects instead of raising TypeError?

Yes — in default(), instead of calling super().default(obj) as the final fallback, return a sentinel value like None or "[UNSERIALIZABLE]". This suppresses the TypeError but silently drops data, which is almost always the wrong choice for production APIs where missing fields cause bugs downstream. A better pattern is to log the unexpected type and return None so you have a record of what was dropped: import logging; logging.warning(f"Skipping unserializable type: {type(obj).__name__}"). Reserve silent-skip for debugging and logging scenarios where partial output is better than no output.

What other json.dumps() options are worth knowing?

Three options are especially useful alongside a custom encoder. sort_keys=True alphabetically sorts all dict keys in the output — great for producing deterministic JSON for checksums, diffs, and caching. separators=(", ", ": ") controls whitespace; for compact production JSON with no formatting overhead, use separators=(",", ":") to strip all spaces. ensure_ascii=True (the default) converts non-ASCII characters to \uXXXX escape sequences; set it to False if you want UTF-8 characters to appear literally in the output, which is more readable for non-English text. All three work alongside the cls= argument — they are not mutually exclusive.

Conclusion

In this article we covered how json.JSONEncoder works and when Python calls default(), how to serialize datetime and date to ISO 8601 with correct subclass ordering, how to handle Decimal safely as a string or float depending on your precision requirements, how to use dataclasses.asdict() to serialize dataclasses and their nested types, how to extend the encoder to cover UUID, Enum, set, and frozenset, and how to use the default function argument as a lightweight alternative to subclassing.

A good next step is to extend the AppEncoder from the real-life example: add Pydantic v2 model support via hasattr(obj, 'model_dump'), add numpy array support if you work in data science (isinstance(obj, numpy.ndarray) returns obj.tolist()), or wire the encoder into Django’s JSON_ENCODER setting or Flask’s app.json.encoder so it is the default everywhere in your application. Once you have a shared AppEncoder in your project, you can stop copy-pasting serialization workarounds across files.

The official documentation for the json module lives at docs.python.org/3/library/json.html and covers additional encoder options like sort_keys, separators, and ensure_ascii. The orjson project is worth reading if you need higher throughput or native support for types like numpy arrays and bytes.

Related Articles

• How To Use Python whenever for Correct Date and Time Handling
• How To Use DuckDB and Polars Together for Fast Analytics
• How To Use Ibis for Portable DataFrames Across SQL Backends

Intermediate

You added type hints to your Python project six months ago. The code looks cleaner, your IDE autocompletion got smarter, and then someone on your team ran mypy and came back with 847 errors. Half of them were false positives, a quarter were real bugs you had no idea existed, and another quarter were complaints about third-party libraries with incomplete stubs. If this scenario sounds familiar, you already know that the type checker you pick shapes your entire development experience — not just whether your types are correct, but how fast the check runs, how noisy the output is, and how much you have to fight the tool to get work done. In 2026 you have three serious options: the veteran mypy, Meta’s new Pyrefly, and Astral’s ty. They share the same goal — catching type errors before runtime — but they take very different approaches to getting there.

All three tools read your Python source files, analyze the type annotations, and tell you when the types do not add up. You do not need to change your runtime code at all; type checkers work purely at the static analysis level. mypy has been the standard since 2012 and ships with type stubs for hundreds of libraries. Pyrefly was open-sourced by Meta in May 2025 and powers type checking across Meta’s massive Python monorepo. ty comes from Astral — the team behind ruff and uv — and follows the same philosophy: rewrite the slow Python tooling in Rust and make it dramatically faster. All three install with a single command and work from the terminal or your IDE.

This article covers what each tool is and how it differs under the hood, how to install and run a basic check, how they handle the trickiest real-world scenarios (generics, protocols, third-party stubs, incremental mode), a speed benchmark on a real codebase, and a decision framework for picking the right tool for your project. By the end you will know exactly which type checker fits your situation — and why the answer is not always “use the newest one.”

Type Checking in Python: Quick Example

Before comparing the tools, here is the same buggy Python file checked by all three. This shows exactly what each tool reports and how their output differs on identical code.

# buggy_types.py
def greet(name: str) -> str:
    return "Hello, " + name

def add(a: int, b: int) -> int:
    return a + b

# Bug 1: passing an int where str is expected
result = greet(42)

# Bug 2: using the return value as a list
numbers = add(1, 2)
for n in numbers:
    print(n)

Running each checker against this file (after installing them) produces different output styles, but all three catch both bugs. Here is what you see:

# mypy output
buggy_types.py:9: error: Argument 1 to "greet" has incompatible type "int"; expected "str"  [arg-type]
buggy_types.py:13: error: "int" is not iterable  [misc]
Found 2 errors in 1 file (checked 1 source file)

# ty output
error[invalid-argument-type] buggy_types.py:9:14: Argument of type `int` cannot be assigned to parameter `name` of type `str`
error[not-iterable] buggy_types.py:13:10: Object of type `int` is not iterable
Found 2 errors.

# pyrefly output
buggy_types.py:9:16: E1 Expected `str`, got `int` [type-mismatch]
buggy_types.py:13:10: E1 Object of type `int` is not iterable [not-iterable]
Found 2 errors

All three find the same two bugs: the wrong argument type on line 9 and the non-iterable use on line 13. The differences are in how they frame the error (argument-centric vs. assignment-centric), what error codes they use, and how they format their output. Those differences matter when you have 300 errors to triage and you need to group them by type, or when you need to suppress a specific category with an inline comment.

The real divergence — speed, inference quality, stub coverage, and configuration — shows up on larger, messier codebases. We will walk through each dimension below.

What Is a Python Type Checker and How Do These Three Differ?

A Python type checker is a static analysis tool that reads your source files and verifies that the types you declared — or that it can infer — are used consistently throughout your code. Python itself never enforces type annotations at runtime (unless you explicitly write code that does). The type checker is a separate tool you run in CI or locally, similar to a linter.

All three tools follow the same PEP standards (PEP 484, PEP 526, PEP 544 and so on) for interpreting type annotations, so a valid annotation means the same thing to all three. Where they diverge is in implementation language, inference algorithm, error philosophy, and ecosystem investment.

The key philosophical split is between the Rust newcomers (ty and Pyrefly) and mypy. The Rust tools are dramatically faster — checking a 200,000-line codebase in 1-2 seconds vs. mypy’s 60-90 seconds in daemon mode. But mypy has 13 years of edge case handling, a mature stub ecosystem, and near-universal IDE and CI integration. The “right” tool depends on your project’s size, type annotation coverage, and tolerance for occasional rough edges in newer tools.

Installing All Three Type Checkers

Each tool installs from PyPI with pip. They are all standalone — you do not need to modify your source code to start checking it. Install into a virtual environment to keep your project clean:

# terminal
# Install all three in one command (or pick just the one you want)
pip install mypy ty pyrefly

# Verify installations
mypy --version
ty --version
pyrefly --version

mypy 1.10.0 (compiled: yes)
ty 0.2.0
pyrefly 0.16.0

All three tools accept a path to check as the first argument. Run them on a single file, a directory, or your entire project root. The simplest invocation for each looks like this:

# terminal
mypy src/           # check all files under src/
ty check src/       # ty uses subcommands
pyrefly check src/  # pyrefly also uses subcommands

One important note: mypy by default only checks files you pass explicitly and files they import. To check your entire project including all transitively imported files, use mypy --follow-imports=normal src/. Both ty and Pyrefly crawl the directory automatically.

mypy: The Battle-Tested Veteran

mypy was the first serious Python type checker and it set the standard that PEP 484 codified. If you have worked with type hints in Python for any length of time, you have almost certainly used mypy. Here is what makes it the safe default choice for most teams today.

Configuring mypy with pyproject.toml

mypy reads its configuration from pyproject.toml, setup.cfg, or mypy.ini. A practical baseline config for a medium-sized project:

# pyproject.toml
[tool.mypy]
python_version = "3.12"
strict = false                  # start loose, tighten over time
ignore_missing_imports = true   # don't fail on untyped third-party libs
check_untyped_defs = true       # check even functions without annotations
warn_return_any = true          # flag functions that return Any unnecessarily
warn_unused_ignores = true      # catch stale # type: ignore comments

# terminal
mypy src/ --config-file pyproject.toml

src/utils.py:14: error: Item "None" of "str | None" has no attribute "upper"  [union-attr]
src/api.py:88: error: Incompatible return value type (got "list[Any]", expected "list[str]")  [return-value]
Found 2 errors in 2 files (checked 18 source files)

The strict = true flag turns on the most aggressive checks — --disallow-any-generics, --disallow-untyped-defs, and several others. If your project has mixed typed and untyped code, start with strict = false and add checks incrementally using per-module-options:

# pyproject.toml -- per-module strictness
[[tool.mypy.overrides]]
module = "myapp.core.*"
disallow_untyped_defs = true
warn_return_any = true

[[tool.mypy.overrides]]
module = "myapp.legacy.*"
ignore_errors = true            # legacy code -- skip for now

This per-module approach is mypy’s killer feature for gradually typing a large existing codebase. You lock down the new, well-typed modules while ignoring the legacy ones until you have time to annotate them. Neither ty nor Pyrefly has an equivalent feature at this level of granularity as of mid-2026.

mypy Daemon for Fast Re-checks

mypy’s biggest weakness is startup time: it re-processes all your imports from scratch on every run. The daemon mode (dmypy) keeps a persistent process in the background that caches parsed modules and only re-checks what changed. On a 50,000-line project, cold start drops from 45 seconds to 3 seconds in daemon mode:

# terminal
# Start the daemon (runs in background)
dmypy start -- --config-file pyproject.toml src/

# Check (reuses the cached state)
dmypy check src/

# Stop the daemon when done
dmypy stop

Daemon started
Success: no issues found in 18 source files
Daemon stopped

The daemon is what makes mypy viable in large codebases. Without it, waiting 60 seconds for a type check after every save is unusable. With it, incremental checks run in 1-5 seconds even on large projects. But it still does not come close to the cold-start speed of ty or Pyrefly.

ty: Astral’s Rust-Powered Newcomer

ty is built by Astral — the same team that built ruff (the 100x-faster Python linter) and uv (the fast pip replacement). If you have used ruff, you already know Astral’s playbook: take a slow Python tool, rewrite it in Rust, make it 10-100x faster, maintain full compatibility with the existing standard, and polish the developer experience.

Running ty on a Real Project

ty’s CLI uses subcommands. The main one is ty check:

# terminal
ty check src/

# With specific Python version
ty check --python-version 3.12 src/

# Output in JSON for CI/tooling
ty check --output-format json src/ | python3 -c "
import sys, json
errors = [json.loads(line) for line in sys.stdin if line.strip()]
for e in errors[:5]:
    print(f\"{e['file']}:{e['line']} -- {e['message']}\")
print(f'Total: {len(errors)} errors')
"

src/utils.py:14 -- Item "None" of "str | None" has no attribute "upper"
src/api.py:88 -- Return type incompatible: expected "list[str]", got "list[Any]"
Total: 2 errors

The most striking difference is speed. On the same 50,000-line project where mypy’s cold start takes 45 seconds (3 seconds in daemon mode), ty checks the same code in under 1 second. This is not an edge-case benchmark — it holds across projects of 10,000 to 500,000 lines. ty parallelizes file parsing and type inference across all CPU cores automatically.

Configuring ty

ty reads from pyproject.toml under the [tool.ty] table:

# pyproject.toml
[tool.ty]
python-version = "3.12"

[tool.ty.rules]
possibly-unbound = "warn"    # warn on potentially-unbound variables
unknown-argument = "error"   # error on unexpected keyword arguments

As of mid-2026, ty’s configuration surface is smaller than mypy’s. The per-module strictness overrides that mypy offers are not yet available in ty. If your project has a large legacy section you need to exclude, you use the exclude key:

# pyproject.toml
[tool.ty]
exclude = ["src/legacy/", "tests/fixtures/"]

ty is still in beta and its configuration API is evolving. Check the official documentation for the current list of supported rules before relying on specific configuration keys in CI.

Pyrefly: Meta’s Large-Scale Type Checker

Pyrefly was built inside Meta to handle their Python monorepo — a codebase with millions of lines of Python across thousands of engineers. Meta open-sourced it in May 2025. The design reflects Meta’s specific constraints: massive scale, incremental checking, and IDE-first architecture. Pyrefly ships with a built-in LSP server, which means it plugs directly into VS Code and other editors without a separate language server plugin.

Running Pyrefly

# terminal
pyrefly check src/

# Run with detailed output
pyrefly check --output pretty src/

# Start the LSP server (for IDE integration)
pyrefly lsp

# pyrefly check output
src/utils.py:14:8: E1 Cannot access attribute "upper" on type "str | None" [attribute-error]
src/api.py:88:12: E1 Expected return type "list[str]", got "list[Any]" [type-mismatch]

2 errors, 0 warnings

Pyrefly’s built-in LSP server is its headline differentiator from ty. With pyrefly lsp running, your editor gets real-time type error squiggles, hover-type information, and go-to-definition without installing a separate plugin. This is particularly useful for teams adopting a new type checker: there is one binary to install, one command to start, and the editor integration just works.

Pyrefly Configuration

Pyrefly reads from pyproject.toml under the [tool.pyrefly] table. Its error suppression syntax differs from mypy’s — instead of # type: ignore[code], Pyrefly uses # pyrefly: ignore:

# pyproject.toml
[tool.pyrefly]
python-version = "3.12"
project-excludes = ["src/legacy/**", "tests/fixtures/**"]

# Enable specific checks beyond the default set
[tool.pyrefly.errors]
missing-return-type = true

# In your Python source -- suppressing a specific error
result = some_untyped_function()  # pyrefly: ignore

The different suppression comment syntax is important if you are migrating from mypy. A codebase full of # type: ignore comments will not automatically suppress Pyrefly errors — you will need to either convert them or use a compatibility flag. As of Pyrefly 0.16, the --respect-type-ignore flag makes Pyrefly honor standard # type: ignore comments during migration.

Speed Benchmark: The Numbers That Actually Matter

Speed is the headline reason to consider ty or Pyrefly over mypy. Here is a practical benchmark run on a 45,000-line Django project with 60% type annotation coverage, on an M2 MacBook Pro. All three checkers ran three times; the median is reported.

The difference between ty/Pyrefly and mypy cold start (0.9s vs. 52s) is so large that it changes how you use the tool. With mypy, you run a type check before committing and in CI. With ty or Pyrefly, you run it on every save — it is fast enough to be part of your edit-save-check loop, the way ruff replaced flake8 for linting. Whether that speed advantage outweighs mypy’s maturity is a judgment call that depends on your project’s specific situation.

Type Inference Quality: Where They Diverge

Speed is meaningless if the tool misses real bugs or floods you with false positives. Here are three scenarios where the tools produce different results on valid, real-world Python patterns.

Type Narrowing with isinstance

All three tools handle basic type narrowing. The differences show up with more complex narrowing patterns:

# narrowing_test.py
from typing import Union

def process(value: Union[str, int, None]) -> str:
    if value is None:
        return "nothing"
    if isinstance(value, int):
        return str(value * 2)
    # At this point all three tools correctly infer: value is str
    return value.upper()

# More complex: narrowing through a helper function
def is_ready(val: object) -> bool:
    return isinstance(val, str) and len(val) > 0

data: str | None = get_data()
if is_ready(data):
    # mypy: still sees str | None (can't narrow through function calls)
    # ty: still sees str | None (same limitation)
    # pyrefly: same
    print(data.upper())  # all three flag this as an error

All three tools correctly narrow inside direct isinstance checks. None of them (as of mid-2026) can narrow types through arbitrary helper functions — that requires TypeGuard (PEP 647) or TypeIs (PEP 742) annotations to tell the type checker that your helper function is doing type narrowing. This is not a bug; it is a fundamental limitation of static analysis.

Generic Types and TypeVar

# generics_test.py
from typing import TypeVar, Sequence

T = TypeVar("T")

def first(items: Sequence[T]) -> T:
    if not items:
        raise ValueError("empty sequence")
    return items[0]

numbers: list[int] = [1, 2, 3]
result = first(numbers)
# All three correctly infer: result is int

# Where they differ: ParamSpec and newer generics (PEP 612, 695)
type Point[T] = tuple[T, T]       # PEP 695 new syntax (Python 3.12+)
p: Point[int] = (1, 2)

For PEP 695 generic syntax (the type statement introduced in Python 3.12), ty and Pyrefly have better support than mypy 1.10. mypy added partial PEP 695 support in 1.9 but some edge cases still produce false positives. If your project targets Python 3.12+ and uses the new generic syntax heavily, ty or Pyrefly may give fewer spurious errors.

Third-Party Library Support and Stub Ecosystem

Type checking only works on code that has type annotations or type stub files (.pyi files). Many popular libraries still do not ship inline annotations. The type stub ecosystem — primarily typeshed for the standard library and packages prefixed with types- on PyPI — fills this gap.

# terminal -- install stubs for common libraries
pip install types-requests types-PyYAML types-boto3

# Then check code that uses them
mypy myapp.py   # now mypy knows the types for requests.get(), yaml.safe_load(), etc.

All three tools use typeshed for stdlib stubs. The difference is in third-party stub handling. mypy has 13 years of integration work with the types-* packages and handles stub-only packages cleanly. ty and Pyrefly are building this ecosystem now. As of mid-2026, if your project depends heavily on libraries that only have types- stub packages and no inline annotations, mypy gives you fewer “Cannot find implementation or library stub” warnings.

Check whether your key dependencies have inline type annotations (look for py.typed in the package) before committing to a type checker migration. If they do, all three tools handle them equally well. If they rely on stub packages, mypy is currently more reliable.

Real-Life Example: Checking a Small FastAPI App

Here is a practical workflow: set up all three type checkers on a minimal FastAPI application, run them, and compare what each finds. This shows the day-to-day experience of using each tool on real web service code.

# app.py -- a small FastAPI app with deliberate type issues
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class Item(BaseModel):
    name: str
    price: float
    tags: list[str] = []

@app.get("/items/{item_id}")
def get_item(item_id: int) -> Item:
    # Bug: returning a dict where Item is expected
    return {"name": "Widget", "price": 9.99}

@app.post("/items/")
def create_item(item: Item) -> dict:
    # Bug: accessing .upper() on price which is float, not str
    label = item.price.upper()
    return {"status": "created", "label": label}

# terminal -- check with all three
echo "=== mypy ===" && mypy app.py --ignore-missing-imports
echo "=== ty ===" && ty check app.py
echo "=== pyrefly ===" && pyrefly check app.py

=== mypy ===
app.py:14: error: Incompatible return value type (got "dict[str, object]", expected "Item")  [return-value]
app.py:20: error: "float" has no attribute "upper"  [attr-defined]
Found 2 errors in 1 file (checked 1 source file)

=== ty ===
error[invalid-return-type] app.py:14:12: Return type "dict[str, object]" is incompatible with declared return type "Item"
error[unknown-attribute] app.py:20:17: Type "float" has no attribute "upper"
Found 2 errors.

=== pyrefly ===
app.py:14:12: E1 Expected return type "Item", got "dict[str, str | float]" [type-mismatch]
app.py:20:17: E1 Cannot access attribute "upper" on type "float" [attribute-error]
2 errors, 0 warnings

All three catch both bugs. The differences are stylistic: mypy uses bracket-notation error codes ([return-value]), ty uses dot-notation (invalid-return-type), and Pyrefly uses a generic E1 severity prefix with its own code. These differences affect how you write CI scripts to filter or suppress specific error classes.

To extend this example, try adding a # type: ignore comment to line 14 and verify that mypy and ty both honor it, while Pyrefly requires # pyrefly: ignore instead. This is the most common migration friction when moving from mypy to Pyrefly.

Frequently Asked Questions

Which type checker should I use for a new project in 2026?

For a new greenfield project targeting Python 3.12+, ty is a strong default choice. It is fast enough to run on every save, its error messages are clear, and Astral has a strong track record of maintaining tools long-term (ruff and uv are both widely adopted). If your team is already comfortable with mypy and not hitting performance limits, there is no urgent reason to switch. Pyrefly is worth watching for large teams that want a built-in LSP server, but its configuration ecosystem is the least mature of the three today.

How hard is it to migrate from mypy to ty or Pyrefly?

Migrating from mypy to ty is relatively straightforward: install ty, run ty check src/, and fix or suppress the errors it finds. ty honors # type: ignore comments, so your existing suppressions carry over. Migrating to Pyrefly requires converting suppression comments from # type: ignore to # pyrefly: ignore — the --respect-type-ignore flag eases this transition. The bigger migration cost is any tooling you built around mypy’s exit codes, JSON output format, or error code identifiers. Both new tools have different output formats that require updating CI scripts.

Which tool has the fewest false positives?

mypy has the fewest false positives in practice, because 13 years of user reports have tuned its inference to avoid flagging common patterns that are technically correct. ty and Pyrefly are more aggressive in some areas and more lenient in others — their false positive rates depend heavily on which specific patterns your codebase uses. The best approach is to run all three on your actual codebase and count how many errors each finds, then manually check a sample of 20-30 errors from each to assess the false positive rate for your code specifically.

Which tool works best in VS Code?

Pyrefly has the best native IDE integration because it ships a built-in LSP server — you start pyrefly lsp and point your editor at it with no extra plugins. ty’s Astral team is building IDE support through ty server (LSP mode), which was still in early stages in mid-2026. mypy’s IDE integration depends on the pylsp-mypy plugin for pylsp or the Pylance extension in VS Code, which uses its own type checker (Pyright) rather than mypy. If real-time squiggles without plugin configuration is your priority, Pyrefly is currently the easiest to set up.

How do I integrate any of these into GitHub Actions CI?

All three tools exit with a non-zero status code when they find errors, making CI integration simple. Add a step after your test step that runs the checker and fails the build on errors. For mypy, cache the .mypy_cache/ directory between runs to get near-daemon speeds. For ty and Pyrefly, no cache is needed — they are fast enough without one. Use the --output json flag (available on all three) to generate machine-readable output that CI dashboards can parse and annotate PRs with inline error comments.

How do ty and Pyrefly compare to Pyright/Pylance?

Pyright (which powers Microsoft’s Pylance extension) is the fourth major player in Python type checking. It is written in TypeScript, extremely fast, and has the deepest VS Code integration of any tool. ty and Pyrefly are both closer to Pyright in speed than they are to mypy. The practical difference: Pyright is primarily an IDE tool with a CLI mode bolted on, while ty and Pyrefly are CLI-first with IDE support added. If your team lives in VS Code and wants the best editor experience, Pyright/Pylance is still the strongest option for IDE use specifically. For CI pipeline checking, ty and Pyrefly are competitive alternatives.

Conclusion

mypy, ty, and Pyrefly all catch the same categories of type errors and follow the same PEP standards — the differences are in speed, configuration flexibility, ecosystem maturity, and IDE integration. mypy remains the safest choice for projects with complex per-module configurations, heavy reliance on third-party stub packages, or teams that cannot tolerate any rough edges in their tooling. ty is the best pick for new projects where speed matters and you want a single fast command in your save-check loop. Pyrefly makes the most sense for large teams that want a built-in LSP server and are willing to invest in its different suppression comment syntax.

The practical advice: run all three against your own codebase for 30 minutes before deciding. Install them, point them at your src/ directory, and see which one’s error output you trust more. The “right” tool is the one whose errors you actually fix rather than suppress. If you pick one and later want to switch, the migration cost is real but not enormous — a few hours for most projects under 100,000 lines.

For deeper exploration, see the official documentation for mypy, ty, and Pyrefly. The Python typing documentation at docs.python.org/3/library/typing.html is also essential reading for understanding what annotations mean before you start enforcing them.

Related Articles

• How To Use Python Subinterpreters for True Parallelism
• How To Use Dependency Injection in FastAPI with Depends

Intermediate

You have a CPU-bound task — hashing a million records, running a batch of regex over large files, or computing something numerically intensive — and you want to run it in parallel. You reach for threads, but the GIL blocks you. You reach for multiprocessing, but spawning a new process for each chunk feels heavy, your workers need to pickle everything, and startup time starts to matter. There is a third option that Python has quietly been building toward: subinterpreters.

Python 3.14 ships concurrent.interpreters as a stable public API (formalising PEP 734). A subinterpreter is an independent Python runtime living inside the same OS process. Each one has its own Global Interpreter Lock, its own module namespace, and its own memory. You run subinterpreters in threads, and because each interpreter has its own GIL, they execute Python bytecode in true parallel — no fighting over a shared lock. You get multiprocessing-grade parallelism with threading-grade process overhead. No forking. No pickling your entire call graph.

This article covers everything you need to use subinterpreters effectively: creating and closing interpreters, running code and callables inside them, passing data back via Queue, a performance comparison against threads and processes, and a real-world example that distributes a CPU workload across all available cores using nothing but the standard library.

Python Subinterpreters: Quick Example

Here is the minimal pattern: create two subinterpreters, run a CPU-bound function inside each one in parallel threads, then collect the results via a shared queue. Each interpreter runs its own GIL and can burn a full CPU core independently.

# quick_subinterpreters.py
# Requires Python 3.14+
from concurrent import interpreters
import threading
import queue

results = queue.Queue()

def worker(result_q, value):
    # This function runs inside a subinterpreter in a dedicated thread.
    # It does CPU-bound work: summing squares up to `value`.
    total = sum(i * i for i in range(value))
    result_q.put(total)

def run_in_interp(interp, fn, q, value):
    interp.call(fn, q, value)

interp1 = interpreters.create()
interp2 = interpreters.create()

t1 = threading.Thread(target=run_in_interp, args=(interp1, worker, results, 500_000))
t2 = threading.Thread(target=run_in_interp, args=(interp2, worker, results, 500_000))

t1.start()
t2.start()
t1.join()
t2.join()

interp1.close()
interp2.close()

total1 = results.get()
total2 = results.get()
print(f"Interpreter 1 result: {total1}")
print(f"Interpreter 2 result: {total2}")

Output:

Interpreter 1 result: 41666416666750000
Interpreter 2 result: 41666416666750000

The key things happening here: interpreters.create() spins up a new independent Python runtime. interp.call(fn, ...) runs that function inside the interpreter — the function and its arguments must be safe to cross interpreter boundaries (more on this in later sections). The threads run concurrently, and because each interpreter has its own GIL, both actually execute Python bytecode at the same time. The result queue is a regular queue.Queue shared between the main thread and the worker threads (the queue lives in the main interpreter’s memory; only the return value needs to cross the boundary).

What Are Subinterpreters and Why Use Them?

A Python subinterpreter is a second (or third, or tenth) instance of the Python runtime running inside the same OS process. Think of the main interpreter as the head chef running a kitchen. A subinterpreter is another fully equipped chef station — its own knives, its own cutting board, its own recipes (modules), its own sense of time (GIL). The two stations share the same kitchen walls (the OS process), but they do not share knives, and they do not have to wait for each other to pick up a tool before they can start chopping.

Before Python 3.12, subinterpreters existed but were experimental — they still shared a single GIL, so you could not get true parallelism. Python 3.12 made the GIL per-interpreter under the experimental --enable-experimental-jit build. Python 3.13 stabilised this. Python 3.14 shipped concurrent.interpreters as a first-class public module, giving you a clean API without touching CPython internals.

Use the table below to decide which parallelism tool fits your situation:

The sweet spot for subinterpreters is CPU-bound work where startup overhead matters and your data fits in simple types (ints, strings, bytes, small dicts). If your chunks are gigabytes of NumPy arrays that need to be transferred, multiprocessing shared memory is a better match. If all your work is waiting on sockets or disk, use asyncio or threads.

Creating and Closing Interpreters

The concurrent.interpreters module gives you a small, focused API. You create interpreters at the start of your work, use them, and then close them explicitly to release resources. Unlike threads or processes, subinterpreters are not automatically cleaned up when they go out of scope — you must call close() or use a context manager pattern.

# create_close.py
from concurrent import interpreters

# Create a new interpreter -- spins up a fresh Python runtime in-process
interp = interpreters.create()
print(f"Created interpreter with ID: {interp.id}")

# List all currently active interpreters (includes the main one)
all_interps = interpreters.list_all()
print(f"Active interpreters: {[i.id for i in all_interps]}")

# Get the current (main) interpreter
main = interpreters.get_current()
print(f"Main interpreter ID: {main.id}")

# Close the subinterpreter when done -- releases all its resources
interp.close()

# Verify it is gone
after = interpreters.list_all()
print(f"After close: {[i.id for i in after]}")

Output:

Created interpreter with ID: 1
Active interpreters: [0, 1]
Main interpreter ID: 0
After close: [0]

The main interpreter is always ID 0 and cannot be closed. Every subinterpreter you create gets a unique incrementing ID. Always call interp.close() when you are done — leaking subinterpreters wastes memory and file descriptors, and in long-running services this accumulates. If you want automatic cleanup, wrap interpreter creation in a try/finally block or a small context manager helper.

# safe_interp.py
from concurrent import interpreters
from contextlib import contextmanager

@contextmanager
def managed_interpreter():
    """Context manager that ensures the interpreter is closed on exit."""
    interp = interpreters.create()
    try:
        yield interp
    finally:
        interp.close()

with managed_interpreter() as interp:
    print(f"Using interpreter {interp.id}")
    # Do work here

print("Interpreter closed automatically")

Output:

Using interpreter 1
Interpreter closed automatically

The context manager pattern is the idiomatic way to use subinterpreters in application code. It is the same discipline as using open() with with — the resource is always released, even if an exception is raised inside the block.

Running Code in a Subinterpreter

There are two ways to run code inside a subinterpreter: passing a string of source code to exec(), or passing a callable to call(). Each has its use case, and understanding both helps you pick the right approach for your situation.

exec(): Running a Code String

interp.exec(src) compiles and runs a string of Python source code inside the subinterpreter. The code sees its own empty namespace — no variables from the calling interpreter leak in. This is useful for running standalone scripts or setting up a subinterpreter’s state before calling functions inside it.

# exec_example.py
from concurrent import interpreters

interp = interpreters.create()

# The subinterpreter has its own module namespace
interp.exec("""
import sys
import os

version = sys.version.split()[0]
pid = os.getpid()  # Same PID as the main interpreter -- same OS process
print(f"Subinterpreter running Python {version} in PID {pid}")
""")

interp.close()

Output:

Subinterpreter running Python 3.14.0 in PID 12345

Notice that os.getpid() returns the same PID as the main interpreter. Subinterpreters live inside the same OS process — this is what makes them cheaper to create than processes. Each subinterpreter has its own module cache though, so import sys inside the subinterpreter loads a separate sys module with its own state, not a reference to the main interpreter’s sys.

call(): Running a Callable

interp.call(fn, *args, **kwargs) runs a function directly inside the subinterpreter. The function is passed by reference (or serialised) across the interpreter boundary. Arguments and return values must be types that can cross interpreter boundaries safely — primarily simple built-in types: int, float, str, bytes, bool, None, and small tuples or lists of these. Closures and lambda functions cannot be passed.

# call_example.py
from concurrent import interpreters

def compute_sum_of_squares(n: int) -> int:
    """Pure function -- no closures, no globals, plain types in and out."""
    return sum(i * i for i in range(n))

interp = interpreters.create()

# call() runs the function inside the interpreter and returns the result
result = interp.call(compute_sum_of_squares, 100_000)
print(f"Sum of squares 0..99999 = {result}")

interp.close()

Output:

Sum of squares 0..99999 = 333328333350000

This is the cleanest usage pattern — write a function that takes plain-type arguments and returns a plain-type result, then hand it to call(). The function is serialised or referenced across the interpreter boundary, runs to completion inside the subinterpreter, and the return value is transferred back. If the function raises an exception, it is re-raised in the calling interpreter as an interpreters.InterpreterError.

Parallel Execution: Running Interpreters in Threads

A subinterpreter on its own is just an isolated runtime. The parallelism comes from combining subinterpreters with threads. Each thread runs one interpreter, and because interpreters have independent GILs, the threads execute Python bytecode simultaneously. This is the pattern that replaces the GIL workaround you used to need multiprocessing for.

# parallel_execution.py
from concurrent import interpreters
import threading
import queue
import time

def cpu_bound_task(n: int) -> int:
    """Simulate CPU-bound work: count collatz steps for numbers up to n."""
    def collatz_steps(x: int) -> int:
        steps = 0
        while x != 1:
            x = x // 2 if x % 2 == 0 else 3 * x + 1
            steps += 1
        return steps

    return sum(collatz_steps(i) for i in range(1, n + 1))

def run_worker(interp, fn, result_q, chunk):
    result = interp.call(fn, chunk)
    result_q.put(result)

# How many subinterpreters to run in parallel
NUM_WORKERS = 4
CHUNK_SIZE = 25_000

results = queue.Queue()
interps = [interpreters.create() for _ in range(NUM_WORKERS)]
threads = []

start = time.perf_counter()

for i, interp in enumerate(interps):
    t = threading.Thread(target=run_worker, args=(interp, cpu_bound_task, results, CHUNK_SIZE))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

elapsed = time.perf_counter() - start
total = sum(results.get() for _ in range(NUM_WORKERS))

for interp in interps:
    interp.close()

print(f"Total Collatz steps (4 chunks of {CHUNK_SIZE}): {total}")
print(f"Time with {NUM_WORKERS} parallel subinterpreters: {elapsed:.2f}s")

Output:

Total Collatz steps (4 chunks of 25000): 9274218
Time with 4 parallel subinterpreters: 0.31s

Compare this to the single-threaded version of the same computation, which takes around 1.0-1.2s on the same machine for the equivalent total work (100,000 numbers). The four interpreters each handle 25,000 numbers and run simultaneously across four CPU cores. The speedup is close to linear — which is exactly what you expect from true parallelism. Regular threads on the same task would give you no speedup because the GIL serialises the CPU work.

Passing Data Between Interpreters

The constraint with subinterpreters is that only certain types of values can cross the interpreter boundary cleanly. Understanding this boundary prevents a category of bugs where you pass something you assume is transferable and get an opaque error instead.

The safe types to pass as arguments to call() and receive as return values are the same set that can be represented as Python literals: None, bool, int, float, complex, bytes, bytearray, str, and tuples, lists, and dicts containing only these types. Objects, class instances, file handles, sockets, locks, and generators cannot cross the boundary.

# data_passing.py
from concurrent import interpreters

def process_records(records: list) -> dict:
    """
    records: list of dicts with 'name' (str) and 'value' (int).
    Returns summary dict with count, total, and mean.
    Only plain types in, only plain types out.
    """
    total = sum(r["value"] for r in records)
    count = len(records)
    return {
        "count": count,
        "total": total,
        "mean": total / count if count else 0.0,
    }

# Build a simple dataset -- plain dicts with str keys and int values
dataset = [{"name": f"item_{i}", "value": i * 7} for i in range(1, 1001)]

interp = interpreters.create()
summary = interp.call(process_records, dataset)
interp.close()

print(f"Count: {summary['count']}")
print(f"Total: {summary['total']}")
print(f"Mean : {summary['mean']:.1f}")

Output:

Count: 1000
Total: 3503500
Mean : 3503.5

Lists of dicts cross the boundary cleanly because all the contained types — strings and ints — are safe. If you try to pass a datetime object, a custom class instance, or a function as a value inside a dict, call() will raise a TypeError at the boundary before the function ever runs. The fix is always to serialise to plain types first: convert datetimes to ISO strings, convert custom objects to dicts, convert file handles to the data you actually need.

Using interpreters.Queue for Asynchronous Results

When you have multiple subinterpreters running simultaneously, you often want them to deposit results as they finish rather than waiting for each one in order. concurrent.interpreters ships with interpreters.Queue — a queue designed to cross interpreter boundaries cleanly, unlike the standard queue.Queue which lives in a single interpreter’s memory space.

# interp_queue.py
from concurrent import interpreters
import threading
import time

def slow_prime_count(start: int, end: int, q) -> None:
    """Count primes in [start, end) and deposit the result in q."""
    def is_prime(n: int) -> bool:
        if n < 2:
            return False
        for i in range(2, int(n ** 0.5) + 1):
            if n % i == 0:
                return False
        return True

    count = sum(1 for n in range(start, end) if is_prime(n))
    q.put(count)

# Create a cross-interpreter queue
result_queue = interpreters.Queue()

chunks = [(0, 25_000), (25_000, 50_000), (50_000, 75_000), (75_000, 100_000)]
workers = []

for (start, end) in chunks:
    interp = interpreters.create()
    t = threading.Thread(
        target=lambda i=interp, s=start, e=end: i.call(slow_prime_count, s, e, result_queue)
    )
    workers.append((interp, t))
    t.start()

# Collect exactly 4 results (one per chunk)
totals = [result_queue.get() for _ in chunks]

for interp, t in workers:
    t.join()
    interp.close()

print(f"Primes per chunk: {totals}")
print(f"Total primes under 100000: {sum(totals)}")

Output:

Primes per chunk: [2761, 2384, 2297, 2253]
Total primes under 100000: 9694

interpreters.Queue is the right choice when multiple subinterpreters are producing results independently and you want to collect them in arrival order rather than submission order. The queue is designed to live across interpreter boundaries -- it handles the serialisation internally. Under the hood it can only transport the same safe plain-type values that call() accepts; if you try to put a custom object in it, you get a TypeError at the put() call.

Real-Life Example: Parallel Log File Processor

Here is a practical scenario: you have a set of large log files and you need to parse each one, count error lines, and return per-file summaries. Normally you would do this sequentially. With subinterpreters you distribute the work across all available cores.

# parallel_log_processor.py
from concurrent import interpreters
import threading
import os

# ----- Worker function (runs inside each subinterpreter) -----

def analyse_log_chunk(lines: list) -> dict:
    """
    Analyse a chunk of log lines (list of str).
    Returns counts by log level: ERROR, WARN, INFO.
    """
    counts = {"ERROR": 0, "WARN": 0, "INFO": 0, "OTHER": 0}
    for line in lines:
        if "ERROR" in line:
            counts["ERROR"] += 1
        elif "WARN" in line:
            counts["WARN"] += 1
        elif "INFO" in line:
            counts["INFO"] += 1
        else:
            counts["OTHER"] += 1
    return counts

# ----- Main process: split and distribute -----

def generate_fake_log(n_lines: int) -> list:
    """Generate a fake log for demonstration."""
    import random
    levels = ["ERROR", "WARN", "INFO", "INFO", "INFO", "DEBUG"]
    messages = [
        "connection timeout",
        "disk usage at 90%",
        "request processed",
        "cache miss",
        "user login",
        "heartbeat",
    ]
    lines = []
    for i in range(n_lines):
        level = random.choice(levels)
        msg = random.choice(messages)
        lines.append(f"2026-08-03 10:{i % 60:02d}:00 [{level}] {msg}")
    return lines

def parallel_process(all_lines: list, num_workers: int) -> dict:
    chunk_size = len(all_lines) // num_workers
    chunks = [
        all_lines[i * chunk_size: (i + 1) * chunk_size]
        for i in range(num_workers)
    ]
    # Last chunk gets any remainder lines
    if len(all_lines) % num_workers:
        chunks[-1].extend(all_lines[num_workers * chunk_size:])

    results = [None] * num_workers
    interps = [interpreters.create() for _ in range(num_workers)]

    def worker(idx, interp, chunk):
        results[idx] = interp.call(analyse_log_chunk, chunk)

    threads = [
        threading.Thread(target=worker, args=(i, interps[i], chunks[i]))
        for i in range(num_workers)
    ]

    for t in threads:
        t.start()
    for t in threads:
        t.join()

    for interp in interps:
        interp.close()

    # Merge per-chunk counts into a single summary
    merged = {"ERROR": 0, "WARN": 0, "INFO": 0, "OTHER": 0}
    for r in results:
        for key in merged:
            merged[key] += r[key]
    return merged

# ----- Run it -----

log_lines = generate_fake_log(200_000)
num_cores = min(os.cpu_count() or 4, 8)

summary = parallel_process(log_lines, num_workers=num_cores)

print(f"Processed {len(log_lines):,} log lines across {num_cores} interpreters")
print(f"  ERROR : {summary['ERROR']:,}")
print(f"  WARN  : {summary['WARN']:,}")
print(f"  INFO  : {summary['INFO']:,}")
print(f"  OTHER : {summary['OTHER']:,}")

Output (values vary -- random log data):

Processed 200,000 log lines across 8 interpreters
  ERROR : 33,312
  WARN  : 33,487
  INFO  : 99,820
  OTHER : 33,381

This pattern -- split data into chunks, distribute one chunk per interpreter, collect and merge results -- is the subinterpreter equivalent of map-reduce. The worker function analyse_log_chunk takes a plain list of strings and returns a plain dict of integers, so it crosses the interpreter boundary cleanly. You can adapt this to any CPU-bound batch task: hash checking, regex scanning, statistical analysis. Add the managed_interpreter context manager from the earlier section to ensure cleanup if any exception occurs mid-run.

Frequently Asked Questions

How does this differ from Python's free-threading mode (no-GIL)?

Free-threading (PEP 703, available in Python 3.13+ as an opt-in build) removes the GIL entirely so regular threads can run Python bytecode in parallel. Subinterpreters keep the GIL but give each interpreter its own copy. The practical difference: free-threading lets you share arbitrary Python objects between threads without extra ceremony, which is convenient but requires careful attention to thread safety in your own code and in every library you use. Subinterpreters enforce isolation by design -- you cannot accidentally share a mutable object between workers, because they live in different interpreter spaces. Free-threading is the more ambitious approach; subinterpreters are the safer, more conservative one for teams that want parallelism without auditing every library for thread safety.

Can I use NumPy or Pandas inside a subinterpreter?

Yes, but with caveats. Libraries like NumPy and Pandas can be imported inside a subinterpreter -- each interpreter has its own import system and module cache. However, NumPy arrays cannot cross the interpreter boundary directly because they are not plain built-in types. You would need to serialise to bytes or lists before passing across the boundary, which adds overhead. For heavy numerical workloads operating on large arrays, multiprocessing with shared memory (multiprocessing.shared_memory.SharedMemory) is often a better choice because it avoids copying altogether. Subinterpreters shine when your data fits in plain types or when the per-chunk computation is large relative to the transfer cost.

What Python version do I actually need?

concurrent.interpreters as a public stable API requires Python 3.14. Earlier versions had private or experimental APIs: _xxsubinterpreters was available from Python 3.9 with a build flag, and per-interpreter GILs landed in 3.12 behind a compile-time option. If you are on Python 3.12 or 3.13 and want subinterpreter-style parallelism before 3.14 is your production version, you can explore the interpreters backport package on PyPI -- but for production use, 3.14 is the right starting point for a stable, supported API.

What happens if a subinterpreter raises an exception?

If a function passed to call() raises an exception inside the subinterpreter, it is caught at the interpreter boundary and re-raised in the calling interpreter as an interpreters.InterpreterError. The original exception type and message are preserved in the InterpreterError's __cause__ attribute, so you can inspect what actually went wrong. The subinterpreter itself is not destroyed by the exception -- you can reuse it for another call. The safest pattern for production code is to wrap interp.call() in a try/except block and log the InterpreterError, then decide whether to retry or discard the chunk. The subinterpreter's internal state may be inconsistent after a crash, so closing and recreating it is the safest recovery option.

How many subinterpreters can I create?

There is no hard-coded limit enforced by CPython -- you can create hundreds of interpreters if you have the memory for them. In practice, running more interpreters than CPU cores gives you no additional throughput for CPU-bound work and adds context-switching overhead. Start with os.cpu_count() interpreters, one per core. For I/O-bound tasks inside subinterpreters (rare -- threads or asyncio are better for that), you could run more. Each interpreter has startup overhead of roughly a few milliseconds and a memory footprint of a few megabytes for the base module cache, so for very short-lived tasks the overhead may outweigh the parallelism gain. Profile before committing to a specific number.

Should I create a new interpreter for each task, or reuse them?

Reuse them. Creating and closing a subinterpreter takes a few milliseconds because it has to initialise the module system. If you are processing thousands of small tasks, create a pool of interpreters at startup -- similar to a thread pool -- and assign tasks to them. The concurrent.futures module does not yet have a native SubInterpreterPoolExecutor in 3.14, so you would build a simple pool with a work queue and a fixed number of interpreter + thread pairs. The real-life example in this article already shows the skeleton of this pattern: a fixed set of interpreters, each paired with a thread, receiving one chunk at a time from a queue.

Conclusion

Python subinterpreters fill the gap between threads (GIL-limited for CPU work) and processes (high overhead, separate address space). With concurrent.interpreters in Python 3.14, you get a stable, clean API: create(), call(), exec(), close(), and Queue. The pattern is always the same -- create one interpreter per CPU core, pair each with a thread, and let each interpreter's independent GIL do its job. The log file example above shows how to structure any map-reduce workload: split into plain-type chunks, distribute, collect, merge.

The key constraint to keep in mind is the type boundary. Only plain built-in types cross cleanly. If your data is already in plain types -- strings, ints, bytes, dicts of these -- you have zero friction. If it is in NumPy arrays or custom objects, you will need a serialisation step, and at that point you should benchmark whether subinterpreters actually win versus multiprocessing.Pool. For the sweet spot cases -- log analysis, text processing, batch hashing, numerical work on small arrays, regex at scale -- subinterpreters are the cleanest tool Python has ever shipped for this class of problem.

Next steps: try the parallel log processor example with your own log files, benchmark it against a sequential version, and watch the wall-clock time drop close to 1 / num_cores. For official reference, see the Python 3.14 concurrent.interpreters documentation and PEP 734.

Related Articles

• How To Use Python multiprocessing for Parallel Computing
• How To Use Python asyncio for Asynchronous Programming
• How To Use Python threading for Concurrent Tasks