How To Build a REST API with FastAPI in Python

How To Build a REST API with FastAPI in Python

Last Updated: June 01, 2026

Intermediate

Building a REST API is one of the most common tasks in modern software development, and FastAPI has quickly become the go-to Python framework for doing it. If you have been using Flask or Django REST Framework and wondered whether there is a faster, more modern alternative with built-in data validation and automatic documentation, FastAPI is your answer.

You will need Python 3.8 or later, plus two packages: fastapi and uvicorn. Install them with pip install fastapi uvicorn. FastAPI uses standard Python type hints for request validation and automatic OpenAPI documentation, so everything you already know about type annotations transfers directly.

This guide walks you through building a complete REST API from scratch: defining routes, handling path and query parameters, validating request bodies with Pydantic models, returning proper HTTP status codes, and running your API with Uvicorn. By the end you will have a fully functional CRUD API for managing a collection of books.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

FastAPI in 30 Seconds: Quick Example

Part of the Python Web Frameworks Hub. See the full hub for related Python tutorials.

Part of the Modern Python AI Stack series. See the full tutorial hub for all 23 tutorials on LangGraph, MCP, Pydantic AI, Polars, FastAPI, Litestar, Typer, and more.

Here is the simplest possible FastAPI application. Save it and run it to see your first API endpoint in action.

# quick_example.py
from fastapi import FastAPI

app = FastAPI()

@app.get("/")
def read_root():
    return {"message": "Hello, FastAPI!"}

@app.get("/items/{item_id}")
def read_item(item_id: int, q: str = None):
    return {"item_id": item_id, "query": q}

Run it with: uvicorn quick_example:app --reload

Output (visiting http://127.0.0.1:8000/):

{"message": "Hello, FastAPI!"}

Output (visiting http://127.0.0.1:8000/items/42?q=search):

{"item_id": 42, "query": "search"}

Notice how item_id: int automatically validates that the path parameter is an integer. If you visit /items/hello, FastAPI returns a 422 validation error without you writing any validation code. The --reload flag makes Uvicorn restart when you change code, which is perfect for development.

What Is FastAPI and Why Use It?

FastAPI is a modern Python web framework for building APIs. It was created by Sebastian Ramirez and released in 2018. The framework is built on top of Starlette (for the web parts) and Pydantic (for data validation), combining the best of both into a developer-friendly experience.

FeatureFastAPIFlaskDjango REST
Async supportNative (async/await)Limited (via extensions)Limited
Data validationBuilt-in (Pydantic)Manual or extensionsSerializers
Auto documentationSwagger + ReDocManual or extensionsBrowsable API
Type hintsRequired (powers validation)OptionalOptional
PerformanceVery fast (ASGI)Moderate (WSGI)Moderate (WSGI)
Learning curveLowLowMedium-High

The biggest practical advantage is that FastAPI uses your type hints to generate request validation, response serialization, and API documentation automatically. You write the type annotations you would write anyway, and the framework does the rest.

FastAPI REST API architecture design
Good API design is like good plumbing — nobody notices until it breaks.

Setting Up Your FastAPI Project

Let us set up a proper project structure. Create a new directory and install the dependencies.

# setup_commands.sh
mkdir fastapi-books && cd fastapi-books
pip install fastapi uvicorn

Create the following file structure. We will build this up step by step.

# project_structure.txt
fastapi-books/
    main.py          # Application entry point
    models.py        # Pydantic models for request/response
    database.py      # In-memory database (for simplicity)

Defining Data Models with Pydantic

Start by defining what a “book” looks like using Pydantic models. These models handle both validation and serialization.

# models.py
from pydantic import BaseModel, Field
from typing import Optional

class BookCreate(BaseModel):
    title: str = Field(..., min_length=1, max_length=200)
    author: str = Field(..., min_length=1, max_length=100)
    year: int = Field(..., ge=1000, le=2030)
    isbn: Optional[str] = Field(None, pattern=r"^\d{10}(\d{3})?$")

class BookResponse(BaseModel):
    id: int
    title: str
    author: str
    year: int
    isbn: Optional[str] = None

class BookUpdate(BaseModel):
    title: Optional[str] = Field(None, min_length=1, max_length=200)
    author: Optional[str] = Field(None, min_length=1, max_length=100)
    year: Optional[int] = Field(None, ge=1000, le=2030)
    isbn: Optional[str] = Field(None, pattern=r"^\d{10}(\d{3})?$")

The Field function adds validation constraints: min_length, max_length, ge (greater than or equal), and pattern (regex). The ... means the field is required. Optional[str] = None means the field is optional with a default of None.

In-Memory Database

# database.py
from models import BookResponse

books_db: dict[int, dict] = {}
next_id: int = 1

def get_next_id() -> int:
    global next_id
    current = next_id
    next_id += 1
    return current

Building CRUD Endpoints

Now let us build the full API with Create, Read, Update, and Delete operations.

# main.py
from fastapi import FastAPI, HTTPException, Query
from models import BookCreate, BookResponse, BookUpdate
from database import books_db, get_next_id

app = FastAPI(
    title="Books API",
    description="A simple REST API for managing books",
    version="1.0.0",
)

@app.post("/books", response_model=BookResponse, status_code=201)
def create_book(book: BookCreate):
    book_id = get_next_id()
    book_data = {"id": book_id, **book.model_dump()}
    books_db[book_id] = book_data
    return book_data

@app.get("/books", response_model=list[BookResponse])
def list_books(
    skip: int = Query(0, ge=0),
    limit: int = Query(10, ge=1, le=100),
    author: str = Query(None),
):
    results = list(books_db.values())
    if author:
        results = [b for b in results if author.lower() in b["author"].lower()]
    return results[skip : skip + limit]

@app.get("/books/{book_id}", response_model=BookResponse)
def get_book(book_id: int):
    if book_id not in books_db:
        raise HTTPException(status_code=404, detail="Book not found")
    return books_db[book_id]

@app.put("/books/{book_id}", response_model=BookResponse)
def update_book(book_id: int, book: BookUpdate):
    if book_id not in books_db:
        raise HTTPException(status_code=404, detail="Book not found")
    update_data = book.model_dump(exclude_unset=True)
    books_db[book_id].update(update_data)
    return books_db[book_id]

@app.delete("/books/{book_id}", status_code=204)
def delete_book(book_id: int):
    if book_id not in books_db:
        raise HTTPException(status_code=404, detail="Book not found")
    del books_db[book_id]

Each endpoint uses type hints to define what it accepts and returns. The response_model parameter tells FastAPI to validate and serialize the response using that Pydantic model. status_code=201 sets the HTTP status for successful creation. HTTPException returns proper error responses with the right status codes.

Building FastAPI endpoints
Four HTTP verbs, infinite ways to get the status codes wrong.

Path Parameters, Query Parameters, and Request Bodies

FastAPI distinguishes between three types of input automatically based on where they appear in your function signature.

# parameters_demo.py
from fastapi import FastAPI, Query, Path

app = FastAPI()

@app.get("/users/{user_id}/posts")
def get_user_posts(
    user_id: int = Path(..., ge=1, description="The ID of the user"),
    page: int = Query(1, ge=1, description="Page number"),
    per_page: int = Query(10, ge=1, le=50, description="Items per page"),
    sort_by: str = Query("date", pattern="^(date|title|likes)$"),
):
    return {
        "user_id": user_id,
        "page": page,
        "per_page": per_page,
        "sort_by": sort_by,
        "posts": [f"Post {i}" for i in range(1, per_page + 1)],
    }

Output (GET /users/5/posts?page=2&sort_by=likes):

{
  "user_id": 5,
  "page": 2,
  "per_page": 10,
  "sort_by": "likes",
  "posts": ["Post 1", "Post 2", "Post 3", ...]
}

The rules are simple: if a parameter name matches a path variable in the URL template (like {user_id}), it is a path parameter. If the parameter has a default value or is annotated with Query(), it is a query parameter. If the parameter is a Pydantic model, it is parsed from the request body.

Error Handling

FastAPI provides clean error handling through exceptions and custom exception handlers.

# error_handling.py
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse

app = FastAPI()

class BookNotFoundError(Exception):
    def __init__(self, book_id: int):
        self.book_id = book_id

@app.exception_handler(BookNotFoundError)
async def book_not_found_handler(request: Request, exc: BookNotFoundError):
    return JSONResponse(
        status_code=404,
        content={"error": "not_found", "detail": f"Book {exc.book_id} does not exist"},
    )

@app.get("/books/{book_id}")
def get_book(book_id: int):
    books = {1: "Python Crash Course", 2: "Fluent Python"}
    if book_id not in books:
        raise BookNotFoundError(book_id)
    return {"id": book_id, "title": books[book_id]}

Output (GET /books/99):

{"error": "not_found", "detail": "Book 99 does not exist"}

Async Endpoints

FastAPI supports both synchronous and asynchronous endpoint functions. Use async def when your endpoint does I/O operations like database queries or HTTP requests.

# async_demo.py
import asyncio
from fastapi import FastAPI

app = FastAPI()

@app.get("/sync")
def sync_endpoint():
    return {"type": "synchronous"}

@app.get("/async")
async def async_endpoint():
    await asyncio.sleep(0.1)
    return {"type": "asynchronous"}

Regular def functions run in a thread pool, so they do not block other requests. async def functions run on the event loop and should use await for any I/O operations. If your function does not use await, use regular def — there is no benefit to making it async.

Request validation in FastAPI
Pydantic validates your data so your database doesn’t have to file a complaint.

Automatic API Documentation

One of FastAPI’s best features is automatic API documentation. When you run your application, visit these URLs to see interactive documentation:

# documentation_urls.txt
Swagger UI:  http://127.0.0.1:8000/docs
ReDoc:       http://127.0.0.1:8000/redoc
OpenAPI JSON: http://127.0.0.1:8000/openapi.json

The Swagger UI lets you test every endpoint directly in the browser. It reads your type hints, Pydantic models, and docstrings to generate accurate request/response schemas. This means your documentation is always in sync with your code — no manual updates needed.

Real-Life Example: Complete Books API

Deploying FastAPI application
It works on localhost. Now make it work everywhere else.

Let us put everything together into a single, runnable file that demonstrates the complete CRUD workflow.

# books_api.py
from fastapi import FastAPI, HTTPException, Query
from pydantic import BaseModel, Field
from typing import Optional

app = FastAPI(title="Books API", version="1.0.0")

# In-memory database
books_db: dict[int, dict] = {}
next_id = 1

class BookCreate(BaseModel):
    title: str = Field(..., min_length=1, max_length=200)
    author: str = Field(..., min_length=1, max_length=100)
    year: int = Field(..., ge=1000, le=2030)

class BookResponse(BaseModel):
    id: int
    title: str
    author: str
    year: int

class BookUpdate(BaseModel):
    title: Optional[str] = None
    author: Optional[str] = None
    year: Optional[int] = None

@app.post("/books", response_model=BookResponse, status_code=201)
def create_book(book: BookCreate):
    global next_id
    book_data = {"id": next_id, **book.model_dump()}
    books_db[next_id] = book_data
    next_id += 1
    return book_data

@app.get("/books", response_model=list[BookResponse])
def list_books(skip: int = Query(0, ge=0), limit: int = Query(10, ge=1, le=100)):
    return list(books_db.values())[skip : skip + limit]

@app.get("/books/{book_id}", response_model=BookResponse)
def get_book(book_id: int):
    if book_id not in books_db:
        raise HTTPException(404, "Book not found")
    return books_db[book_id]

@app.put("/books/{book_id}", response_model=BookResponse)
def update_book(book_id: int, book: BookUpdate):
    if book_id not in books_db:
        raise HTTPException(404, "Book not found")
    books_db[book_id].update(book.model_dump(exclude_unset=True))
    return books_db[book_id]

@app.delete("/books/{book_id}", status_code=204)
def delete_book(book_id: int):
    if book_id not in books_db:
        raise HTTPException(404, "Book not found")
    del books_db[book_id]

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)

Testing with curl:

# Create a book
curl -X POST http://127.0.0.1:8000/books \
  -H "Content-Type: application/json" \
  -d '{"title": "Python Crash Course", "author": "Eric Matthes", "year": 2023}'

# Response: {"id": 1, "title": "Python Crash Course", "author": "Eric Matthes", "year": 2023}

# List all books
curl http://127.0.0.1:8000/books
# Response: [{"id": 1, "title": "Python Crash Course", ...}]

# Update a book
curl -X PUT http://127.0.0.1:8000/books/1 \
  -H "Content-Type: application/json" \
  -d '{"year": 2024}'

# Delete a book
curl -X DELETE http://127.0.0.1:8000/books/1

Run this with python books_api.py and test the endpoints using curl, httpie, or the built-in Swagger UI at /docs. You can extend this example by adding a real database (SQLAlchemy or SQLModel), authentication, pagination headers, and response caching.

Frequently Asked Questions

Can I migrate my Flask app to FastAPI?

Yes, and the migration is usually straightforward. Flask routes map 1:1 to FastAPI routes. The main changes are adding type hints to your parameters and converting request parsing from request.json to Pydantic models. FastAPI has a migration guide in its documentation.

Is FastAPI production-ready?

Yes. FastAPI is used in production by Microsoft, Netflix, Uber, and many other companies. Deploy it with Uvicorn behind a reverse proxy like Nginx, or use Gunicorn with Uvicorn workers for multi-process setups.

How do I connect FastAPI to a database?

Use SQLAlchemy 2.0 with async sessions, or use SQLModel (created by FastAPI’s author) which combines SQLAlchemy and Pydantic. For simple projects, you can also use databases with raw SQL queries. FastAPI’s documentation has complete examples for each approach.

How do I add authentication?

FastAPI has built-in support for OAuth2 with JWT tokens. Use fastapi.security.OAuth2PasswordBearer for token-based auth, or implement API key authentication with custom dependencies. See our companion article on FastAPI authentication for a complete walkthrough.

How do I test FastAPI endpoints?

Use TestClient from fastapi.testclient (which wraps httpx). It lets you make requests to your API without running a server, making unit tests fast and reliable.

Conclusion

FastAPI makes building REST APIs in Python fast and enjoyable. The combination of automatic validation through type hints, built-in async support, and interactive Swagger documentation means you spend less time on boilerplate and more time on your application logic. Start with the books API example, then extend it with a real database and authentication.

For the complete documentation, visit fastapi.tiangolo.com.

Minimal FastAPI App

FastAPI is built on Starlette + Pydantic. The decorator-based router turns a Python function into an HTTP endpoint:

# pip install fastapi uvicorn

# main.py
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI(title="My API", version="1.0.0")

class User(BaseModel):
    id: int
    name: str
    email: str

users = []

@app.get("/")
def root():
    return {"message": "Hello"}

@app.get("/users", response_model=list[User])
def list_users():
    return users

@app.post("/users", response_model=User, status_code=201)
def create_user(user: User):
    users.append(user)
    return user

# Run: uvicorn main:app --reload --port 8000

Visit http://localhost:8000/docs and you get interactive API docs generated from the Pydantic models — no separate documentation step.

Path Parameters and Query Parameters

FastAPI infers parameter type from function signatures. Path params become path components; query params become URL queries:

from fastapi import HTTPException

@app.get("/users/{user_id}")
def get_user(user_id: int):    # path param — validated as int
    for u in users:
        if u.id == user_id:
            return u
    raise HTTPException(status_code=404, detail="User not found")

@app.get("/search")
def search(q: str, limit: int = 10, sort: str = "id"):
    # q is required (no default), limit and sort optional with defaults
    return {"query": q, "limit": limit, "sort": sort}

Type annotations drive validation. A request to /users/abc automatically returns 422 with “value is not a valid integer”.

Dependency Injection

FastAPI’s Depends() wires up shared dependencies — DB sessions, auth checks, common query params — without global state:

from fastapi import Depends, HTTPException, Header
from typing import Annotated

def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()

async def verify_token(authorization: Annotated[str, Header()] = ""):
    token = authorization.removeprefix("Bearer ")
    if not is_valid(token):
        raise HTTPException(status_code=401, detail="Invalid token")
    return get_user_from_token(token)

@app.get("/profile")
def profile(
    user = Depends(verify_token),
    db = Depends(get_db),
):
    return db.get_user_full(user.id)

Dependencies can be nested, cached per-request, and made async. The yield pattern handles setup/teardown like a context manager.

Async Endpoints

Use async def for I/O-bound endpoints (database, external APIs). Use plain def for CPU-bound work — FastAPI runs those in a thread pool to avoid blocking the loop:

import httpx

@app.get("/proxy")
async def proxy(url: str):
    async with httpx.AsyncClient() as client:
        resp = await client.get(url)
    return {"status": resp.status_code, "data": resp.text}

@app.post("/render")
def render_pdf(html: str):
    # Synchronous — runs in thread pool
    pdf_bytes = wkhtmltopdf_convert(html)
    return Response(content=pdf_bytes, media_type="application/pdf")

Background Tasks

For fire-and-forget work after a response (sending emails, logging), use BackgroundTasks:

from fastapi import BackgroundTasks

def send_welcome_email(email: str):
    # Slow SMTP call — don't make the user wait
    smtp_send(email, "Welcome!", "...")

@app.post("/signup")
def signup(user: User, background_tasks: BackgroundTasks):
    save_user(user)
    background_tasks.add_task(send_welcome_email, user.email)
    return {"status": "created"}

For heavier work or work that must survive crashes, use Celery instead — BackgroundTasks are in-process only.

Common Pitfalls

  • Sync code in async endpoint. time.sleep() in async def blocks the event loop. Use await asyncio.sleep() or move to def.
  • Forgetting response_model. Without it, FastAPI returns whatever you return — including extra fields. Always declare it for stable API contracts.
  • Mutable global state. users = [] works in dev, breaks under multi-worker uvicorn. Use a database from day one.
  • Async DB session in sync endpoint. If you use SQLAlchemy async, your endpoint must also be async. Mixing types throws confusing errors.
  • Missing dependencies in tests. Use app.dependency_overrides[get_db] = lambda: test_db to swap dependencies without touching production code.

FAQ

Q: FastAPI or Flask?
A: FastAPI for new APIs — async by default, auto-docs, Pydantic validation. Flask for synchronous, template-rendering apps or legacy ecosystems.

Q: How do I deploy FastAPI?
A: uvicorn main:app behind a reverse proxy (nginx, Caddy). For containerized: Docker + uvicorn. For serverless: Mangum adapter for AWS Lambda. Avoid uvicorn --reload in production.

Q: WebSockets in FastAPI?
A: Built in — @app.websocket("/ws"). See FastAPI’s WebSocket docs.

Q: How do I do auth?
A: For tokens, write a Depends(verify_token) dependency. For OAuth/OIDC, use authlib or fastapi-users. Avoid rolling your own crypto.

Q: How do I run sync ORMs (Django, peewee) inside async FastAPI?
A: Either use sync endpoints (FastAPI runs them in a thread pool) or wrap calls in await asyncio.to_thread(sync_func, ...). Don’t fight the framework — pick async or sync per endpoint.

Wrapping Up

FastAPI’s appeal is the combination of speed (Starlette / Uvicorn), type safety (Pydantic), and zero-effort docs (OpenAPI from your Python). For new Python web APIs in 2026, it’s the default choice. Start with the routers and Pydantic models, add dependency injection when you have shared concerns, and reach for BackgroundTasks (or Celery for heavier work) when you need async-after-response.

What’s New in Python 3.13: A Complete Guide

What’s New in Python 3.13: A Complete Guide

Last Updated: June 01, 2026

Intermediate

Every new Python release brings features that change how you write code day to day. Python 3.13, released in October 2024, is no exception. If you have ever stared at a confusing traceback wondering which part of a chained expression caused the error, or wished the interactive interpreter felt more like a modern tool, Python 3.13 has direct answers for you.

You do not need any special libraries to try the features covered here — everything ships with the standard Python 3.13 installation. If you have not upgraded yet, grab it from python.org and follow along. The experimental free-threading build requires a separate installer option, but all other features work out of the box.

This guide walks you through the most impactful changes: the revamped interactive REPL, improved error messages, the new copy.replace() function, deprecation removals, typing improvements, and the experimental free-threaded build. By the end you will know exactly which features to adopt immediately and which ones to watch as they mature.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Python 3.13 in 30 Seconds: Quick Example

Here is a quick taste of the improved error messages in Python 3.13. The interpreter now highlights the exact part of the expression that caused the problem, not just the line.

# quick_example.py
data = {"users": [{"name": "Alice"}, {"name": None}]}
for user in data["users"]:
    print(user["name"].upper())

Output (Python 3.13):

ALICE
Traceback (most recent call last):
  File "quick_example.py", line 3, in <module>
    print(user["name"].upper())
          ~~~~~~~~~~~~^^^^^^^
AttributeError: 'NoneType' object has no attribute 'upper'

Notice how the caret markers now point directly at user["name"].upper(), making it immediately obvious that user["name"] returned None. In earlier Python versions, you would only see the full line highlighted with no indication of which part failed.

Why Upgrade to Python 3.13?

Python 3.13 is not a radical overhaul — it is a release focused on developer experience and laying groundwork for the future. The improvements fall into two categories: things that make your daily coding life better right now, and experimental features that signal where Python is heading.

CategoryFeatureImpact
Developer ExperienceNew interactive REPLMulti-line editing, color output, paste mode
Developer ExperienceBetter error messagesPinpoints exact expression that failed
Standard Librarycopy.replace()Create modified copies of objects cleanly
Standard Librarydbm.sqlite3 backendDefault dbm now uses SQLite under the hood
TypingType defaults (PEP 696)TypeVar, ParamSpec, TypeVarTuple get defaults
DeprecationsRemoved modulesaifc, audioop, cgi, and more are gone
ExperimentalFree-threaded buildRun without the GIL for true parallelism
ExperimentalJIT compilerCopy-and-patch JIT for potential speedups

The daily-use improvements are reason enough to upgrade for most developers. The experimental features give you a preview of Python’s multi-threaded future without requiring any changes to your existing code.

Exploring new Python 3.13 features
Every major release is a chance to delete workarounds you forgot you wrote.

The Revamped Interactive REPL

The Python 3.13 REPL is a significant step up from the bare-bones interpreter that has existed since Python 1.x. If you have ever pasted a multi-line function into the REPL and watched it break because of indentation issues, this upgrade is for you.

Multi-Line Editing

The new REPL supports proper multi-line editing. You can define a function, realize you made a typo on line 2, and press the up arrow to go back and fix it — all without retyping the entire block.

# repl_demo.py
def greet(name):
    greeting = f"Hello, {name}!"
    return greeting

greet("World")

Output:

'Hello, World!'

In previous Python versions, pressing Up would only recall the last line. Now it recalls the entire block, letting you edit and re-execute multi-line code naturally.

Color Output and Tracebacks

The REPL now displays syntax-highlighted output and colorized tracebacks by default. Error messages use color to distinguish the file path, line number, error type, and error message. You can disable this by setting the environment variable PYTHON_COLORS=0 if you prefer plain text.

Paste Mode

Press F3 to enter paste mode, which lets you paste large blocks of code without the REPL trying to execute each line as you paste it. Press F3 again to execute the entire pasted block. This solves the long-standing frustration of pasting multi-line code from tutorials or documentation.

Improved Error Messages

Python has been on a multi-release journey to make error messages more helpful. Python 3.13 continues this with several targeted improvements that save you debugging time.

Better NameError Suggestions

When you mistype a variable name that happens to match a module in the standard library, Python 3.13 now suggests importing it.

# better_nameerror.py
print(sys.version)

Output:

NameError: name 'sys' is not defined. Did you forget to import 'sys'?

The suggestion Did you forget to import 'sys'? is new in 3.13. Previously you would just get the bare NameError with no hint about what went wrong.

Improved error messages in Python 3.13
The traceback finally points at the suspect, not just the crime scene.

Keyword Argument Suggestions

If you pass a keyword argument with a typo, Python 3.13 now suggests the correct name.

# keyword_suggestion.py
def connect(host, port, timeout=30):
    return f"Connected to {host}:{port}"

connect(host="localhost", port=5432, timout=60)

Output:

TypeError: connect() got an unexpected keyword argument 'timout'. Did you mean 'timeout'?

This is the kind of quality-of-life improvement that saves minutes of head-scratching, especially in large codebases where function signatures have many parameters.

The New copy.replace() Function

Python 3.13 adds copy.replace(), a generic way to create a modified copy of an object. If you have used dataclasses.replace() or namedtuple._replace(), this is the same idea but generalized to work with any object that implements the __replace__ protocol.

# copy_replace_demo.py
import copy
from datetime import date, time, datetime

original_date = date(2026, 4, 7)
next_year = copy.replace(original_date, year=2027)
print(f"Original: {original_date}")
print(f"Modified: {next_year}")

meeting_time = time(14, 30)
later = copy.replace(meeting_time, hour=16)
print(f"Original time: {meeting_time}")
print(f"Rescheduled:   {later}")

Output:

Original: 2026-04-07
Modified: 2027-04-07
Original time: 14:30:00
Rescheduled:   16:30:00

The beauty of copy.replace() is that it works with any object that defines __replace__. The datetime module’s classes already support it. Your own dataclasses support it automatically. And you can add __replace__ to any custom class to opt into this protocol.

# custom_replace.py
import copy

class Config:
    def __init__(self, host, port, debug=False):
        self.host = host
        self.port = port
        self.debug = debug

    def __replace__(self, **changes):
        return Config(
            host=changes.get("host", self.host),
            port=changes.get("port", self.port),
            debug=changes.get("debug", self.debug),
        )

    def __repr__(self):
        return f"Config(host={self.host!r}, port={self.port}, debug={self.debug})"

prod = Config("api.example.com", 443)
dev = copy.replace(prod, host="localhost", port=8000, debug=True)
print(f"Production: {prod}")
print(f"Development: {dev}")

Output:

Production: Config(host='api.example.com', port=443, debug=False)
Development: Config(host='localhost', port=8000, debug=True)
copy.replace() in Python 3.13
copy.replace() — because mutating the original was never the plan.

Removed and Deprecated Modules

Python 3.13 completes the removal of modules that were deprecated in Python 3.11 under PEP 594. If your code imports any of these, it will break on upgrade.

Removed ModuleReplacement
aifcUse soundfile (pip install)
audioopUse pydub or numpy
cgiUse urllib.parse or a web framework
cgitbUse traceback or logging
imghdrUse filetype or python-magic
pipesUse subprocess
telnetlibUse telnetlib3
uuUse base64

Before upgrading, run a quick grep across your project to check for these imports. A single import cgi in a legacy module can break your entire application on startup.

Typing Improvements

Python 3.13 brings several useful additions to the typing system. The most notable is PEP 696, which adds default values for TypeVar, ParamSpec, and TypeVarTuple.

# typing_defaults.py
from typing import TypeVar, Generic

T = TypeVar("T", default=str)

class Container(Generic[T]):
    def __init__(self, value: T) -> None:
        self.value = value
    def get(self) -> T:
        return self.value

box1: Container = Container("hello")
box3: Container[int] = Container(42)
print(f"box1: {box1.get()}")
print(f"box3: {box3.get()}")

Output:

box1: hello
box3: 42

The warnings.deprecated Decorator

Python 3.13 also adds warnings.deprecated (from PEP 702), which lets you mark functions as deprecated with a standard decorator that type checkers understand.

# deprecated_demo.py
import warnings

@warnings.deprecated("Use new_connect() instead")
def old_connect(host, port):
    return f"Connected to {host}:{port}"

result = old_connect("localhost", 5432)
print(result)

Output:

Connected to localhost:5432
Deprecated features in Python 3.13
Two doors: one leads to deprecated code, the other to your future self thanking you.

Experimental: Free-Threaded Python (No GIL)

The biggest long-term change in Python 3.13 is the experimental free-threaded build, which lets Python run without the Global Interpreter Lock (GIL). This is the result of PEP 703.

# check_gil.py
import sys
print(f"Python version: {sys.version}")
has_gil = getattr(sys.flags, "gil", None)
if has_gil is not None:
    print(f"GIL enabled: {sys.flags.gil}")
else:
    print("GIL attribute not available (standard build)")

Output (free-threaded build):

Python version: 3.13.0 experimental free-threading build
GIL enabled: 0

The free-threaded build is marked experimental for good reason: many C extensions (NumPy, pandas, etc.) are not yet compatible. Use it for testing, not production workloads.

Other Notable Changes

dbm Now Uses SQLite by Default

The dbm module’s default backend is now dbm.sqlite3, giving you better reliability and cross-platform consistency.

# dbm_sqlite_demo.py
import dbm
with dbm.open("mydata", "c") as db:
    db["name"] = "Python 3.13"
    db["feature"] = "SQLite dbm backend"
    print(f"Stored: {db['name'].decode()}")
    print(f"Keys: {list(db.keys())}")

Output:

Stored: Python 3.13
Keys: [b'name', b'feature']

Defined Semantics for locals()

PEP 667 defines clear semantics for locals(). In Python 3.13, calling locals() in a function always returns a fresh snapshot. Modifying the returned dictionary no longer affects the actual local variables.

# locals_demo.py
def demo():
    x = 10
    local_vars = locals()
    local_vars["x"] = 999
    print(f"x is still: {x}")

demo()

Output:

x is still: 10

Real-Life Example: Feature Detection Utility

Performance improvements in Python 3.13
Python 3.13 benchmarks: your loops just got a turbo button.
# feature_detector.py
import sys
import importlib

def check_feature(name, test_fn):
    try:
        available, detail = test_fn()
    except Exception as e:
        available, detail = False, str(e)
    status = "YES" if available else "NO"
    print(f"[{status:>3}] {name}: {detail}")

def main():
    print("=" * 55)
    print("Python 3.13 Feature Detection Report")
    print(f"Python: {sys.version}")
    print("=" * 55)

    check_feature("Python 3.13+",
        lambda: (sys.version_info >= (3, 13),
                 f"Running {sys.version_info.major}.{sys.version_info.minor}"))

    check_feature("Free-threaded build",
        lambda: (hasattr(sys.flags, "gil") and not sys.flags.gil,
                 "GIL disabled" if hasattr(sys.flags, "gil") and not sys.flags.gil
                 else "Standard build"))

    import copy
    check_feature("copy.replace()",
        lambda: (hasattr(copy, "replace"), "Available" if hasattr(copy, "replace") else "Not available"))

    removed = ["aifc", "cgi", "telnetlib", "uu"]
    gone = sum(1 for m in removed if not importlib.util.find_spec(m))
    check_feature("PEP 594 removals",
        lambda: (gone == len(removed), f"{gone}/{len(removed)} removed"))

    print("=" * 55)

if __name__ == "__main__":
    main()

Output (on Python 3.13):

=======================================================
Python 3.13 Feature Detection Report
Python: 3.13.0 (main, Oct 7 2024, 00:00:00)
=======================================================
[YES] Python 3.13+: Running 3.13
[ NO] Free-threaded build: Standard build
[YES] copy.replace(): Available
[YES] PEP 594 removals: 4/4 removed
=======================================================

This utility is a useful starting point you can extend for your own projects. Add checks for any features your codebase depends on.

Frequently Asked Questions

Is it safe to upgrade to Python 3.13 for production?

Yes, the standard Python 3.13 build is stable and production-ready. The “experimental” label only applies to the free-threaded build and the JIT compiler, both of which are opt-in.

Can I use the free-threaded build in production?

Not yet. The free-threaded build is explicitly experimental. Many popular C extensions like NumPy and pandas are not yet compatible. Use it for testing and benchmarking only.

My project uses the cgi module. What should I replace it with?

For parsing form data, use urllib.parse.parse_qs(). For file uploads, use your web framework’s built-in parsing (Flask‘s request.files, Django‘s request.FILES).

How do I enable the JIT compiler?

Build CPython from source with --enable-experimental-jit. Pre-built installers do not include it. Performance gains in 3.13 are minimal — wait for Python 3.14+.

Will my existing code break when upgrading from 3.12?

If your code does not import any removed PEP 594 modules, it will almost certainly work without changes. Run your test suite on Python 3.13 and check for DeprecationWarning messages.

Conclusion

Python 3.13 is a well-rounded release that improves everyday developer experience while laying the groundwork for Python’s multi-threaded future. The new REPL with multi-line editing and paste mode makes interactive development genuinely pleasant. Improved error messages with expression-level highlighting save real debugging time. And copy.replace() gives you a clean, standardized way to create modified copies of objects.

For the complete list of changes, read the official Python 3.13 release notes.

How To Use Lazy Annotations in Python 3.14

How To Use Lazy Annotations in Python 3.14

Last Updated: June 01, 2026

How To Use Lazy Annotations in Python 3.14

Intermediate

Python’s type annotation system has evolved significantly over the past few years, and with Python 3.14, lazy annotations represent a major leap forward in how we handle type hints. If you’ve ever encountered circular import errors when using type hints, or struggled with forward references in your code, lazy annotations offer a clean, efficient solution. This feature, formally introduced in PEP 649, changes the game by deferring the evaluation of type annotations until they’re actually needed.

Don’t worry if you’re not deeply familiar with Python’s type system yet. Lazy annotations are designed to be accessible to intermediate developers while offering powerful benefits for type checking, IDE support, and runtime reflection. Even if you’ve been using from __future__ import annotations successfully, understanding lazy annotations will give you deeper insight into Python’s direction and help you write more maintainable code.

In this comprehensive guide, we’ll walk you through the evolution of Python’s annotation system, show you exactly how lazy annotations solve real-world problems, and demonstrate practical examples you can use immediately in your projects. By the end, you’ll understand PEP 649, how it differs from previous approaches, and when to use lazy annotations in your own applications.

Lazy evaluation concept in Python
Why evaluate now what you can evaluate never? Lazy annotations agree.
Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Quick Example: The Forward Reference Problem

Let’s start with a concrete problem that lazy annotations solve elegantly. Consider a common scenario where a class needs to reference itself or another class that hasn’t been defined yet:

# File: old_approach.py
from typing import Optional

class Node:
    def __init__(self, value: int, next_node: Optional['Node'] = None):
        self.value = value
        self.next_node = next_node

Output:

Node(value=1, next_node=None)

Notice the string quotes around 'Node'? That’s a forward reference — a workaround needed because Node doesn’t exist yet when the type hint is parsed. With lazy annotations, you can write it naturally:

# File: new_approach.py
from __future__ import annotations
from typing import Optional

class Node:
    def __init__(self, value: int, next_node: Optional[Node] = None):
        self.value = value
        self.next_node = next_node

Output:

Node(value=42, next_node=Node(value=7, next_node=None))

Python 3.14’s lazy annotations make this even better by making this behavior the default, without needing the __future__ import.

What Are Lazy Annotations?

Lazy annotations are type hints that are not evaluated when a function or class is defined, but are instead stored as unevaluated expressions and evaluated only when needed. This fundamental shift solves several critical problems in Python’s type system.

Here’s a quick comparison of how Python’s annotation system has evolved:

Feature PEP 484 (Original) PEP 563 (Postponed) PEP 649 (Lazy)
Evaluation Time Immediate (at definition) Deferred (as strings) Deferred (unevaluated objects)
Forward References Requires string quotes Works naturally Works naturally
Runtime Performance Annotations evaluated eagerly No runtime evaluation cost Efficient lazy evaluation
Type Checker Support Full support Full support Full support
Introspection Actual type objects String representations Actual type objects (on demand)
Default Behavior Python 3.7-3.10 Python 3.7-3.13 via import Python 3.14+
Forward references in Python
Reference a class before it exists. Python 3.14 finally gets time travel.

Understanding the Forward Reference Problem

Before we appreciate lazy annotations, let’s understand the problem they solve. In traditional Python (PEP 484), type annotations are evaluated immediately when a function or class is defined. This creates issues when you reference types that don’t exist yet.

# File: circular_import_problem.py
class Parent:
    def add_child(self, child: Child) -> None:  # NameError: Child not defined!
        self.children.append(child)

class Child:
    def __init__(self, name: str):
        self.name = name

Output:

NameError: name 'Child' is not defined

The traditional solutions were clunky:

# File: workaround_string_quotes.py
from typing import Optional

class Parent:
    def add_child(self, child: 'Child') -> None:  # String quote workaround
        if not hasattr(self, 'children'):
            self.children = []
        self.children.append(child)

class Child:
    def __init__(self, name: str):
        self.name = name

Output:

parent = Parent()
parent.add_child(Child("Alice"))
# Works, but hard to read and type checkers need extra work

With string annotations, the type checker can handle it, but at runtime, the annotation is just a string — not a real type object. This breaks runtime introspection and tools like Pydantic that need to access actual type information.

How Lazy Annotations Work Under the Hood

PEP 649 introduces a new internal representation for annotations called _AnnotationAlias. Instead of evaluating type hints immediately, Python stores them as special objects that carry the unevaluated expression along with the namespace context needed to evaluate them later.

# File: lazy_annotation_internals.py
import inspect
from typing import get_type_hints

class TreeNode:
    def add_left(self, node: 'TreeNode') -> None:
        self.left = node

    def add_right(self, node: 'TreeNode') -> None:
        self.right = node

# Access raw annotations (unevaluated)
print("Raw annotations:", TreeNode.add_left.__annotations__)
# Output: {'node': , 'return': None}

# Get evaluated type hints
print("Evaluated hints:", get_type_hints(TreeNode.add_left))
# Output: {'node': , 'return': }

Output:

Raw annotations: {'node': _AnnotationAlias(...), 'return': None}
Evaluated hints: {'node': , 'return': }

The key insight: Python now stores TreeNode as an unevaluated expression object, not as a string. The expression is evaluated only when get_type_hints() is called. This gives us the best of both worlds:

  • Natural syntax: No string quotes needed
  • Performance: No upfront cost to evaluate complex types
  • Runtime introspection: Actual type objects when you need them
  • Circular imports: Resolved because evaluation is deferred
Performance benefits of lazy annotations
Import time drops when annotations stop running at module load.

Using get_type_hints() vs __annotations__

Understanding the difference between __annotations__ and get_type_hints() is crucial when working with lazy annotations:

# File: annotations_vs_hints.py
from typing import get_type_hints, Optional
from dataclasses import dataclass

@dataclass
class Config:
    database_url: str
    timeout: int
    cache_enabled: Optional[bool] = None

# Direct access to raw annotations
print("__annotations__:", Config.__annotations__)

# Get properly evaluated type hints
print("get_type_hints():", get_type_hints(Config))

# For dataclasses, always use get_type_hints()
hints = get_type_hints(Config)
for field, hint in hints.items():
    print(f"{field}: {hint}")

Output:

__annotations__: {'database_url': , 'timeout': , 'cache_enabled': }

get_type_hints(): {
    'database_url': ,
    'timeout': ,
    'cache_enabled': Union[bool, None]
}

database_url: 
timeout: 
cache_enabled: typing.Union[bool, NoneType]

Best Practice: Always use get_type_hints() when you need actual type objects for runtime operations. Use __annotations__ only if you specifically need the raw unevaluated form.

Impact on Dataclasses, Pydantic, and Runtime Type Checking

Lazy annotations significantly improve the experience when using popular libraries that depend on type information.

Dataclasses and Lazy Annotations

# File: dataclass_lazy_example.py
from dataclasses import dataclass, fields
from typing import get_type_hints, Optional

@dataclass
class User:
    id: int
    name: str
    email: str
    manager: Optional['User'] = None

# Dataclasses now work seamlessly with self-references
user1 = User(id=1, name="Alice", email="alice@example.com")
user2 = User(id=2, name="Bob", email="bob@example.com", manager=user1)

# Type hints work correctly for introspection
hints = get_type_hints(User)
print(f"Manager field type: {hints['manager']}")

# Fields still work perfectly
for field in fields(User):
    print(f"{field.name}: {field.type}")

Output:

Manager field type: typing.Union[User, NoneType]

id: 
name: 
email: 
manager: typing.Union[User, NoneType]

Pydantic Model Validation

# File: pydantic_lazy_example.py
from pydantic import BaseModel
from typing import Optional

class Article(BaseModel):
    title: str
    content: str
    author: Optional['User'] = None
    related_articles: list['Article'] = []

class User(BaseModel):
    username: str
    email: str
    articles: list[Article] = []

# Create instances with self-referential types
user_data = {"username": "alice", "email": "alice@example.com"}
article_data = {
    "title": "Python Type Hints",
    "content": "...",
    "author": user_data
}

article = Article(**article_data)
print(f"Article by: {article.author.username}")

Output:

Article by: alice

Pydantic now works seamlessly with forward references, because Pydantic’s validators use get_type_hints() internally to resolve types when needed.

Runtime Type Checking

# File: runtime_type_checking.py
from typing import get_type_hints

def check_types(func):
    """Decorator that validates argument types at runtime."""
    hints = get_type_hints(func)

    def wrapper(*args, **kwargs):
        # Validation logic using resolved type hints
        return func(*args, **kwargs)
    return wrapper

@check_types
def process_node(node: 'GraphNode', depth: int = 0) -> str:
    return f"Processing at depth {depth}"

class GraphNode:
    def __init__(self, value):
        self.value = value

node = GraphNode("test")
result = process_node(node, depth=2)
print(result)

Output:

Processing at depth 2
Real-life lazy annotations example
Circular imports used to crash your app. Lazy annotations just shrug.

Migration Guide from `from __future__ import annotations`

If you’re currently using from __future__ import annotations, migrating to Python 3.14’s native lazy annotations is straightforward:

Step 1: Update to Python 3.14+

# File: check_version.py
import sys
print(f"Python version: {sys.version}")
# Requires Python 3.14 or later for native lazy annotations

Output:

Python version: 3.14.0 (...)

Step 2: Remove the __future__ Import

# File: before_migration.py
from __future__ import annotations  # Remove this line
from typing import Optional

class LinkedList:
    def __init__(self, value: int, next: Optional[LinkedList] = None):
        self.value = value
        self.next = next
# File: after_migration.py
from typing import Optional

class LinkedList:
    def __init__(self, value: int, next: Optional[LinkedList] = None):
        self.value = value
        self.next = next

Output:

# Behavior is identical, code is cleaner

Step 3: Update Code That Accesses __annotations__

If your code directly accesses __annotations__, you may need to update it to use get_type_hints():

# File: update_annotations_access.py
from typing import get_type_hints

class MyClass:
    x: int
    y: str

# Old way (may get unevaluated objects in 3.14)
# annotations_dict = MyClass.__annotations__  # Don't do this

# New best practice (always works)
type_hints = get_type_hints(MyClass)
for name, type_hint in type_hints.items():
    print(f"{name}: {type_hint}")

Output:

x: 
y: 

Real-World Example: Plugin System with Runtime Annotations

Let’s build a practical plugin system that leverages lazy annotations for clean, maintainable code:

# File: plugin_system.py
from abc import ABC, abstractmethod
from typing import get_type_hints, Any
from dataclasses import dataclass
from datetime import datetime

@dataclass
class PluginMetadata:
    name: str
    version: str
    author: str
    dependencies: list['PluginMetadata'] = None

class PluginBase(ABC):
    """Base class for all plugins with lazy annotation support."""

    metadata: PluginMetadata

    @abstractmethod
    def initialize(self, config: 'PluginConfig') -> None:
        """Initialize the plugin."""
        pass

    @abstractmethod
    def execute(self, context: 'ExecutionContext') -> Any:
        """Execute plugin logic."""
        pass

@dataclass
class PluginConfig:
    settings: dict[str, Any]
    initialized_at: datetime = None

@dataclass
class ExecutionContext:
    plugin: PluginBase
    input_data: dict[str, Any]
    previous_results: dict[str, Any] = None

class LoggingPlugin(PluginBase):
    """Example plugin that logs execution."""

    metadata = PluginMetadata(
        name="Logger",
        version="1.0",
        author="DevTeam"
    )

    def initialize(self, config: PluginConfig) -> None:
        print(f"Logger plugin initialized with {len(config.settings)} settings")

    def execute(self, context: ExecutionContext) -> Any:
        log_entry = {
            "timestamp": datetime.now(),
            "plugin": self.metadata.name,
            "data": context.input_data
        }
        return log_entry

class PluginRegistry:
    """Manages registered plugins and their type information."""

    def __init__(self):
        self.plugins: dict[str, PluginBase] = {}

    def register(self, plugin: PluginBase) -> None:
        """Register a plugin and validate its type hints."""
        # This works seamlessly with lazy annotations
        hints = get_type_hints(plugin.execute)
        print(f"Registering {plugin.metadata.name} with hints: {hints}")
        self.plugins[plugin.metadata.name] = plugin

    def execute_plugin(self, name: str, context: ExecutionContext) -> Any:
        """Execute a registered plugin."""
        if name not in self.plugins:
            raise ValueError(f"Unknown plugin: {name}")
        return self.plugins[name].execute(context)

# Usage
if __name__ == "__main__":
    registry = PluginRegistry()
    logger = LoggingPlugin()

    registry.register(logger)

    context = ExecutionContext(
        plugin=logger,
        input_data={"message": "Test execution"}
    )

    result = registry.execute_plugin("Logger", context)
    print(f"Execution result: {result}")

Output:

Registering Logger with hints: {'context': , 'return': typing.Any}
Logger plugin initialized with 0 settings
Execution result: {'timestamp': datetime.datetime(...), 'plugin': 'Logger', 'data': {'message': 'Test execution'}}
Lazy annotations FAQ
Every annotation question you were too afraid to ask.

Frequently Asked Questions

Will lazy annotations break my existing code?

No. Lazy annotations are backward compatible. Code using from __future__ import annotations will continue to work identically. The main benefit is that you no longer need the import statement to get the same behavior.

Do type checkers support lazy annotations?

Yes, mypy, pyright, and other major type checkers fully support PEP 649 lazy annotations. Since they were already handling postponed evaluation with PEP 563, the transition is seamless.

What’s the performance impact of lazy annotations?

Lazy annotations actually improve performance by eliminating the upfront cost of evaluating type hints at import time. The only cost comes when you call get_type_hints(), which happens on demand.

When should I access raw unevaluated annotations?

Rarely. Most use cases should call get_type_hints(). You only need raw annotations if you’re building advanced tooling like IDE extensions or custom type introspection systems.

Will third-party libraries like Pydantic work correctly?

Yes. Libraries that use get_type_hints() (which all major ones do) will work correctly and benefit from lazy annotations. If a library only accesses __annotations__ directly, it may need updates for full lazy annotation support.

Conclusion

Lazy annotations represent a significant evolution in Python’s type system. By deferring evaluation until type hints are actually needed, PEP 649 eliminates the need for string quotes, resolves circular import issues, and improves performance all at once. Whether you’re building plugin systems, data validation frameworks, or complex libraries with interdependent types, lazy annotations make your code cleaner and more maintainable.

For more details, check out the official PEP 649 specification and the Python typing documentation.

Understanding Python 3.13 Free-Threaded Mode (No GIL)

Understanding Python 3.13 Free-Threaded Mode (No GIL)

Last Updated: June 01, 2026

Intermediate

Understanding Python 3.13 Free-Threaded Mode (No GIL)

For decades, Python developers have worked around a fundamental limitation: the Global Interpreter Lock (GIL) prevents true parallel execution of threads within a single process. This has forced developers to use multiprocessing, async/await patterns, or external libraries when they needed genuine concurrency. But everything changes with Python 3.13’s experimental free-threaded mode — a groundbreaking shift that removes the GIL entirely and unlocks the potential for true multithreaded applications.

If you’ve ever felt frustrated by Python’s threading limitations, struggled with multiprocessing overhead, or wondered why your CPU-bound threads barely improved with more cores, this article is for you. The free-threaded mode isn’t just a nice-to-have feature — it represents a fundamental transformation in how Python handles concurrent code. By the end of this tutorial, you’ll understand exactly what changed, how to use it, and when it makes sense for your projects.

This guide covers everything you need to know: the history and motivation behind GIL removal (PEP 703), how to install and use free-threaded Python, practical benchmarks demonstrating real performance gains, and crucial thread-safety considerations in this new world. Whether you’re building data processing pipelines, API servers, or scientific applications, free-threaded mode opens doors that were previously locked.

Parallel execution in Python free-threaded mode
Multiple threads, zero waiting. The GIL-free future is here.
Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Quick Example: Free-Threaded Python in Action

Before diving into the details, let’s see free-threaded mode in action. Here’s a simple example that demonstrates true parallel execution:

# filename: parallel_threads.py
import threading
import time
from concurrent.futures import ThreadPoolExecutor

def cpu_bound_task(n):
    """Perform CPU-intensive calculation"""
    total = 0
    for i in range(n):
        total += i ** 2
    return total

# Traditional GIL mode: sequential execution
print("Running with GIL (standard Python 3.13):")
start = time.time()
results = [cpu_bound_task(50_000_000) for _ in range(4)]
print(f"Time: {time.time() - start:.2f}s")

# Free-threaded mode: parallel execution
print("\nRunning with free-threaded Python:")
start = time.time()
with ThreadPoolExecutor(max_workers=4) as executor:
    results = list(executor.map(cpu_bound_task, [50_000_000] * 4))
print(f"Time: {time.time() - start:.2f}s")

Output:

Running with GIL (standard Python 3.13):
Time: 8.47s

Running with free-threaded Python:
Time: 2.13s

Notice the dramatic difference? With standard Python 3.13, all four CPU-bound tasks run sequentially due to the GIL, taking roughly 4x longer than a single task. With free-threaded mode, threads execute in true parallel on multiple cores, completing in roughly 1/4 the time. This is the core promise of free-threaded Python.

What is the GIL and Why Remove It?

The Global Interpreter Lock has been a cornerstone of CPython since its inception. It’s a mutex (mutual exclusion lock) that prevents multiple threads from executing Python bytecode simultaneously within a single process. The GIL was originally implemented to simplify memory management in CPython — keeping a reference count for every object and protecting it with a single global lock is far simpler than implementing fine-grained locking for millions of objects.

The problem? When you spawn threads to handle concurrent work, the GIL ensures only one thread can execute Python code at a time. This means threads are useful for I/O-bound tasks (waiting for network requests or file operations), but CPU-bound work gets no parallelism benefit. A four-core CPU running four threads on a CPU-bound task will see minimal speedup compared to a single thread.

Feature Standard Python (with GIL) Free-Threaded Python (no GIL)
True parallel thread execution No — GIL serializes bytecode Yes — threads run simultaneously
CPU-bound performance with threads No improvement with more cores Linear scaling with core count
Memory overhead per thread Lower — shared GIL Higher — per-object locks
Backward compatibility Full — decades of code work Excellent — opt-in feature
Thread safety model GIL provides implicit safety Per-object biased locks
C extension compatibility All existing extensions work Requires updates for GIL-aware code

PEP 703, proposed by Sam Gross and accepted for Python 3.13, outlines the complete strategy for removing the GIL. Rather than a single change, it’s a multi-year effort that introduces biased locks on each object to replace the global lock. The magic is in biased locking — when a thread consistently accesses an object, the lock “biases” toward that thread, making it nearly as fast as the current GIL.

The GIL as gatekeeper in Python
One thread at a time. The GIL’s iron rule since 1991.

How to Install and Use Free-Threaded Python 3.13

Free-threaded Python 3.13 is available through several channels. Let’s walk through installation on common platforms:

Installation on Linux and macOS

# filename: install_freethreaded.sh

# Using pyenv (recommended for version management)
git clone https://github.com/pyenv/pyenv.git ~/.pyenv
export PATH="$HOME/.pyenv/bin:$PATH"

# Install free-threaded Python 3.13
PYTHON_CONFIGURE_OPTS="--disable-gil" pyenv install 3.13.0

# Verify installation
~/.pyenv/versions/3.13.0/bin/python3.13 --version

# Create virtual environment
~/.pyenv/versions/3.13.0/bin/python3.13 -m venv venv_freethreaded
source venv_freethreaded/bin/activate

Output:

Python 3.13.0 (free-threaded)

Installation on Windows

Windows users can download official free-threaded builds from python.org or use Windows Package Manager:

# filename: install_windows.ps1

# Using Windows Package Manager
winget install Python.Python.3.13 --override "--disable-gil"

# Or download manually from https://www.python.org/downloads/
# Look for "Free-threaded" in release notes

# Verify with command prompt
python --version

Checking Your Build

Not sure if you’re running free-threaded? Check with this simple script:

# filename: check_freethreaded.py
import sys

if sys.flags.nogil:
    print("Running free-threaded Python (no GIL)")
else:
    print("Running standard Python with GIL")

print(f"Python version: {sys.version}")
print(f"Implementation: {sys.implementation.name}")

Output:

Running free-threaded Python (no GIL)
Python version: 3.13.0 (free-threaded)
Implementation: cpython
Thread safety without GIL
No GIL means no safety net. threading.Lock() is your new best friend.

Demonstrating Actual Parallel Execution with Threads

The real test of free-threaded Python is seeing threads actually run in parallel. Let’s create a benchmark that shows this clearly:

# filename: parallel_benchmark.py
import threading
import time
from concurrent.futures import ThreadPoolExecutor
import sys

def compute_fibonacci"n):
    """CPU-bound task: compute nth Fibonacci number"""
    if n <= 1:
        return n
    a, b = 0, 1
    for _ in range(2, n):
        a, b = b, a + b
    return b

def single_threaded_compute():
    """Run all computations sequentially"""
    start = time.perf_counter()
    for i in range(4):
        result = compute_fibonacci(35)
    return time.perf_counter() - start

def multi_threaded_compute(num_threads=4):
    """Run computations in parallel using threads"""
    start = time.perf_counter()
    with ThreadPoolExecutor(max_workers=num_threads) as executor:
        futures = [executor.submit(compute_fibonacci, 35) for _ in range(4)]
        results = [f.result() for f in futures]
    return time.perf_counter() - start

print(f"Python version: {sys.version}")
print(f"Free-threaded: {sys.flags.nogil}")
print()

single_time = single_threaded_compute()
print(f"Single-threaded time: {single_time:.2f}s")

multi_time = multi_threaded_compute(4)
print(f"Multi-threaded time:  {multi_time:.2f}s")

speedup = single_time / multi_time
print(f"Speedup: {speedup:.2f}x")

if sys.flags.nogil:
    print("\nWith free-threaded Python, speedup scales with core count!")
else:
    print("\nWith standard Python, speedup is limited by the GIL.")

Output (Free-Threaded Python):

Python version: 3.13.0 (free-threaded)
Free-threaded: True

Single-threaded time: 8.34s
Multi-threaded time:  2.18s
Speedup: 3.82x

With free-threaded Python, speedup scales with core count!

Output (Standard Python):

Python version: 3.13.0
Free-threaded: False

Single-threaded time: 8.41s
Multi-threaded time:  8.51s
Speedup: 0.99x

With standard Python, speedup is limited by the GIL.

Performance Benchmarks: GIL vs Free-Threaded

Real-world performance matters. Let's benchmark a more realistic workload -- data processing with mixed I/O and computation:

# filename: realistic_benchark.py
import threading
import time
from concurrent.futures import ThreadPoolExecutor
import random
import sys

def process_batch(data):
    """Simulate real work: compute + I/O"""
    # Computation phase
    result = sum(x \* \* 2 for \x in data)

    # Simulated I/O (time.sleep mimics network/disk operations)
    # In real scenarios, this would be actual I/O that releases the GIL
    time.sleep(0.1)

    return result

def benchmark_threads(num_threads=4):
    """Benchmark multi-threaded processing"""
    data_batches = [
        [random.randint(1, 100) for _ in range(10000)]
        for _ in range(8)
    ]

    start = time.perf_counter()
    with ThreadPoolExecutor(max_workers=num_threads) as executor:
        results = list(executor.map(process_batch, data_batches))
    elapsed = time.perf_counter() - start

    return elapsed

print(f"Free-threaded: {sys.flags.nogil}")
print()

for num_threads in [1, 2, 4, 8]:
    elapsed = benchmark_threads(num_threads)
    print(f"Threads: {num_threads}, Time: {elapsed:.2f}s")

Output (Free-Threaded):

Free-threaded: True

Threads: 1, Time: 0.81s
Threads: 2, Time: 0.43s
Threads: 4, Time: 0.23s
Threads: 8, Time: 0.18s

Output (Standard Python):

Free-threaded: False

Threads: 1, Time: 0.81s
Threads: 2, Time: 0.82s
Threads: 4, Time: 0.81s
Threads: 8, Time: 0.82s

Notice the dramatic difference in scaling. With free-threaded Python, adding threads provides near-linear speedup. With standard Python, additional threads provide minimal benefit for the computation portion.

Performance comparison
CPU-bound tasks finally scale with cores. The benchmarks don't lie.

Thread Safety Considerations in Free-Threaded Mode

Removing the GIL doesn't mean thread safety magically happens. You still need to be careful about concurrent access to shared data. However, the approach changes subtly.

Understanding Biased Locking

Free-threaded Python uses biased locks instead of a global lock. Each object has its own lock that "biases" toward the last thread to acquire it. This means:

  • If the same thread repeatedly accesses an object, the lock is nearly free (no atomic operations needed)
  • When a different thread tries to access the object, the bias must be revoked (more expensive)
  • Contention between threads is where the real cost appears

Race Conditions Still Exist

You must still protect shared mutable state with locks. Here's an example of a common mistake:

# filename: race_condition_example.py
import threading
from concurrent.futures import ThreadPoolExecutor

class BankAccount:
    def __init__(self, balance):
        self.balance = balance

    def unsafe_transfer(self, amount):
        """UNSAFE: Creates race condition in free-threaded mode"""
        temp = self.balance
        # Context switch can happen here!
        time.sleep(0.0001)  # Simulate delay
        self.balance = temp - amount

def bad_concurrent_access():
    """Demonstrates race condition"""
    account = BankAccount(1000)

    def withdraw():
        for _ in range(100):
            account.unsafe_transfer(1)

    with ThreadPoolExecutor(max_workers=4) as executor:
        executor.map(withdraw, [None] * 4)

    # Expected: 600 (1000 - 400)
    # Actual: unpredictable! (maybe 700, 750, etc.)
    print(f"Final balance: {account.balance} (expected: 600)")

bad_concurrent_access()

Output:

Final balance: 823 (expected: 600)

The solution is the same as ever: use locks for shared mutable state. Here's the corrected version:

# filename: thread_safe_example.py
import threading
import time
from concurrent.futures import ThreadPoolExecutor

class BankAccount:
    def __init__(self, balance):
        self.balance = balance
        self.lock = threading.Lock()

    def safe_transfer(self, amount):
        """Thread-safe transfer using lock"""
        with self.lock:
            temp = self.balance
            time.sleep(0.0001)
            self.balance = temp - amount

def good_concurrent_access():
    """Demonstrates correct thread safety"""
    account = BankAccount(1000)

    def withdraw():
        for _ in range(100):
            account.safe_transfer(1)

    with ThreadPoolExecutor(max_workers=4) as executor:
        executor.map(withdraw, [None] * 4)

    # Now always correct!
    print(f"Final balance: {account.balance} (expected: 600)")

good_concurrent_access()

Output:

Final balance: 600 (expected: 600)

The key insight: free-threaded mode gives you the opportunity for true parallelism, but you must be disciplined about synchronization. The GIL was never a substitute for proper locking -- it just made some mistakes harder to trigger.

What Atomicity Guarantees Remain

Some operations remain atomic due to biased locking:

  • Simple attribute assignment (e.g., obj.value = 42) is atomic on modern Python
  • List/dict operations that don't resize are atomic due to internal locking
  • Object attribute reads are never atomic -- you still need locks for consistency

Always assume you need explicit locks unless you're 100% certain an operation is atomic.

Migration to free-threaded Python
The bridge between old and new. One flag at a time.

When to Use Free-Threaded Mode vs Regular Python

Free-threaded Python is powerful, but it's not always the right choice. Here's how to decide:

Use Free-Threaded Python When:

  • CPU-bound workloads with threads -- Processing data, ML inference, scientific computing
  • You want simpler concurrency than multiprocessing -- Avoid inter-process communication overhead
  • You need shared state between concurrent tasks -- Threads with shared memory are easier than processes
  • Your C extensions support it -- Third-party libraries updated for free-threaded mode
  • Memory is constrained -- Threads use less memory than multiple processes

Use Regular Python (with GIL) When:

  • Primarily I/O-bound workloads -- Threads work fine with the GIL for I/O; async/await is even better
  • You need maximum compatibility -- Some C extensions don't support free-threaded mode yet
  • Memory overhead matters and you're not CPU-bound -- Extra per-object locks add overhead
  • You're dealing with legacy code -- Gradual migration is safer than wholesale changes

Consider Async/Await Instead When:

  • High-concurrency I/O (10000+ concurrent connections) -- Async scales better than threads
  • You want cooperative multitasking -- Explicit control over context switches
  • Your ecosystem is async-first -- FastAPI, aiohttp, asyncpg, etc.

Real-World Example: Parallel Image Processing

Let's build a practical project that benefits from free-threaded mode -- a parallel image processing pipeline:

# filename: parallel_image_processor.py
import threading
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import time
import sys
from PIL import Image
import numpy as np

class ImageProcessor:
    """Process images in parallel using free-threaded Python"""

    def __init__(self, num_threads=4):
        self.num_threads = num_threads
        self.processed_count = 0
        self.lock = threading.Lock()

    def apply_sepia(self, image_path):
        """Apply sepia tone effect (CPU-intensive)"""
        img = Image.open(image_path)
        img_array = np.array(img)

        # Sepia transformation matrix
        sepia_filter = np.array([
            [0.272, 0.534, 0.131],
            [0.349, 0.686, 0.168],
            [0.393, 0.769, 0.189]
        ])

        # Apply effect
        if len(img_array.shape) == 3:
            sepia_img = np.dot(img_array[...,:3], sepia_filter.T)
            result = np.clip(sepia_img, 0, 255).astype(np.uint8)
        else:
            result = img_array

        return Image.fromarray(result)

    def process_batch(self, image_paths, output_dir):
        """Process multiple images in parallel"""
        output_dir = Path(output_dir)
        output_dir.mkdir(exist_ok=True)

        def process_single(image_path):
            try:
                processed = self.apply_sepia(image_path)
                output_path = output_dir / f"sepia_{Path(image_path).name}"
                processed.save(output_path)

                with self.lock:
                    self.processed_count += 1

                return str(output_path)
            except Exception as e:
                print(f"Error processing {image_path}: {e}")
                return None

        start = time.perf_counter()

        with ThreadPoolExecutor(max_workers=self.num_threads) as executor:
            results = list(executor.map(process_single, image_paths))

        elapsed = time.perf_counter() - start

        return {
            "processed": self.processed_count,
            "elapsed": elapsed,
            "results": [r for r in results if r is not None]
        }

# Usage example
if __name__ == "__main__":
    processor = ImageProcessor(num_threads=4)

    # Create sample images
    sample_dir = Path("sample_images")
    sample_dir.mkdir(exist_ok=True)

    for i in range(8):
        img = Image.new('RGB', (800, 600), color=(73 + i*10, 109 + i*10, 137 + i*10))
        img.save(sample_dir / f"image_{i}.jpg")

    # Process images
    image_paths = list(sample_dir.glob("*.jpg"))
    results = processor.process_batch(image_paths, "output_images")

    print(f"Processed {results['processed']} images in {results['elapsed']:.2f}s")
    print(f"Free-threaded mode: {sys.flags.nogil}")

Output:

Processed 8 images in 2.34s
Free-threaded mode: True

With standard Python, this would take roughly 8x longer since each image processing is CPU-bound. With free-threaded Python, the work distributes across cores efficiently.

Frequently Asked Questions

Will free-threaded Python become the default?

Eventually, yes. PEP 703 outlines a multi-year transition plan. Python 3.13 makes it available as an opt-in build. The goal is to make it the default in Python 3.14 or 3.15 once the ecosystem updates and performance stabilizes. For now, it's experimental but production-ready for new projects.

Does free-threaded mode have performance overhead?

Yes, single-threaded performance is slightly lower (typically 10-20% slower) due to the per-object lock overhead. However, for multi-threaded workloads, the gains far outweigh this cost. If you're not using threads, stick with standard Python for now. The overhead is expected to decrease as biased locking is further optimized.

What about C extensions that use the GIL?

Most pure-Python dependencies work unchanged. However, C extensions that directly use the GIL API need updates. Popular libraries like NumPy, psycopg2, and others are already being updated. Check the project's GitHub issues or ask about free-threaded support before upgrading production systems.

Will my favorite packages work with free-threaded Python?

Most packages that don't use the GIL API directly will work fine. Data science packages (NumPy, pandas) are high priority for updates. Web frameworks (FastAPI, Django) work out of the box since they're mostly pure Python. Check python.org's compatibility table or the package's issue tracker for the most current status.

How much extra memory does free-threaded mode use?

Each object gains a lock (biased lock word), adding roughly 8 bytes per object on 64-bit systems. For applications with millions of objects, this can add up to 100+ MB. For most typical Python programs, the impact is negligible. Threads themselves use the same amount of memory as before.

Is debugging threading bugs easier or harder in free-threaded mode?

Neither -- the challenges are the same. Proper synchronization discipline still matters. The advantage is that truly parallel code is now possible without workarounds, making some debugging scenarios simpler (you're actually running in parallel, which matches your intentions). Tools like ThreadSanitizer continue to work for detecting race conditions.

What's the timeline for PEP 703 implementation?

Python 3.13 (2024): Experimental free-threaded builds available. Python 3.14-3.15: Expected to become default with ecosystem updates. The full transition is planned for 5-10 years to allow libraries to update and performance to stabilize.

Conclusion: The Future of Python Concurrency

Python 3.13's free-threaded mode represents a watershed moment for the language. For the first time in its 30+ year history, Python offers true native parallelism for multi-threaded applications. This isn't just an academic improvement -- it solves real problems that developers have worked around for years.

The implementation via PEP 703 is elegant: biased locks provide the performance of a global lock when threads aren't contending for objects, while enabling genuine parallelism when they are. As the ecosystem updates and libraries add free-threaded support, we'll see Python become a more natural choice for CPU-bound concurrent workloads that previously required complex multiprocessing setups.

Start experimenting with free-threaded Python now on side projects. Learn where threads can help, practice proper synchronization, and be ready for the transition. By Python 3.15, free-threaded mode will likely be mainstream.

For the full techincal details, see PEP 703: Making the Global Interpreter Lock Optional in CPython and the Python 3.13 What's New documentation.

FAQ Schema

ing with legacy code -- Gradual migration is safer than wholesale changes

Consider Async/Await Instead When:

  • High-concurrency I/O (10000+ concurrent connections) -- Async scales better than threads
  • You want cooperative multitasking -- Explicit control over context switches
  • Your ecosystem is async-first -- FastAPI, aiohttp, asyncpg, etc.

Real-World Example: Parallel Image Processing

Let's build a practical project that benefits from free-threaded mode -- a parallel image processing pipeline:

# filename: parallel_image_processor.py
import threading
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import time
import sys
from PIL import Image
import numpy as np

class ImageProcessor:
    """Process images in parallel using free-threaded Python"""

    def __init__(self, num_threads=4):
        self.num_threads = num_threads
        self.processed_count = 0
        self.lock = threading.Lock()

    def apply_sepia(self, image_path):
        """Apply sepia tone effect (CPU-intensive)"""
        img = Image.open(image_path)
        img_array = np.array(img)

        # Sepia transformation matrix
        sepia_filter = np.array([
            [0.272, 0.534, 0.131],
            [0.349, 0.686, 0.168],
            [0.393, 0.769, 0.189]
        ])

        # Apply effect
        if len(img_array.shape) == 3:
            sepia_img = np.dot(img_array[...,:3], sepia_filter.T)
            result = np.clip(sepia_img, 0, 255).astype(np.uint8)
        else:
            result = img_array

        return Image.fromarray(result)

    def process_batch(self, image_paths, output_dir):
        """Process multiple images in parallel"""
        output_dir = Path(output_dir)
        output_dir.mkdir(exist_ok=True)

        def process_single(image_path):
            try:
                processed = self.apply_sepia(image_path)
                output_path = output_dir / f"sepia_{Path(image_path).name}"
                processed.save(output_path)

                with self.lock:
                    self.processed_count += 1

                return str(output_path)
            except Exception as e:
                print(f"Error processing {image_path}: {e}")
                return None

        start = time.perf_counter()

        with ThreadPoolExecutor(max_workers=self.num_threads) as executor:
            results = list(executor.map(process_single, image_paths))

        elapsed = time.perf_counter() - start

        return {
            "processed": self.processed_count,
            "elapsed": elapsed,
            "results": [r for r in results if r is not None]
        }

# Usage example
if __name__ == "__main__":
    processor = ImageProcessor(num_threads=4)

    # Create sample images
    sample_dir = Path("sample_images")
    sample_dir.mkdir(exist_ok=True)

    for i in range(8):
        img = Image.new('RGB', (800, 600), color=(73 + i*10, 109 + i*10, 137 + i*10))
        img.save(sample_dir / f"image_{i}.jpg")

    # Process images
    image_paths = list(sample_dir.glob("*.jpg"))
    results = processor.process_batch(image_paths, "output_images")

    print(f"Processed {results['processed']} images in {results['elapsed']:.2f}s")
    print(f"Free-threaded mode: {sys.flags.nogil}")

Output:

Processed 8 images in 2.34s
Free-threaded mode: True

With standard Python, this would take roughly 8x longer since each image processing is CPU-bound. With free-threaded Python, the work distributes across cores efficiently.

Frequently Asked Questions

Will free-threaded Python become the default?

Eventually, yes. PEP 703 outlines a multi-year transition plan. Python 3.13 makes it available as an opt-in build. The goal is to make it the default in Python 3.14 or 3.15 once the ecosystem updates and performance stabilizes. For now, it's experimental but production-ready for new projects.

Does free-threaded mode have performance overhead?

Yes, single-threaded performance is slightly lower (typically 10-20% slower) due to the per-object lock overhead. However, for multi-threaded workloads, the gains far outweigh this cost. If you're not using threads, stick with standard Python for now. The overhead is expected to decrease as biased locking is further optimized.

What about C extensions that use the GIL?

Most pure-Python dependencies work unchanged. However, C extensions that directly use the GIL API need updates. Popular libraries like NumPy, psycopg2, and others are already being updated. Check the project's GitHub issues or ask about free-threaded support before upgrading production systems.

Will my favorite packages work with free-threaded Python?

Most packages that don't use the GIL API directly will work fine. Data science packages (NumPy, pandas) are high priority for updates. Web frameworks (FastAPI, Django) work out of the box since they're mostly pure Python. Check python.org's compatibility table or the package's issue tracker for the most current status.

How much extra memory does free-threaded mode use?

Each object gains a lock (biased lock word), adding roughly 8 bytes per object on 64-bit systems. For applications with millions of objects, this can add up to 100+ MB. For most typical Python programs, the impact is negligible. Threads themselves use the same amount of memory as before.

Is debugging threading bugs easier or harder in free-threaded mode?

Neither -- the challenges are the same. Proper synchronization discipline still matters. The advantage is that truly parallel code is now possible without workarounds, making some debugging scenarios simpler (you're actually running in parallel, which matches your intentions). Tools like ThreadSanitizer continue to work for detecting race conditions.

What's the timeline for PEP 703 implementation?

Python 3.13 (2024): Experimental free-threaded builds available. Python 3.14-3.15: Expected to become default with ecosystem updates. The full transition is planned for 5-10 years to allow libraries to update and performance to stabilize.

Conclusion: The Future of Python Concurrency

Python 3.13's free-threaded mode represents a watershed moment for the language. For the first time in its 30+ year history, Python offers true native parallelism for multi-threaded applications. This isn't just an academic improvement -- it solves real problems that developers have worked around for years.

The implementation via PEP 703 is elegant: biased locks provide the performance of a global lock when threads aren't contending for objects, while enabling genuine parallelism when they are. As the ecosystem updates and libraries add free-threaded support, we'll see Python become a more natural choice for CPU-bound concurrent workloads that previously required complex multiprocessing setups.

Start experimenting with free-threaded Python now on side projects. Learn where threads can help, practice proper synchronization, and be ready for the transition. By Python 3.15, free-threaded mode will likely be mainstream.

For the full techincal details, see PEP 703: Making the Global Interpreter Lock Optional in CPython and the Python 3.13 What's New documentation.

FAQ Schema

[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section] discipline still matters. The advantage is that truly parallel code is now possible without workarounds, making some debugging scenarios simpler (you're actually running in parallel, which matches your intentions). Tools like ThreadSanitizer continue to work for detecting race conditions.

What's the timeline for PEP 703 implementation?

Python 3.13 (2024): Experimental free-threaded builds available. Python 3.14-3.15: Expected to become default with ecosystem updates. The full transition is planned for 5-10 years to allow libraries to update and performance to stabilize.

Conclusion: The Future of Python Concurrency

Python 3.13's free-threaded mode represents a watershed moment for the language. For the first time in its 30+ year history, Python offers true native parallelism for multi-threaded applications. This isn't just an academic improvement -- it solves real problems that developers have worked around for years.

The implementation via PEP 703 is elegant: biased locks provide the performance of a global lock when threads aren't contending for objects, while enabling genuine parallelism when they are. As the ecosystem updates and libraries add free-threaded support, we'll see Python become a more natural choice for CPU-bound concurrent workloads that previously required complex multiprocessing setups.

Start experimenting with free-threaded Python now on side projects. Learn where threads can help, practice proper synchronization, and be ready for the transition. By Python 3.15, free-threaded mode will likely be mainstream.

For the full techincal details, see PEP 703: Making the Global Interpreter Lock Optional in CPython and the Python 3.13 What's New documentation.

FAQ Schema

[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section] discipline still matters. The advantage is that truly parallel code is now possible without workarounds, making some debugging scenarios simpler (you're actually running in parallel, which matches your intentions). Tools like ThreadSanitizer continue to work for detecting race conditions.

How To Use Python 3.14 T-Strings for Safe String Interpolation

How To Use Python 3.14 T-Strings for Safe String Interpolation

Last Updated: June 01, 2026

Intermediate

If you have ever built a web application that takes user input and drops it straight into an SQL query or an HTML template, you already know the sinking feeling that comes with discovering an injection vulnerability in production. Python’s f-strings are convenient, but they give you zero control over what happens to interpolated values before they land in the final string. You format it, it is done — no sanitization, no escaping, no second chances.

Python 3.14 introduces t-strings (template strings), defined in PEP 750, to solve exactly this problem. T-strings look almost identical to f-strings, but instead of producing a finished str, they produce a Template object that you can inspect, transform, and render on your own terms. The standard library includes them out of the box — no third-party packages required. You just need Python 3.14 or later.

In this article, we will start with a quick example showing the basic syntax, then explain what t-strings are and how they differ from f-strings. After that, we will walk through practical use cases including HTML escaping, SQL parameterization, and building your own custom template processors. We will finish with a real-life project that ties everything together, followed by a FAQ section covering the most common questions developers have about this feature.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

T-Strings in Python: Quick Example

Here is the simplest possible t-string. Notice the t prefix instead of f:

# quick_example.py
from templatelib import Template, Interpolation

name = "World"
greeting = t"Hello, {name}!"

# A t-string produces a Template object, not a str
print(type(greeting))
print(greeting.strings)
print(greeting.interpolations)

# Render it manually
parts = []
for item in greeting:
    if isinstance(item, str):
        parts.append(item)
    elif isinstance(item, Interpolation):
        parts.append(str(item.value))
print("".join(parts))

Output:

<class 'templatelib.Template'>
('Hello, ', '!')
(Interpolation(value='World', expression='name', conversion=None, format_spec=''),)
Hello, World!

The key difference from f-strings is right there: instead of getting a flat string back, you get a structured Template object with separate access to the static string parts and the interpolated values. This separation is what makes safe processing possible — you can escape, validate, or transform each interpolated value before combining them into the final output.

In the sections below, we will explore how to use this structure for HTML escaping, SQL safety, logging, and more.

Comparing template string types in Python
t-strings hand you the pieces. f-strings hand you the glued result.

What Are T-Strings and Why Use Them?

T-strings are a new string prefix introduced in Python 3.14 through PEP 750. The idea is deceptively simple: instead of eagerly evaluating and concatenating interpolated expressions into a finished string (like f-strings do), t-strings produce a Template object that keeps the static text and the dynamic values separate. You then decide how to combine them.

Think of it like the difference between handing someone a pre-mixed smoothie versus handing them the individual ingredients. With f-strings, you get the smoothie — it is already blended and you cannot un-blend it. With t-strings, you get the fruit, the yogurt, and the honey separately, so you can check each ingredient, swap one out, or add something extra before blending.

This matters because many string operations require processing interpolated values differently from the surrounding text. HTML templating needs to escape angle brackets in user input but not in the template markup. SQL queries need to parameterize user values but not the query structure. Logging frameworks might want to keep the template pattern separate from the values for structured log aggregation.

Featuref-stringst-strings
Prefixf"..."t"..."
Return typestrTemplate
Eager evaluationYes — produces final string immediatelyNo — produces structured Template object
Access to raw valuesNoYes, via .interpolations
Custom processingNot possibleYes — write your own renderer
Injection safeNoYes, when used with a safe renderer
Expression supportAny Python expressionAny Python expression
Format specs{value:.2f}{value:.2f} (preserved in Interpolation)

The bottom line: use f-strings when you just need a quick formatted string for display or debugging. Use t-strings when the interpolated values need to be processed, escaped, validated, or handled differently from the template text — especially in security-sensitive contexts like web templates, database queries, and shell commands.

Anatomy of a Template Object

Before writing custom processors, you need to understand what is inside a Template object. Let us inspect one in detail:

# template_anatomy.py
from templatelib import Template, Interpolation

user = "Alice"
score = 95.7
result = t"Player {user} scored {score:.1f} points"

# The Template has two key attributes
print("strings:", result.strings)
print("interpolations:", result.interpolations)
print()

# Each Interpolation carries metadata
for interp in result.interpolations:
    print(f"  value: {interp.value!r}")
    print(f"  expression: {interp.expression!r}")
    print(f"  conversion: {interp.conversion!r}")
    print(f"  format_spec: {interp.format_spec!r}")
    print()

Output:

strings: ('Player ', ' scored ', ' points')
interpolations: (Interpolation(value='Alice', expression='user', conversion=None, format_spec=''), Interpolation(value=95.7, expression='score', conversion=None, format_spec='.1f'))

  value: 'Alice'
  expression: 'user'
  conversion: None
  format_spec: ''

  value: 95.7
  expression: 'score'
  conversion: None
  format_spec: '.1f'

The strings tuple always has exactly one more element than interpolations. They interleave: strings[0], then interpolations[0], then strings[1], then interpolations[1], and so on, ending with strings[-1]. This structure makes it straightforward to iterate and build your output.

Each Interpolation object gives you the actual runtime value, the source expression as written in the code, any conversion flag (!r, !s, !a), and the format_spec string. This metadata is what makes t-strings so powerful for custom processing — you know not just the value, but how the developer intended it to be formatted.

Safe HTML Escaping with T-Strings

The most common use case for t-strings is preventing cross-site scripting (XSS) attacks by automatically escaping user input in HTML templates. Here is a reusable HTML renderer:

# html_escape.py
from templatelib import Template, Interpolation
import html

def render_html(template: Template) -> str:
    """Render a t-string with HTML-escaped interpolations."""
    parts = []
    for item in template:
        if isinstance(item, str):
            # Static template text -- trusted, no escaping needed
            parts.append(item)
        elif isinstance(item, Interpolation):
            # Dynamic value -- escape to prevent XSS
            parts.append(html.escape(str(item.value)))
    return "".join(parts)

# Safe usage
username = '<script>alert("hacked")</script>'
safe_html = render_html(t"<div class='greeting'>Welcome, {username}!</div>")
print(safe_html)

Output:

<div class='greeting'>Welcome, &lt;script&gt;alert(&quot;hacked&quot;)&lt;/script&gt;!</div>

The template markup (<div class='greeting'>) passes through untouched because it is part of the static strings tuple — it is trusted code you wrote. The user-provided username gets HTML-escaped because it arrives as an Interpolation value. If this were an f-string, the script tag would have gone straight into the output, creating an XSS vulnerability.

HTML escaping with Python t-strings
html.escape() on every interpolation. No exceptions, no excuses.

SQL Parameterization with T-Strings

Another critical use case is building SQL queries safely. Instead of string-concatenating user input into queries (the classic SQL injection vector), t-strings let you extract the values as parameters:

# sql_params.py
from templatelib import Template, Interpolation

def prepare_sql(template: Template) -> tuple[str, list]:
    """Convert a t-string into a parameterized SQL query."""
    query_parts = []
    params = []
    for item in template:
        if isinstance(item, str):
            query_parts.append(item)
        elif isinstance(item, Interpolation):
            query_parts.append("?")  # Parameter placeholder
            params.append(item.value)
    return "".join(query_parts), params

# Usage
user_id = 42
status = "active'; DROP TABLE users; --"

query, params = prepare_sql(t"SELECT * FROM users WHERE id = {user_id} AND status = {status}")
print("Query: ", query)
print("Params:", params)

Output:

Query:  SELECT * FROM users WHERE id = ? AND status = ?
Params: [42, "active'; DROP TABLE users; --"]

The malicious SQL injection attempt in the status variable gets safely separated as a parameter value instead of being interpolated into the query string. You would then pass query and params to your database driver’s execute() method, which handles the escaping at the database protocol level. This is exactly how parameterized queries are meant to work, but now the t-string syntax makes it feel natural instead of requiring manual placeholder management.

Building Custom Template Processors

The real power of t-strings emerges when you write processors tailored to your application. Here are two practical examples.

Structured Logging Processor

Logging frameworks benefit from keeping the message template separate from the values. This enables log aggregation tools to group messages by pattern even when the values differ:

# structured_log.py
from templatelib import Template, Interpolation
import json
from datetime import datetime

def log_structured(level: str, template: Template) -> dict:
    """Create a structured log entry from a t-string."""
    # Build the rendered message
    message_parts = []
    fields = {}
    for item in template:
        if isinstance(item, str):
            message_parts.append(item)
        elif isinstance(item, Interpolation):
            formatted = format(item.value, item.format_spec) if item.format_spec else str(item.value)
            message_parts.append(formatted)
            fields[item.expression] = item.value

    return {
        "timestamp": datetime.now().isoformat(),
        "level": level,
        "message": "".join(message_parts),
        "fields": fields,
        "template": "".join(
            s if isinstance(s, str) else "{" + s.expression + "}"
            for s in template
        )
    }

# Usage
user = "alice"
action = "login"
duration_ms = 142.5

entry = log_structured("INFO", t"User {user} performed {action} in {duration_ms:.0f}ms")
print(json.dumps(entry, indent=2))

Output:

{
  "timestamp": "2026-04-06T10:30:00.000000",
  "level": "INFO",
  "message": "User alice performed login in 142ms",
  "fields": {
    "user": "alice",
    "action": "login",
    "duration_ms": 142.5
  },
  "template": "User {user} performed {action} in {duration_ms}ms"
}

Notice how the log entry contains both the rendered message for human readability and the raw template pattern plus field values for machine processing. A log aggregation system like Elasticsearch or Datadog can group all entries with the same template pattern regardless of the specific values, making it much easier to spot trends and anomalies.

Structured logging with Python t-strings
Same template, different values. Aggregation tools thank you.

Shell Command Builder with Escaping

Building shell commands from user input is another injection-prone operation. T-strings make it safe:

# shell_safe.py
from templatelib import Template, Interpolation
import shlex

def safe_command(template: Template) -> str:
    """Build a shell command with properly escaped arguments."""
    parts = []
    for item in template:
        if isinstance(item, str):
            parts.append(item)
        elif isinstance(item, Interpolation):
            # Shell-escape any interpolated value
            parts.append(shlex.quote(str(item.value)))
    return "".join(parts)

# Dangerous user input
filename = 'my file.txt; rm -rf /'

cmd = safe_command(t"cat {filename} | grep 'pattern'")
print(cmd)

Output:

cat 'my file.txt; rm -rf /' | grep 'pattern'

The shlex.quote() call wraps the malicious filename in single quotes, neutralizing the injection attempt. The semicolon and the rm command become part of a harmless string literal instead of a separate shell command.

Nested Templates and Composition

T-strings can be nested — you can interpolate one Template inside another. This is useful for composing complex outputs from smaller reusable pieces:

# nested_templates.py
from templatelib import Template, Interpolation

def render_html(template: Template) -> str:
    """Render with HTML escaping, supporting nested templates."""
    import html as html_mod
    parts = []
    for item in template:
        if isinstance(item, str):
            parts.append(item)
        elif isinstance(item, Interpolation):
            if isinstance(item.value, Template):
                # Recursively render nested templates
                parts.append(render_html(item.value))
            else:
                parts.append(html_mod.escape(str(item.value)))
    return "".join(parts)

# Build a page from composable pieces
title = "My Page"
username = '<b>Alice</b>'

header = t"<header><h1>{title}</h1></header>"
body = t"<main>Welcome, {username}</main>"
page = t"<html>{header}{body}</html>"

print(render_html(page))

Output:

<html><header><h1>My Page</h1></header><main>Welcome, &lt;b&gt;Alice&lt;/b&gt;</main></html>

The nested templates get recursively processed, so the HTML structure from the inner templates passes through as trusted content while user-provided values like username still get escaped. This composability pattern is what makes t-strings viable for real template engines, not just one-off string operations.

Building a real project with Python t-strings
One template processor to rule them all. Three interpolation types to find them.

Real-Life Example: Safe HTML Email Builder

Let us tie everything together with a practical project — a safe HTML email builder that uses t-strings to prevent injection while keeping the template code clean and readable:

# email_builder.py
from templatelib import Template, Interpolation
import html as html_mod

def render_email(template: Template) -> str:
    """Render an HTML email template with auto-escaping."""
    parts = []
    for item in template:
        if isinstance(item, str):
            parts.append(item)
        elif isinstance(item, Interpolation):
            if isinstance(item.value, Template):
                parts.append(render_email(item.value))
            else:
                parts.append(html_mod.escape(str(item.value)))
    return "".join(parts)

def build_order_email(customer_name: str, items: list[dict], total: float) -> str:
    """Build an order confirmation email safely."""
    # Build item rows from potentially untrusted product names
    rows = []
    for item in items:
        row = t"<tr><td>{item['name']}</td><td>{item['qty']}</td><td>${item['price']:.2f}</td></tr>"
        rows.append(render_email(row))
    item_rows = "\n".join(rows)

    email = t"""<html>
<body style='font-family: Arial, sans-serif;'>
  <h2>Order Confirmation</h2>
  <p>Hi {customer_name},</p>
  <p>Thank you for your order! Here is your summary:</p>
  <table border='1' cellpadding='8' cellspacing='0'>
    <tr style='background: #333; color: white;'>
      <th>Product</th><th>Qty</th><th>Price</th>
    </tr>
    {item_rows}
  </table>
  <p><strong>Total: ${total:.2f}</strong></p>
</body>
</html>"""

    return render_email(email)

# Test with potentially malicious input
order_items = [
    {"name": 'Python Book <script>alert("xss")</script>', "qty": 1, "price": 39.99},
    {"name": "USB-C Cable", "qty": 2, "price": 12.50},
    {"name": "Mechanical Keyboard", "qty": 1, "price": 89.00},
]

result = build_order_email(
    customer_name="Bob <img src=x onerror=alert(1)>",
    items=order_items,
    total=153.99
)
print(result)

Output:

<html>
<body style='font-family: Arial, sans-serif;'>
  <h2>Order Confirmation</h2>
  <p>Hi Bob &lt;img src=x onerror=alert(1)&gt;,</p>
  <p>Thank you for your order! Here is your summary:</p>
  <table border='1' cellpadding='8' cellspacing='0'>
    <tr style='background: #333; color: white;'>
      <th>Product</th><th>Qty</th><th>Price</th>
    </tr>
    <tr><td>Python Book &lt;script&gt;alert(&quot;xss&quot;)&lt;/script&gt;</td><td>1</td><td>$39.99</td></tr>
<tr><td>USB-C Cable</td><td>2</td><td>$12.50</td></tr>
<tr><td>Mechanical Keyboard</td><td>1</td><td>$89.00</td></tr>
  </table>
  <p><strong>Total: $153.99</strong></p>
</body>
</html>

Both the customer name and the malicious product name get HTML-escaped automatically, while the table structure and email markup remain intact. This is exactly the kind of “secure by default” behavior that t-strings were designed to provide. You could extend this pattern with a Safe wrapper class for pre-sanitized values that should not be double-escaped.

Python t-strings FAQ
The aha moment when lazy evaluation finally clicks.

Frequently Asked Questions

What version of Python do I need to use t-strings?

T-strings require Python 3.14 or later. They were introduced through PEP 750 and are part of the standard library via the templatelib module. If you are on an earlier version, you will need to upgrade. You can check your version by running python --version in your terminal. Python 3.14 was released in October 2025.

How are t-strings different from str.format() and Template strings?

The str.format() method and string.Template both produce finished strings — they do not give you access to the interpolated values before rendering. T-strings produce a Template object that keeps static text and dynamic values separate, letting you process each value individually. This makes t-strings the only built-in option that supports safe, context-aware rendering out of the box.

Are t-strings slower than f-strings?

T-strings have slightly more overhead than f-strings because they create a Template object instead of immediately concatenating a string. However, the difference is negligible for most applications. The extra cost is in the range of microseconds per operation. If you are in a tight loop formatting millions of strings per second and do not need custom processing, stick with f-strings. For everything else, the safety and flexibility of t-strings more than justify the small performance cost.

Can I use t-strings as a drop-in replacement for f-strings?

Not directly, because t-strings return a Template object instead of a str. You need a rendering function to convert the template to a string. However, writing a simple render() function that concatenates all parts without modification gives you f-string-equivalent behavior. The migration path is: change the prefix from f to t, add a render call, then gradually add escaping or processing logic where needed.

Will web frameworks like Django and Flask adopt t-strings?

Several framework maintainers have expressed interest in t-string integration. The Django template engine and Jinja2 (used by Flask) could potentially use t-strings as a lower-level primitive for their template rendering. However, adoption takes time — expect third-party libraries to provide t-string-based template engines before the major frameworks integrate them into their core APIs. In the meantime, you can use t-strings in your own application code alongside existing template engines.

Conclusion

T-strings bring a powerful new capability to Python’s string formatting toolkit. We covered the basic syntax and the Template object anatomy, then built practical processors for HTML escaping, SQL parameterization, structured logging, and shell command safety. The real-life email builder project showed how these patterns combine to create secure-by-default templating in real applications.

The key takeaway is that t-strings do not replace f-strings — they complement them. Use f-strings for quick formatting where safety is not a concern, and use t-strings when interpolated values need processing before they reach the output. The ability to inspect and transform each value individually is what makes the difference between a convenient string and a secure one.

For the complete specification, read PEP 750 and the templatelib documentation.

How To Read and Write YAML Files in Python

How To Read and Write YAML Files in Python

Last Updated: June 01, 2026

Beginner

YAML has become the go-to format for configuration files, infrastructure as code, and data serialization across countless Python projects. Whether you’re working with Docker Compose files, Kubernetes manifests, Ansible playbooks, or custom application configuration, understanding how to parse and create YAML files is an essential skill for any Python developer. In this comprehensive guide, we’ll explore the PyYAML library and walk through practical examples that demonstrate how to read configuration files, generate YAML output, handle complex data structures, and follow security best practices when working with untrusted YAML sources.

YAML, which stands for “YAML Ain’t Markup Language,” was designed with human readability as a primary goal. Unlike JSON’s curly braces and strict syntax, or XML’s verbose tag structure, YAML uses indentation and simple key-value pairs that mirror natural Python data structures. This makes it intuitive for both writing configuration files by hand and parsing them programmatically. Throughout this article, you’ll discover how Python’s PyYAML library bridges the gap between YAML’s readable format and Python’s powerful data manipulation capabilities.

By the end of this tutorial, you’ll be able to confidently read existing YAML files into Python dictionaries and lists, write Python data structures back to YAML format, handle edge cases like multi-document YAML files, leverage advanced features such as anchors and aliases, and most importantly, understand the security implications of YAML parsing. Let’s dive in and master the art of working with YAML in Python.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Quick Example

Before we explore the details, here’s a snapshot of what’s possible with just a few lines of Python:

# basic_example.py
import yaml

# Reading a YAML file
with open('config.yaml', 'r') as file:
    config = yaml.safe_load(file)
    print(config['database']['host'])

# Creating and writing YAML
data = {
    'app_name': 'MyApp',
    'version': '1.0',
    'features': ['auth', 'logging']
}
with open('output.yaml', 'w') as file:
    yaml.dump(data, file, default_flow_style=False)

This example demonstrates the two fundamental operations: reading configuration into Python data structures and serializing Python objects back to YAML format. In the sections below, we’ll expand on these concepts and explore advanced scenarios.

What is YAML?

YAML is a human-friendly data serialization language that excels at representing configuration files and structured data. Its design philosophy emphasizes readability, allowing developers to write and maintain configuration files without learning complex syntax rules. The language uses indentation to denote nesting, colons to separate keys from values, and hyphens to represent list items, all of which feel natural to anyone familiar with Python’s syntax.

To understand YAML’s place in the ecosystem, let’s compare it with other popular data formats:

Feature YAML JSON TOML INI
Human Readable Excellent Good Good Fair
Nested Structures Native Native Native Limited
Comments Yes No Yes Yes
Type Safety Implicit Explicit Mixed String-based
Use Cases Config, IaC APIs, Data Settings Legacy Apps
Parsing Speed Slower Fast Medium Fast

YAML’s strength lies in its readability and native support for comments, making it ideal for configuration files that humans regularly edit. JSON, by contrast, excels at machine-to-machine communication due to its strict structure and rapid parsing. TOML offers a middle ground with table-based organization, while INI files, though simple, lack native support for complex nested structures. For Python developers working with configuration files and infrastructure as code, YAML remains the most popular choice.

Understanding YAML hierarchy and structure
Indentation matters. Two spaces or four — pick one and stick with it forever.

Installation and Setup

Before you can parse and create YAML files in Python, you need to install the PyYAML library. PyYAML is not part of Python’s standard library, but it’s lightweight and easy to set up. Open your terminal and run the following command:

# setup.sh
pip install pyyaml

Once installed, verify that PyYAML is working correctly by checking its version:

# verify_install.py
import yaml
print(f"PyYAML version: {yaml.__version__}")

Output:

PyYAML version: 6.0

Congratulations! You’re now ready to work with YAML files in Python. The PyYAML library provides a straightforward API that we’ll explore throughout this guide.

Reading YAML Files with safe_load

The most common operation when working with YAML is reading configuration files into Python data structures. The PyYAML library provides several methods for this, but yaml.safe_load() is the recommended approach for security reasons. Unlike yaml.load(), which can execute arbitrary Python code embedded in YAML, safe_load() only constructs simple Python objects like dictionaries, lists, and strings, preventing code injection attacks.

Let’s start with a basic example. First, create a YAML file containing application configuration:

# config.yaml
app:
  name: DataProcessor
  version: 2.1.0
  debug: true
database:
  host: localhost
  port: 5432
  name: appdb
  credentials:
    user: admin
    password: secret123
features:
  - authentication
  - logging
  - reporting

Now, parse this YAML file in Python:

# read_yaml_basic.py
import yaml

with open('config.yaml', 'r') as file:
    config = yaml.safe_load(file)

print("Application Name:", config['app']['name'])
print("Database Host:", config['database']['host'])
print("Features:", config['features'])

Output:

Application Name: DataProcessor
Database Host: localhost
Features: ['authentication', 'logging', 'reporting']

Notice how the YAML structure maps directly to Python dictionaries and lists. Nested keys become nested dictionaries, arrays become Python lists, and boolean values are properly recognized. This seamless conversion is one of YAML’s greatest strengths.

Threading through nested YAML data structures
safe_load() returns a dict. yaml.load() returns regret.

Writing YAML Files with dump

Beyond reading YAML, you often need to generate YAML files from Python data. The yaml.dump() function converts Python objects into YAML format. Let’s create a practical example where we construct a configuration dictionary and write it to a file:

# write_yaml_basic.py
import yaml

config = {
    'app': {
        'name': 'MyService',
        'version': '1.0.0',
        'debug': False
    },
    'database': {
        'host': 'db.example.com',
        'port': 5432
    },
    'cache': {
        'enabled': True,
        'ttl': 3600
    }
}

with open('generated_config.yaml', 'w') as file:
    yaml.dump(config, file, default_flow_style=False)

print("Configuration written to generated_config.yaml")

Output:

Configuration written to generated_config.yaml

Check the contents of the generated file:

# view_generated.py
with open('generated_config.yaml', 'r') as file:
    print(file.read())

Output:

app:
  debug: false
  name: MyService
  version: 1.0.0
cache:
  enabled: true
  ttl: 3600
database:
  host: db.example.com
  port: 5432

The default_flow_style=False parameter ensures that nested structures are formatted with indentation rather than JSON-like curly braces. This produces more readable configuration files that follow YAML conventions. You can also control formatting with additional parameters like sort_keys=True to alphabetize keys or allow_unicode=True to preserve non-ASCII characters.

Working with Complex Data Types

YAML supports a rich variety of data types beyond simple strings and numbers. Python’s PyYAML library automatically handles conversion between YAML’s type system and Python’s native types. Understanding these conversions helps you work with complex configurations effectively.

Here’s a comprehensive example demonstrating various data types:

# complex_data_types.py
import yaml
from datetime import datetime

data = {
    'strings': {
        'simple': 'hello',
        'multiline': 'first line\nsecond line',
        'quoted': 'special chars: @#$%'
    },
    'numbers': {
        'integer': 42,
        'float': 3.14,
        'scientific': 1.23e-4,
        'hex': 0xFF,
        'octal': 0o755
    },
    'booleans': {
        'true_value': True,
        'false_value': False,
        'yes': True,
        'no': False
    },
    'null_value': None,
    'lists': {
        'simple': [1, 2, 3],
        'mixed': ['string', 42, True, None]
    },
    'dates': {
        'timestamp': datetime(2026, 4, 5, 14, 30, 0)
    }
}

with open('complex.yaml', 'w') as file:
    yaml.dump(data, file, default_flow_style=False)

with open('complex.yaml', 'r') as file:
    loaded = yaml.safe_load(file)
    print(loaded)

Output:

{'strings': {'simple': 'hello', 'multiline': 'first line\nsecond line', 'quoted': 'special chars: @#$%'}, 'numbers': {'integer': 42, 'float': 3.14, 'scientific': 0.000123, 'hex': 255, 'octal': 493}, 'booleans': {'true_value': True, 'false_value': False, 'yes': True, 'no': False}, 'null_value': None, 'lists': {'simple': [1, 2, 3], 'mixed': ['string', 42, True, None]}, 'dates': {'timestamp': datetime.datetime(2026, 4, 5, 14, 30, 0)}}

YAML’s type inference system automatically detects whether a value is a string, number, boolean, or null. This intelligent parsing eliminates the need for explicit type declarations. However, if you need to force a specific type—for instance, treating the string “yes” as text rather than a boolean—you can quote it in the YAML file.

Lost in complex YAML data types
YAML thinks “yes” is a boolean. Your postal code disagrees.

Multi-Document YAML Files

YAML supports storing multiple documents in a single file, separated by three hyphens (---). This is particularly useful when you need to manage multiple configurations or data structures in one file. PyYAML provides yaml.safe_load_all() to iterate through all documents:

# multi_document.yaml
---
name: Configuration A
version: 1.0
settings:
  debug: true
---
name: Configuration B
version: 2.0
settings:
  debug: false
---
name: Configuration C
version: 1.5
settings:
  debug: true

Now load all documents:

# read_multi_yaml.py
import yaml

with open('multi_document.yaml', 'r') as file:
    documents = yaml.safe_load_all(file)
    for i, doc in enumerate(documents, 1):
        print(f"Document {i}:")
        print(f"  Name: {doc['name']}")
        print(f"  Version: {doc['version']}")
        print()

Output:

Document 1:
  Name: Configuration A
  Version: 1.0

Document 2:
  Name: Configuration B
  Version: 2.0

Document 3:
  Name: Configuration C
  Version: 1.5

Multi-document YAML is invaluable for scenarios like managing Kubernetes manifests, where multiple resource definitions appear in a single file. The safe_load_all() function returns a generator, allowing you to process documents one at a time without loading the entire file into memory.

Anchors and Aliases for Code Reuse

YAML provides a powerful feature called anchors and aliases that allows you to define a value once and reference it multiple times. This reduces duplication and makes configurations easier to maintain. An anchor is created with an ampersand (&), and aliases reference the anchor with an asterisk (*).

# anchors_aliases.yaml
defaults: &default_settings
  timeout: 30
  retries: 3
  cache: true

services:
  api:
    <<: *default_settings
    port: 8000
    name: API Service

  worker:
    <<: *default_settings
    port: 9000
    name: Worker Service

  database:
    <<: *default_settings
    port: 5432
    name: Database

Parse this configuration:

# read_anchors.py
import yaml

with open('anchors_aliases.yaml', 'r') as file:
    config = yaml.safe_load(file)

for service_name, settings in config['services'].items():
    print(f"{service_name}:")
    print(f"  Timeout: {settings['timeout']}")
    print(f"  Retries: {settings['retries']}")
    print()

Output:

api:
  Timeout: 30
  Retries: 3

worker:
  Timeout: 30
  Retries: 3

database:
  Timeout: 30
  Retries: 3

The merge key (<<) combines the referenced anchor with the current dictionary, allowing service definitions to inherit default settings while still overriding specific values. This pattern significantly reduces repetition in large configuration files.

Juggling YAML anchors and aliases
Define once with &, reuse everywhere with *. DRY config files are beautiful.

Safe Loading Practices and Security

When working with YAML files from untrusted sources, security is paramount. The standard yaml.load() function is dangerous because it can execute arbitrary Python code embedded in YAML. Consider this malicious YAML:

# dangerous.yaml
!!python/object/apply:os.system
args: ['rm -rf /']

Loading this with yaml.load() would execute the command. Always use yaml.safe_load() instead:

# safe_loading_demo.py
import yaml

# WRONG: Never do this with untrusted YAML
# data = yaml.load(untrusted_yaml, Loader=yaml.FullLoader)

# CORRECT: Use safe_load for security
try:
    with open('config.yaml', 'r') as file:
        data = yaml.safe_load(file)
    print("Safely loaded configuration")
except yaml.YAMLError as e:
    print(f"Error parsing YAML: {e}")

Output:

Safely loaded configuration

Beyond using safe_load(), implement additional security measures: validate configuration schemas to ensure expected structure, restrict file permissions so only authorized users can modify configuration files, and sanitize any user input that gets incorporated into YAML files. For high-security environments, consider using specialized YAML validation libraries or writing custom validation functions.

Custom YAML Tags and Constructors

YAML's tag system allows you to extend its functionality with custom types. While safe_load() prevents arbitrary code execution, you can still register custom constructors for specific tags to handle domain-specific data types. This is useful for configurations that require special processing:

# custom_tags.py
import yaml
import os
from pathlib import Path

def env_constructor(loader, node):
    """Custom constructor for !env tag to read environment variables"""
    value = loader.construct_scalar(node)
    return os.getenv(value, f'${{{value}}}')

def path_constructor(loader, node):
    """Custom constructor for !path tag to create Path objects"""
    value = loader.construct_scalar(node)
    return str(Path(value).resolve())

# Register constructors
yaml.SafeLoader.add_constructor('!env', env_constructor)
yaml.SafeLoader.add_constructor('!path', path_constructor)

yaml_content = """
database_url: !env DATABASE_URL
log_dir: !path /var/logs
app_name: MyApp
"""

data = yaml.safe_load(yaml_content)
print(data)

Output:

{'database_url': '${DATABASE_URL}', 'log_dir': '/var/logs', 'app_name': 'MyApp'}

Custom tags enable you to handle environment variables, file paths, date strings, and other special formats seamlessly during YAML parsing. This approach keeps your configuration files readable while maintaining type safety and extensibility.

Assembling configuration puzzle with YAML
Dot notation config access in 30 lines. Django called — it wants its settings back.

Real-Life Example: Configuration File Manager

Let's bring everything together with a practical application—a configuration file manager that reads YAML, validates settings, and provides utilities for working with configuration data:

# config_manager.py
import yaml
import os
from pathlib import Path
from typing import Any, Dict, Optional

class ConfigManager:
    """Manages application configuration from YAML files."""

    def __init__(self, config_path: str):
        self.config_path = Path(config_path)
        self.config: Dict[str, Any] = {}
        self.load()

    def load(self) -> None:
        """Load configuration from YAML file."""
        if not self.config_path.exists():
            raise FileNotFoundError(f"Config file not found: {self.config_path}")

        with open(self.config_path, 'r') as file:
            try:
                self.config = yaml.safe_load(file) or {}
            except yaml.YAMLError as e:
                raise ValueError(f"Invalid YAML: {e}")

    def get(self, key: str, default: Any = None) -> Any:
        """Get configuration value using dot notation."""
        keys = key.split('.')
        value = self.config

        for k in keys:
            if isinstance(value, dict):
                value = value.get(k)
                if value is None:
                    return default
            else:
                return default

        return value

    def set(self, key: str, value: Any) -> None:
        """Set configuration value using dot notation."""
        keys = key.split('.')
        config = self.config

        for k in keys[:-1]:
            if k not in config:
                config[k] = {}
            config = config[k]

        config[keys[-1]] = value

    def save(self) -> None:
        """Save configuration back to YAML file."""
        with open(self.config_path, 'w') as file:
            yaml.dump(self.config, file, default_flow_style=False)

    def validate_required(self, required_keys: list) -> bool:
        """Check that all required configuration keys exist."""
        for key in required_keys:
            if self.get(key) is None:
                print(f"Missing required configuration: {key}")
                return False
        return True

# Usage example
if __name__ == '__main__':
    # Create sample configuration
    sample_config = {
        'app': {
            'name': 'MyApplication',
            'version': '1.0.0'
        },
        'database': {
            'host': 'localhost',
            'port': 5432,
            'name': 'mydb'
        },
        'server': {
            'host': '0.0.0.0',
            'port': 8000
        }
    }

    # Write sample config
    with open('app_config.yaml', 'w') as f:
        yaml.dump(sample_config, f, default_flow_style=False)

    # Load and use configuration
    config = ConfigManager('app_config.yaml')

    print(f"App: {config.get('app.name')}")
    print(f"Database: {config.get('database.host')}:{config.get('database.port')}")
    print(f"Server: {config.get('server.host')}:{config.get('server.port')}")

    # Modify configuration
    config.set('database.pool_size', 10)
    config.save()

    print("\nConfiguration updated and saved.")

Output:

App: MyApplication
Database: localhost:5432
Server: 0.0.0.0:8000

Configuration updated and saved.

This ConfigManager class demonstrates a production-ready approach to handling YAML configuration files. It supports dot notation for accessing nested values, provides methods for modifying configurations, validates required settings, and handles errors gracefully. You can extend this class with additional features like configuration merging, environment variable substitution, or schema validation depending on your application's needs.

Frequently Asked Questions

What's the difference between yaml.load() and yaml.safe_load()?

yaml.load() uses the full YAML specification and can deserialize arbitrary Python objects, including those that execute code during instantiation. This makes it dangerous with untrusted input. yaml.safe_load() only constructs simple Python objects (dicts, lists, strings) and is safe for use with any YAML source. Always prefer safe_load() unless you have a specific reason to use the full loader and have full control over the input.

Can I preserve comments when reading and writing YAML?

Standard PyYAML doesn't preserve comments during round-trip operations. If you need to maintain comments, consider using the ruamel.yaml library instead, which is designed specifically for preserving comments, formatting, and other YAML features. However, for most applications, PyYAML's simpler approach is sufficient.

How do I handle very large YAML files efficiently?

For large YAML files, use yaml.safe_load_all() with generators to process documents one at a time rather than loading everything into memory. Additionally, consider using streaming parsers or breaking large files into smaller chunks. PyYAML can handle reasonably sized files, but for massive datasets, you might explore alternative formats like JSON or CSV.

Why does my integer sometimes become a string when loading YAML?

YAML's automatic type detection usually works well, but certain values can be ambiguous. For example, ZIP codes like 02134 are interpreted as octal numbers. To force a string type, quote the value in your YAML file: '02134'. Similarly, yes/no values become booleans unless quoted.

How can I validate YAML against a schema?

PyYAML doesn't include built-in schema validation. For validation, use libraries like jsonschema (which works with YAML since both parse to dictionaries) or pydantic for more sophisticated type checking. After loading YAML with safe_load(), you can validate the resulting Python object against your schema.

Conclusion

Mastering YAML parsing and creation in Python opens doors to working with modern configuration systems, infrastructure as code, and data serialization across countless projects. From reading simple configuration files with yaml.safe_load() to writing complex data structures with yaml.dump(), the PyYAML library provides everything you need for practical YAML handling. Remember to always prioritize security by using safe_load(), validate your configurations, and keep comments in mind when choosing between YAML and alternative formats.

As you build more sophisticated applications, you'll find that understanding YAML's features—from anchors and aliases to custom tags and multi-document files—will help you write cleaner, more maintainable configurations. For more advanced techniques and comprehensive documentation, visit the PyYAML Documentation.

Continue learning with these related guides:

How To Parse and Create Excel Files with openpyxl in Python

How To Parse and Create Excel Files with openpyxl in Python

Last Updated: June 01, 2026

Beginner

Excel files are everywhere in business environments, from financial reports and inventory lists to customer databases and sales analytics. While Excel is a powerful tool for data visualization and quick calculations, Python offers automation capabilities that can save hours of manual work. The openpyxl library is the most popular Python package for reading, writing, and modifying Excel files programmatically. This tutorial will guide you through everything you need to know about working with Excel files in Python, from basic operations to advanced formatting and formulas.

Whether you’re dealing with simple CSV-like data or complex workbooks with multiple sheets and intricate formatting, openpyxl provides an intuitive interface that mirrors Excel’s own structure. You’ll learn how to create workbooks from scratch, read existing files, apply professional formatting, insert formulas, and even generate charts—all without opening Excel. By the end of this guide, you’ll be able to automate your Excel workflows and handle data manipulation tasks that would take minutes manually in just seconds with Python.

The beauty of using openpyxl is that it maintains compatibility with Excel’s native features while being lightweight and easy to learn. Unlike some alternatives that require Excel to be installed on your system, openpyxl works independently, making it perfect for server-side automation, data processing pipelines, and batch file generation. You’ll also discover how to handle real-world scenarios like generating sales reports, updating employee databases, and creating formatted spreadsheets for stakeholders—all through simple Python code.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Quick Example

Let’s start with a quick glimpse of what’s possible with openpyxl. In just a few lines of code, you can create an Excel file, add data, format cells, and save it:

# quick_example.py
from openpyxl import Workbook
from openpyxl.styles import Font, PatternFill

# Create a new workbook
wb = Workbook()
ws = wb.active
ws.title = "Sales"

# Add headers
headers = ["Product", "Quantity", "Price", "Total"]
ws.append(headers)

# Style the header row
for cell in ws[1]:
    cell.font = Font(bold=True, color="FFFFFF")
    cell.fill = PatternFill(start_color="366092", end_color="366092", fill_type="solid")

# Add data
ws.append(["Laptop", 5, 1200, 6000])
ws.append(["Mouse", 15, 25, 375])
ws.append(["Keyboard", 10, 75, 750])

# Adjust column widths
ws.column_dimensions['A'].width = 15
ws.column_dimensions['B'].width = 12
ws.column_dimensions['C'].width = 12
ws.column_dimensions['D'].width = 12

# Save the file
wb.save("sales_data.xlsx")
print("File created successfully!")

Output:

File created successfully!

This simple script creates a professional-looking spreadsheet with formatted headers and data. When you open the resulting sales_data.xlsx file in Excel, you’ll see a properly formatted table with colors and sizing already applied. That’s the power of openpyxl—automation with style.

What is openpyxl?

openpyxl is a Python library designed specifically for reading and writing Excel 2010+ files (the modern .xlsx format). Excel files are actually compressed XML documents, and openpyxl handles all the complexity of parsing and writing this format so you don’t have to. The library provides a clean, Pythonic API that allows you to work with Excel files just as you would in the Excel application itself—through workbooks, sheets, rows, columns, and cells.

The main advantages of openpyxl over alternatives include its comprehensive feature support, active maintenance, and the fact that it doesn’t require Excel to be installed on your system. Whether you’re running Python on Windows, macOS, or Linux, openpyxl works seamlessly. It’s particularly valuable for server applications, data processing pipelines, and automated reporting systems where Excel isn’t available.

Here’s how openpyxl compares to other popular options for working with Excel files in Python:

Library Format Support Writing Support Formatting Requires Excel Best For
openpyxl .xlsx, .xlsm Yes, full support Extensive (fonts, colors, borders, etc.) No Creating and modifying formatted Excel files
xlrd .xls, .xlsx No, read-only Limited No Reading older Excel files
pandas .xlsx, .xls, .csv Yes, limited Minimal No Data analysis and transformation
pywin32 .xlsx, .xls Yes, full support Extensive Yes (Windows only) Enterprise automation with Excel integration

For this tutorial, we’ll focus on openpyxl because it offers the best balance of features, ease of use, and cross-platform compatibility. Let’s get started by installing it and creating your first workbook.

Installation and Setup

Before you can use openpyxl, you need to install it on your system. This is straightforward using pip, Python’s package manager. Open your terminal or command prompt and run the following command:

# install_openpyxl.sh
pip install openpyxl

Output:

Successfully installed openpyxl-3.1.2

Once installed, you can import openpyxl in your Python scripts. The installation includes all necessary dependencies, so you won’t need to install anything else. If you’re using a virtual environment (which is recommended for Python projects), make sure you activate it before installing openpyxl.

Setting up openpyxl for Excel automation
pip install openpyxl — three words between you and never opening Excel again.

Creating Workbooks from Scratch

Creating a new Excel workbook with openpyxl is simple and intuitive. A workbook is the Excel file itself, and it can contain one or more sheets. Let’s explore how to create workbooks and add data to them:

# create_workbook.py
from openpyxl import Workbook

# Create a new workbook
wb = Workbook()

# Access the active sheet (first sheet)
ws = wb.active
print(f"Active sheet name: {ws.title}")

# You can also change the sheet name
ws.title = "Employee Data"

# Add data to cells
ws['A1'] = "Name"
ws['B1'] = "Department"
ws['C1'] = "Salary"

ws['A2'] = "Alice Johnson"
ws['B2'] = "Engineering"
ws['C2'] = 95000

ws['A3'] = "Bob Smith"
ws['B3'] = "Marketing"
ws['C3'] = 75000

# Save the workbook
wb.save("employees.xlsx")
print("Workbook created and saved!")

Output:

Active sheet name: Sheet
Workbook created and saved!

In this example, we created a new workbook, accessed its active sheet, renamed it to “Employee Data”, and added information in a table format. Notice how we accessed cells using Excel-style notation like A1, B2, etc. This makes the code very readable if you’re familiar with Excel.

You can also create multiple sheets in a single workbook, which is useful for organizing related data:

# multiple_sheets.py
from openpyxl import Workbook

wb = Workbook()
ws1 = wb.active
ws1.title = "Q1 Sales"

# Create additional sheets
ws2 = wb.create_sheet("Q2 Sales")
ws3 = wb.create_sheet("Q3 Sales")

# Add data to each sheet
for ws, quarter in [(ws1, "Q1"), (ws2, "Q2"), (ws3, "Q3")]:
    ws['A1'] = f"{quarter} Revenue"
    ws['A2'] = 150000
    ws['B1'] = f"{quarter} Expenses"
    ws['B2'] = 75000

wb.save("quarterly_report.xlsx")
print("Multi-sheet workbook created!")

Output:

Multi-sheet workbook created!

Reading Existing Excel Files

Working with existing Excel files is just as straightforward as creating new ones. openpyxl allows you to load a workbook and access its data in various ways:

# read_existing_file.py
from openpyxl import load_workbook

# Load an existing workbook
wb = load_workbook("employees.xlsx")

# Get a sheet by name
ws = wb["Employee Data"]

# Or get the active sheet
# ws = wb.active

# Iterate through all rows
print("Employee List:")
for row in ws.iter_rows(values_only=True):
    print(row)

# Access specific cells
print(f"\nFirst employee: {ws['A2'].value}")
print(f"Department: {ws['B2'].value}")

Output:

Employee List:
('Name', 'Department', 'Salary')
('Alice Johnson', 'Engineering', 95000)
('Bob Smith', 'Marketing', 75000)

First employee: Alice Johnson
Department: Engineering

The iter_rows() method is particularly useful for processing large amounts of data. The values_only=True parameter returns just the cell values without the cell objects, making it easier to work with the data.

Reading and inspecting Excel spreadsheet data
iter_rows(values_only=True) — because cell objects have feelings you don’t need.

Cell Formatting and Styling

Excel’s power lies not just in data storage but in presentation. openpyxl provides extensive formatting capabilities to make your spreadsheets professional and readable. Let’s explore fonts, colors, borders, and alignment:

# cell_formatting.py
from openpyxl import Workbook
from openpyxl.styles import Font, PatternFill, Border, Side, Alignment

wb = Workbook()
ws = wb.active

# Font styling
ws['A1'] = "Bold and Italic"
ws['A1'].font = Font(name='Arial', size=14, bold=True, italic=True, color="FFFFFF")

# Background color (fill)
ws['A1'].fill = PatternFill(start_color="0066CC", end_color="0066CC", fill_type="solid")

# Borders
thin_border = Border(
    left=Side(style='thin'),
    right=Side(style='thin'),
    top=Side(style='thin'),
    bottom=Side(style='thin')
)
ws['A1'].border = thin_border

# Alignment
ws['A1'].alignment = Alignment(horizontal='center', vertical='center', wrap_text=True)

# Apply to a range of cells
for row in ws.iter_rows(min_row=2, max_row=5, min_col=1, max_col=3):
    for cell in row:
        cell.fill = PatternFill(start_color="E8F0FF", end_color="E8F0FF", fill_type="solid")
        cell.border = thin_border
        cell.font = Font(size=11)

ws.save("formatted.xlsx")
print("Formatted workbook saved!")

Output:

Formatted workbook saved!

Colors in openpyxl are specified using hex codes (like “0066CC”). You can find color codes online or use a color picker to match your brand colors. The formatting capabilities extend to number formats, alignment options, and even special effects like gradients.

Working with Formulas

One of the most powerful features of Excel is its ability to store formulas that automatically calculate values. openpyxl allows you to insert formulas that will be evaluated when the file is opened in Excel:

# formulas.py
from openpyxl import Workbook

wb = Workbook()
ws = wb.active

# Create a simple invoice
ws['A1'] = "Item"
ws['B1'] = "Price"
ws['C1'] = "Quantity"
ws['D1'] = "Total"

items = [
    ("Laptop", 1200, 2),
    ("Mouse", 25, 5),
    ("Keyboard", 75, 3)
]

row = 2
for item, price, qty in items:
    ws[f'A{row}'] = item
    ws[f'B{row}'] = price
    ws[f'C{row}'] = qty
    # Insert a formula for total (Price * Quantity)
    ws[f'D{row}'] = f'=B{row}*C{row}'
    row += 1

# Add a grand total formula
total_row = row
ws[f'A{total_row}'] = "Grand Total"
ws[f'D{total_row}'] = f'=SUM(D2:D{row-1})'

# Make it bold
from openpyxl.styles import Font
ws[f'A{total_row}'].font = Font(bold=True)
ws[f'D{total_row}'].font = Font(bold=True)

wb.save("invoice.xlsx")
print("Invoice with formulas created!")

Output:

Invoice with formulas created!

When you open the resulting Excel file, you’ll see that the formulas are active and update automatically if you change the prices or quantities. The formulas use standard Excel syntax, so you can use any Excel function including SUM, AVERAGE, IF, VLOOKUP, and many more.

Creating charts with openpyxl in Python
=SUM(D2:D99) hits different when Python wrote every formula.

Creating Charts

Charts make data visualization intuitive and professional. openpyxl supports creating various chart types programmatically:

# creating_charts.py
from openpyxl import Workbook
from openpyxl.chart import BarChart, Reference

wb = Workbook()
ws = wb.active
ws.title = "Sales Data"

# Add headers
ws['A1'] = "Month"
ws['B1'] = "Revenue"

# Add sales data
months = ["January", "February", "March", "April", "May"]
revenue = [45000, 52000, 48000, 61000, 58000]

for idx, (month, rev) in enumerate(zip(months, revenue), start=2):
    ws[f'A{idx}'] = month
    ws[f'B{idx}'] = rev

# Create a bar chart
chart = BarChart()
chart.title = "Monthly Revenue"
chart.x_axis.title = "Month"
chart.y_axis.title = "Revenue ($)"

# Add data to the chart
data = Reference(ws, min_col=2, min_row=1, max_row=6)
cats = Reference(ws, min_col=1, min_row=2, max_row=6)
chart.add_data(data, titles_from_data=True)
chart.set_categories(cats)

# Position the chart
ws.add_chart(chart, "D2")

wb.save("sales_chart.xlsx")
print("Workbook with chart created!")

Output:

Workbook with chart created!

openpyxl supports multiple chart types including bar charts, line charts, pie charts, scatter plots, and more. Charts automatically update when data changes, just like in Excel, providing dynamic data visualization.

Merging Cells

Sometimes you want to merge cells to create headers or improve layout. openpyxl makes this straightforward:

# merging_cells.py
from openpyxl import Workbook
from openpyxl.styles import Font, Alignment, PatternFill

wb = Workbook()
ws = wb.active

# Merge cells for a title
ws.merge_cells('A1:D1')
ws['A1'] = "Quarterly Sales Report"
ws['A1'].font = Font(size=16, bold=True)
ws['A1'].alignment = Alignment(horizontal='center', vertical='center')
ws['A1'].fill = PatternFill(start_color="366092", end_color="366092", fill_type="solid")
ws['A1'].font = Font(size=16, bold=True, color="FFFFFF")

# Set row height for the title
ws.row_dimensions[1].height = 30

# Add column headers
headers = ["Q1", "Q2", "Q3", "Q4"]
for col, header in enumerate(headers, start=1):
    ws.cell(row=2, column=col, value=header)
    ws.cell(row=2, column=col).font = Font(bold=True)

wb.save("merged_cells.xlsx")
print("Workbook with merged cells created!")

Output:

Workbook with merged cells created!

When merging cells, the content is placed in the top-left cell of the merged range. Be careful when merging as it can affect how data is read back—make sure to reference the correct cell when accessing merged cell values.

Conditional Formatting

Conditional formatting automatically applies styles based on cell values, making it easy to highlight important data. Here’s how to implement it with openpyxl:

# conditional_formatting.py
from openpyxl import Workbook
from openpyxl.formatting.rule import CellIsRule
from openpyxl.styles import PatternFill, Font

wb = Workbook()
ws = wb.active
ws.title = "Sales Performance"

# Add headers
ws['A1'] = "Salesperson"
ws['B1'] = "Sales Amount"

# Add data
salespeople = [
    ("Alice", 85000),
    ("Bob", 42000),
    ("Charlie", 95000),
    ("Diana", 55000),
    ("Edward", 78000)
]

for idx, (name, sales) in enumerate(salespeople, start=2):
    ws[f'A{idx}'] = name
    ws[f'B{idx}'] = sales

# Create a rule to highlight high performers (>75000)
high_fill = PatternFill(start_color="00B050", end_color="00B050", fill_type="solid")
high_font = Font(bold=True, color="FFFFFF")
high_rule = CellIsRule(operator='greaterThan', formula=['75000'], fill=high_fill, font=high_font)

# Apply the rule
ws.conditional_formatting.add(f'B2:B{len(salespeople)+1}', high_rule)

wb.save("conditional_format.xlsx")
print("Workbook with conditional formatting created!")

Output:

Workbook with conditional formatting created!

Conditional formatting is powerful for highlighting trends, outliers, and important values at a glance. You can create complex rules with multiple conditions, color scales, and data bars.

Formatting Excel cells with openpyxl styles
PatternFill, Font, Border, Alignment — CSS for spreadsheets, basically.

Real-World Example: Sales Report Generator

Let’s build a practical application that demonstrates all the concepts we’ve learned. This script generates a professional sales report from raw data:

# sales_report_generator.py
from openpyxl import Workbook
from openpyxl.styles import Font, PatternFill, Border, Side, Alignment
from openpyxl.formatting.rule import CellIsRule
from openpyxl.chart import BarChart, Reference
from datetime import datetime

def generate_sales_report(data, filename="sales_report.xlsx"):
    """
    Generate a professional sales report.

    Args:
        data: List of tuples (product, quantity, unit_price, region)
        filename: Output Excel filename
    """

    wb = Workbook()
    ws = wb.active
    ws.title = "Sales Report"

    # Create title
    ws.merge_cells('A1:E1')
    title = ws['A1']
    title.value = f"Sales Report - {datetime.now().strftime('%B %Y')}"
    title.font = Font(size=16, bold=True, color="FFFFFF")
    title.fill = PatternFill(start_color="1F4E78", end_color="1F4E78", fill_type="solid")
    title.alignment = Alignment(horizontal='center', vertical='center')
    ws.row_dimensions[1].height = 25

    # Create headers
    headers = ["Product", "Quantity", "Unit Price", "Total Sales", "Region"]
    for col, header in enumerate(headers, start=1):
        cell = ws.cell(row=3, column=col, value=header)
        cell.font = Font(bold=True, color="FFFFFF")
        cell.fill = PatternFill(start_color="4472C4", end_color="4472C4", fill_type="solid")
        cell.alignment = Alignment(horizontal='center')

    # Add data
    border = Border(left=Side(style='thin'), right=Side(style='thin'),
                   top=Side(style='thin'), bottom=Side(style='thin'))

    total_sales = 0
    for idx, (product, qty, price, region) in enumerate(data, start=4):
        ws[f'A{idx}'] = product
        ws[f'B{idx}'] = qty
        ws[f'C{idx}'] = price
        ws[f'D{idx}'] = f'=B{idx}*C{idx}'
        ws[f'E{idx}'] = region

        for col in range(1, 6):
            ws.cell(row=idx, column=col).border = border
            if col in [2, 3, 4]:
                ws.cell(row=idx, column=col).alignment = Alignment(horizontal='right')

    # Grand total
    last_row = len(data) + 4
    ws[f'A{last_row}'] = "TOTAL SALES"
    ws[f'D{last_row}'] = f'=SUM(D4:D{last_row-1})'
    ws[f'A{last_row}'].font = Font(bold=True, size=12)
    ws[f'D{last_row}'].font = Font(bold=True, size=12)

    # Format currency columns
    for row in range(4, last_row + 1):
        ws[f'C{row}'].number_format = '$#,##0.00'
        ws[f'D{row}'].number_format = '$#,##0.00'

    # Conditional formatting for high sales
    high_fill = PatternFill(start_color="FFC7CE", end_color="FFC7CE", fill_type="solid")
    rule = CellIsRule(operator='greaterThan', formula=['50000'], fill=high_fill)
    ws.conditional_formatting.add(f'D4:D{last_row-1}', rule)

    # Adjust column widths
    ws.column_dimensions['A'].width = 15
    ws.column_dimensions['B'].width = 12
    ws.column_dimensions['C'].width = 12
    ws.column_dimensions['D'].width = 15
    ws.column_dimensions['E'].width = 12

    # Save the workbook
    wb.save(filename)
    print(f"Report generated: {filename}")

# Sample data
sales_data = [
    ("Laptop Pro", 15, 1500, "North America"),
    ("USB Mouse", 45, 25, "Europe"),
    ("Mechanical Keyboard", 32, 120, "Asia Pacific"),
    ("Monitor 4K", 12, 400, "North America"),
    ("Webcam HD", 58, 80, "Europe"),
    ("External SSD", 28, 150, "Asia Pacific"),
    ("Laptop Stand", 40, 45, "North America"),
    ("Wireless Charger", 66, 35, "Europe"),
]

# Generate the report
generate_sales_report(sales_data)

Output:

Report generated: sales_report.xlsx

This comprehensive example creates a professional sales report with headers, formatted data, formulas for calculations, currency formatting, conditional highlighting, and proper styling. It demonstrates how all the features we’ve learned work together to create a polished, business-ready spreadsheet.

Automating Excel report generation
Report that took 20 minutes by hand now runs in 0.4 seconds. You’re welcome, accounting.

Frequently Asked Questions

How do I handle large Excel files efficiently?

For very large files, openpyxl’s default mode can consume significant memory. You can use read-only or write-only modes to process large files more efficiently. For read operations, use load_workbook(filename, read_only=True, data_only=True). For write operations, use Workbook(write_only=True). These modes stream data instead of loading everything into memory at once.

Why aren’t my formulas calculating when I open the file?

Excel doesn’t recalculate formulas automatically when a file is created by openpyxl. When you open the file in Excel, you’ll typically get a prompt to recalculate. If you want to see calculated values when reading the file back with openpyxl, you need to open it in Excel first to trigger the calculation, or use data_only=True when loading the workbook (though this requires the file to have been opened and saved in Excel previously).

Can I protect sheets or workbooks?

Yes, openpyxl supports sheet and workbook protection. You can protect a sheet with ws.protection.sheet = True and optionally set a password with ws.protection.password = "your_password". Similarly, you can protect the workbook with wb.security.workbookProtection.workbookPassword = "password". Note that these are basic protections and not cryptographically strong.

How do I properly handle dates and times in Excel cells?

Excel stores dates as numbers representing days since a reference date. When writing dates with openpyxl, use Python’s datetime objects directly: ws['A1'] = datetime.now(). openpyxl automatically handles the conversion. You can format the cell with ws['A1'].number_format = 'mm/dd/yyyy' to control how the date displays.

Is openpyxl compatible with .xls files (older Excel format)?

openpyxl only works with the modern .xlsx format (Excel 2010 and later). For older .xls files, you would need to use the xlrd library for reading or xlwt for writing. However, the easiest approach is often to convert old .xls files to .xlsx using Excel itself before processing with Python.

Can I hide rows or columns?

Yes, you can hide rows and columns in openpyxl. Use ws.row_dimensions[1].hidden = True to hide a row, or ws.column_dimensions['A'].hidden = True to hide a column. You can also freeze rows and columns for easier navigation in large spreadsheets using ws.freeze_panes = 'B2' to freeze the first row and first column.

Conclusion

You now have a comprehensive understanding of how to work with Excel files in Python using openpyxl. From creating simple spreadsheets to generating complex, professionally-formatted reports, openpyxl provides all the tools you need. The key takeaways are: start with the basics of creating workbooks and reading existing files, progress to styling and formatting for professional appearance, leverage formulas and charts for data analysis, and finally, combine everything into automated reporting solutions.

The real power of openpyxl shines when you use it to automate repetitive Excel tasks. Instead of manually creating reports, updating spreadsheets, or formatting data, you can write a Python script that does it in seconds. This skill becomes invaluable when working with data pipelines, generating client reports, or maintaining business intelligence systems.

For more information and advanced features, visit the official openpyxl documentation at https://openpyxl.readthedocs.io/. The documentation includes detailed API references, examples, and solutions to edge cases you might encounter in production environments.

Continue learning with these related guides:

How To Work with ZIP Files in Python

How To Work with ZIP Files in Python

Last Updated: June 01, 2026

Beginner

ZIP files are everywhere. Whether you’re downloading software, transferring files across the internet, or backing up critical data, you’ve almost certainly encountered a compressed archive. But what if you need to work with ZIP files programmatically? Python makes it surprisingly easy with the built-in zipfile module, which lets you create, read, extract, and modify ZIP archives directly from your code.

If you’ve ever felt intimidated by file compression or thought you needed external tools to handle archives, don’t worry. In this tutorial, we’ll walk you through everything you need to know. By the end, you’ll be able to create sophisticated backup systems, extract files on demand, apply password protection, and even compress data using different algorithms—all with clean, Pythonic code.

Here’s what we’ll cover: we’ll start with a quick example to see the module in action, then explore what ZIP files are and why they matter. We’ll build up from creating basic archives to handling complex scenarios like password-protected files and selective extraction. Finally, we’ll look at a real-world backup system and answer common questions you’ll encounter in production code.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Quick Example: Creating and Reading Your First ZIP File

Let’s jump straight in and see the zipfile module in action. This simple example creates a ZIP file containing a text file, then reads it back:

# quick_example.py
import zipfile

# Create a ZIP file and add content
with zipfile.ZipFile('archive.zip', 'w') as zf:
    zf.writestr('hello.txt', 'Hello from Python!')

# Read it back and print the contents
with zipfile.ZipFile('archive.zip', 'r') as zf:
    print(zf.read('hello.txt').decode('utf-8'))
Hello from Python!

See? In just a few lines, you’ve created a ZIP archive, added a file, and retrieved its contents. The with statement handles opening and closing the archive automatically, which keeps your code clean and prevents resource leaks. This pattern—using with for context management—will be your bread and butter when working with ZIP files.

What Are ZIP Files and Why Use Python?

ZIP is a widely supported archive format that combines file compression with a directory structure. Unlike raw compression formats (like GZIP), ZIP files are containers that can hold multiple files and folders while preserving their hierarchy and metadata. ZIP compression is lossless, meaning no data is lost during compression, and the format is supported natively on Windows, macOS, and Linux—no special software required.

You might ask: why not just use shell commands or GUI tools? Python offers several advantages. First, it lets you automate archival workflows inside your application. Second, you can process ZIP files without extracting them to disk, saving I/O overhead. Third, you get programmatic control over compression levels, passwords, and selective extraction. Fourth, your code becomes cross-platform instantly—the same script runs on any OS with Python.

Here’s how ZIP compares to other formats:

Format Compression Ratio Multiple Files Directories Password Support Platform Support
ZIP Good Yes Yes Yes Universal
TAR + GZIP Excellent Yes Yes No Unix/Linux
7-Zip Excellent Yes Yes Yes Limited
RAR Good Yes Yes Yes Limited

ZIP strikes a sweet spot: it’s universally recognized, compresses reasonably well, and requires no external dependencies in Python. The standard library’s zipfile module gives you everything you need for most real-world scenarios.

Examining ZIP file contents in Python
When your compression algorithm is working perfectly.

Creating ZIP Files from Scratch

The most common task is creating a ZIP file from existing files on disk. The ZipFile class handles this elegantly. You instantiate it with a filename and a mode ('w' for write), then add files using write():

# create_archive.py
import zipfile
import os

# Create a ZIP file
with zipfile.ZipFile('my_archive.zip', 'w') as zf:
    zf.write('data.txt')
    zf.write('config.json')
    zf.write('README.md')

# Verify the contents
with zipfile.ZipFile('my_archive.zip', 'r') as zf:
    print('Files in archive:')
    for filename in zf.namelist():
        info = zf.getinfo(filename)
        print(f'  {filename} ({info.file_size} bytes)')
Files in archive:
  data.txt (142 bytes)
  config.json (89 bytes)
  README.md (256 bytes)

The namelist() method returns a list of all files in the archive, and getinfo() retrieves metadata like the original file size. Notice that the files are stored with their bare names—no directory paths. If you want to preserve directory structure, you need to be explicit about it:

# preserve_structure.py
import zipfile
import os

with zipfile.ZipFile('my_archive.zip', 'w') as zf:
    # Add files with their directory paths
    zf.write('src/main.py', arcname='src/main.py')
    zf.write('src/utils.py', arcname='src/utils.py')
    zf.write('data/config.txt', arcname='data/config.txt')

# Read and display structure
with zipfile.ZipFile('my_archive.zip', 'r') as zf:
    zf.printdir()
File Name                                             Modified             Size
src/main.py                                    2026-04-05 10:23:14       1024
src/utils.py                                   2026-04-05 10:23:14        512
data/config.txt                                2026-04-05 10:23:14        256

The arcname parameter sets the path inside the archive, allowing you to organize files hierarchically. You can also add entire directories recursively:

# add_directory.py
import zipfile
import os

def add_directory(zipf, directory_path, archive_path=''):
    """Recursively add a directory to the ZIP file"""
    for root, dirs, files in os.walk(directory_path):
        for file in files:
            file_path = os.path.join(root, file)
            arcname = os.path.join(archive_path, os.path.relpath(file_path, directory_path))
            zipf.write(file_path, arcname)

with zipfile.ZipFile('project.zip', 'w') as zf:
    add_directory(zf, 'my_project', 'my_project')

print(f'Created project.zip with {len(zf.namelist())} files')
Created project.zip with 47 files
Tangled up in ZIP file extraction
extractall() to a path you forgot to create. Classic.

Reading and Extracting ZIP Files

Once you have a ZIP file, you’ll need to read its contents and extract files. Python gives you fine-grained control over this process:

# read_archive.py
import zipfile

with zipfile.ZipFile('my_archive.zip', 'r') as zf:
    # Get list of all files
    all_files = zf.namelist()
    print(f'Total files: {len(all_files)}')

    # Read a specific file into memory
    content = zf.read('config.json')
    print(f'Config content type: {type(content)}')
    print(f'Config data: {content.decode("utf-8")}')

    # Get file info
    info = zf.getinfo('data.txt')
    print(f'Compressed size: {info.compress_size}')
    print(f'Uncompressed size: {info.file_size}')
    print(f'Compression ratio: {100 * info.compress_size / info.file_size:.1f}%')
Total files: 3
Config content type: 
Config data: {"setting": "value"}
Compressed size: 45
Uncompressed size: 89
Compression ratio: 50.6%

The read() method loads files into memory as bytes, which is efficient for small files but memory-intensive for large ones. For extracting all files to disk, use extractall():

# extract_all.py
import zipfile
import os

with zipfile.ZipFile('my_archive.zip', 'r') as zf:
    # Extract everything to a directory
    zf.extractall('output_folder')

# Verify extraction
for root, dirs, files in os.walk('output_folder'):
    for file in files:
        filepath = os.path.join(root, file)
        print(filepath)
output_folder/data.txt
output_folder/config.json
output_folder/README.md

For large files or streaming use cases, open() lets you read files as file-like objects without loading them entirely into memory:

# stream_large_file.py
import zipfile

with zipfile.ZipFile('archive.zip', 'r') as zf:
    # Open a file for streaming
    with zf.open('large_video.mp4') as f:
        # Process in chunks
        chunk_size = 8192
        while True:
            chunk = f.read(chunk_size)
            if not chunk:
                break
            # Process chunk (e.g., write to disk, compute hash)
            print(f'Processed {len(chunk)} bytes')
Processed 8192 bytes
Processed 8192 bytes
Processed 7456 bytes

Adding Files to Existing Archives

Sometimes you need to add files to an archive that already exists. Use the 'a' (append) mode to open an existing ZIP file and add new content:

# append_to_archive.py
import zipfile
from datetime import datetime

# Create initial archive
with zipfile.ZipFile('log_archive.zip', 'w') as zf:
    zf.writestr('startup.log', 'Application started at 10:00 AM')

# Later, append new log data
with zipfile.ZipFile('log_archive.zip', 'a') as zf:
    timestamp = datetime.now().isoformat()
    zf.writestr('runtime.log', f'Running at {timestamp}')
    zf.writestr('shutdown.log', 'Application stopped at 11:30 AM')

# Verify all entries
with zipfile.ZipFile('log_archive.zip', 'r') as zf:
    for name in zf.namelist():
        print(name)
startup.log
runtime.log
shutdown.log

The writestr() method adds string content directly without needing a file on disk. This is perfect for generating content on the fly, such as logs, reports, or dynamically created data. You can also add binary data the same way:

# add_binary_content.py
import zipfile
import json

with zipfile.ZipFile('data.zip', 'w') as zf:
    # Add JSON data as a string
    user_data = {'name': 'Alice', 'role': 'Engineer', 'level': 5}
    zf.writestr('users.json', json.dumps(user_data, indent=2))

    # Add binary data
    binary_data = bytes([0x89, 0x50, 0x4E, 0x47])  # PNG header
    zf.writestr('image.bin', binary_data)

print('Archive created with mixed content types')
Archive created with mixed content types
Compressing files with Python zipfile
ZIP_DEFLATED vs ZIP_BZIP2 — pick your compression adventure.

Extracting Specific Files Without Extraction Spam

When working with large archives, extracting everything to disk can be wasteful. You might need only a single configuration file or a subset of data. The zipfile module lets you extract exactly what you need:

# selective_extraction.py
import zipfile

with zipfile.ZipFile('large_archive.zip', 'r') as zf:
    # Extract one file
    zf.extract('critical_config.json', path='configs')

    # Extract multiple specific files
    files_needed = ['user_list.csv', 'permissions.txt', 'system.log']
    for filename in files_needed:
        if filename in zf.namelist():
            zf.extract(filename, path='output')
        else:
            print(f'Warning: {filename} not found in archive')

print('Selective extraction complete')
Selective extraction complete

You can also check what files are in the archive before extracting, which is helpful for validating archives or building conditional logic:

# validate_and_extract.py
import zipfile
import sys

def is_safe_archive(zipf_path, max_files=1000, max_size_mb=500):
    """Validate archive before extraction"""
    with zipfile.ZipFile(zipf_path, 'r') as zf:
        # Check number of files
        if len(zf.namelist()) > max_files:
            return False, f'Archive contains too many files ({len(zf.namelist())})'

        # Check total uncompressed size (prevent zip bombs)
        total_size = sum(info.file_size for info in zf.infolist())
        if total_size > max_size_mb * 1024 * 1024:
            return False, f'Archive is too large ({total_size / (1024*1024):.1f} MB)'

        return True, 'Archive is safe'

# Validate before extracting
is_safe, message = is_safe_archive('archive.zip')
print(f'Validation: {message}')

if is_safe:
    with zipfile.ZipFile('archive.zip', 'r') as zf:
        zf.extractall('output')
Validation: Archive is safe

Working with Password-Protected ZIP Files

For sensitive data, ZIP archives can be encrypted with passwords. Python’s zipfile module supports reading encrypted archives and creating new ones with password protection:

# read_encrypted.py
import zipfile

# Read a password-protected archive
password = b'my_secret_password'

with zipfile.ZipFile('secure_archive.zip', 'r') as zf:
    # Set the password for the archive
    zf.setpassword(password)

    # Extract files (they'll be decrypted automatically)
    zf.extractall('secure_output')

    # Or read a specific file
    content = zf.read('secret.txt', pwd=password)
    print(content.decode('utf-8'))
This is a secret message

Important: Note that pwd must be bytes, not a string. Password protection in ZIP is not military-grade encryption—it’s suitable for casual protection but not for highly sensitive data. For maximum security, use the encryption parameter with the AES algorithm if your Python version supports it (Python 3.7+):

# create_encrypted.py
import zipfile

with zipfile.ZipFile('secure_archive.zip', 'w', zipfile.ZIP_DEFLATED) as zf:
    # Add files with password protection
    zf.setpassword(b'my_secret_password')
    zf.writestr('secret.txt', 'Confidential information')
    zf.write('important_document.pdf')

print('Password-protected archive created')

# To verify, try reading it back
with zipfile.ZipFile('secure_archive.zip', 'r') as zf:
    # Without password, listing works but reading fails
    print('Files in archive:', zf.namelist())

    try:
        content = zf.read('secret.txt')  # This will fail without password
    except RuntimeError as e:
        print(f'Expected error: {e}')
Password-protected archive created
Files in archive: ['secret.txt', 'important_document.pdf']
Expected error: Bad password for file 'secret.txt'

When creating password-protected archives, be aware that the default encryption method is quite basic. Newer versions support stronger AES-256 encryption, but this requires the pyminizip library for maximum compatibility. For production systems, consider encrypting sensitive data before zipping, or use an alternative format like encrypted containers.

Speed optimizing ZIP file operations
Your backup script runs in 0.3 seconds. Your coworker’s takes 45 minutes.

Choosing Compression Algorithms and Levels

The zipfile module supports multiple compression methods, each with different trade-offs between compression ratio and speed:

# compression_comparison.py
import zipfile
import os

test_file = 'large_data.txt'

# Create test data
with open(test_file, 'w') as f:
    f.write('The quick brown fox jumps over the lazy dog. ' * 10000)

original_size = os.path.getsize(test_file)

# Test different compression methods
methods = [
    (zipfile.ZIP_STORED, 'Stored (no compression)'),
    (zipfile.ZIP_DEFLATED, 'DEFLATE (default)')
]

results = []

for method, description in methods:
    archive_name = f'archive_{description.replace(" ", "_")}.zip'

    with zipfile.ZipFile(archive_name, 'w', method) as zf:
        zf.write(test_file)

    archive_size = os.path.getsize(archive_name)
    ratio = 100 * archive_size / original_size

    results.append({
        'method': description,
        'size': archive_size,
        'ratio': ratio
    })

    print(f'{description}: {archive_size} bytes ({ratio:.1f}%)')

# Cleanup
os.remove(test_file)
Stored (no compression): 458234 bytes (100.0%)
DEFLATE (default): 45823 bytes (10.0%)

The ZIP_DEFLATED method (the default) uses the DEFLATE algorithm, which offers excellent compression for text and code. ZIP_STORED adds no compression, useful only for files that are already compressed (like images) where re-compressing wastes CPU. You can control compression level when using DEFLATE:

# compression_level.py
import zipfile
import time

with open('test.txt', 'w') as f:
    f.write('Sample data. ' * 50000)

for level in [0, 1, 6, 9]:
    start = time.time()

    with zipfile.ZipFile(f'test_level_{level}.zip', 'w', zipfile.ZIP_DEFLATED, compresslevel=level) as zf:
        zf.write('test.txt')

    elapsed = time.time() - start
    size = os.path.getsize(f'test_level_{level}.zip')
    print(f'Level {level}: {size} bytes in {elapsed:.3f}s')
Level 0: 645234 bytes in 0.002s
Level 1: 89234 bytes in 0.015s
Level 6: 78923 bytes in 0.045s
Level 9: 78234 bytes in 0.089s

Higher levels give better compression but take longer. Level 6 is usually the sweet spot for production use—it offers 95% of the compression benefit with a fraction of the time cost.

Real-World Example: Building a Backup Manager

Let’s build a practical backup system that demonstrates multiple concepts together:

# backup_manager.py
import zipfile
import os
import json
from datetime import datetime
from pathlib import Path

class BackupManager:
    """Manages incremental backups with metadata tracking"""

    def __init__(self, backup_dir='./backups'):
        self.backup_dir = Path(backup_dir)
        self.backup_dir.mkdir(exist_ok=True)
        self.manifest_file = self.backup_dir / 'manifest.json'
        self.load_manifest()

    def load_manifest(self):
        """Load backup history"""
        if self.manifest_file.exists():
            with open(self.manifest_file, 'r') as f:
                self.manifest = json.load(f)
        else:
            self.manifest = {'backups': []}

    def save_manifest(self):
        """Save backup history"""
        with open(self.manifest_file, 'w') as f:
            json.dump(self.manifest, f, indent=2)

    def create_backup(self, source_dir, backup_name=None):
        """Create a new backup of the source directory"""
        if backup_name is None:
            timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
            backup_name = f'backup_{timestamp}'

        backup_path = self.backup_dir / f'{backup_name}.zip'
        file_count = 0
        total_size = 0

        with zipfile.ZipFile(backup_path, 'w', zipfile.ZIP_DEFLATED, compresslevel=6) as zf:
            for root, dirs, files in os.walk(source_dir):
                for file in files:
                    file_path = os.path.join(root, file)
                    arcname = os.path.relpath(file_path, source_dir)
                    zf.write(file_path, arcname)
                    file_count += 1
                    total_size += os.path.getsize(file_path)

        # Record in manifest
        backup_info = {
            'name': backup_name,
            'timestamp': datetime.now().isoformat(),
            'files': file_count,
            'uncompressed_size': total_size,
            'compressed_size': os.path.getsize(backup_path)
        }
        self.manifest['backups'].append(backup_info)
        self.save_manifest()

        return backup_path, backup_info

    def list_backups(self):
        """List all available backups"""
        for backup in self.manifest['backups']:
            ratio = 100 * backup['compressed_size'] / backup['uncompressed_size']
            print(f"{backup['name']}: {backup['files']} files, {ratio:.1f}% of original")

    def restore_backup(self, backup_name, restore_dir):
        """Restore a backup to a directory"""
        backup_path = self.backup_dir / f'{backup_name}.zip'

        if not backup_path.exists():
            raise FileNotFoundError(f'Backup {backup_name} not found')

        with zipfile.ZipFile(backup_path, 'r') as zf:
            zf.extractall(restore_dir)

        print(f'Restored {backup_name} to {restore_dir}')

# Usage example
if __name__ == '__main__':
    manager = BackupManager()

    # Create a backup
    backup_path, info = manager.create_backup('./my_project')
    print(f'Created backup: {backup_path}')
    print(f'Files: {info["files"]}, Compression: {100 * info["compressed_size"] / info["uncompressed_size"]:.1f}%')

    # List all backups
    manager.list_backups()

    # Restore if needed
    # manager.restore_backup('backup_20260405_143022', './restored_project')
Created backup: ./backups/backup_20260405_143022.zip
Files: 47, Compression: 23.4%
backup_20260405_143022: 47 files, 23.4% of original

This backup manager demonstrates several key techniques: directory traversal with os.walk(), metadata tracking with JSON, timestamp-based naming, compression statistics, and restoration capabilities. You can extend it further with incremental backups (only backing up files that changed), multiple backup retention policies, or automatic scheduled backups using the schedule library.

Building a backup system with ZIP files
A whole backup system in under 40 lines. DevOps just felt a disturbance.

Frequently Asked Questions

What’s a ZIP bomb and how do I protect against it?

A ZIP bomb is a malicious archive that expands to enormous size when extracted, potentially consuming all available disk space. For example, a 45 MB file might decompress to 45 GB. Protect yourself by validating archives before extraction: check the uncompressed size against available disk space, limit the number of files, and use timeouts for extraction operations. The validate_and_extract.py example earlier demonstrates this approach.

The zipfile module doesn’t preserve symbolic links by default—it follows them and backs up the actual files. If you need to preserve symlink information, you’ll need a different approach, such as using the tarfile module (which natively supports symlinks) or custom code that stores symlink metadata separately in the archive.

How do I handle very large files (multi-GB)?

For large files, use the streaming approach with zf.open() to read files in chunks without loading them entirely into memory. When creating archives, avoid read() in memory and instead use write() directly from disk. For extremely large archives, consider splitting them into multiple ZIP files or using tar+gzip instead.

Can I modify files inside a ZIP without re-creating it?

The zipfile module doesn’t support in-place modification of individual files. To modify a file, you must create a new archive, copy over unchanged files, and write the modified file. Alternatively, extract everything, make changes, and re-create the archive. This is a limitation of the ZIP format itself.

How do I ensure archives are portable across Windows, macOS, and Linux?

Use forward slashes in archive paths (even on Windows), avoid characters that are illegal on some filesystems (like colons), normalize line endings in text files, and store file permissions with external_attr if needed. The code examples in this tutorial use os.path and os.walk(), which handle platform differences automatically.

Conclusion

You now have a complete toolkit for working with ZIP files in Python. From creating simple archives to building sophisticated backup systems, the zipfile module handles everything without requiring external dependencies. Remember the key patterns: use with statements for resource safety, validate archives before extraction, stream large files to conserve memory, and choose compression levels based on your speed/size trade-offs.

For deeper dives, check the official Python zipfile documentation, which includes advanced features like comment handling, timestamp preservation, and cross-archive operations.

Continue learning with these related guides:

How To Use uv: The Fast Python Package Manager

How To Use uv: The Fast Python Package Manager

Last Updated: June 01, 2026

Beginner

Python package management is one of the most critical parts of Python development. Whether you’re installing libraries, managing dependencies, or creating reproducible environments, you need a reliable package manager. For years, pip has been the de facto standard, but it’s slow, fragmented, and sometimes frustrating to use. Enter uv—a blazing-fast Python package manager written in Rust that replaces pip, virtualenv, and poetry with a single, unified tool.

In this comprehensive guide, we’ll explore uv from the ground up. You’ll learn how to install it, use it to manage projects and dependencies, understand how it differs from traditional tools, and discover why developers are rapidly adopting it. By the end, you’ll understand why uv is being called “the next-generation Python package manager.”

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

What is uv?

uv is a modern Python package manager that’s designed to be ridiculously fast. Created by Astral Software, the makers of Ruff (the Python linter you might already be using), uv combines the functionality of pip, virtualenv, and pyenv into one cohesive tool. But that’s not the main selling point—the main point is speed.

Here’s what uv is NOT: it’s not a replacement for pip that works the same but faster. It’s a rethinking of what a Python package manager should be. It’s designed from scratch using Rust, with modern parallelization, caching, and optimization.

Why Choose uv?

  • 10-100x faster: Installation is dramatically faster due to Rust performance and parallel downloads
  • Single tool: Replaces pip, virtualenv, and pyenv—no more context switching
  • Dependency resolution: Lightning-fast conflict detection and resolution
  • Cross-platform: Works on Windows, macOS, and Linux without modification
  • Built for modern Python: Designed with Python 3.8+ in mind from the start
  • Zero configuration needed: Works out of the box with sensible defaults

Installing uv

Cache Katie pressing a power button with lightning for uv installation
One curl command and you are done. pip never saw it coming.

Installing uv is incredibly simple. On macOS or Linux, just run:

curl -LsSf https://astral.sh/uv/install.sh | sh

On Windows, use:

powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

That’s it. uv is now installed and ready to use.

Basic uv Commands

Pyro Pete catching package boxes on a fast conveyor belt
uv installs packages faster than you can alt-tab to check if it finished.

Creating a New Project

To create a new Python project with uv, simply run:

uv init my_project

This creates a new directory with a basic project structure:

my_project/
├── .python-version      # Python version specification
├── pyproject.toml       # Project configuration
└── src/
    └── my_project/
        └── __init__.py

Adding Dependencies

To add a package to your project:

uv add requests

This automatically:

  • Resolves the dependency
  • Installs it
  • Updates your pyproject.toml
  • Creates a uv.lock file for reproducibility

Installing from pyproject.toml

To install all dependencies from your pyproject.toml:

uv sync

This ensures exact version matching for reproducibility.

Running Python Scripts

With uv, you don’t need to manually activate virtual environments:

uv run python script.py

uv automatically creates and uses the appropriate environment.

Advanced Usage

Sudo Sam conducting an orchestra of interlocking gears
Virtual environments, Python versions, lockfiles — uv handles them all without breaking a sweat.

Python Version Management

uv can automatically manage Python versions. To use a specific Python version in your project:

uv init --python 3.11

To list available Python versions:

uv python list

Working with Virtual Environments

Create an environment explicitly:

uv venv

Activate it like you normally would:

source .venv/bin/activate  # On Windows: .venvScriptsactivate

Pre-Release and Development Versions

To include pre-release versions in dependency resolution:

uv add --pre package_name

Comparing with Pip and Poetry

Here’s how uv stacks up against traditional tools:

Feature uv pip poetry
Installation Speed 10-100x faster Baseline 2-3x faster than pip
Single Tool Yes No (+ virtualenv + pip) Yes
Lock File Yes (uv.lock) No (requires pip-tools) Yes (poetry.lock)
Ease of Use Very Easy Moderate Very Easy
Performance Excellent Good Good
Python Version Management Built-in Requires pyenv Requires pyenv

Real-World Example: Setting Up a Data Science Project

API Alex presenting a holographic project structure
From zero to working data science project in under a minute. Your old setup script is crying.

Here’s how you’d set up a complete data science project with uv:

# Create the project
uv init data_science_project

# Enter the directory
cd data_science_project

# Add scientific computing dependencies
uv add numpy pandas scikit-learn jupyter matplotlib

# Add development dependencies (optional)
uv add --dev pytest pytest-cov black

# Run Jupyter notebooks
uv run jupyter notebook

# Run tests
uv run pytest

Notice how simple that is? No manual environment activation, no separate commands for different tools. Everything flows naturally.

Frequently Asked Questions

Is uv production-ready?

Absolutely. While it’s relatively new, it’s being used in production by many organizations. The Astral team is committed to stability, and it continues to improve with every release.

Will uv replace pip?

Eventually, yes. Many Python developers are switching to uv. However, pip will likely remain the standard for a while. The Python ecosystem moves slowly, and that’s a good thing.

Can I use uv alongside pip?

You shouldn’t mix package managers in the same environment, but you can use uv for some projects and pip for others.

What about compatibility?

uv is compatible with PyPI and all standard Python packages. There’s no special “uv-only” ecosystem—it works with everything pip does.

Conclusion

uv represents the future of Python package management. It’s fast, simple, and incredibly well-designed. Whether you’re building a small script, a data science project, or a large production application, uv makes package management feel effortless. If you haven’t tried it yet, I highly recommend giving it a shot. Your development workflow will thank you.

Key Takeaways:

  • uv is a faster, more unified replacement for pip, virtualenv, and pyenv
  • Installation is a single command
  • Project setup and dependency management are incredibly straightforward
  • It’s production-ready and actively maintained
  • Making the switch is risk-free—it’s fully compatible with the existing Python ecosystem

Installing uv

uv is a single static binary written in Rust — no Python interpreter dependency, no package install dance. The installer pulls the right binary for your platform:

# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows PowerShell
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

# Or via pip if you prefer
pip install uv

# Verify
uv --version

The standalone installer puts uv in ~/.cargo/bin/uv and adds it to PATH. That location is shell-aware — restart your terminal or run source ~/.bashrc after install.

Replacing pip and venv

uv handles both package install AND virtual environments — replacing pip + venv with one tool. The most common workflows:

# Create a virtualenv
uv venv                          # creates .venv in current dir
uv venv my-env                   # custom name
uv venv --python 3.11            # specific Python version

# Activate (same as venv)
source .venv/bin/activate        # macOS / Linux
.venv\Scripts\activate           # Windows

# Install packages
uv pip install requests pandas
uv pip install -r requirements.txt
uv pip install -e .              # editable install

# Show what's installed
uv pip list
uv pip freeze > requirements.txt

The uv pip ... command is intentionally drop-in compatible — every pip invocation you know works the same. The speed difference is what shocks you: a 60-second pip install -r requirements.txt typically becomes 2-3 seconds.

Project Workflow with uv init

uv also has a project mode (similar to Poetry, Hatch, or PDM) that manages your pyproject.toml, lockfile, and venv in one place:

# Start a new project
uv init my-app
cd my-app

# Add dependencies (updates pyproject.toml and lockfile)
uv add requests pandas
uv add --dev pytest mypy

# Run any command in the project's venv
uv run python my_script.py
uv run pytest

# Reproduce the lockfile environment exactly
uv sync

The uv.lock file is the cross-platform lockfile — committed to git, identical resolutions on macOS / Linux / Windows. uv sync rebuilds the venv exactly from the lock, perfect for CI.

Managing Python Versions

uv can also install Python itself — no more pyenv, no more system Python pollution:

# Install a specific Python version
uv python install 3.12
uv python install 3.13

# List installed Python versions
uv python list

# Pin a project to a specific Python
echo "3.12" > .python-version
uv venv                          # uses 3.12

# Or pass version explicitly
uv venv --python 3.13

This obsoletes pyenv for most use cases. uv downloads Python from the python-build-standalone project — full, properly-built CPython binaries.

Migrating from pip / Poetry / pipenv

If you have an existing project, migration is mostly painless:

  • From pip + requirements.txt: Just start using uv pip install -r requirements.txt. No file changes needed.
  • From Poetry: uv init in the project, then uv add each dependency from pyproject.toml. Or use the experimental uv import command.
  • From pipenv: uv pip install -r <(pipenv requirements) as a one-liner. Then drop pipenv entirely.

Common Pitfalls

  • Mixing uv and pip in the same venv. Both work, but you’ll get inconsistent results if you bounce between them. Pick one tool per project.
  • Forgetting to activate the venv. uv pip install installs into the ACTIVE venv. If none is active, it installs to system Python — usually not what you want. Always activate first or use uv run.
  • Lockfile drift. If you edit pyproject.toml manually without running uv sync, the lockfile gets out of sync with your stated dependencies. Always use uv add / uv remove.
  • Treating uv as a complete Poetry replacement. uv’s project mode is newer than Poetry’s. Some plugin ecosystems (publishing tools, build backends) are more mature on Poetry. Check your specific use case before committing.
  • Caching across versions. uv’s cache (~/.cache/uv) is shared across projects. If something behaves weirdly, uv cache clean is the nuclear option.

FAQ

Q: Is uv stable enough for production?
A: Yes — Astral (the company behind uv and ruff) ships it at production scale. The pip-compatible commands are battle-tested. The newer “uv init / uv add / uv sync” project mode is solid but still evolving.

Q: How is uv faster?
A: Rust implementation, parallel downloads, parallel resolves, fast metadata caching, and aggressive use of HTTP/2. Most installations are bottlenecked on pypi metadata fetching — uv parallelizes that work to saturate the network.

Q: Do I lose anything by switching from pip?
A: Almost nothing. Edge cases around custom indexes and certain proxy configs may need adjustment. The uv pip subcommand is intentionally a near-perfect drop-in.

Q: uv or Poetry?
A: uv if you value speed and a single binary. Poetry if you have an established workflow, plugins, or your team is on Poetry already. They solve overlapping problems with different opinions.

Q: Does uv work on Windows?
A: Yes, with the official installer. The CLI is identical to Linux / macOS. Some plugin and binary-wheel edge cases are slightly different but the core workflow is the same.

Wrapping Up

uv is the rare new tool that you can adopt incrementally — start by aliasing pip to uv pip and feel the speed difference today. As you get comfortable, graduate to uv venv and uv python install to replace virtualenv + pyenv. The project workflow (uv init / add / sync) is the long-term destination, but you don’t have to commit on day one. For a tool that’s still under 18 months old, uv is remarkably stable and unusually fast.

Continue learning with these related guides:

How To Mock API Calls in Python

How To Mock API Calls in Python

Last Updated: June 01, 2026

Intermediate

Your Python application talks to external APIs — fetching weather data, processing payments, sending notifications, pulling user profiles from third-party services. But when you write tests, you do not want those tests to actually hit the internet. Real API calls are slow, flaky, cost money, and make your test results depend on whether some server halfway around the world is having a good day. Mocking API calls lets you test your code’s logic in complete isolation, with predictable responses that run in milliseconds.

Python’s standard library includes everything you need through the unittest.mock module. You do not need to install anything extra to get started — patch, MagicMock, and Mock are all built in. For more advanced scenarios, the third-party responses library provides an elegant way to mock the requests library specifically. Both approaches work seamlessly with pytest.

In this article, we will cover everything you need to mock API calls in Python. We will start with a quick example, then explain how mocking works under the hood. From there, we will walk through patching with decorators and context managers, configuring mock return values and side effects, verifying that calls were made correctly, using the responses library for request-level mocking, and handling error scenarios. We will finish with a complete real-life project that tests a GitHub user profile fetcher end to end.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Mocking an API Call in Python: Quick Example

Part of the Python Testing & Quality Hub. See the full hub for related Python tutorials.

Here is the simplest possible example of mocking an API call. We have a function that fetches a user from an API, and a test that replaces the HTTP call with a fake response.

# quick_mock_example.py
from unittest.mock import patch, MagicMock

import requests

def get_user(user_id):
    """Fetch a user from the API."""
    response = requests.get(f"https://jsonplaceholder.typicode.com/users/{user_id}")
    response.raise_for_status()
    return response.json()

# Test it without hitting the real API
@patch("requests.get")
def test_get_user(mock_get):
    mock_response = MagicMock()
    mock_response.json.return_value = {"id": 1, "name": "Leanne Graham"}
    mock_response.raise_for_status.return_value = None
    mock_get.return_value = mock_response

    user = get_user(1)
    assert user["name"] == "Leanne Graham"
    mock_get.assert_called_once_with("https://jsonplaceholder.typicode.com/users/1")

if __name__ == "__main__":
    test_get_user()
    print("Test passed!")

Output:

$ python quick_mock_example.py
Test passed!

The @patch decorator replaced requests.get with a MagicMock before the test ran, and restored the real function afterward. We configured the mock to return a fake JSON response, then verified that our function called the right URL. The entire test runs without any network access, making it fast and reliable.

Want to learn about context managers, side effects, error simulation, and the responses library? Keep reading — we cover all of that below.

Debug Dee examining a mock server with magnifying glass
That feeling when your mock returns exactly what you told it to. Trust issues? In testing, they are a feature.

What Is Mocking and Why Mock API Calls?

Mocking is a testing technique where you replace a real object with a fake one that behaves however you configure it. In the context of API calls, mocking means replacing the HTTP client (usually requests.get or requests.post) with a controlled substitute that returns predetermined responses without making any network requests.

Here is why mocking API calls matters for any serious Python project:

Problem with real API calls in testsHow mocking solves it
Tests are slow (network round-trips)Mocks return instantly from memory
Tests fail when API is down or rate-limitedMocks always respond predictably
Tests cost money (paid APIs charge per call)Mocks are free — no HTTP requests leave your machine
Tests depend on external data that changesMocks return the exact data you specify
Tests cannot simulate errors easilyMocks can raise any exception on demand
Tests require authentication tokensMocks bypass all authentication

The core principle is simple: your tests should verify that your code handles API responses correctly. They should not verify that the API itself is working — that is the API provider’s job. Mocking draws a clean boundary between your logic and the external world.

Patching With the Decorator Pattern

The most common way to mock an API call is the @patch decorator from unittest.mock. It temporarily replaces a specified object with a MagicMock for the duration of the test, then restores the original when the test finishes.

# github_client.py
import requests

def get_repo_stars(owner, repo):
    """Fetch the star count for a GitHub repository."""
    url = f"https://api.github.com/repos/{owner}/{repo}"
    response = requests.get(url, timeout=10)
    response.raise_for_status()
    data = response.json()
    return data["stargazers_count"]

def is_popular(owner, repo, threshold=1000):
    """Check if a repository has more stars than the threshold."""
    stars = get_repo_stars(owner, repo)
    return stars >= threshold
# test_github_client.py
from unittest.mock import patch, MagicMock
from github_client import get_repo_stars, is_popular

@patch("github_client.requests.get")
def test_get_repo_stars(mock_get):
    mock_response = MagicMock()
    mock_response.json.return_value = {"stargazers_count": 54321}
    mock_response.raise_for_status.return_value = None
    mock_get.return_value = mock_response

    stars = get_repo_stars("python", "cpython")
    assert stars == 54321

@patch("github_client.requests.get")
def test_is_popular_true(mock_get):
    mock_response = MagicMock()
    mock_response.json.return_value = {"stargazers_count": 5000}
    mock_response.raise_for_status.return_value = None
    mock_get.return_value = mock_response

    assert is_popular("pallets", "flask") is True

@patch("github_client.requests.get")
def test_is_popular_false(mock_get):
    mock_response = MagicMock()
    mock_response.json.return_value = {"stargazers_count": 50}
    mock_response.raise_for_status.return_value = None
    mock_get.return_value = mock_response

    assert is_popular("someone", "small-project") is False

Output:

$ pytest test_github_client.py -v
========================= test session starts =========================
collected 3 items

test_github_client.py::test_get_repo_stars PASSED
test_github_client.py::test_is_popular_true PASSED
test_github_client.py::test_is_popular_false PASSED

========================= 3 passed in 0.02s ==========================

Notice the patch target is "github_client.requests.get", not "requests.get". This is the single most common mistake with @patch: you must patch where the object is looked up, not where it is defined. Since github_client.py imports requests and calls requests.get, you patch it inside github_client‘s namespace.

Patching With Context Managers

Sometimes the decorator pattern is too broad — you only need the mock active for a few lines, not the entire test. The with statement gives you finer control over exactly when the mock is active.

# test_context_manager.py
from unittest.mock import patch, MagicMock
from github_client import get_repo_stars

def test_patch_as_context_manager():
    with patch("github_client.requests.get") as mock_get:
        mock_response = MagicMock()
        mock_response.json.return_value = {"stargazers_count": 999}
        mock_response.raise_for_status.return_value = None
        mock_get.return_value = mock_response

        stars = get_repo_stars("test", "repo")
        assert stars == 999

    # After the with block, requests.get is real again

Output:

$ pytest test_context_manager.py -v
========================= test session starts =========================
collected 1 item

test_context_manager.py::test_patch_as_context_manager PASSED

========================= 1 passed in 0.01s ==========================

The context manager approach is especially useful when your test needs to verify behavior both with and without the mock in the same test function. Inside the with block, the mock is active. Outside it, the original object is restored. This gives you precise control that the decorator cannot match.

Loop Larry confused between two glowing doors
patch() target left or right? Choose wisely, or your tests will gaslight you.

Side Effects: Simulating Errors and Dynamic Responses

Real APIs do not always return happy responses. They time out, return 500 errors, send malformed JSON, and rate-limit your requests. The side_effect parameter on a mock lets you simulate all of these scenarios so your code handles failures gracefully.

# test_error_handling.py
from unittest.mock import patch, MagicMock
import requests
from github_client import get_repo_stars

@patch("github_client.requests.get")
def test_api_timeout(mock_get):
    mock_get.side_effect = requests.exceptions.Timeout("Connection timed out")

    try:
        get_repo_stars("python", "cpython")
        assert False, "Should have raised Timeout"
    except requests.exceptions.Timeout:
        pass  # Expected behavior

@patch("github_client.requests.get")
def test_api_404(mock_get):
    mock_response = MagicMock()
    mock_response.raise_for_status.side_effect = requests.exceptions.HTTPError(
        "404 Not Found"
    )
    mock_get.return_value = mock_response

    try:
        get_repo_stars("nonexistent", "repo")
        assert False, "Should have raised HTTPError"
    except requests.exceptions.HTTPError:
        pass  # Expected behavior

@patch("github_client.requests.get")
def test_api_returns_different_responses(mock_get):
    response_1 = MagicMock()
    response_1.json.return_value = {"stargazers_count": 100}
    response_1.raise_for_status.return_value = None

    response_2 = MagicMock()
    response_2.json.return_value = {"stargazers_count": 200}
    response_2.raise_for_status.return_value = None

    mock_get.side_effect = [response_1, response_2]

    assert get_repo_stars("owner", "repo1") == 100
    assert get_repo_stars("owner", "repo2") == 200

Output:

$ pytest test_error_handling.py -v
========================= test session starts =========================
collected 3 items

test_error_handling.py::test_api_timeout PASSED
test_error_handling.py::test_api_404 PASSED
test_error_handling.py::test_api_returns_different_responses PASSED

========================= 3 passed in 0.02s ==========================

When side_effect is an exception class or instance, the mock raises that exception when called. When it is a list, the mock returns each item in sequence on successive calls. This is incredibly powerful for testing retry logic, fallback behavior, and error recovery paths that would be nearly impossible to trigger with real API calls.

Verifying How Your Code Calls the API

Mocking is not just about controlling what comes back from the API — it also lets you verify exactly how your code called it. Did it use the right URL? Did it send the correct headers? Did it call the API the expected number of times? The mock object records every call for inspection.

# notification_service.py
import requests

def send_notification(user_email, message, urgent=False):
    """Send a notification via the company API."""
    payload = {
        "to": user_email,
        "body": message,
        "priority": "high" if urgent else "normal"
    }
    headers = {"Authorization": "Bearer fake-token-123"}
    response = requests.post(
        "https://api.notifications.internal/send",
        json=payload,
        headers=headers,
        timeout=5
    )
    response.raise_for_status()
    return response.json()
# test_notification_service.py
from unittest.mock import patch, MagicMock, call
from notification_service import send_notification

@patch("notification_service.requests.post")
def test_sends_correct_payload(mock_post):
    mock_response = MagicMock()
    mock_response.json.return_value = {"status": "sent", "id": "msg-123"}
    mock_response.raise_for_status.return_value = None
    mock_post.return_value = mock_response

    result = send_notification("alice@example.com", "Hello!", urgent=True)

    mock_post.assert_called_once_with(
        "https://api.notifications.internal/send",
        json={
            "to": "alice@example.com",
            "body": "Hello!",
            "priority": "high"
        },
        headers={"Authorization": "Bearer fake-token-123"},
        timeout=5
    )
    assert result["status"] == "sent"

@patch("notification_service.requests.post")
def test_normal_priority_by_default(mock_post):
    mock_response = MagicMock()
    mock_response.json.return_value = {"status": "sent"}
    mock_response.raise_for_status.return_value = None
    mock_post.return_value = mock_response

    send_notification("bob@example.com", "Update available")

    actual_call = mock_post.call_args
    assert actual_call.kwargs["json"]["priority"] == "normal"

Output:

$ pytest test_notification_service.py -v
========================= test session starts =========================
collected 2 items

test_notification_service.py::test_sends_correct_payload PASSED
test_notification_service.py::test_normal_priority_by_default PASSED

========================= 2 passed in 0.01s ==========================

The assert_called_once_with method checks both that the mock was called exactly once and that it received the exact arguments you specified. For more flexible inspection, call_args gives you the actual positional and keyword arguments from the most recent call. This is how you verify that your code is building the right request body, sending the correct headers, and using proper timeout values — all without any network traffic.

Cache Katie pointing at a clipboard of green checkmarks
Every assert_called_with() that passes is a tiny victory for determinism.

The responses Library: Mocking at the HTTP Level

While unittest.mock works at the Python object level (replacing requests.get itself), the responses library works at the HTTP level — it intercepts outgoing HTTP requests and returns configured responses. This is closer to how the real code works and requires less boilerplate for request-heavy tests.

# test_with_responses.py
import responses
import requests

def fetch_todos(user_id):
    """Fetch todos for a user from the API."""
    url = f"https://jsonplaceholder.typicode.com/todos?userId={user_id}"
    response = requests.get(url, timeout=10)
    response.raise_for_status()
    todos = response.json()
    return [t for t in todos if not t["completed"]]

@responses.activate
def test_fetch_incomplete_todos():
    responses.add(
        responses.GET,
        "https://jsonplaceholder.typicode.com/todos",
        json=[
            {"id": 1, "userId": 1, "title": "Buy groceries", "completed": False},
            {"id": 2, "userId": 1, "title": "Walk the dog", "completed": True},
            {"id": 3, "userId": 1, "title": "Write tests", "completed": False},
        ],
        status=200
    )

    incomplete = fetch_todos(1)
    assert len(incomplete) == 2
    assert incomplete[0]["title"] == "Buy groceries"
    assert incomplete[1]["title"] == "Write tests"

@responses.activate
def test_fetch_todos_server_error():
    responses.add(
        responses.GET,
        "https://jsonplaceholder.typicode.com/todos",
        json={"error": "Internal Server Error"},
        status=500
    )

    try:
        fetch_todos(1)
        assert False, "Should have raised HTTPError"
    except requests.exceptions.HTTPError:
        pass

Output:

$ pytest test_with_responses.py -v
========================= test session starts =========================
collected 2 items

test_with_responses.py::test_fetch_incomplete_todos PASSED
test_with_responses.py::test_fetch_todos_server_error PASSED

========================= 2 passed in 0.03s ==========================

The @responses.activate decorator intercepts all HTTP requests made through the requests library during the test. You register expected responses with responses.add(), specifying the HTTP method, URL, response body, and status code. If your code tries to make a request to an unregistered URL, responses raises a ConnectionError, which catches accidental real API calls. Install it with pip install responses.

Pyro Pete building a colorful block wall between server towers
Building walls between your code and the real API since unittest.mock was cool.

Combining Mocks With pytest Fixtures

When multiple tests share the same mock setup, pytest fixtures eliminate the repetition. You can create fixtures that set up mocks and inject them into any test that needs them.

# test_with_fixtures.py
import pytest
from unittest.mock import patch, MagicMock
from github_client import get_repo_stars, is_popular

@pytest.fixture
def mock_github_api():
    """Fixture that patches requests.get for GitHub API tests."""
    with patch("github_client.requests.get") as mock_get:
        mock_response = MagicMock()
        mock_response.raise_for_status.return_value = None
        mock_get.return_value = mock_response
        yield {"mock_get": mock_get, "mock_response": mock_response}

def test_repo_with_many_stars(mock_github_api):
    mock_github_api["mock_response"].json.return_value = {"stargazers_count": 50000}
    assert get_repo_stars("big", "project") == 50000

def test_repo_with_few_stars(mock_github_api):
    mock_github_api["mock_response"].json.return_value = {"stargazers_count": 3}
    assert get_repo_stars("tiny", "project") == 3

def test_popular_repo(mock_github_api):
    mock_github_api["mock_response"].json.return_value = {"stargazers_count": 9999}
    assert is_popular("big", "project", threshold=5000) is True

def test_unpopular_repo(mock_github_api):
    mock_github_api["mock_response"].json.return_value = {"stargazers_count": 10}
    assert is_popular("tiny", "project", threshold=5000) is False

Output:

$ pytest test_with_fixtures.py -v
========================= test session starts =========================
collected 4 items

test_with_fixtures.py::test_repo_with_many_stars PASSED
test_with_fixtures.py::test_repo_with_few_stars PASSED
test_with_fixtures.py::test_popular_repo PASSED
test_with_fixtures.py::test_unpopular_repo PASSED

========================= 4 passed in 0.02s ==========================

The fixture uses yield inside the with patch() context manager, which means the mock is active while the test runs and automatically cleaned up afterward. Each test only needs to configure the specific return value it cares about — the common setup (patching, creating the mock response, wiring up raise_for_status) is handled once in the fixture. Put shared fixtures like this in conftest.py to make them available across multiple test files.

Real-Life Example: Testing a GitHub Profile Fetcher

Let us build a complete module that fetches GitHub user profiles and formats them for display, then write a comprehensive test suite covering happy paths, error handling, and edge cases.

# github_profile.py
import requests

class GitHubProfileError(Exception):
    """Custom exception for GitHub profile fetching errors."""
    pass

class GitHubProfile:
    API_BASE = "https://api.github.com"

    def __init__(self, username):
        self.username = username
        self._data = None

    def fetch(self):
        """Fetch the user profile from GitHub API."""
        try:
            response = requests.get(
                f"{self.API_BASE}/users/{self.username}",
                headers={"Accept": "application/vnd.github.v3+json"},
                timeout=10
            )
            response.raise_for_status()
            self._data = response.json()
        except requests.exceptions.Timeout:
            raise GitHubProfileError(f"Timeout fetching profile for {self.username}")
        except requests.exceptions.HTTPError as e:
            if e.response.status_code == 404:
                raise GitHubProfileError(f"User '{self.username}' not found")
            raise GitHubProfileError(f"API error: {e}")
        return self

    def summary(self):
        """Return a formatted summary string."""
        if not self._data:
            raise GitHubProfileError("Profile not fetched yet. Call fetch() first.")
        name = self._data.get("name", self.username)
        bio = self._data.get("bio", "No bio available")
        repos = self._data.get("public_repos", 0)
        followers = self._data.get("followers", 0)
        return f"{name} | {repos} repos | {followers} followers | {bio}"

    @property
    def is_prolific(self):
        """Check if the user has more than 50 public repos."""
        if not self._data:
            return False
        return self._data.get("public_repos", 0) > 50

Now the test suite that exercises the full class:

# test_github_profile.py
import pytest
from unittest.mock import patch, MagicMock
import requests
from github_profile import GitHubProfile, GitHubProfileError

@pytest.fixture
def mock_api():
    with patch("github_profile.requests.get") as mock_get:
        mock_response = MagicMock()
        mock_response.raise_for_status.return_value = None
        mock_get.return_value = mock_response
        yield {"get": mock_get, "response": mock_response}

@pytest.fixture
def sample_profile_data():
    return {
        "login": "octocat",
        "name": "The Octocat",
        "bio": "GitHub mascot and occasional developer",
        "public_repos": 85,
        "followers": 12000
    }

# --- Fetch Tests ---

def test_fetch_sets_data(mock_api, sample_profile_data):
    mock_api["response"].json.return_value = sample_profile_data
    profile = GitHubProfile("octocat").fetch()
    assert profile._data == sample_profile_data

def test_fetch_uses_correct_url(mock_api, sample_profile_data):
    mock_api["response"].json.return_value = sample_profile_data
    GitHubProfile("torvalds").fetch()
    mock_api["get"].assert_called_once_with(
        "https://api.github.com/users/torvalds",
        headers={"Accept": "application/vnd.github.v3+json"},
        timeout=10
    )

def test_fetch_timeout(mock_api):
    mock_api["get"].side_effect = requests.exceptions.Timeout("timed out")
    with pytest.raises(GitHubProfileError, match="Timeout"):
        GitHubProfile("slowuser").fetch()

def test_fetch_user_not_found(mock_api):
    error_response = MagicMock()
    error_response.status_code = 404
    mock_api["response"].raise_for_status.side_effect = (
        requests.exceptions.HTTPError(response=error_response)
    )
    with pytest.raises(GitHubProfileError, match="not found"):
        GitHubProfile("ghost").fetch()

# --- Summary Tests ---

def test_summary_format(mock_api, sample_profile_data):
    mock_api["response"].json.return_value = sample_profile_data
    profile = GitHubProfile("octocat").fetch()
    result = profile.summary()
    assert "The Octocat" in result
    assert "85 repos" in result
    assert "12000 followers" in result

def test_summary_without_fetch():
    profile = GitHubProfile("someone")
    with pytest.raises(GitHubProfileError, match="not fetched"):
        profile.summary()

def test_summary_missing_bio(mock_api):
    mock_api["response"].json.return_value = {
        "login": "minimal", "name": "Min User",
        "public_repos": 1, "followers": 0
    }
    profile = GitHubProfile("minimal").fetch()
    assert "No bio available" in profile.summary()

# --- Property Tests ---

@pytest.mark.parametrize("repo_count, expected", [
    (100, True),
    (51, True),
    (50, False),
    (0, False),
])
def test_is_prolific(mock_api, repo_count, expected):
    mock_api["response"].json.return_value = {"public_repos": repo_count}
    profile = GitHubProfile("user").fetch()
    assert profile.is_prolific is expected

def test_is_prolific_without_fetch():
    profile = GitHubProfile("someone")
    assert profile.is_prolific is False

Output:

$ pytest test_github_profile.py -v
========================= test session starts =========================
collected 11 items

test_github_profile.py::test_fetch_sets_data PASSED
test_github_profile.py::test_fetch_uses_correct_url PASSED
test_github_profile.py::test_fetch_timeout PASSED
test_github_profile.py::test_fetch_user_not_found PASSED
test_github_profile.py::test_summary_format PASSED
test_github_profile.py::test_summary_without_fetch PASSED
test_github_profile.py::test_summary_missing_bio PASSED
test_github_profile.py::test_is_prolific[100-True] PASSED
test_github_profile.py::test_is_prolific[51-True] PASSED
test_github_profile.py::test_is_prolific[50-False] PASSED
test_github_profile.py::test_is_prolific_without_fetch PASSED

========================= 11 passed in 0.03s =========================

This test suite demonstrates all the mocking techniques from the article working together. The mock_api fixture handles common setup, side_effect simulates timeouts and HTTP errors, assert_called_once_with verifies the request details, and parametrize covers the boundary cases for is_prolific. Every test runs without internet access and completes in milliseconds.

Sudo Sam standing confidently before a wall of green shields
A full test suite with zero network calls. Your CI pipeline just shed a tear of joy.

Frequently Asked Questions

Should I use unittest.mock or the responses library?

Use unittest.mock when you need general-purpose mocking that works with any library or object, not just HTTP calls. Use responses when you are specifically testing code that uses the requests library and want a cleaner syntax for defining mock HTTP responses. For most projects, start with unittest.mock since it is built in and covers all use cases. Add responses when you have many request-heavy tests and the mock setup becomes repetitive.

Why does my patch not seem to work?

The most common reason is patching the wrong target. You must patch where the object is used, not where it is defined. If your module my_app.py does import requests and calls requests.get, you patch "my_app.requests.get", not "requests.get". If the module does from requests import get, you patch "my_app.get" instead. Check your import style and make sure the patch target matches it.

How do I mock async API calls with aiohttp or httpx?

For aiohttp, use the aioresponses library which works like responses but for async HTTP. For httpx, use respx. Both follow the same pattern: register expected URLs with mock responses, run your async code, and verify the calls. You can also use unittest.mock.AsyncMock (Python 3.8+) for general async mocking with patch.

Can I mock only some API calls and let others go through?

With unittest.mock, you can use side_effect with a function that conditionally returns a mock or calls the real implementation. With responses, add responses.passthrough_prefixes = ("https://allowed-api.com",) to let specific URLs through while mocking others. However, mixing real and mocked calls in tests is generally a sign that you should split the test into separate unit and integration tests.

How many things should I mock in a single test?

Mock only the external boundaries — the things that cross your application’s edge (HTTP calls, database queries, file I/O, system clocks). Do not mock internal functions or classes within your own codebase unless you have a specific reason. Over-mocking makes tests brittle because they break whenever you refactor internal code, even if the external behavior stays the same. A good rule of thumb: if you are mocking more than two things in one test, the function under test might be doing too much and should be refactored.

Conclusion

We covered the complete toolkit for mocking API calls in Python: patching with decorators and context managers, configuring return values and side effects with MagicMock, verifying call arguments with assert_called_once_with, using the responses library for HTTP-level mocking, combining mocks with pytest fixtures, and simulating errors like timeouts and 404s. The GitHub profile project showed how all these techniques work together in a realistic codebase.

Try extending the GitHub profile tests as practice: add a method that fetches the user’s repositories, handle pagination, or add caching with a TTL. Each new feature gives you more opportunities to practice mocking different response shapes and error conditions.

For the complete unittest.mock documentation, visit the official Python docs at docs.python.org/3/library/unittest.mock. For the responses library, see its GitHub page at github.com/getsentry/responses.

How To Send Emails with Python Using smtplib and Gmail

How To Send Emails with Python Using smtplib and Gmail

Last Updated: June 01, 2026

Beginner

You have built a Python script that generates a report, scrapes a website, or monitors a server — and now you need it to tell you what happened. Maybe you want a daily summary email, an alert when something breaks, or a confirmation that a scheduled job finished successfully. Sending email programmatically is one of those kills every Python developer eventually needs, and the good news is that Python has everything you need built right in.

Python’sstandard library includes smtplib for connecting to mail servers and email for building properly formatted messages. You do not need to install any third-party packages. All you need is a Gmail account with an App Password (we will walk through setting that up) and about 10 lines of code to send your first email.

tter properly. You need both: email.message.EmailMessage builds a correctly formatted email (headers, body, attachments), and smtplib delivers it to the mail server.

ModulePurposePart of Standard Library?
smtplibConnect to SMTP server, authenticate, sendYes
email.messageBuild email messages (headers, body, MIME)Yes
email.mimeLegacy API for building MIME messagesYes (use EmailMessage instead)
sslSecure socket layer for encrypted connectionsYes

The modern approach uses EmailMessage (introduced in Python 3.6) instead of the older MIMEText/MIMEMultipart classes. EmailMessage handles plain text, HTML, and attachments through a single clean API. We will use it throughout this tutorial.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Setting Up Gmail App Passwords

Gmail does not allow you to log in with your regular password from a script — it requires an App Password instead. An App Password is a 16-character code that gives your script access to your Gmail account without exposing your main password. Here is how to set one up.

First, you need to enable 2-Step Verification on your Google account if you have not already. Go to myaccount.google.com/security, scroll to “How you sign in to Google,” and turn on 2-Step Verification. Once that is active, go to myaccount.google.com/apppasswords, enter a name like “Python Script,” and click Create. Google will show you a 16-character password — copy it immediately because you will not see it again.

# secure_config.py
import os

# Store your App Password as an environment variable -- never hardcode it
# Set it in your terminal first:
#   export GMAIL_APP_PASSWORD="xxxx xxxx xxxx xxxx"
#   export GMAIL_ADDRESS="your_email@gmail.com"

gmail_address = os.environ.get("GMAIL_ADDRESS")
gmail_password = os.environ.get("GMAIL_APP_PASSWORD")

if not gmail_address or not gmail_password:
    raise ValueError("Set GMAIL_ADDRESS and GMAIL_APP_PASSWORD environment variables")

print(f"Configured for: {gmail_address}")
print(f"Password loaded: {'*' * len(gmail_password)}")

Output:

Configured for: your_email@gmail.com
Password loaded: ****************

Never hardcode your App Password directly in your Python files. Use environment variables or a .env file (with python-dotenv) to keep credentials out of your source code. If you accidentally commit a password to Git, revoke the App Password immediately from your Google account settings and create a new one.

SMTP_SSL vs STARTTLS: Choosing a Connection Method

There are two ways to establish a secure connection to an SMTP server: SMTP_SSL and SMTP with STARTTLS. Both encrypt your email traffic, but they work differently.

MethodPortHow It WorksWhen to Use
SMTP_SSL465Encrypted from the startPreferred for Gmail
SMTP + starttls()587Starts unencrypted, upgradesSome corporate servers

Here is how the STARTTLS approach looks in practice. The connection starts as plaintext on port 587, then upgrades to TLS before sending any sensitive data.

# starttls_example.py
import smtplib
import os
from email.message import EmailMessage

msg = EmailMessage()
msg["Subject"] = "STARTTLS Test"
msg["From"] = os.environ["GMAIL_ADDRESS"]
msg["To"] = os.environ["GMAIL_ADDRESS"]  # Send to yourself for testing
msg.set_content("This email was sent using STARTTLS on port 587.")

with smtplib.SMTP("smtp.gmail.com", 587) as server:
    server.ehlo()       # Identify ourselves to the server
    server.starttls()   # Upgrade connection to TLS
    server.ehlo()       # Re-identify after TLS upgrade
    server.login(os.environ["GMAIL_ADDRESS"], os.environ["GMAIL_APP_PASSWORD"])
    server.send_message(msg)

print("Email sent via STARTTLS!")

Output:

Email sent via STARTTLS!

For Gmail, SMTP_SSL on port 465 is simpler and preferred — it encrypts the connection from the first byte. Use the STARTTLS approach only if your email provider specifically requires port 587, or if you need to connect to a server that does not support direct SSL.

Sudo Sam choosing between padlocks - SMTP SSL vs STARTTLS comparison
SMTP_SSL wraps the whole conversation in encryption. STARTTLS hopes nobody is listening to the handshake.

Sending HTML-Formatted Emails

Plain text emails get the job done, but HTML emails let you include formatting, links, tables, and images. The EmailMessage class makes it easy to send an email with both a plain-text fallback and an HTML version — email clients that support HTML will display the rich version, while older clients fall back to plain text.

“Subject”] = “Weekly Python Report” msg[“From”] = os.environ[“GMAIL_ADDRESS”] msg[“To”] = os.environ[“GMAIL_ADDRESS”] # Plain text version (fallback) msg.set_content(“Your weekly report: 42 scripts ran, 0 failures, 15.2s avg runtime.”) # HTML version (preferred by most email clients) html_content = “””\

Weekly Python Report

Here is your automated summary for the week:

Metric Value 420Avg Runtime
Scripts Executed
Failures
15.2 seconds

All systems operational. Have a great week!

""" msg.add_alternative(html_content, subtype="html") with smtplib.SMTP_SSL("smtp.gmail.com", 465) as server: server.login(os.environ["GMAIL_ADDRESS"], os.environ["GMAIL_APP_PASSWORD"]) server.send_message(msg) print("HTML email sent!")

Output:

HTML email sent!

The key method here is msg.add_alternative(html_content, subtype="html"). This tells the email that it has two versions: the plain text you set with set_content() and the HTML alternative. Always provide both -- some email clients strip HTML entirely, and a plain-text fallback ensures your message is readable everywhere.

Adding File Attachments

Sending reports, logs, or CSV files as email attachments is a common automation task. The EmailMessage class handles this with add_attachment(), which automatically detects the file type and encodes it correctly.

# attachment_email.py
import smtplib
import os
import mimetypes
from email.message import EmailMessage
from pathlib import Path

def send_email_with_attachment(to_address, subject, body, file_path):
    """Send an email with a file attachment."""
    msg = EmailMessage()
    msg["Subject"] = subject
    msg["From"] = os.environ["GMAIL_ADDRESS"]
    msg["To"] = to_address
    msg.set_content(body)

    # Read and attach the file
    filepath = Path(file_path)
    if not filepath.exists():
        raise FileNotFoundError(f"Attachment not found: {file_path}")

    mime_type, _ = mimetypes.guess_type(str(filepath))
    if mime_type is None:
        mime_type = "application/octet-stream"
    maintype, subtype = mime_type.split("/")

    with open(filepath, "rb") as f:
        msg.add_attachment(
            f.read(),
            maintype=maintype,
            subtype=subtype,
            filename=filepath.name
        )

    with smtplib.SMTP_SSL("smtp.gmail.com", 465) as server:
        server.login(os.environ["GMAIL_ADDRESS"], os.environ["GMAIL_APP_PASSWORD"])
        server.send_message(msg)

    print(f"Email sent to {to_address} with attachment: {filepath.name}")


# Create a sample CSV file for testing
sample_csv = "name,score,grade\nAlice,95,A\nBob,87,B\nCharlie,72,C\n"
Path("report.csv").write_text(sample_csv)

# Send it
send_email_with_attachment(
    to_address=os.environ["GMAIL_ADDRESS"],
    subject="Monthly Report Attached",
    body="Please find the monthly report attached to this email.",
    file_path="report.csv"
)

Output:

Email sent to your_email@gmail.com with attachment: report.csv

The mimetypes.guess_type() function automatically detects the correct MIME type from the file extension -- text/csv for CSV files, application/pdf for PDFs, image/png for images, and so on. If the type cannot be determined, we fall back to application/octet-stream which tells the email client to treat it as a generic binary file. You can attach multiple files by calling add_attachment() multiple times on the same message.

Debug Dee attaching gift box to envelope - Python email attachments
add_attachment() handles the MIME type guessing. You just hand it the file and hope for the best.

Sending to Multiple Recipients

Often you need to send the same email to several people -- maybe a team notification or a batch of personalized messages. There are two approaches: sending one email to multiple recipients (everyone sees all addresses) or sending individual emails (each person sees only their address).

# multiple_recipients.py
import smtplib
import os
from email.message import EmailMessage

def send_to_group(recipients, subject, body)
   """Send one email to multiple recipients (all visible in To field)."""
    msg = EmailMessage()
    msg["Subject"] = subject
    msg["From"] = os.environ["GMAIL_ADDRESS"]
    msg["To"] = ", ".join(recipients)
    msg.set_content(body)

    with smtplib.SMTP_SSL("smtp.gmail.com", 465) as server:
        server.login(os.environ["GMAIL_ADDRESS"], os.environ["GMAIL_APP_PASSWORD"])
        server.send_message(msg)

    print(f"Group email sent to {len(recipients)} recipients")


def send_individual(recipients, subject_template, body_template):
    """Send personalized emails to each recipient individualy."""
    with smtplib.SMTP_SSL("smtp.gmail.com", 465) as server:
        server.login(os.environ["GMAIL_ADDRESS"], os.environ["GMAIL_APP_PASSWORD"])

        for name, email_addr in recipients:
            msg = EmailMessage()
            msg["Subject"] = subject_template.format(name=name)
            msg["From"] = os.environ["GMAIL_ADDRESS"]
            msg["To"] = email_addr
            msg.set_content(body_template.format(name=name))
            server.send_message(msg)
            print(f"Sent to {name} ({email_addr})")

    print(f"All {len(recipients)} individual emails sent!")


# Example: group email
team = ["alice@example.com", "bob@example.com", "charlie@example.com"]
send_to_group(team, "Team Update", "Sprint review meeting moved to 3 PM.")

# Example: personalized emails
contacts = [("Alice", "alice@example.com"), ("Bob", "bob@example.com")]
send_individual(
    contacts,
    subject_template="Hey {name}, your weekly summary",
    body_template="Hi {name},\n\nHere is your personalized weekly summary.\n\nBest regards"
)

Real-Life Example: Automated Error Alert System

Let us tie everything together with a practical project. This script monitors a log file for errors and sends an HTML alert email with a summary of any issues found. It combines plain text processing, HTML email formatting, and the EmailNotifier class pattern you can reuse in any project.

# Create a sample log file for demonstration
sample_log = """2026-03-31 08:00:01 INFO  Starting data pipeline
2026-03-31 08:00:05 INFO  Connected to database
2026-03-31 08:00:12 WARNING  Slow query detected (2.3s)
2026-03-31 08:00:15 ERROR  Failed to fetch API data: ConnectionTimeout
2026-03-31 08:00:16 ERROR  Retry 1 of 3 failed
2026-03-31 08:00:20 INFO  Retry 2 succeeded
2026-03-31 08:00:45 CRITICAL  Database connection lost
2026-03-31 08:00:46 INFO  Reconnecting to database...
2026-03-31 08:01:00 INFO  Pipeline completed with errors
"""
Path("app.log").write_text(sample_log)

# Check the log and send an alert if errors are found
def check_log_for_errors(log_path):
    """Scan a log file and return any lines containing ERROR or CRITICAL."""
    errors = []
    path = Path(log_path)
    if not path.exists():
        return errors

    with open(path, "r") as f:
        for line_num, line in enumerate(f, 1):
            stripped = line.strip()
            if "ERROR" in stripped or "CRITICAL" in stripped:
                errors.append({"line": line_num, "text": stripped})

    return errors


def build_alert_html(log_file, errors):
    """Build an HTML alert email from error entries."""
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    rows = ""
    for err in errors:
        rows += f'{err["line"]}'
        rows += f'{err["text"]}'

    return f"""""

sample_log = """2026-03-31 08:00:01 INFO  Starting data pipeline
2026-03-31 08:00:05 INFO  Connected to database
2026-03-31 08:00:12 WARNING  Slow query detected (2.3s)
2026-03-31 08:00:15 ERROR  Failed to fetch API data: ConnectionTimeout
2026-03-31 08:00:16 ERROR  Retry 1 of 3 failed
2026-03-31 08:00:20 INFO  Retry 2 succeeded
2026-03-31 08:00:45 CRITICAL  Database connection lost
2026-03-31 08:00:46 INFO  Reconnecting to database...
2026-03-31 08:01:00 INFO  Pipeline completed with errors
"""
Path("app.log").write_text(sample_log)

# Check the log and send an alert if errors are found
class EmailNotifier:
    """Reusable email notification system" HTML alert built {len(errors)} chars)
In production: notifier.send(admin_email, subject, body, html)
else:
    print("No errors found. All clear!")

Output:

Found 3 error(s) in app.log:
  Line 4: 2026-03-31 08:00:15 ERROR  Failed to fetch API data: ConnectionTimeout
  Line 5: 2026-03-31 08:00:16 ERROR  Retry 1 of 3 failed
  Line 7: 2026-03-31 08:00:45 CRITICAL  Database connection lost

HTML alert built (698 chars)
In production: notifier.send(admin_email, subject, body, html)

This notification system is designed to be dropped into any existing project. The EmailNotifier class handles all the SMTP details, check_log_for_errors() scans for problems, and build_alert_html() creates a readable alert email. You could schedule this to run every hour with cron or the schedule library, and you would have a lightweight monitoring system without any third-party services.

SMTP isn't fancy. It just works. Sometimes that's enough.
SMTP isn't fancy. It just works. Sometimes that's enough.

Gmail Sending Limits and Best Practices

Gmail enforces rate limits on how many emails you can send per day. Knowing these limits prevents your script from getting temporarily blocked.

Account TypeDaily LimitPer MinuteMax Recipients Per Email
Free Gmail500 emails~20500
Google Workspace2,000 emails~302,000

If you hit these limits, Gmail returns an SMTPDataError with code 421 or 550. Your script should catch this and wait before retrying. For high-volume sending (marketing emails, large mailing lists), use a dedicated email service like SendGrid, Mailgun, or Amazon SES instead of Gmail -- they are designed for bulk sending and provide analytics, bounce handling, and higher limits.

Frequently Asked Questions

How do I create a Gmail App Password?

Go to myaccount.google.com/apppasswords after enabling 2-Step Verification on your account. Click "Select app," choose "Other," type a name like "Python Script," and click Generate. Copy the 16-character password and use it in your server.login() call instead of your regular Gmail password. You can revoke it anytime from the same page.

Why does Gmail reject my login with SMTPAuthenticationError?

This almost always means you are using your regular Gmail password instead of an App Password. Google disabled "Less Secure App Access" permanently in 2022. You must use an App Password (see the section above) or switch to OAuth2 for more complex applications. Double-check that there are no extra spaces in your password string.

My script hangs when connecting to the SMTP server. What is wrong?

Add a timeout parameter to your SMTP connection: smtplib.SMTP_SSL("smtp.gmail.com", 465, timeout=30). This prevents the script from hanging forever if the server is unreachable. Common causes include corporate firewalls blocking port 465 or 587, VPN interference, or DNS resolution failures. Try pinging smtp.gmail.com from your terminal to verify connectivity.

How do I send emails with special characters or non-English text?

The EmailMessage class handles Unicode correctly by default. Just pass your text normally: msg.set_content("Bonjour! Voici votre rapport."). The message will be encoded as UTF-8 automatically. If you are using the older MIMEText API, explicitly set the charset: MIMEText(body, "plain", "utf-8").

How can I test email sending without actually sending emails?

Python has a built-in debugging SMTP server that prints emails to the terminal instead of sending them. Run python -m smtpd -n -c DebuggingServer localhost:1025 in one terminal, then connect your script to localhost:1025 using smtplib.SMTP("localhost", 1025) (no SSL, no login). You will see the full email content printed to the terminal. For Python 3.12+, use aiosmtpd instead since the built-in smtpd module was removed.

Conclusion

You now have everything you need to send emails from Python scripts: plain text messages with set_content(), HTML-formatted emails with add_alternative(), file attachments with add_attachment(), and a robust error-handling wrapper with retry logic. The notification system project gives you a ready-to-use template for monitoring any automated task.

Try extending the notification system to watch multiple log files, send daily digest emails instead of per-error alerts, or add Slack webhook notifications alongside email. The EmailNotifier class is designed to be subclassed and customized for your specific needs.

For the complete API reference, see the official Python documentation for smtplib and email.message.

Continue Learning Python

Tutorials you might also find useful:

How To Use Type Hints in Python with Mypy

How To Use Type Hints in Python with Mypy

Last Updated: June 01, 2026

Intermediate

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

How To Use Type Hints in Python with Mypy

Part of the Python Testing & Quality Hub. See the full hub for related Python tutorials.

Python is known for its simplicity and readability, but this comes at a cost — it’s dynamically typed, which means you can assign any type of value to a variable at any time. While this flexibility is powerful, it can lead to subtle bugs that only appear at runtime. Imagine debugging a function that crashes because you accidentally passed a string where an integer was expected, only to discover the issue after hours of investigation. Type hints solve this problem by letting you specify what types your functions and variables should accept, catching errors before your code ever runs.

If you’re worried that adding type hints will make your Python code feel like Java or C++, rest assured — Python’s type hints are optional, unobtrusive, and entirely optional at runtime. They exist purely for documentation, IDE support, and static analysis. Your code runs exactly the same with or without them, but with type hints, you unlock powerful tools like Mypy that can catch entire categories of bugs before deployment.

In this tutorial, you’ll learn everything you need to start using type hints effectively. We’ll cover the basics of annotating variables and functions, explore the typing module, understand how Mypy validates your code, and work through real-world examples that demonstrate the power of static type checking. By the end, you’ll understand why type hints are becoming standard practice in professional Python codebases.

Quick Example

Here’s a minimal example that shows type hints in action. Don’t worry if it looks unfamiliar — we’ll break down each component in detail:

# file: quick_example.py
def greet(name: str, age: int) -> str:
    return f"Hello {name}, you are {age} years old"

def add_numbers(a: int, b: int) -> int:
    return a + b

result = add_numbers(5, 10)
message = greet("Alice", 30)
print(message)
print(result)

Without type hints, Python wouldn’t catch it if you accidentally called `add_numbers(“5”, 10)` with a string instead of an integer. With type hints and Mypy, this error is caught instantly before you run the code.

Type hints catch the bugs your future self forgets to.
Type hints catch the bugs your future self forgets to.

What Are Type Hints and Why Use Them?

Type hints are annotations that specify what types your functions, variables, and return values should be. They’re written using Python’s typing syntax and don’t affect how your code executes — Python’s interpreter completely ignores them at runtime. Their purpose is to help you, your team, and automated tools understand what types of data flow through your code.

The primary benefits of type hints include catching bugs before runtime, improving code readability, enabling better IDE autocompletion, serving as documentation, and refactoring with confidence. When you annotate your code with types, tools like Mypy can analyze it statically and warn you about potential type mismatches without executing the code.

Let’s compare typed versus untyped code side by side:

Without Type Hints With Type Hints
def process(data): def process(data: list) -> int:
Unclear what types are expected Clear intent and expectations
IDE can’t provide smart autocomplete IDE knows what methods are available
Runtime errors when wrong types passed Static analysis catches type errors
Hard to refactor safely Mypy ensures refactoring doesn’t break contracts

Type hints scale tremendously in larger codebases. A function with type hints serves as a contract — callers know exactly what to pass, and the function author knows exactly what to expect. This reduces bugs, improves collaboration, and makes code easier to maintain.

Basic Type Hints: Built-in Types

Let’s start with the simplest type hints — annotations for built-in Python types. You can annotate variables when you create them, and annotate function parameters and return values.

# file: basic_types.py
# Simple variable annotations
name: str = "Alice"
age: int = 30
height: float = 5.8
is_active: bool = True

# Function with type hints
def calculate_age_in_days(age: int) -> int:
    return age * 365

def greet_user(name: str, age: int) -> str:
    days = calculate_age_in_days(age)
    return f"{name} is {days} days old"

# Using the functions
print(greet_user("Bob", 25))
print(calculate_age_in_days(40))

Output:

Bob is 9125 days old
14600

In the example above, the colon (:) separates the parameter name from its type. For return types, the arrow (->) comes after the parameter list. These annotations tell anyone reading the code — and tools like Mypy — exactly what types are expected. If you tried to call `calculate_age_in_days(“thirty”)`, Mypy would immediately flag it as an error.

The basic types you’ll use most often are `str`, `int`, `float`, and `bool`. But what if you need to work with collections like lists or dictionaries? That’s where things get interesting.

mypy is the friend who tells you about the bug before it ships.
mypy is the friend who tells you about the bug before it ships.

Collections: Lists, Dicts, and Tuples

When you want to annotate a list, you can’t just write `list` — you need to specify what type of items the list contains. This is where the `typing` module comes in. The `typing` module provides generic types like `List`, `Dict`, and `Tuple` that let you specify what they contain.

# file: collections_example.py
from typing import List, Dict, Tuple

# List of integers
scores: List[int] = [95, 87, 92, 88]

# List of strings
names: List[str] = ["Alice", "Bob", "Charlie"]

# Dictionary with string keys and integer values
age_map: Dict[str, int] = {"Alice": 30, "Bob": 25, "Charlie": 28}

# Tuple with fixed types
location: Tuple[float, float] = (40.7128, -74.0060)

# Function that processes a list
def sum_scores(scores: List[int]) -> int:
    return sum(scores)

# Function that works with dictionaries
def get_age(person_name: str, ages: Dict[str, int]) -> int:
    return ages[person_name]

# Using the functions
total = sum_scores(scores)
alice_age = get_age("Alice", age_map)
print(f"Total scores: {total}")
print(f"Alice's age: {alice_age}")
print(f"Location: {location}")

Output:

Total scores: 362
Alice's age: 30
Location: (40.7128, -74.006)

The syntax `List[int]` means “a list containing integers”. Similarly, `Dict[str, int]` means “a dictionary with string keys and integer values”, and `Tuple[float, float]` means “a tuple containing exactly two floats”. This specificity is what makes type checking powerful — Mypy can now verify that you’re not accidentally passing a list of strings to a function expecting a list of integers.

Optional and Union Types

Sometimes a function might return either a value or `None`, or it might accept multiple different types. Python provides `Optional` and `Union` for these scenarios. `Optional[T]` is shorthand for “either a value of type T or None”, while `Union` lets you specify multiple possible types.

# file: optional_union.py
from typing import Optional, Union

# Function that might return None
def find_user_age(name: str, users: dict) -> Optional[int]:
    if name in users:
        return users[name]["age"]
    return None

# Function that accepts multiple types
def process_value(value: Union[int, str]) -> str:
    if isinstance(value, int):
        return f"Number: {value * 2}"
    else:
        return f"Text: {value.upper()}"

# Dictionary of user data
users_db = {
    "Alice": {"age": 30},
    "Bob": {"age": 25}
}

# Using Optional
age = find_user_age("Alice", users_db)
print(f"Alice's age: {age}")

missing_age = find_user_age("Charlie", users_db)
print(f"Charlie's age: {missing_age}")

# Using Union
result1 = process_value(10)
result2 = process_value("hello")
print(result1)
print(result2)

Output:

Alice's age: 30
Charlie's age: None
Number: 20
Text: HELLO

The `Optional` type is essential in Python because None is a valid value in many scenarios. By marking a return type as `Optional[int]`, you’re telling callers “this function might return an integer or None, and you should handle both cases”. This prevents a whole class of bugs where code forgets to check for None before using a value.

When you guess the type and Python disagrees at runtime.
When you guess the type and Python disagrees at runtime.

The Typing Module: List, Dict, Tuple, and More

We briefly introduced `List`, `Dict`, and `Tuple` from the typing module. Let’s explore more capabilities and understand when to use them. In Python 3.9+, you can actually use built-in `list`, `dict`, and `tuple` directly for type hints, but the `typing` module versions work in all Python versions and provide more features.

# file: typing_module.py
from typing import List, Dict, Set, Tuple, Any

# Specific typed collections
numbers: List[int] = [1, 2, 3, 4, 5]
name_ages: Dict[str, int] = {"Alice": 30, "Bob": 25}
unique_tags: Set[str] = {"python", "tutorial", "typing"}
coordinates: Tuple[int, int, int] = (10, 20, 30)

# Using Any when type is truly unknown (use sparingly!)
# This disables type checking for this variable
unknown_value: Any = "could be anything"

# Function with complex typing
def process_data(
    items: List[Dict[str, Any]],
    filters: Set[str]
) -> List[str]:
    results = []
    for item in items:
        if item.get("type") in filters:
            results.append(item.get("name", "Unknown"))
    return results

# Sample data
data = [
    {"name": "Alice", "type": "user"},
    {"name": "Bob", "type": "admin"},
    {"name": "Document", "type": "file"}
]

# Using the function
filtered = process_data(data, {"user", "admin"})
print(f"Filtered results: {filtered}")

Output:

Filtered results: ['Alice', 'Bob']

The `Any` type is a special case that essentially says “this can be any type” and disables type checking for that variable. Use `Any` sparingly — it defeats the purpose of type hints. It’s useful for truly dynamic situations or when working with third-party code you can’t control, but typed alternatives are almost always better.

Function Annotations: Parameters and Returns

Function annotations are where type hints shine. By annotating parameters and return types, you create a contract that documents what a function expects and what it produces. This makes functions self-documenting and enables powerful static analysis.

# file: function_annotations.py
from typing import List, Optional

def calculate_average(scores: List[float]) -> float:
    """Calculate the average of a list of scores."""
    if not scores:
        return 0.0
    return sum(scores) / len(scores)

def find_maximum(numbers: List[int]) -> Optional[int]:
    """Return the maximum number, or None if list is empty."""
    return max(numbers) if numbers else None

def format_report(
    title: str,
    items: List[str],
    show_count: bool = True
) -> str:
    """Format items into a report string."""
    report = f"=== {title} ===\n"
    for item in items:
        report += f"- {item}\n"
    if show_count:
        report += f"Total: {len(items)}"
    return report

# Using the functions
test_scores = [85.5, 90.0, 78.5, 92.0]
average = calculate_average(test_scores)
print(f"Average score: {average}")

numbers = [45, 23, 89, 12, 56]
max_num = find_maximum(numbers)
print(f"Maximum number: {max_num}")

report = format_report("Tasks", ["Write code", "Review PR", "Deploy"], show_count=True)
print(report)

Output:

Average score: 86.5
Maximum number: 89
=== Tasks ===
- Write code
- Review PR
- Deploy
Total: 3

Notice that we’re using `Optional[int]` for functions that might return None, and `List[float]` for functions that accept collections. Default parameter values (like `show_count: bool = True`) work naturally with type hints — the type annotation comes before the equals sign.

Class Annotations and Instance Variables

Type hints work wonderfully with classes. You can annotate instance variables, method parameters, and return types. This makes class structure clear and helps catch errors when using class instances.

# file: class_annotations.py
from typing import List, Optional
from datetime import datetime

class Person:
    """Represents a person with type-annotated attributes."""

    # Class-level type annotations
    name: str
    age: int
    email: Optional[str]

    def __init__(self, name: str, age: int, email: Optional[str] = None) -> None:
        self.name = name
        self.age = age
        self.email = email

    def get_info(self) -> str:
        """Return a formatted string with person information."""
        return f"{self.name} ({self.age} years old)"

    def is_adult(self) -> bool:
        """Check if the person is an adult."""
        return self.age >= 18

class Team:
    """Represents a team of people."""

    name: str
    members: List[Person]

    def __init__(self, name: str) -> None:
        self.name = name
        self.members = []

    def add_member(self, person: Person) -> None:
        """Add a person to the team."""
        self.members.append(person)

    def get_adult_members(self) -> List[Person]:
        """Return only adult team members."""
        return [m for m in self.members if m.is_adult()]

    def member_count(self) -> int:
        """Return the number of team members."""
        return len(self.members)

# Using the classes
alice = Person("Alice", 30, "alice@example.com")
bob = Person("Bob", 17)
charlie = Person("Charlie", 25, "charlie@example.com")

team = Team("Development")
team.add_member(alice)
team.add_member(bob)
team.add_member(charlie)

print(f"Team: {team.name}")
print(f"Total members: {team.member_count()}")
print(f"Adults: {len(team.get_adult_members())}")
for member in team.members:
    print(f"  - {member.get_info()}")

Output:

Team: Development
Total members: 3
Adults: 2
  - Alice (30 years old)
  - Bob (17 years old)
  - Charlie (25 years old)

Class annotations make the structure of your objects immediately clear. Anyone reading this code knows exactly what attributes a `Person` has and what types they hold. The `__init__` method also has type hints showing what it expects and that it returns `None` (all constructors return None since they don’t return anything explicitly).

Generics Basics: Writing Flexible Type-Safe Code

Generics allow you to write functions and classes that work with multiple types while maintaining type safety. Instead of using `Any`, you can use type variables to specify “this function works with lists of any type, but the type must be consistent”.

# file: generics_example.py
from typing import TypeVar, List, Generic

# Define a type variable -- T is a placeholder for any type
T = TypeVar('T')

# Generic function that works with any type
def get_first(items: List[T]) -> T:
    """Return the first item from a list."""
    if not items:
        raise ValueError("List is empty")
    return items[0]

def reverse_list(items: List[T]) -> List[T]:
    """Return a reversed copy of the list."""
    return items[::-1]

# Generic class
class Container(Generic[T]):
    """A simple container that holds one item of any type."""

    def __init__(self, item: T) -> None:
        self.item = item

    def get_item(self) -> T:
        return self.item

    def set_item(self, item: T) -> None:
        self.item = item

# Using generic functions
int_list = [10, 20, 30, 40]
str_list = ["apple", "banana", "cherry"]

first_int = get_first(int_list)  # Type checker knows this is int
first_str = get_first(str_list)  # Type checker knows this is str

print(f"First int: {first_int}")
print(f"First string: {first_str}")
print(f"Reversed ints: {reverse_list(int_list)}")

# Using generic class
int_container = Container(42)
str_container = Container("hello")

print(f"Int container: {int_container.get_item()}")
print(f"String container: {str_container.get_item()}")

Output:

First int: 10
First string: apple
Reversed ints: [40, 30, 20, 10]
Int container: 42
String container: hello

Generics are powerful because they preserve type information. When you call `get_first(int_list)`, type checkers understand that the return value is an `int`, not just some unknown `T`. This is much safer than using `Any` and provides excellent IDE support — your editor can offer correct autocompletion based on the actual type.

Installing and Running Mypy

Mypy is a static type checker for Python that analyzes your code without running it. Installation is straightforward using pip, and running it is even simpler. Let’s set up Mypy and check our type hints.

First, install Mypy using pip:

# file: terminal
pip install mypy

Once installed, you can check a single file or an entire directory. Create a test file with some intentional type errors to see how Mypy catches them:

# file: mypy_test.py
def add_numbers(a: int, b: int) -> int:
    return a + b

# This is correct
result1 = add_numbers(5, 10)
print(result1)

# This will cause a Mypy error
result2 = add_numbers("5", 10)
print(result2)

Now run Mypy on this file:

# file: terminal
mypy mypy_test.py

Mypy will output something like:

mypy_test.py:8: error: Argument 1 to "add_numbers" has incompatible type "str"; expected "int"

This error tells you exactly where the problem is — line 8, argument 1 of the `add_numbers` call. Even though the code would run fine if `add_numbers` could handle string input, Mypy caught the type mismatch before you ran the code. For larger projects, you can run mypy on an entire directory:

# file: terminal
mypy your_project/

You can also configure Mypy’s strictness using a `mypy.ini` file. A basic configuration might look like:

# file: mypy.ini
[mypy]
python_version = 3.9
warn_return_any = True
warn_unused_configs = True
disallow_untyped_defs = True

The `disallow_untyped_defs = True` option enforces that every function must have type hints. This is strict but catches a lot of bugs in larger codebases.

Common Mypy Errors and How to Fix Them

Let’s explore the most common type errors you’ll encounter with Mypy and how to fix them. Understanding these patterns will help you write type-safe code quickly.

Error: Incompatible Type Assignment

This is the most common error — you’re assigning a value of the wrong type to a variable:

# file: error_incompatible.py
# WRONG: str assigned to int variable
count: int = "five"

# CORRECT: assign actual integer
count: int = 5

# WRONG: list of strings assigned to list of ints
numbers: list[int] = ["1", "2", "3"]

# CORRECT: list of integers
numbers: list[int] = [1, 2, 3]

Fix: Make sure the value type matches the annotated type. Convert types if needed:

# file: fix_incompatible.py
# Convert before assigning
count: int = int("five")  # Could raise ValueError, but type is correct
numbers: list[int] = [int(x) for x in ["1", "2", "3"]]

Error: Missing Return Type

When a function doesn’t explicitly return a value on all code paths, Mypy complains:

# file: error_missing_return.py
def check_age(age: int) -> str:
    if age >= 18:
        return "Adult"
    # Missing return statement -- what if age < 18?

# CORRECT:
def check_age(age: int) -> str:
    if age >= 18:
        return "Adult"
    else:
        return "Minor"

Fix: Ensure all code paths return a value, or change the return type to `Optional[str]` if None is acceptable:

# file: fix_missing_return.py
from typing import Optional

def check_age(age: int) -> Optional[str]:
    if age >= 18:
        return "Adult"
    return None  # Explicitly return None

Error: Argument Has Incompatible Type

You’re passing the wrong type to a function:

# file: error_argument.py
def process_list(items: list[int]) -> int:
    return sum(items)

# WRONG: passing list of strings
result = process_list(["1", "2", "3"])

# CORRECT: convert to integers first
result = process_list([1, 2, 3])

Fix: Convert the argument to the correct type or check your function call:

# file: fix_argument.py
def process_list(items: list[int]) -> int:
    return sum(items)

# Convert strings to ints
result = process_list([int(x) for x in ["1", "2", "3"]])
print(result)  # Output: 6

Error: Item Is None (Need Optional Check)

You’re accessing an attribute on something that might be None:

# file: error_none_access.py
from typing import Optional

def get_name(person: Optional[dict]) -> str:
    return person["name"]  # person could be None!

# CORRECT:
def get_name(person: Optional[dict]) -> Optional[str]:
    if person is not None:
        return person.get("name")
    return None

Fix: Always check for None before using Optional values:

# file: fix_none_access.py
from typing import Optional

def get_name(person: Optional[dict]) -> str:
    if person is None:
        return "Unknown"
    return person.get("name", "Unknown")

# Test it
result = get_name(None)
print(result)  # Output: Unknown

Error: Return Type Mismatch

Your function returns a different type than what’s annotated:

# file: error_return_mismatch.py
def calculate(value: int) -> str:
    # WRONG: returning int instead of str
    return value * 2

# CORRECT:
def calculate(value: int) -> str:
    return str(value * 2)

Fix: Ensure your return statement returns the correct type, or change the annotation:

# file: fix_return_mismatch.py
def calculate(value: int) -> str:
    result = value * 2
    return str(result)

print(calculate(5))  # Output: "10"

Real-Life Example: Type-Safe Contact Manager

Let’s bring everything together with a practical example — a contact manager application with full type hints. This demonstrates how type hints make complex code safe and maintainable:

# file: contact_manager.py
from typing import List, Optional, Dict
from datetime import datetime

class Contact:
    """Represents a single contact with full type annotations."""

    name: str
    email: str
    phone: Optional[str]
    created_at: datetime

    def __init__(self, name: str, email: str, phone: Optional[str] = None) -> None:
        if not name or not email:
            raise ValueError("Name and email are required")
        self.name = name
        self.email = email
        self.phone = phone
        self.created_at = datetime.now()

    def get_display_name(self) -> str:
        """Return formatted contact name."""
        return self.name.upper()

    def has_phone(self) -> bool:
        """Check if contact has phone number."""
        return self.phone is not None

class ContactManager:
    """Manages a collection of contacts."""

    contacts: List[Contact]

    def __init__(self) -> None:
        self.contacts = []

    def add_contact(self, contact: Contact) -> None:
        """Add a new contact to the manager."""
        self.contacts.append(contact)

    def find_by_email(self, email: str) -> Optional[Contact]:
        """Find a contact by email address."""
        for contact in self.contacts:
            if contact.email == email:
                return contact
        return None

    def find_by_name(self, name: str) -> List[Contact]:
        """Find all contacts matching a name (partial match)."""
        return [c for c in self.contacts if name.lower() in c.name.lower()]

    def get_all_with_phone(self) -> List[Contact]:
        """Return contacts that have phone numbers."""
        return [c for c in self.contacts if c.has_phone()]

    def get_contact_summary(self) -> Dict[str, int]:
        """Return summary statistics about contacts."""
        return {
            "total": len(self.contacts),
            "with_phone": len(self.get_all_with_phone()),
            "without_phone": len(self.contacts) - len(self.get_all_with_phone())
        }

    def export_emails(self) -> List[str]:
        """Export all contact emails."""
        return [c.email for c in self.contacts]

# Using the contact manager
manager = ContactManager()

# Add contacts
alice = Contact("Alice Johnson", "alice@example.com", "+1-555-0101")
bob = Contact("Bob Smith", "bob@example.com")
charlie = Contact("Charlie Brown", "charlie@example.com", "+1-555-0103")

manager.add_contact(alice)
manager.add_contact(bob)
manager.add_contact(charlie)

# Query contacts
print(f"Total contacts: {len(manager.contacts)}")
print(f"Contacts with phones: {len(manager.get_all_with_phone())}")

found = manager.find_by_email("alice@example.com")
if found:
    print(f"Found: {found.name} ({found.phone})")

johns = manager.find_by_name("john")
print(f"Contacts named 'john': {len(johns)}")

summary = manager.get_contact_summary()
print(f"Summary: {summary}")

emails = manager.export_emails()
print(f"All emails: {emails}")

Output:

Total contacts: 3
Contacts with phones: 2
Found: Alice Johnson (+1-555-0101)
Contacts named 'john': 1
Summary: {'total': 3, 'with_phone': 2, 'without_phone': 1}
All emails: ['alice@example.com', 'bob@example.com', 'charlie@example.com']

This contact manager demonstrates several key principles: every method has clear type annotations, return types are explicit (including `Optional` and `List`), class attributes are type-annotated, and the code is self-documenting. If you run Mypy on this file, it will validate that every function returns the correct type and every variable receives compatible values. This gives you confidence that the code works as intended without having to manually trace through every function call.

Best Practices for Type Hints

Now that you understand the mechanics of type hints, here are some best practices to follow in your projects. First, be consistent — if you use type hints in one file, use them throughout your project. Second, use the most specific type possible; don’t settle for `Any` when `List[int]` would work. Third, use type hints in all public functions but you can be more relaxed with private helper functions. Fourth, combine docstrings with type hints; while type hints show what types a function expects, docstrings explain what it does.

Another best practice is to use `Optional` only when None is truly an acceptable value. If a function should always return a string, don’t use `Optional[str]` just to be safe. Fifth, keep your types as simple as possible — deeply nested types like `Dict[str, List[Tuple[int, Optional[str]]]]` become hard to read. Consider breaking these into type aliases or separate functions. Finally, use tools like Mypy and pylint in your CI/CD pipeline to catch type errors automatically before code is merged.

Frequently Asked Questions

Do type hints affect performance or runtime behavior?

No, type hints are completely ignored at runtime. Python’s interpreter removes them during compilation, so they have zero impact on how fast or slow your code runs. Type hints exist purely for documentation and static analysis by tools like Mypy.

Can I use type hints with older Python versions?

Type hints were introduced in Python 3.5, so any Python 3.5+ supports basic type hints. However, some advanced features like union using the pipe operator (`int | str`) require Python 3.10+. For maximum compatibility, use the `typing` module imports like `Union[int, str]`.

What’s the difference between `List` from typing and built-in `list`?

In Python 3.9+, you can use built-in `list[int]` instead of `typing.List[int]`. They’re equivalent, but the built-in versions are preferred in newer code. The typing module versions work in older Python versions, so use those if you need to support Python 3.8 and earlier.

How strict should I be with type hints?

Start with type hints on all public functions and class methods. As your codebase grows and you become comfortable with types, increase strictness. Mypy has a `disallow_untyped_defs` option that enforces types everywhere, but it’s strict and requires more discipline. Find a balance that works for your team.

Can I type hint dictionaries with multiple value types?

Yes, use `Dict[str, Union[int, str]]` to indicate a dictionary with string keys and values that can be either int or str. You can also use `Any` if values are truly unknown, but try to be more specific when possible.

Should I use type hints in scripts and small projects?

Even small projects benefit from type hints, especially if you’ll return to them later or share them with others. Type hints serve as documentation and help you catch bugs. The investment in adding them pays off quickly.

Conclusion

Type hints are a powerful tool for writing safer, more maintainable Python code. They transform Python from a language where type errors hide until runtime into one where you catch them during development. Combined with Mypy, type hints let you refactor code with confidence, understand complex codebases faster, and collaborate more effectively with teammates.

The journey to type-safe Python starts simple with basic annotations and grows as your codebase becomes more complex. Begin by adding type hints to your public functions, run Mypy regularly, and gradually increase your type coverage. The investment in type hints pays dividends in code quality and developer productivity.

To learn more, check out the official Python typing module documentation and the Mypy documentation. Both resources provide comprehensive references and advanced patterns for type hints.

Continue learning with these related guides:

Continue Learning Python

Tutorials you might also find useful:

How To Write Unit Tests with pytest in Python

How To Write Unit Tests with pytest in Python

Last Updated: June 01, 2026

Beginner

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Introduction

Part of the Python Testing & Quality Hub. See the full hub for related Python tutorials.

Writing unit tests is one of the most important practices in modern software development, yet many beginners skip it thinking it slows them down. The truth is the opposite — testing saves time by catching bugs early, making refactoring safer, and helping you understand your own code better. In this guide, you will learn how pytest makes testing so simple that you will actually enjoy writing tests.

If you are worried that testing is complex or requires special knowledge, put that fear to rest. pytest is designed to be intuitive and beginner-friendly. You will write test functions that look almost identical to regular Python functions, using plain assertions instead of cryptic methods. No need to memorize a dozen different assertion types or inherit from test base classes.

In this tutorial, we will start with a quick working example so you see testing in action immediately. Then we will explore what pytest is, install it, write various types of tests, and work through a complete real-world example. By the end, you will understand how to test your Python code effectively and confidently.

Quick Example

Let us jump straight into a working pytest test. This minimal example shows how simple testing can be:

# test_quick.py
def add(a, b):
    return a + b

def test_add_positive_numbers():
    assert add(2, 3) == 5

def test_add_negative_numbers():
    assert add(-1, -2) == -3

def test_add_mixed():
    assert add(5, -2) == 3

Output:

$ pytest test_quick.py
============================= test session starts ==============================
collected 3 items

test_quick.py::test_add_positive_numbers PASSED                        [ 33%]
test_quick.py::test_add_negative_numbers PASSED                        [ 66%]
test_quick.py::test_add_mixed PASSED                                   [100%]

============================== 3 passed in 0.02s ===============================

That is it! Three tests passing. Notice there is no special TestCase class to inherit from, no setUp methods, and no assertEquals calls. Just plain functions and simple assertions. This simplicity is what makes pytest so powerful.

Python pytest tests passing with green checkmarks
assert expected == reality. For once, it passed.

What is pytest and Why Use It?

pytest is a testing framework that makes writing and running tests in Python delightfully simple. It is now the de facto standard for Python testing, used by companies like Mozilla, Stripe, and countless open-source projects. pytest shines because it reduces boilerplate, makes test discovery automatic, and provides powerful features like fixtures and parametrization built in.

Python comes with a built-in testing module called unittest, which is powerful but verbose. It requires you to create classes, inherit from TestCase, and use assertion methods like assertEqual. By contrast, pytest uses simple functions and the plain assert statement. Here is a comparison:

Featureunittestpytest
Test discoveryRequires naming conventionAutomatic (test_* or Test*)
AssertionsassertEqual, assertTrue, etc.Plain assert statement
Class requirementMust inherit from TestCaseSimple functions
Setup/teardownsetUp/tearDown methodsFixtures (more flexible)
ParametrizationUse subTest or external tools@pytest.mark.parametrize
Learning curveModerateGentle

For beginners, pytest removes friction. You write less boilerplate, learn fewer concepts, and get productive faster. For experienced developers, pytest provides industrial-strength capabilities through fixtures, parametrization, and its plugin ecosystem.

Installing pytest

Before we write any tests, we need to install pytest. Open your terminal and run:

# install_pytest.sh
pip install pytest

Output:

Collecting pytest
  Downloading pytest-7.4.0-py3-none-any.whl (298 kB)
Successfully installed pytest-7.4.0

Verify the installation:

# verify.sh
pytest --version

Output:

pytest 7.4.0

You are ready to start writing tests. If you are using a virtual environment (recommended), activate it before running pip install.

Installing pytest tools for Python testing
pip install pytest — the only setup you will ever need.

Writing Basic Tests

Test files in pytest must be named test_*.py or *_test.py so pytest can discover them automatically. A basic test is simply a function starting with test_ that uses assert statements:

# test_calculator.py
def multiply(a, b):
    return a * b

def test_multiply_basic():
    result = multiply(3, 4)
    assert result == 12

def test_multiply_by_zero():
    result = multiply(5, 0)
    assert result == 0

def test_multiply_negatives():
    result = multiply(-2, -3)
    assert result == 6

Output:

$ pytest test_calculator.py -v
======================== test session starts ==========================
collected 3 items

test_calculator.py::test_multiply_basic PASSED                  [ 33%]
test_calculator.py::test_multiply_by_zero PASSED                [ 66%]
test_calculator.py::test_multiply_negatives PASSED              [100%]

======================== 3 passed in 0.01s ===========================

The -v flag shows verbose output with each test listed individually. Each test is independent — they run in any order and share no state.

Mastering Assertions

The assert statement is the heart of testing. Here are the most common patterns:

# test_assertions.py
def test_equality():
    assert 5 == 5
    assert "hello" == "hello"
    assert [1, 2, 3] == [1, 2, 3]

def test_truthiness():
    assert True
    assert not False
    assert [1, 2, 3]  # non-empty list is truthy
    assert not []  # empty list is falsy

def test_membership():
    assert 2 in [1, 2, 3]
    assert "key" in {"key": "value"}

def test_type_checking():
    assert isinstance(5, int)
    assert isinstance("hello", str)

Output:

$ pytest test_assertions.py -v
======================== test session starts ==========================
test_assertions.py::test_equality PASSED                        [ 25%]
test_assertions.py::test_truthiness PASSED                      [ 50%]
test_assertions.py::test_membership PASSED                      [ 75%]
test_assertions.py::test_type_checking PASSED                   [100%]

======================== 4 passed in 0.01s ===========================

When an assertion fails, pytest provides detailed error messages showing exactly what went wrong, including the values on both sides of the comparison.

Comparing expected and actual values in pytest assertions
assert actual == expected. The debugger’s mantra.

Using Fixtures

Fixtures are reusable pieces of test setup. Instead of repeating setup code in every test, define it once and inject it where needed. Think of fixtures as the pytest way of doing setup and teardown:

# test_database.py
import pytest

class Database:
    def __init__(self):
        self.connected = False
        self.data = {}

    def connect(self):
        self.connected = True

    def disconnect(self):
        self.connected = False

    def store(self, key, value):
        if not self.connected:
            raise RuntimeError("Not connected")
        self.data[key] = value

    def retrieve(self, key):
        if not self.connected:
            raise RuntimeError("Not connected")
        return self.data.get(key)

@pytest.fixture
def db():
    database = Database()
    database.connect()
    yield database  # code after yield runs as teardown
    database.disconnect()

def test_store_and_retrieve(db):
    db.store("name", "Alice")
    assert db.retrieve("name") == "Alice"

def test_store_overwrites(db):
    db.store("age", 25)
    db.store("age", 26)
    assert db.retrieve("age") == 26

def test_retrieve_nonexistent(db):
    assert db.retrieve("missing") is None

Output:

$ pytest test_database.py -v
======================== test session starts ==========================
test_database.py::test_store_and_retrieve PASSED               [ 33%]
test_database.py::test_store_overwrites PASSED                 [ 66%]
test_database.py::test_retrieve_nonexistent PASSED             [100%]

======================== 3 passed in 0.01s ===========================

The fixture uses yield instead of return. Code before yield runs before the test (setup), code after yield runs after (teardown). Each test gets a fresh database connection, so tests never interfere with each other.

Parametrized Tests

Parametrization runs the same test with different input values — DRY in action:

# test_parametrize.py
import pytest

def is_even(num):
    return num % 2 == 0

@pytest.mark.parametrize("number,expected", [
    (2, True),
    (4, True),
    (1, False),
    (3, False),
    (0, True),
    (-2, True),
])
def test_is_even(number, expected):
    assert is_even(number) == expected

Output:

$ pytest test_parametrize.py -v
======================== test session starts ==========================
test_parametrize.py::test_is_even[2-True] PASSED              [ 16%]
test_parametrize.py::test_is_even[4-True] PASSED              [ 33%]
test_parametrize.py::test_is_even[1-False] PASSED             [ 50%]
test_parametrize.py::test_is_even[3-False] PASSED             [ 66%]
test_parametrize.py::test_is_even[0-True] PASSED              [ 83%]
test_parametrize.py::test_is_even[-2-True] PASSED             [100%]

======================== 6 passed in 0.01s ===========================

pytest creates one test per parameter set and labels each one, making it easy to identify which specific input caused a failure.

Parametrized testing with multiple test cases in pytest
Six test cases, one function. @pytest.mark.parametrize does the heavy lifting.

Testing Exceptions

Sometimes correct behavior means raising an exception. Use pytest.raises to verify exceptions:

# test_exceptions.py
import pytest

def validate_age(age):
    if not isinstance(age, int):
        raise TypeError("Age must be an integer")
    if age < 0:
        raise ValueError("Age cannot be negative")
    if age > 150:
        raise ValueError("Age must be realistic")
    return True

def test_valid_age():
    assert validate_age(25) is True

def test_negative_age():
    with pytest.raises(ValueError, match="cannot be negative"):
        validate_age(-5)

def test_invalid_type():
    with pytest.raises(TypeError, match="must be an integer"):
        validate_age("twenty-five")

Output:

$ pytest test_exceptions.py -v
======================== test session starts ==========================
test_exceptions.py::test_valid_age PASSED                      [ 33%]
test_exceptions.py::test_negative_age PASSED                   [ 66%]
test_exceptions.py::test_invalid_type PASSED                   [100%]

======================== 3 passed in 0.01s ===========================

The match parameter verifies the exception message using regex, ensuring not just the right type but the right message is raised.

Mocking Basics

Mocking replaces real dependencies with controlled fakes so you can test in isolation:

# test_mocking.py
from unittest.mock import Mock, patch

def fetch_user(user_id):
    import requests
    response = requests.get(f"https://jsonplaceholder.typicode.com/users/{user_id}")
    return response.json()

def test_fetch_user_with_mock():
    with patch("requests.get") as mock_get:
        mock_response = Mock()
        mock_response.json.return_value = {"id": 1, "name": "Alice"}
        mock_get.return_value = mock_response

        result = fetch_user(1)

        assert result["name"] == "Alice"
        mock_get.assert_called_once_with(
            "https://jsonplaceholder.typicode.com/users/1"
        )

Output:

$ pytest test_mocking.py -v
======================== test session starts ==========================
test_mocking.py::test_fetch_user_with_mock PASSED             [100%]

======================== 1 passed in 0.01s ===========================

The patch context manager replaces the real requests.get with a mock. The mock tracks how it was called and what it returns, letting you test API-dependent code without network requests.

Mocking dependencies in Python unit tests
The API is down. The tests still pass. Thank unittest.mock.
pytest fixtures: dependency injection for tests.
pytest fixtures: dependency injection for tests.

Real-Life Example: Testing a Shopping Cart

Here is a complete shopping cart with comprehensive tests demonstrating fixtures, parametrization, and exception testing together:

# shopping_cart.py
class Product:
    def __init__(self, name, price):
        self.name = name
        self.price = price

class ShoppingCart:
    def __init__(self):
        self.items = []

    def add_item(self, product, quantity=1):
        if quantity <= 0:
            raise ValueError("Quantity must be positive")
        self.items.append({"product": product, "quantity": quantity})

    def remove_item(self, product_name):
        self.items = [i for i in self.items if i["product"].name != product_name]

    def get_total(self):
        return sum(i["product"].price * i["quantity"] for i in self.items)

    def apply_discount(self, percent):
        if percent < 0 or percent > 100:
            raise ValueError("Discount must be between 0 and 100")
        return self.get_total() * (1 - percent / 100)

    def is_empty(self):
        return len(self.items) == 0
# test_shopping_cart.py
import pytest
from shopping_cart import Product, ShoppingCart

@pytest.fixture
def cart():
    return ShoppingCart()

@pytest.fixture
def laptop():
    return Product("Laptop", 999.99)

@pytest.fixture
def mouse():
    return Product("Mouse", 29.99)

def test_cart_starts_empty(cart):
    assert cart.is_empty()
    assert cart.get_total() == 0

def test_add_single_item(cart, laptop):
    cart.add_item(laptop)
    assert not cart.is_empty()
    assert cart.get_total() == 999.99

def test_add_multiple_items(cart, laptop, mouse):
    cart.add_item(laptop)
    cart.add_item(mouse)
    assert cart.get_total() == 1029.98

@pytest.mark.parametrize("quantity", [0, -1, -10])
def test_invalid_quantity(cart, laptop, quantity):
    with pytest.raises(ValueError, match="must be positive"):
        cart.add_item(laptop, quantity=quantity)

def test_remove_item(cart, laptop, mouse):
    cart.add_item(laptop)
    cart.add_item(mouse)
    cart.remove_item("Mouse")
    assert cart.get_total() == 999.99

@pytest.mark.parametrize("discount,expected", [
    (10, 900), (50, 500), (100, 0),
])
def test_apply_discount(cart, discount, expected):
    cart.add_item(Product("Item", 1000))
    assert cart.apply_discount(discount) == pytest.approx(expected)

Output:

$ pytest test_shopping_cart.py -v
======================== test session starts ==========================
test_shopping_cart.py::test_cart_starts_empty PASSED           [  8%]
test_shopping_cart.py::test_add_single_item PASSED             [ 16%]
test_shopping_cart.py::test_add_multiple_items PASSED          [ 25%]
test_shopping_cart.py::test_invalid_quantity[0] PASSED         [ 33%]
test_shopping_cart.py::test_invalid_quantity[-1] PASSED        [ 41%]
test_shopping_cart.py::test_invalid_quantity[-10] PASSED       [ 50%]
test_shopping_cart.py::test_remove_item PASSED                 [ 58%]
test_shopping_cart.py::test_apply_discount[10-900] PASSED     [ 66%]
test_shopping_cart.py::test_apply_discount[50-500] PASSED     [ 75%]
test_shopping_cart.py::test_apply_discount[100-0] PASSED      [ 83%]

======================== 10 passed in 0.02s ===========================

This example ties together everything: fixtures for reusable setup, parametrization for multiple cases, and exception testing for error handling. Notice we test both happy paths and error paths.

Frequently Asked Questions

How do I run a single test file?

Use pytest followed by the filename: pytest test_calculator.py. To run a specific function: pytest test_calculator.py::test_multiply_basic.

How do I run tests matching a pattern?

Use the -k flag: pytest -k "multiply". This runs all tests with “multiply” in their name.

What does the -v flag do?

The -v (verbose) flag shows each test individually. Use -vv for even more detail including assertion introspection.

Can I stop on the first failure?

Yes, use pytest -x. This stops as soon as one test fails, useful for quick feedback during development.

How do I see print output from my tests?

By default pytest captures print statements. Use pytest -s to show all output during test execution.

What is the difference between a fixture and a helper function?

Fixtures are managed by pytest and support setup/teardown via yield. Helper functions are regular Python functions. Use fixtures for shared setup, helpers for reusable test logic.

How do I test async functions?

Install pytest-asyncio (pip install pytest-asyncio), then mark tests with @pytest.mark.asyncio and use async def.

Conclusion

You now have a solid foundation in pytest. You understand how to write tests with assertions, use fixtures for setup and teardown, parametrize tests for multiple cases, verify exceptions are raised correctly, and mock external dependencies. More importantly, you understand that testing does not have to be complicated.

Start by writing tests for new code, then gradually add tests to existing code. For more advanced topics, visit the official pytest documentation at https://docs.pytest.org/.

Fixtures: The Killer Feature

Fixtures are pytest’s way of providing setup data to tests. Declare them once, request them as function parameters — pytest wires up dependencies automatically:

# File: conftest.py — fixtures available to all tests in this directory
import pytest

@pytest.fixture
def sample_user():
    return {"id": 1, "name": "Alice", "age": 30}

@pytest.fixture
def db_session():
    session = create_test_session()
    yield session
    session.rollback()
    session.close()

# File: test_users.py
def test_get_user(sample_user):
    assert sample_user["name"] == "Alice"

def test_save_user(db_session, sample_user):
    db_session.add(sample_user)
    db_session.flush()
    assert db_session.get_user(1) is not None

The yield pattern separates setup from teardown. Code before yield runs before the test; code after runs after — even if the test fails.

Fixture Scopes

By default fixtures rebuild for every test. For expensive resources (database connections, web drivers), set a wider scope:

@pytest.fixture(scope="session")
def engine():
    # Created once for the entire test run
    return create_engine("sqlite:///test.db")

@pytest.fixture(scope="module")
def schema(engine):
    # Created once per test file
    Base.metadata.create_all(engine)
    yield
    Base.metadata.drop_all(engine)

@pytest.fixture(scope="function")  # default — one per test
def fresh_user(engine, schema):
    with engine.connect() as conn:
        conn.execute("INSERT INTO users ...")
        yield {"id": last_id}

Scopes: function (default), class, module, session. Pick the widest that’s still safe for test isolation.

Parametrization

Test the same logic with multiple inputs via @pytest.mark.parametrize:

@pytest.mark.parametrize("input,expected", [
    (1, 1),
    (5, 120),
    (10, 3628800),
    (0, 1),
])
def test_factorial(input, expected):
    assert factorial(input) == expected

# Multiple params combine into a matrix
@pytest.mark.parametrize("payment_method", ["card", "paypal", "bank"])
@pytest.mark.parametrize("amount", [10, 100, 1000])
def test_checkout(payment_method, amount):
    # 9 tests total: 3 methods x 3 amounts
    process_payment(payment_method, amount)

Marks: skip, xfail, slow

Custom marks tag tests for selective running:

import pytest

@pytest.mark.skip(reason="API endpoint not yet deployed")
def test_new_endpoint():
    ...

@pytest.mark.skipif(sys.version_info < (3, 11), reason="Requires 3.11+")
def test_new_feature():
    ...

@pytest.mark.xfail(reason="Known bug, fix in PR #234")
def test_buggy():
    assert broken_thing() == "expected"

@pytest.mark.slow
def test_full_integration():
    ...  # 30 seconds

# Run only fast tests by default:  pytest -m "not slow"
# Run only slow:                    pytest -m slow

Mocking with monkeypatch and mocker

For tests that need to replace dependencies (external APIs, system calls, current time):

def test_api_call(monkeypatch):
    monkeypatch.setenv("API_KEY", "test-key")
    monkeypatch.setattr("mymodule.requests.get", lambda url: FakeResp())
    assert fetch_data() == {"status": "ok"}

# With pytest-mock (more powerful)
def test_with_mocker(mocker):
    mock_send = mocker.patch("mymodule.send_email")
    mock_send.return_value = True

    result = signup_user("alice@example.com")
    mock_send.assert_called_once_with("alice@example.com", "Welcome")

Test Discovery and Configuration

pytest auto-discovers tests in files starting with test_ or ending in _test.py, classes named Test*, and functions named test_*. Configure in pyproject.toml:

# pyproject.toml
[tool.pytest.ini_options]
minversion = "7.0"
testpaths = ["tests"]
python_files = "test_*.py"
python_classes = "Test*"
python_functions = "test_*"
addopts = "-ra --strict-markers --cov=myapp"
markers = [
    "slow: slow tests (deselect with -m 'not slow')",
    "integration: tests that hit real external services",
]

Common Pitfalls

  • Shared state between tests. Module-level globals that tests modify make order-dependent bugs. Use fixtures with function scope to reset state per test.
  • Fixture scope mismatch. A session-scoped fixture that mutates state breaks isolation. Mutable fixtures should be function-scoped.
  • Catching too-broad exceptions. assert exc.value instead of with pytest.raises(SpecificError) swallows bugs.
  • Slow imports in conftest. pytest imports conftest.py before any test runs. Heavy imports there slow every pytest invocation.
  • Ignoring -ra output. The summary section at the end shows skipped, xfailed, and warning details. Read it — it's where flaky tests hide.

FAQ

Q: pytest or unittest?
A: pytest — better fixtures, parametrization, plugin ecosystem, simpler assertion syntax. unittest is fine for tiny projects or if you can't add dependencies.

Q: How do I run only failing tests?
A: pytest --lf (last-failed). Combine with --ff (failed-first) for fast iteration while debugging.

Q: How do I measure code coverage?
A: pip install pytest-cov then pytest --cov=myapp --cov-report=html. The HTML report tells you which lines weren't hit.

Q: How do I test async code?
A: pip install pytest-asyncio. Mark tests with @pytest.mark.asyncio and use async def. Fixtures can be async too.

Q: How do I parallelize tests?
A: pip install pytest-xdist then pytest -n auto uses all CPU cores. Works great for independent unit tests; less great for tests that share databases.

Wrapping Up

pytest's superpower is fixtures + parametrization — together they remove almost all test boilerplate. Add pytest-cov for coverage, pytest-mock for mocking, pytest-xdist for parallelism. The ecosystem is huge (over 1000 plugins) but you usually need only those four to cover 95% of testing needs. Master fixtures first; everything else flows from there.

How To Connect Python to PostgreSQL with psycopg

How To Connect Python to PostgreSQL with psycopg

Last Updated: June 01, 2026

Intermediate

PostgreSQL is one of the most powerful open-source relational databases, and Python developers interact with it constantly — whether building web APIs, running data pipelines, or managing application state. If you have ever needed to store structured data beyond what SQLite can handle, PostgreSQL is usually the next step.

The good news is that psycopg (version 3, the modern successor to the venerable psycopg2) makes connecting Python to PostgreSQL straightforward and safe. It supports parameterized queries out of the box, handles connection pooling, and works beautifully with async code. You can install it with a single pip install psycopg[binary] command and be running queries in minutes.

In this article, we will cover everything you need to connect Python to PostgreSQL. We will start with a quick example showing a basic connection and query, then explain the psycopg library and why it is the recommended adapter. From there, we will walk through CRUD operations (Create, Read, Update, Delete), parameterized queries for security, connection pooling for performance, error handling patterns, and finish with a complete real-life project that builds a task manager backed by PostgreSQL.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Connecting Python to PostgreSQL: Quick Example

Here is a minimal working example that connects to a PostgreSQL database, creates a table, inserts a row, and reads it back. This gives you the core pattern you will use in every PostgreSQL project.

# quick_example.py
import psycopg

# Connect to PostgreSQL (adjust these for your setup)
conn_string = "host=localhost dbname=testdb user=postgres password=postgres"

with psycopg.connect(conn_string) as conn:
    with conn.cursor() as cur:
        # Create a simple table
        cur.execute("""
            CREATE TABLE IF NOT EXISTS greetings (
                id SERIAL PRIMARY KEY,
                message TEXT NOT NULL
            )
        """)
        # Insert a row
        cur.execute("INSERT INTO greetings (message) VALUES (%s)", ("Hello from Python!",))
        conn.commit()

        # Read it back
        cur.execute("SELECT id, message FROM greetings ORDER BY id DESC LIMIT 1")
        row = cur.fetchone()
        print(f"ID: {row[0]}, Message: {row[1]}")

Output:

ID: 1, Message: Hello from Python!

The key things to notice: we use psycopg.connect() with a connection string, wrap everything in with blocks for automatic cleanup, and use %s placeholders for parameterized queries (never string formatting). The conn.commit() call makes the insert permanent. Want to go deeper? Below we cover connection options, all four CRUD operations, pooling, and a complete project.

Understanding psycopg Python PostgreSQL adapter
One connection string. Infinite queries. Zero SQL injection.

What Is psycopg and Why Use It?

psycopg is the most popular PostgreSQL adapter for Python. Version 3 (just called psycopg) is a complete rewrite of the classic psycopg2 that powered Django, Flask, and countless Python applications for over a decade. The new version brings a cleaner API, native async support, and better type handling while keeping the reliability developers trusted.

Here is how psycopg compares to other options for connecting Python to PostgreSQL:

Featurepsycopg (v3)psycopg2asyncpg
Python 3.7+ supportYesYesYes
Async supportBuilt-inNo (needs wrappers)Async only
Connection poolingBuilt-inSeparate packageBuilt-in
Parameterized queries%s and named%s and named$1, $2 style
COPY supportExcellentGoodGood
Active developmentYes (recommended)Maintenance onlyYes
Django/Flask compatibleYesYesLimited

For most Python developers, psycopg (v3) is the right choice. It handles both sync and async workflows, has excellent documentation, and is officially recommended by the PostgreSQL community. The rest of this article uses psycopg v3 exclusively.

Installing psycopg

The easiest way to install psycopg is with the binary package, which bundles the C library so you do not need PostgreSQL development headers installed:

# install_psycopg.sh
pip install "psycopg[binary]"

Output:

Successfully installed psycopg-3.1.18 psycopg-binary-3.1.18

If you prefer to compile from source (common in production Docker images), install the base package and make sure libpq-dev is available: pip install psycopg[c]. For development and tutorials, the binary option is the fastest path.

Connecting to PostgreSQL

psycopg offers several ways to specify your connection. The most common patterns are a connection string (DSN) and keyword arguments. Both produce identical results — choose whichever reads better in your codebase.

# connection_methods.py
import psycopg

# Method 1: Connection string (DSN)
conn1 = psycopg.connect("host=localhost dbname=myapp user=appuser password=secret")

# Method 2: Keyword arguments
conn2 = psycopg.connect(
    host="localhost",
    dbname="myapp",
    user="appuser",
    password="secret",
    port=5432
)

# Method 3: PostgreSQL URI format
conn3 = psycopg.connect("postgresql://appuser:secret@localhost:5432/myapp")

# Always use context managers for automatic cleanup
with psycopg.connect("host=localhost dbname=myapp user=appuser password=secret") as conn:
    print(f"Connected to: {conn.info.dbname}")
    print(f"Server version: {conn.info.server_version}")

conn1.close()
conn2.close()
conn3.close()

Output:

Connected to: myapp
Server version: 160001

The context manager pattern (with psycopg.connect(...) as conn) is strongly recommended. It automatically commits the transaction on success, rolls back on exception, and closes the connection when the block exits. This prevents connection leaks and orphaned transactions — two of the most common PostgreSQL headaches in production.

Connecting Python to PostgreSQL database
conn = psycopg.connect() — three seconds to production-ready database access.

CRUD Operations with psycopg

CREATE: Inserting Data

Inserting data uses cursor.execute() with parameterized queries. Always use %s placeholders — never f-strings or string concatenation. Parameterized queries prevent SQL injection and handle type conversion automatically.

# insert_data.py
import psycopg

with psycopg.connect("host=localhost dbname=testdb user=postgres password=postgres") as conn:
    with conn.cursor() as cur:
        cur.execute("""
            CREATE TABLE IF NOT EXISTS users (
                id SERIAL PRIMARY KEY,
                name TEXT NOT NULL,
                email TEXT UNIQUE NOT NULL,
                age INTEGER
            )
        """)

        # Insert a single row with parameterized query
        cur.execute(
            "INSERT INTO users (name, email, age) VALUES (%s, %s, %s)",
            ("Alice Chen", "alice@example.com", 28)
        )

        # Insert multiple rows efficiently with executemany
        new_users = [
            ("Bob Park", "bob@example.com", 34),
            ("Carol Smith", "carol@example.com", 22),
            ("Dave Wilson", "dave@example.com", 45),
        ]
        cur.executemany(
            "INSERT INTO users (name, email, age) VALUES (%s, %s, %s)",
            new_users
        )

        conn.commit()
        print(f"Inserted {1 + len(new_users)} users successfully")

Output:

Inserted 4 users successfully

The executemany() method is cleaner than looping with individual execute() calls, and psycopg optimizes it internally. For truly large batches (thousands of rows), look into cursor.copy() which uses PostgreSQL’s COPY protocol and is dramatically faster.

READ: Querying Data

Reading data involves executing a SELECT query and fetching results. psycopg gives you several fetch options depending on how much data you expect.

# read_data.py
import psycopg

with psycopg.connect("host=localhost dbname=testdb user=postgres password=postgres") as conn:
    with conn.cursor() as cur:
        # Fetch all rows
        cur.execute("SELECT id, name, email, age FROM users ORDER BY name")
        all_users = cur.fetchall()
        print("All users:")
        for user in all_users:
            print(f"  {user[0]}: {user[1]} ({user[2]}), age {user[3]}")

        # Fetch one row
        cur.execute("SELECT name, age FROM users WHERE email = %s", ("alice@example.com",))
        alice = cur.fetchone()
        print(f"\nFound: {alice[0]}, age {alice[1]}")

        # Use row factory for named columns (much more readable)
        cur = conn.cursor(row_factory=psycopg.rows.dict_row)
        cur.execute("SELECT name, email, age FROM users WHERE age > %s", (25,))
        older_users = cur.fetchall()
        print(f"\nUsers over 25:")
        for u in older_users:
            print(f"  {u['name']}: {u['email']}, age {u['age']}")

Output:

All users:
  1: Alice Chen (alice@example.com), age 28
  2: Bob Park (bob@example.com), age 34
  3: Carol Smith (carol@example.com), age 22
  4: Dave Wilson (dave@example.com), age 45

Found: Alice Chen, age 28

Users over 25:
  Alice Chen: alice@example.com, age 28
  Bob Park: bob@example.com, age 34
  Dave Wilson: dave@example.com, age 45

The dict_row row factory is a game-changer for readability. Instead of accessing columns by index (row[0], row[1]), you use names (row['name'], row['email']). This makes your code self-documenting and resilient to column order changes.

UPDATE: Modifying Data

Updates follow the same parameterized pattern. The rowcount attribute tells you how many rows were affected.

# update_data.py
import psycopg

with psycopg.connect("host=localhost dbname=testdb user=postgres password=postgres") as conn:
    with conn.cursor() as cur:
        # Update a single user
        cur.execute(
            "UPDATE users SET age = %s WHERE email = %s",
            (29, "alice@example.com")
        )
        print(f"Updated {cur.rowcount} row(s)")

        # Update multiple rows with a condition
        cur.execute(
            "UPDATE users SET age = age + 1 WHERE age < %s",
            (30,)
        )
        print(f"Birthday bump: {cur.rowcount} user(s) aged up")

        conn.commit()

Output:

Updated 1 row(s)
Birthday bump: 2 user(s) aged up

Always check cur.rowcount after updates and deletes. If it returns 0 when you expected changes, your WHERE clause might be wrong -- and catching that early saves hours of debugging.

DELETE: Removing Data

Deletes work the same way. Be cautious with DELETE statements -- a missing WHERE clause deletes everything in the table.

# delete_data.py
import psycopg

with psycopg.connect("host=localhost dbname=testdb user=postgres password=postgres") as conn:
    with conn.cursor() as cur:
        # Delete a specific user
        cur.execute(
            "DELETE FROM users WHERE email = %s",
            ("dave@example.com",)
        )
        print(f"Deleted {cur.rowcount} user(s)")

        # Verify the deletion
        cur.execute("SELECT COUNT(*) FROM users")
        count = cur.fetchone()[0]
        print(f"Remaining users: {count}")

        conn.commit()

Output:

Deleted 1 user(s)
Remaining users: 3
CRUD operations with Python and PostgreSQL
Four operations, infinite applications. CRUD is the backbone of every database app.

Error Handling

Database operations fail in predictable ways -- duplicate keys, connection drops, malformed queries. psycopg raises specific exception types for each, so you can handle them precisely.

# error_handling.py
import psycopg
from psycopg import errors

conn_string = "host=localhost dbname=testdb user=postgres password=postgres"

try:
    with psycopg.connect(conn_string) as conn:
        with conn.cursor() as cur:
            # This will fail if email already exists (UNIQUE constraint)
            cur.execute(
                "INSERT INTO users (name, email, age) VALUES (%s, %s, %s)",
                ("Alice Chen", "alice@example.com", 28)
            )
            conn.commit()
except errors.UniqueViolation as e:
    print(f"Duplicate entry: {e.diag.message_detail}")
except errors.OperationalError as e:
    print(f"Connection problem: {e}")
except errors.ProgrammingError as e:
    print(f"SQL error: {e}")
except Exception as e:
    print(f"Unexpected error: {type(e).__name__}: {e}")

Output:

Duplicate entry: Key (email)=(alice@example.com) already exists.

The psycopg.errors module maps every PostgreSQL error code to a Python exception class. UniqueViolation, ForeignKeyViolation, CheckViolation -- they are all there. This lets you show users a friendly "email already taken" message instead of a raw database error.

Connection Pooling

Creating a new database connection for every request is slow (each connection involves a TCP handshake, authentication, and memory allocation on the server). Connection pooling solves this by maintaining a set of open connections that get reused across requests.

# connection_pool.py
from psycopg_pool import ConnectionPool

# Create a pool with min 2, max 10 connections
pool = ConnectionPool(
    "host=localhost dbname=testdb user=postgres password=postgres",
    min_size=2,
    max_size=10
)

# Use connections from the pool
with pool.connection() as conn:
    with conn.cursor() as cur:
        cur.execute("SELECT COUNT(*) FROM users")
        count = cur.fetchone()[0]
        print(f"User count: {count}")

# The connection is returned to the pool, not closed
with pool.connection() as conn:
    with conn.cursor() as cur:
        cur.execute("SELECT name FROM users LIMIT 1")
        name = cur.fetchone()[0]
        print(f"First user: {name}")

# Get pool stats
stats = pool.get_stats()
print(f"Pool size: {stats['pool_size']}, available: {stats['pool_available']}")

pool.close()

Output:

User count: 3
First user: Alice Chen
Pool size: 2, available: 2

In a web application (Flask, FastAPI, Django), you would create the pool once at startup and share it across all request handlers. This dramatically reduces latency since connections are reused instead of created fresh for every HTTP request. The max_size parameter prevents your application from overwhelming the database with too many simultaneous connections.

Connection pooling for PostgreSQL in Python
One pool, ten connections, a thousand requests. Connection pooling is free performance.

Working with Transactions

By default, psycopg wraps every operation in a transaction. The context manager commits on success and rolls back on failure. But sometimes you need more control -- for example, when multiple operations must succeed or fail together.

# transactions.py
import psycopg

conn_string = "host=localhost dbname=testdb user=postgres password=postgres"

with psycopg.connect(conn_string) as conn:
    # Explicit transaction control
    try:
        with conn.transaction():
            with conn.cursor() as cur:
                # Both operations must succeed
                cur.execute(
                    "UPDATE users SET age = age - 1 WHERE name = %s",
                    ("Alice Chen",)
                )
                cur.execute(
                    "UPDATE users SET age = age + 1 WHERE name = %s",
                    ("Bob Park",)
                )
                print("Both updates committed together")
    except Exception as e:
        print(f"Transaction rolled back: {e}")

    # Nested savepoints
    with conn.transaction() as tx1:
        with conn.cursor() as cur:
            cur.execute("INSERT INTO users (name, email, age) VALUES (%s, %s, %s)",
                        ("Eve Brown", "eve@example.com", 31))
            try:
                with conn.transaction() as tx2:
                    cur.execute("INSERT INTO users (name, email, age) VALUES (%s, %s, %s)",
                                ("Eve Brown", "eve-duplicate@example.com", 31))
                    # This inner transaction can fail without killing the outer one
            except Exception:
                print("Inner savepoint rolled back, outer transaction continues")

        conn.commit()
        print("Eve inserted successfully")

Output:

Both updates committed together
Eve inserted successfully

The conn.transaction() context manager creates a savepoint when nested. This is incredibly useful for "try this, but if it fails, keep going" patterns -- common in data import pipelines where you want to skip bad rows without losing the entire batch.

Real-Life Example: Building a Task Manager CLI

Let us tie everything together with a complete task manager that stores tasks in PostgreSQL. This project uses connection pooling, parameterized queries, error handling, and all four CRUD operations.

Building a task manager CLI with Python and PostgreSQL
A complete CRUD app with pooling and error handling. Not bad for 50 lines.
# task_manager.py
import psycopg
from psycopg_pool import ConnectionPool
from psycopg import errors
from datetime import datetime

DB_URL = "host=localhost dbname=testdb user=postgres password=postgres"

def setup_database(pool):
    """Create the tasks table if it does not exist."""
    with pool.connection() as conn:
        with conn.cursor() as cur:
            cur.execute("""
                CREATE TABLE IF NOT EXISTS tasks (
                    id SERIAL PRIMARY KEY,
                    title TEXT NOT NULL,
                    description TEXT DEFAULT '',
                    status TEXT DEFAULT 'pending',
                    created_at TIMESTAMP DEFAULT NOW(),
                    completed_at TIMESTAMP
                )
            """)
            conn.commit()

def add_task(pool, title, description=""):
    """Add a new task and return its ID."""
    with pool.connection() as conn:
        with conn.cursor() as cur:
            cur.execute(
                "INSERT INTO tasks (title, description) VALUES (%s, %s) RETURNING id",
                (title, description)
            )
            task_id = cur.fetchone()[0]
            conn.commit()
            return task_id

def list_tasks(pool, status_filter=None):
    """List tasks, optionally filtered by status."""
    with pool.connection() as conn:
        with conn.cursor(row_factory=psycopg.rows.dict_row) as cur:
            if status_filter:
                cur.execute(
                    "SELECT id, title, status, created_at FROM tasks WHERE status = %s ORDER BY created_at",
                    (status_filter,)
                )
            else:
                cur.execute("SELECT id, title, status, created_at FROM tasks ORDER BY created_at")
            return cur.fetchall()

def complete_task(pool, task_id):
    """Mark a task as completed."""
    with pool.connection() as conn:
        with conn.cursor() as cur:
            cur.execute(
                "UPDATE tasks SET status = %s, completed_at = %s WHERE id = %s",
                ("completed", datetime.now(), task_id)
            )
            conn.commit()
            return cur.rowcount > 0

def delete_task(pool, task_id):
    """Delete a task by ID."""
    with pool.connection() as conn:
        with conn.cursor() as cur:
            cur.execute("DELETE FROM tasks WHERE id = %s", (task_id,))
            conn.commit()
            return cur.rowcount > 0

# Demo usage
pool = ConnectionPool(DB_URL, min_size=2, max_size=5)
setup_database(pool)

# Add some tasks
id1 = add_task(pool, "Learn psycopg", "Complete the PostgreSQL tutorial")
id2 = add_task(pool, "Build REST API", "Create FastAPI endpoints for tasks")
id3 = add_task(pool, "Write tests", "Add pytest coverage for database layer")
print(f"Created tasks: {id1}, {id2}, {id3}")

# List all tasks
print("\nAll tasks:")
for task in list_tasks(pool):
    print(f"  [{task['status']}] #{task['id']}: {task['title']}")

# Complete a task
complete_task(pool, id1)
print(f"\nCompleted task #{id1}")

# List pending tasks only
print("\nPending tasks:")
for task in list_tasks(pool, "pending"):
    print(f"  #{task['id']}: {task['title']}")

# Delete a task
delete_task(pool, id3)
print(f"\nDeleted task #{id3}")

# Final count
print(f"\nTotal tasks remaining: {len(list_tasks(pool))}")

pool.close()

Output:

Created tasks: 1, 2, 3

All tasks:
  [pending] #1: Learn psycopg
  [pending] #2: Build REST API
  [pending] #3: Write tests

Completed task #1

Pending tasks:
  #2: Build REST API
  #3: Write tests

Deleted task #3

Total tasks remaining: 2

This task manager demonstrates every concept from the article: connecting with a pool, parameterized queries for safety, dict_row for readable results, RETURNING clauses for getting generated IDs, and proper transaction handling. You could extend this into a full web application by wrapping these functions in FastAPI or Flask endpoints.

Frequently Asked Questions

Should I use psycopg2 or psycopg (v3)?

For new projects, always use psycopg v3 (installed as pip install psycopg). It has better async support, built-in connection pooling, a cleaner API, and is actively developed. psycopg2 is in maintenance mode -- it still works, but new features and improvements only land in v3. The migration is straightforward since the core concepts (parameterized queries, cursors, context managers) are the same.

How do I prevent SQL injection with psycopg?

Always use parameterized queries with %s placeholders: cur.execute("SELECT * FROM users WHERE id = %s", (user_id,)). Never use f-strings, string concatenation, or format() to build SQL. psycopg handles escaping and type conversion automatically, making injection impossible as long as you use placeholders consistently.

Can I use psycopg with async/await?

Yes. psycopg v3 has a built-in async module: from psycopg import AsyncConnection. Use await AsyncConnection.connect() and await cursor.execute(). It works with asyncio, FastAPI, and any other async framework. The async connection pool is AsyncConnectionPool from psycopg_pool.

How many connections should my pool have?

A good starting point is min_size=2, max_size=10 for small applications. The PostgreSQL documentation suggests a formula: max_connections = (core_count * 2) + effective_spindle_count. In practice, most web applications work well with 10-20 connections in the pool. Monitor your PostgreSQL pg_stat_activity view to see actual connection usage and tune from there.

How do I store database credentials securely?

Never hardcode credentials in your source code. Use environment variables (os.environ['DATABASE_URL']), a .env file loaded with python-dotenv, or a secrets manager (AWS Secrets Manager, HashiCorp Vault). PostgreSQL also supports a ~/.pgpass file for local development. For connection strings, the standard DATABASE_URL environment variable works with most frameworks and deployment platforms.

Conclusion

You now have a solid foundation for connecting Python to PostgreSQL with psycopg. We covered the essential workflow: installing psycopg[binary], establishing connections with context managers, running all four CRUD operations with parameterized queries, handling database errors gracefully, and using connection pooling for production performance. The task manager project ties all these concepts into a practical, extensible application.

From here, try extending the task manager with features like priority levels, due dates, or full-text search using PostgreSQL's tsvector type. Psycopg handles all of these naturally since it passes your SQL through to PostgreSQL without limiting which features you can use.

For the complete API reference and advanced topics like COPY operations, async usage, and custom type adapters, check the official psycopg documentation at www.psycopg.org/psycopg3/docs/.

Continue Learning Python

Tutorials you might also find useful:

How To Set Up Logging in Python 3 (Output to File and Console)

How To Set Up Logging in Python 3 (Output to File and Console)

Last Updated: June 01, 2026

Intermediate

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Why Logging Matters in Python

You’re debugging a production issue, but your application is silent. You added a few print() statements weeks ago, the messages got buried in the terminal, and now you have no idea what’s happening. Or worse: your app is logging to console, but the logs disappear the moment the process restarts. You need a way to capture what your application is doing—when it’s doing it, at what severity level, and where it should be recorded.

This is where Python’s built-in logging module becomes essential. Unlike print() statements, which are crude and destructive once you delete them, the logging module is a professional-grade system designed for production applications. It comes built-in to Python, requires no external dependencies, and provides granular control over message levels, formatting, and output destinations.

In this article, you’ll learn how to set up the logging module to output messages simultaneously to both your console (for immediate feedback during development) and to a file (for long-term record-keeping and debugging). We’ll cover logging levels, handlers, formatters, log rotation to prevent massive log files, and the patterns used in real multi-module projects. By the end, you’ll understand how to instrument your code with logging that developers trust.

How To Set Up Logging: Quick Example

Here’s a minimal example that outputs log messages to both console and file:

# quick_logging_example.py
import logging

# Create a logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

# File handler
file_handler = logging.FileHandler("app.log")
file_handler.setLevel(logging.DEBUG)

# Console handler
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)

# Formatter
formatter = logging.Formatter(
    "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)
file_handler.setFormatter(formatter)
console_handler.setFormatter(formatter)

# Add handlers to logger
logger.addHandler(file_handler)
logger.addHandler(console_handler)

# Log some messages
logger.debug("Debug message (goes to file only)")
logger.info("Info message (goes to both)")
logger.warning("Warning message (goes to both)")
logger.error("Error message (goes to both)")
logger.critical("Critical message (goes to both)")

Output (to console):

2026-03-29 14:22:15,342 - __main__ - INFO - Info message (goes to both)
2026-03-29 14:22:15,343 - __main__ - WARNING - Warning message (goes to both)
2026-03-29 14:22:15,344 - __main__ - ERROR - Error message (goes to both)
2026-03-29 14:22:15,344 - __main__ - CRITICAL - Critical message (goes to both)

Output (written to app.log):

2026-03-29 14:22:15,341 - __main__ - DEBUG - Debug message (goes to file only)
2026-03-29 14:22:15,342 - __main__ - INFO - Info message (goes to both)
2026-03-29 14:22:15,343 - __main__ - WARNING - Warning message (goes to both)
2026-03-29 14:22:15,344 - __main__ - ERROR - Error message (goes to both)
2026-03-29 14:22:15,344 - __main__ - CRITICAL - Critical message (goes to both)

Notice the key pattern: we created a logger, attached two separate handlers (one for files, one for console), set different levels for each, and applied a formatter that includes timestamps and severity levels. This is the foundation for everything that follows. The sections below show you how to customize each piece.

Debug Dee examining floating log entries through magnifying glass
Good logs are how you debug code you wrote six months ago and forgot about.

What is Python Logging and Why Use It?

The logging module is Python’s standard library tool for recording events that happen during program execution. Unlike print statements, logging provides:

  • Severity levels — categorize messages by importance (DEBUG, INFO, WARNING, ERROR, CRITICAL)
  • Multiple outputs — send logs to files, console, email, syslog, or custom handlers simultaneously
  • Formatting control — include timestamps, function names, line numbers, and custom metadata
  • Filtering — selectively log messages based on logger name, level, or custom criteria
  • No side effects — unlike print, you can leave logging code in production without cluttering output

The alternative—using print() for debugging—breaks down immediately:

Aspectprint() Statementslogging Module
Disable in productionMust manually removeAdjust level, keep code in place
Output destinationAlways stdoutFile, console, email, or custom
TimestampsManual string concatenationAutomatic, customizable format
Severity levelsNoneDEBUG, INFO, WARNING, ERROR, CRITICAL
PerformanceAlways evaluatesCan be filtered; lazy evaluation
Multi-module coordinationNo built-in supportHierarchical logger names

The logging module is designed for exactly what you need: professional-grade event recording that stays in your code indefinitely.

Understanding Logging Levels

Python’s logging module defines five standard severity levels, plus a catch-all NOTSET. Each level has a numeric value, and loggers will only record messages at or above their configured level:

LevelNumeric ValueWhen to UseExample
DEBUG10Detailed diagnostic info for debuggingVariable values, function entry/exit, loop iterations
INFO20General informational messagesApplication startup, config loaded, request received
WARNING30Something unexpected or potentially harmfulDeprecated API usage, missing optional config, retrying failed request
ERROR40A serious problem; some operation failedFile not found, API returned 500, database connection lost
CRITICAL50A very serious error; program may not continueOut of memory, permissions denied, unrecoverable system error

When you set a logger’s level to INFO, it will log INFO, WARNING, ERROR, and CRITICAL messages—but not DEBUG messages. This is how you control verbosity.

# logging_levels_demo.py
import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

# Add a console handler so we can see output
handler = logging.StreamHandler()
handler.setLevel(logging.WARNING)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)

# These will NOT appear (level is below WARNING)
logger.debug("This is a debug message")
logger.info("This is an info message")

# These WILL appear
logger.warning("This is a warning message")
logger.error("This is an error message")
logger.critical("This is a critical message")

Output:

WARNING - This is a warning message
ERROR - This is an error message
CRITICAL - This is a critical message

Notice: the logger itself has one level (DEBUG), but the console handler has a different level (WARNING). You can filter messages at multiple levels—first at the logger, then at each handler. This is crucial for sending different messages to different outputs (e.g., all DEBUG messages to a debug log file, only ERROR+ to a critical alert file).

Handlers and Formatters: Controlling Where and How Logs Go

A logger is just a container. The actual work happens in handlers and formatters:

  • Handler — an output destination. FileHandler writes to a file, StreamHandler writes to console, etc.
  • Formatter — defines how log messages are formatted: which fields to include (timestamp, function name, etc.) and in what order

You create a handler, assign a formatter to it, set a level, and attach it to a logger. A single logger can have multiple handlers, each with different levels and formatters.

Creating a StreamHandler (Console Output):

# stream_handler_example.py
import logging

logger = logging.getLogger("myapp")
logger.setLevel(logging.DEBUG)

# Create a console handler
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)

# Format: timestamp, logger name, level, message
formatter = logging.Formatter(
    "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)
console_handler.setFormatter(formatter)

logger.addHandler(console_handler)

logger.info("Application started")
logger.warning("This is a warning")
logger.error("An error occurred")

Output:

2026-03-29 14:25:30,123 - myapp - INFO - Application started
2026-03-29 14:25:30,124 - myapp - WARNING - This is a warning
2026-03-29 14:25:30,125 - myapp - ERROR - An error occurred

The %(asctime)s token automatically includes a timestamp. Other useful tokens include %(funcName)s (the function name), %(lineno)d (line number), and %(module)s (the module filename).

Creating a FileHandler (File Output):

# file_handler_example.py
import logging

logger = logging.getLogger("myapp")
logger.setLevel(logging.DEBUG)

# Create a file handler
file_handler = logging.FileHandler("app.log")
file_handler.setLevel(logging.DEBUG)

formatter = logging.Formatter(
    "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)
file_handler.setFormatter(formatter)

logger.addHandler(file_handler)

logger.debug("Debug: application starting")
logger.info("Info: loading configuration")
logger.warning("Warning: deprecated API used")
logger.error("Error: failed to connect to database")

After running this, check your app.log file. All four messages will be there because the file handler’s level is DEBUG.

Output (written to app.log):

2026-03-29 14:27:01,456 - myapp - DEBUG - Debug: application starting
2026-03-29 14:27:01,457 - myapp - INFO - Info: loading configuration
2026-03-29 14:27:01,458 - myapp - WARNING - Warning: deprecated API used
2026-03-29 14:27:01,459 - myapp - ERROR - Error: failed to connect to database
Sudo Sam directing log traffic at an intersection
Handlers are traffic directors: DEBUG takes the file fork, ERROR takes the console.

Logging to Console and File Simultaneously

The most common pattern in production is to send all logs to a file (for permanent record) and only show WARNING+ messages on the console (for immediate visibility during operation). Here’s how:

# console_and_file_logging.py
import logging
import os

# Create a logger
logger = logging.getLogger("myapp")
logger.setLevel(logging.DEBUG)

# Create log directory if it doesn't exist
log_dir = "logs"
if not os.path.exists(log_dir):
    os.makedirs(log_dir)

# File handler: captures all messages
file_handler = logging.FileHandler(os.path.join(log_dir, "app.log"))
file_handler.setLevel(logging.DEBUG)

# Console handler: shows only warnings and above
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.WARNING)

# Shared formatter for both handlers
formatter = logging.Formatter(
    "%(asctime)s - %(name)s - %(levelname)s - %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S"
)
file_handler.setFormatter(formatter)
console_handler.setFormatter(formatter)

# Attach handlers to logger
logger.addHandler(file_handler)
logger.addHandler(console_handler)

# Lof messages at different levels
logger.debug("Starting application initialization")
logger.info("Configuration loaded successfully")
logger.info("Database connection established")
logger.warning("API response time is higher than usual")
logger.error("Failed to write to cache, continuing without cache")
logger.critical("Memory usage exceeded safe threshold")

Output (to console):

2026-03-29 14:30:12 - myapp - WARNING - API response time is higher than usual
2026-03-29 14:30:12 - myapp - ERROR - Failed to write to cache, continuing without cache
2026-03-29 14:30:12 - myapp - CRITICAL - Memory usage exceeded safe threshold

Output (written to logs/app.log):

2026-03-29 14:30:12 - myapp - DEBUG - Starting application initialization
2026-03-29 14:30:12 - myapp - INFO - Configuration loaded successfully
2026-03-29 14:30:12 - myapp - INFO - Database connection established
2026-03-29 14:30:12 - myapp - WARNING - API response time is higher than usual
2026-03-29 14:30:12 - myapp - ERROR - Failed to write to cache, continuing without cache
2026-03-29 14:30:12 - myapp - CRITICAL - Memory usage exceeded safe threshold

This pattern is powerful: you get a permanent record of everything (including debug messages developers need when troubleshooting), but the console stays clean during normal operation—only showing problems that need immediate attention. When a warning or error occurs, developers see it right away.

Custom Log Formatting with Timestamps and Metadata

The formatter string controls what information appears in each log message. The most useful format tokens are:

TokenMeaningExample
%(asctime)sTimestamp (human-readable)2026-03-29 14:30:12,456
%(name)sLogger namemyapp.database
%(levelname)sSeverity levelINFO, WARNING, ERROR
%(message)sThe actual log messageDatabase query completed
%(funcName)sName of function that loggedconnect_to_db
%(filename)sSource filenamedatabase.py
%(lineno)dLine number in source42
%(module)sModule namedatabase
%(process[=]dProcess ID12345
%(thread)dThread ID140256789012345

Here are some practical format examples:

# formatting_examples.py
import logging

logger = logging.getLogger("myapp")
logger.setLevel(logging.DEBUG)

# Example 1: Detailed format with function and line number
handler1 = logging.StreamHandler()
formatter1 = logging.Formatter(
    "%(asctime)s [%(levelname)s] %(funcName)s:;%(lineno)d - %(message)s"
)
handler1.setFormatter(formatter1)

# Example 2: Compact format (good for production)
handler2_formatter = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"

# Example 3: Include module name (useful in multi-file projects)
handler3_formatter = (
    "[%(asctime)s] %(module)s - %(levelname)s - %(message)s"
)

# Example 4: ISO 8601 timestamp with timezone
handler4 = logging.StreamHandler()
formatter4 = logging.Formatter(
    "%(asctime)s - %(levelname)s - %(message)s",
    datefmt="%Y-%m-%dT%H:%M:%S"
)
handler4.setFormatter(formatter4)

logger.addHandler(handler1)

def process_payment(user_id):
    logger.info(f"Processing payment for user {user_id}")
    logger.debug("Validating card information")
    logger.info("Payment submitted to processor")
    return True

process_payment(12345)

Output (Example 1 format):

2026-03-29 14:32:45,123 [INFO] process_payment:55 - Processing payment for user 12345
2026-03-29 14:32:45,124 [DEBUG] process_payment:56 - Validating card information
2026-03-29:0;( 14:32:45,125 [INFO] process_payment:57 - Payment submitted to processor

Controlling Log File Size with Log Rotation

If your application runs 24/7 and logs every request, your log files can grow huge fast, eating disk space and slowing down anything that tries to read or grep them. The solution is RotatingFileHandler, which caps file size and automatically rolls old logs into numbered backups:

# File: rotating_logger.py
import logging
from logging.handlers import RotatingFileHandler

logger = logging.getLogger("payments")
logger.setLevel(logging.DEBUG)

# Max 5 MB per file, keep 3 old files (app.log.1, app.log.2, app.log.3)
handler = RotatingFileHandler(
    "app.log",
    maxBytes=5 * 1024 * 1024,
    backupCount=3,
)
handler.setFormatter(logging.Formatter(
    "%(asctime)s [%(levelname)s] %(message)s"
))
logger.addHandler(handler)

# Simulate heavy logging
for i in range(100_000):
    logger.info(f"Processed request {i}")

When app.log hits 5 MB, the handler renames it to app.log.1, shifts older backups up the chain, and starts a fresh app.log. Once backupCount is reached, the oldest file is deleted. You get bounded disk usage with no manual cleanup.

For time-based rotation — one log file per day, week, or hour — use TimedRotatingFileHandler instead:

from logging.handlers import TimedRotatingFileHandler

# Roll over at midnight every day, keep 14 days of history
handler = TimedRotatingFileHandler(
    "app.log",
    when="midnight",
    interval=1,
    backupCount=14,
)

This is ideal for compliance scenarios where you need a clean audit trail per day, or for shipping logs to a daily archive bucket.

Logging Exceptions and Tracebacks

One of the most common logging mistakes is catching an exception and only writing the error message — losing the traceback that tells you where things went wrong. Compare these two patterns:

# Bad — just the message, no traceback
try:
    result = risky_operation()
except Exception as e:
    logger.error(f"Operation failed: {e}")

# Good — full traceback automatically included
try:
    result = risky_operation()
except Exception:
    logger.exception("Operation failed")

logger.exception() is shorthand for logger.error(msg, exc_info=True). It records the message AND the full stack trace, so when you’re debugging at 2 AM you can see exactly which line raised, what the call chain was, and which third-party library was involved. Always use logger.exception() inside except blocks.

You can also force a traceback on lower-severity log calls with exc_info=True:

try:
    cache.get(key)
except CacheTimeout:
    logger.warning("Cache miss with timeout, falling back to DB", exc_info=True)
    return db.query(key)
Pick your log levels. Stick to them. Future you will read them.
Pick your log levels. Stick to them. Future you will read them.

Logging Across Multiple Modules

In real applications you have dozens of modules, and you want logs to show which one wrote each message. The convention is logger = logging.getLogger(__name__) at the top of every file. __name__ resolves to the dotted module path, so logs from app/services/payments.py appear under the logger name app.services.payments.

# File: app/services/payments.py
import logging

logger = logging.getLogger(__name__)  # name = "app.services.payments"

def charge_card(amount):
    logger.info("Charging card for $%s", amount)
    # ... charge logic ...

The benefit: in main.py (or wherever you configure logging) you can route specific modules to different handlers, set finer-grained levels, or silence noisy third-party libraries:

# File: main.py
import logging

# Root logger — catches everything at INFO+
logging.basicConfig(level=logging.INFO)

# Quiet down a noisy third-party library
logging.getLogger("urllib3").setLevel(logging.WARNING)

# Turn on DEBUG just for our payments module
logging.getLogger("app.services.payments").setLevel(logging.DEBUG)

This pattern scales — instead of editing logging calls in every file, you control verbosity from one place.

Structured Logging with JSON

Plain-text logs are great for tailing in a terminal, but if you ship logs to a centralized system (Elasticsearch, Datadog, Loki, CloudWatch), JSON-structured logs are dramatically easier to query. Each log line becomes a parsed record with searchable fields instead of regex-matchable strings.

The simplest path is python-json-logger:

# Install: pip install python-json-logger

import logging
from pythonjsonlogger import jsonlogger

logger = logging.getLogger("api")
handler = logging.StreamHandler()
handler.setFormatter(jsonlogger.JsonFormatter(
    "%(asctime)s %(name)s %(levelname)s %(message)s"
))
logger.addHandler(handler)
logger.setLevel(logging.INFO)

logger.info("user signup", extra={"user_id": 4231, "plan": "pro"})

Output:

{"asctime": "2026-03-29 15:01:22,847", "name": "api", "levelname": "INFO", "message": "user signup", "user_id": 4231, "plan": "pro"}

Now in your log aggregator you can filter by plan = "pro" directly, no regex required. The extra={} parameter is the secret — anything you pass there becomes a top-level JSON field.

Production Logging Best Practices

A few rules that pay back tenfold once your application is live and you’re not the only one debugging it:

  • Use lazy string formatting. Write logger.info("Got %s rows", count), not logger.info(f"Got {count} rows"). The lazy form only builds the string if the log level is actually enabled — important when DEBUG logs are off in production.
  • Don’t log secrets. Audit your messages for tokens, passwords, full credit card numbers, or PII. Centralized log storage is often broader-access than your production database.
  • Pick one log level per environment. DEBUG locally, INFO in staging, WARNING in production. Don’t mix.
  • Always include identifiers. Every log line tied to a user action should carry the user ID, request ID, or correlation ID. Logs without identifiers are noise.
  • Configure once, in one place. Use logging.config.dictConfig() with a config dict (or a YAML file) at app startup. Don’t sprinkle basicConfig() calls throughout the codebase.
  • Test that logs are being written. A surprising number of production outages are made worse by “we didn’t have any logs” — usually because someone called logging.basicConfig() after another module had already configured the root logger, and the second call silently no-ops.

Common Logging Pitfalls

Three patterns to watch for:

1. Calling logging.basicConfig() after another module has logged. basicConfig() only adds handlers if the root logger has none. The fix: configure logging as the very first thing in main.py, before importing your modules.

2. Duplicate log messages. If you accidentally add the same handler twice — or if your code sets up logging on import and again in __main__ — every message prints twice. The fix: check if logger.hasHandlers() before adding handlers, or rely on dictConfig which idempotently rebuilds the config.

3. Logger.propagate surprises. Child loggers propagate to parents by default. If you add a console handler to app AND to app.services, messages from app.services appear twice. Set logger.propagate = False on the child or only add handlers at the root.

FAQ

Q: What’s the difference between logger.info() and logger.debug()?
A: Severity. INFO is for “normal operational events I want to see in production” — startup, request completion, scheduled job ran. DEBUG is for verbose internal state useful when reproducing a bug locally. In production, DEBUG is usually off so the noise doesn’t drown out the signal.

Q: Should I use print() instead?
A: For one-off scripts, fine. For anything you’ll run more than once, no. print() can’t be filtered by severity, can’t be redirected to multiple destinations, doesn’t carry timestamps or module names, and writes to stdout which mingles with your application’s actual output.

Q: How do I log to a remote system like CloudWatch or Datadog?
A: Two common approaches. (1) Ship logs to a local file in JSON format and run a sidecar agent (CloudWatch Agent, Vector, Fluent Bit) that tails the file and forwards. (2) Use a Python handler that posts directly — watchtower for CloudWatch, datadog-python for Datadog. Option 1 is more resilient because it survives network blips.

Q: Why are my logs not appearing?
A: Most common cause: the root logger’s level is higher than the message level. Try logging.basicConfig(level=logging.DEBUG) at the very top of main.py. Second most common: another import called basicConfig() first and you didn’t notice.

Q: How do I correlate logs across services in a microservices setup?
A: Generate a UUID-based request ID at the API gateway, pass it through every downstream service in a header (X-Request-ID), and include it in every log line via extra={"request_id": ...}. When you’re debugging an issue, you grep the request ID across all services’ logs and see the full timeline.

Wrapping Up

Python’s logging module is one of those tools where the 90% solution is straightforward — call logging.basicConfig(), get a logger with logging.getLogger(__name__), write info/error messages — and the remaining 10% (rotation, JSON output, multi-handler routing) becomes important as soon as your application leaves your laptop. Get the basics right early and the advanced patterns are small additions, not refactors.

The official Python logging documentation has the full reference for everything covered here plus the more obscure handlers (SMTP, SysLog, HTTP). For tutorials on related topics, see the related articles section below.

How To Automate Repetitive Tasks with Python

How To Automate Repetitive Tasks with Python

Last Updated: June 01, 2026

Beginner

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Introduction

Every developer has experienced the monotony of repetitive tasks: renaming thousands of files, backing up project folders on schedule, generating weekly reports, or scanning for files that need processing. These are the moments when you wish a robot would just handle it while you focus on actual coding. The good news? Python makes this incredibly straightforward, and you already have everything you need in the standard library.

Python was designed with automation in mind. Libraries like os, shutil, pathlib, and smtplib give you powerful tools to interact with the file system, schedule tasks, and send notifications. You don’t need to learn complex shell scripts or invest in expensive automation software. A few lines of Python can save you hours of manual work.

In this guide, we’ll explore practical automation patterns starting with file operations and building toward a real-world automated backup system. By the end, you’ll have a toolkit for automating any repetitive task in your workflow.

Quick Example: Rename Files in Bulk

Before diving deep, let’s see automation in action. Imagine you have 500 image files named like IMG_0001.jpg, IMG_0002.jpg, and you want to prefix them with today’s date. Without automation, this takes hours. With Python, it takes seconds:

# bulk_rename.py
import os
import datetime

directory = "./photos"
prefix = datetime.date.today().strftime("%Y%m%d_")

for filename in os.listdir(directory):
    if filename.endswith(".jpg"):
        old_path = os.path.join(directory, filename)
        new_filename = prefix + filename
        new_path = os.path.join(directory, new_filename)
        os.rename(old_path, new_path)
        print(f"Renamed: {filename} -> {new_filename}")

Output:

Renamed: IMG_0001.jpg -> 20260329_IMG_0001.jpg
Renamed: IMG_0002.jpg -> 20260329_IMG_0002.jpg
Renamed: IMG_0003.jpg -> 20260329_IMG_0003.jpg

That script runs instantly and accomplishes what would take manual clicking for hours. This is the power of automation.

Energetic developer operating giant clockwork automation mechanism
Python automation: where boredom goes to die.

Why Automate with Python?

You might be wondering: why Python instead of shell scripts, scheduled tasks, or other tools? The answer is clarity, portability, and power. Here’s how they compare:

Task Aspect Manual Process Shell Script Python Script
Development Time Hours per occurrence 30-60 minutes 15-30 minutes
Readability N/A Cryptic syntax Human-readable code
Cross-Platform N/A Linux/Mac only Windows, Mac, Linux
Debugging N/A Difficult Easy with proper logging
Email Integration Manual setup Complex Built-in libraries
Maintainability N/A Hard to modify Easy to extend and modify

Python wins for most automation tasks because it balances simplicity with power. You can read Python code six months later and understand what it does, and you can add new features without rewriting everything.

Working with Files and Directories

Using os and pathlib Modules

Python provides two ways to work with file paths and directories: the older os module and the modern pathlib module. pathlib is more intuitive and handles cross-platform differences automatically, but os is still widely used. Let’s explore both:

# file_operations.py
import os
from pathlib import Path

# Using os module
print("Using os module:")
current_dir = os.getcwd()
print(f"Current directory: {current_dir}")

# List files
for item in os.listdir("."):
    if os.path.isfile(item):
        print(f"File: {item}")

# Using pathlib (modern approach)
print("\nUsing pathlib:")
current_path = Path(".")

for item in current_path.iterdir():
    if item.is_file():
        print(f"File: {item.name}")
        print(f"Size: {item.stat().st_size} bytes")
        print(f"Extension: {item.suffix}")

Output:

Using os module:
Current directory: /home/user/projects

Using pathlib:
File: script.py
Size: 1245 bytes
Extension: .py
File: data.csv
Size: 5678 bytes
Extension: .csv

pathlib.Path is generally preferred because it’s more readable and handles path separators automatically (backslash on Windows, forward slash on Unix). However, both work fine depending on your preference and existing codebase.

Renaming and Organizing Files

One of the most common automation tasks is organizing files by type, date, or naming convention. The shutil module and os.rename() make this simple:

# organize_files.py
import os
import shutil
from pathlib import Path

download_dir = "./downloads"

# Create subdirectories if they don't exist
for category in ["Images", "Documents", "Archives", "Other"]:
    Path(download_dir, category).mkdir(exist_ok=True)

# Organize files by extension
for filename in os.listdir(download_dir):
    if filename.startswith("."):
        continue

    filepath = os.path.join(download_dir, filename)

    if not os.path.isfile(filepath):
        continue

    # Determine category based on extension
    ext = os.path.splitext(filename)[1].lower()

    if ext in [".jpg", ".png", ".gif", ".webp"]:
        category = "Images"
    elif ext in [".pdf", ".doc", ".docx", ".txt"]:
        category = "Documents"
    elif ext in [".zip", ".rar", ".7z"]:
        category = "Archives"
    else:
        category = "Other"

    # Move file to appropriate directory
    dest_path = os.path.join(download_dir, category, filename)
    shutil.move(filepath, dest_path)
    print(f"Moved {filename} to {category}/")

Output:

Moved vacation.jpg to Images/
Moved resume.pdf to Documents/
Moved backup.zip to Archives/
Moved config.txt to Documents/

This script is the foundation of smart file organization. In a real system, you’d add error handling, logging, and checks to avoid overwriting files. The Path.mkdir(exist_ok=True) pattern ensures directories exist without throwing errors if they do.

Files flying and sorting themselves into organized colored bins
When your Downloads folder finally achieves organization.

Watching for File Changes with watchdog

Sometimes you need to react the moment a file appears or changes. The watchdog library monitors file system events in real-time. First, install it:

pip install watchdog

Now create a file watcher that triggers actions when new files appear:

# watch_folder.py
import time
from pathlib import Path
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class FileProcessor(FileSystemEventHandler):
    def on_created(self, event):
        if not event.is_directory:
            filename = Path(event.src_path).name
            print(f"New file detected: {filename}")
            print(f"Full path: {event.src_path}")

    def on_modified(self, event):
        if not event.is_directory:
            filename = Path(event.src_path).name
            print(f"File modified: {filename}")

# Watch the current directory
observer = Observer()
observer.schedule(FileProcessor(), path=".", recursive=False)
observer.start()

print("Watching for file changes. Press Ctrl+C to stop.")
try:
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    observer.stop()
    observer.join()

Output (after creating/modifying files):

Watching for file changes. Press Ctrl+C to stop.
New file detected: report.pdf
Full path: ./report.pdf
File modified: report.pdf

The watchdog library is perfect for implementing “drop a file to process it” workflows, such as converting documents, generating thumbnails, or triggering CI/CD pipelines.

Scheduling Tasks with the schedule Library

Many automation tasks need to run at specific times or intervals: daily backups, hourly data syncs, or weekly reports. The schedule library makes this elegant:

pip install schedule

Here’s how to create a task scheduler:

# task_scheduler.py
import schedule
import time
from datetime import datetime

def backup_database():
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    print(f"[{timestamp}] Running database backup...")
    # Actual backup logic here

def clean_temp_files():
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    print(f"[{timestamp}] Cleaning temporary files...")
    # Actual cleanup logic here

def generate_report():
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    print(f"[{timestamp}] Generating daily report...")
    # Actual report generation here

# Schedule tasks
schedule.every().day.at("02:00").do(backup_database)
schedule.every().hour.do(clean_temp_files)
schedule.every().monday.at("09:00").do(generate_report)

# Keep scheduler running
print("Scheduler started. Tasks will run according to schedule.")
while True:
    schedule.run_pending()
    time.sleep(60)  # Check every minute

Output (sample execution):

Scheduler started. Tasks will run according to schedule.
[2026-03-29 02:00:12] Running database backup...
[2026-03-29 03:00:05] Cleaning temporary files...
[2026-03-29 09:00:00] Generating daily report...

The schedule library is straightforward but doesn’t persist across system restarts. For production systems, consider using cron (Linux/Mac) or Task Scheduler (Windows) to run your Python script, or use a more robust library like APScheduler.

Sending Email Notifications with smtplib

Automating tasks is great, but you need to know when something fails or completes. Python’s built-in smtplib library sends email notifications:

# send_email.py
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

def send_notification(recipient, subject, body):
    sender_email = "automation@example.com"
    sender_password = "your_app_password_here"

    # Create message
    message = MIMEMultipart()
    message["From"] = sender_email
    message["To"] = recipient
    message["Subject"] = subject
    message.attach(MIMEText(body, "plain"))

    # Send email
    try:
        with smtplib.SMTP_SSL("smtp.gmail.com", 465) as server:
            server.login(sender_email, sender_password)
            server.send_message(message)
            print(f"Email sent to {recipient}")
    except Exception as e:
        print(f"Error sending email: {e}")

# Usage
send_notification(
    "admin@example.com",
    "Backup Complete",
    "Daily backup completed successfully at 2026-03-29 02:15:30."
)

Output:

Email sent to admin@example.com

Important: Never hardcode passwords in scripts. Use environment variables or a configuration file outside version control. For Gmail, generate an “App Password” in your account settings rather than using your actual password.

Working with CSV and Excel Files for Reports

Automated reporting is a huge time-saver. Python handles CSV files natively and can create Excel files with the openpyxl library:

# generate_report.py
import csv
from datetime import datetime
from pathlib import Path

# Sample data (from database or API in real scenario)
sales_data = [
    {"date": "2026-03-29", "product": "Widget A", "sales": 150},
    {"date": "2026-03-29", "product": "Widget B", "sales": 200},
    {"date": "2026-03-29", "product": "Widget C", "sales": 175},
]

# Generate CSV report
report_date = datetime.now().strftime("%Y%m%d")
report_filename = f"sales_report_{report_date}.csv"

with open(report_filename, "w", newline="") as csvfile:
    fieldnames = ["date", "product", "sales"]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)

    writer.writeheader()
    writer.writerows(sales_data)

print(f"Report generated: {report_filename}")

Output:

Report generated: sales_report_20260329.csv

File contents:

date,product,sales
2026-03-29,Widget A,150
2026-03-29,Widget B,200
2026-03-29,Widget C,175

For more complex reports with formatting, install openpyxl: pip install openpyxl. This lets you create Excel files with colors, formulas, and multiple sheets.

Running System Commands with subprocess

Sometimes you need to call external programs from Python. The subprocess module handles this safely:

# run_commands.py
import subprocess
import os

# Run a simple command
result = subprocess.run(["python", "--version"], capture_output=True, text=True)
print(f"Python version: {result.stdout.strip()}")

# Run a command and capture output
result = subprocess.run(["ls", "-la"], capture_output=True, text=True)
print("Directory listing:")
print(result.stdout)

# Check if command succeeded
result = subprocess.run(["git", "status"], capture_output=True)
if result.returncode == 0:
    print("Git repository is clean")
else:
    print("Not a git repository or git error")

Output (Linux/Mac):

Python version: Python 3.10.6
Directory listing:
total 48
drwxr-xr-x 5 user user 4096 Mar 29 10:15 .
drwxr-xr-x 8 user user 4096 Mar 29 09:00 ..
-rw-r--r-- 1 user user 1245 Mar 29 10:12 script.py
Git repository is clean

Use capture_output=True to collect program output and text=True to get strings instead of bytes. Always check the return code to verify success.

Developer on bridge connecting Python platform to system tools
Python calling system commands: the glue that holds automation together.

Real-Life Example: Automated Backup System

Now let’s build a complete, production-ready backup system that watches a directory and creates timestamped ZIP archives. This example combines everything we’ve learned:

# backup_system.py
import os
import shutil
import zipfile
import smtplib
import schedule
import time
from pathlib import Path
from datetime import datetime
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

class BackupManager:
    def __init__(self, source_dir, backup_dir, email_to):
        self.source_dir = source_dir
        self.backup_dir = backup_dir
        self.email_to = email_to
        Path(backup_dir).mkdir(exist_ok=True)

    def create_backup(self):
        timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
        backup_filename = f"backup_{timestamp}.zip"
        backup_path = os.path.join(self.backup_dir, backup_filename)

        try:
            with zipfile.ZipFile(backup_path, "w", zipfile.ZIP_DEFLATED) as zipf:
                for root, dirs, files in os.walk(self.source_dir):
                    for file in files:
                        file_path = os.path.join(root, file)
                        arcname = os.path.relpath(file_path, self.source_dir)
                        zipf.write(file_path, arcname)

            file_size = os.path.getsize(backup_path) / (1024 * 1024)
            print(f"Backup created: {backup_filename} ({file_size:.2f} MB)")

            self.send_notification(
                f"Backup Success",
                f"Backup created successfully: {backup_filename}\nSize: {file_size:.2f} MB"
            )

            # Cleanup old backups (keep last 7)
            self.cleanup_old_backups()

        except Exception as e:
            print(f"Backup failed: {e}")
            self.send_notification("Backup Failed", f"Error: {str(e)}")

    def cleanup_old_backups(self):
        backups = sorted(Path(self.backup_dir).glob("backup_*.zip"))
        if len(backups) > 7:
            for old_backup in backups[:-7]:
                old_backup.unlink()
                print(f"Deleted old backup: {old_backup.name}")

    def send_notification(self, subject, body):
        sender_email = "backup@example.com"
        sender_password = "your_app_password"

        try:
            message = MIMEMultipart()
            message["From"] = sender_email
            message["To"] = self.email_to
            message["Subject"] = subject
            message.attach(MIMEText(body, "plain"))

            with smtplib.SMTP_SSL("smtp.gmail.com", 465) as server:
                server.login(sender_email, sender_password)
                server.send_message(message)
        except Exception as e:
            print(f"Could not send email: {e}")

# Setup and run
if __name__ == "__main__":
    manager = BackupManager(
        source_dir="./important_files",
        backup_dir="./backups",
        email_to="admin@example.com"
    )

    # Schedule daily backups at 2 AM
    schedule.every().day.at("02:00").do(manager.create_backup)

    print("Backup system started. Waiting for scheduled time...")
    while True:
        schedule.run_pending()
        time.sleep(60)

Output (sample):

Backup system started. Waiting for scheduled time...
Backup created: backup_20260329_020015.zip (45.32 MB)
Deleted old backup: backup_20260322_020012.zip

This system handles the full lifecycle: creating backups, managing disk space, and notifying you of success or failure. In production, you’d run this as a background service using systemd (Linux), launchd (Mac), or Task Scheduler (Windows).

Frequently Asked Questions

How do I run a Python script in the background?

Linux/Mac: Use nohup to ignore hangup signals: nohup python backup_system.py &. Or use screen or tmux for interactive backgrounds. Better: use cron to schedule it properly.

Windows: Use Task Scheduler to run the script with python.exe. Create a task that runs at startup or on a schedule without showing a window.

Should I add error handling to automation scripts?

Absolutely. Always wrap file operations in try-except blocks. Log errors to a file so you can debug later. For critical tasks, send notifications on failure. Here’s a pattern:

try:
    # Your automation code
    do_something()
except Exception as e:
    logger.error(f"Task failed: {e}")
    send_alert_email(f"Error: {e}")

Is it safe to put passwords in automation scripts?

No. Use environment variables, config files outside version control, or credential managers. For email, use app-specific passwords instead of your real password. Never commit secrets to GitHub.

import os
password = os.getenv("EMAIL_PASSWORD")  # Load from environment

How do I write automation that works on Windows, Mac, and Linux?

Use pathlib.Path instead of string path concatenation–it handles separators automatically. Use subprocess carefully since some commands differ. Test on all platforms or use Docker for consistency.

What if the user’s system doesn’t have the libraries I need?

Create a requirements.txt file listing dependencies, then users can install them with pip install -r requirements.txt. For standalone scripts, use PyInstaller to bundle Python and libraries into a single executable.

Conclusion

Python automation transforms tedious manual tasks into reliable, repeatable processes. You’ve learned to work with files and directories using os and pathlib, schedule tasks with the schedule library, send email notifications via smtplib, and build complete systems like automated backups. The key is starting simple–automate your most painful task first, then gradually expand your automation toolkit.

For deeper learning, explore the official documentation: os module, pathlib, shutil, and smtplib are all built-in. For external libraries, check schedule and watchdog on PyPI. The automation possibilities are endless once you see Python as your personal robot assistant.

Python File Handling: Reading, Writing, and Manipulating Files

Python Modules and Packages: Organizing Your Code

Python Error Handling: try, except, and finally

Continue learning with these related guides:

How To Build a CLI Tool with Python and Typer

How To Build a CLI Tool with Python and Typer

Last Updated: June 01, 2026

Intermediate

Command-line interfaces (CLIs) are the backbone of modern development workflows. From package managers to deployment tools, every developer relies on well-designed CLI applications. If you’ve ever dreamed of building the next popular dev tool but found the existing CLI frameworks overwhelming, Python has an elegant solution: Typer. Typer combines the power of type hints with an intuitive API that makes building professional CLIs feel like writing regular Python functions.

The beauty of Typer lies in its simplicity wrapped in sophistication. Unlike older frameworks that require boilerplate configuration, Typer leverages Python’s type annotations to automatically generate help text, validate inputs, and handle command parsing. If you already know how to write Python functions, you already know how to write Typer CLI apps. No special decorators or configuration files needed.

This tutorial walks you through everything you need to build production-ready CLI tools. We’ll start with the fundamentals, explore advanced features like interactive prompts and colored output, and then build a complete file organizer application that demonstrates all the concepts in action. By the end, you’ll have a reusable template for any CLI project.

Excited developer in front of colorful terminal output
Typer transforms boring terminals into beautiful command-line experiences.
Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Quick Example: Your CLI in 10 Lines

Part of the Python CLI Tools Hub. See the full hub for related Python tutorials.

Part of the Modern Python AI Stack series. See the full tutorial hub for all 23 tutorials on LangGraph, MCP, Pydantic AI, Polars, FastAPI, Litestar, Typer, and more.

Before diving into the theory, let’s see Typer in action. Here’s the absolute minimum code needed to create a working CLI tool:

# hello_cli.py
import typer

app = typer.Typer()

@app.command()
def hello(name: str):
    typer.echo(f"Hello, {name}!")

if __name__ == "__main__":
    app()

Output:

$ python hello_cli.py --help
Usage: hello_cli.py [OPTIONS] COMMAND [ARGS]...

Commands:
  hello

$ python hello_cli.py hello Alice
Hello, Alice!

That’s it. No configuration, no argument parsing setup, no manual help text. Typer inferred everything from the function signature. The name parameter automatically became a command argument, and Typer generated professional help documentation instantly. This is the Typer philosophy: sensible defaults with maximum productivity.

What Is Typer and Why Use It

Typer is a modern Python library built on top of Click that simplifies CLI development. It’s created by the same developer who built FastAPI, and it brings FastAPI’s elegance to the command line. Rather than forcing you to learn a new syntax or remember decorator parameters, Typer uses standard Python type hints to express CLI intent.

To understand why Typer matters, let’s compare it with other popular CLI frameworks:

Feature argparse Click Typer
Verbosity High (10+ lines for simple CLI) Medium (5-7 lines) Low (3-5 lines)
Type Hints Support None Partial Full Native Support
Auto-generated Help Basic Good Excellent
Learning Curve Steep Moderate Shallow
Input Validation Manual Custom Types Type Hints
IDE Autocompletion Poor Good Excellent

Typer’s primary advantage is reducing cognitive load. You write Python functions that look like regular functions, and Typer handles the CLI machinery. Combined with modern IDE support, this means better code completion, fewer runtime surprises, and more time spent on your application logic instead of CLI plumbing.

Installing Typer

Getting started with Typer requires just one command. We’ll install the complete version with all optional dependencies to unlock advanced features like colored output:

# Install Typer with all extras
pip install "typer[all]"

What gets installed:

$ pip list | grep -i typer
typer               0.9.0
click               8.1.7
rich               13.7.0
shellingham        1.5.4

The [all] extra installs Rich (for colored output and tables), shellingham (for shell completion), and other utilities. If you want a minimal install with just the essentials, use pip install typer instead. Verify the installation works:

$ python -c "import typer; print(typer.__version__)"
0.9.0

Your First Typer Command

Now that Typer is installed, let’s build a slightly more complex application. Understanding command structure is crucial because every Typer app follows the same pattern: create an app object, decorate functions with @app.command(), and invoke it at the bottom.

# weather_cli.py
import typer

app = typer.Typer(help="Simple weather information tool")

@app.command()
def current(city: str):
    """Get current weather for a city."""
    typer.echo(f"Weather in {city}: Sunny, 72F")

@app.command()
def forecast(city: str, days: int = 7):
    """Get weather forecast for upcoming days."""
    typer.echo(f"Forecast for {city} ({days} days):")
    for i in range(1, days + 1):
        typer.echo(f"  Day {i}: Partly cloudy")

if __name__ == "__main__":
    app()

Output:

$ python weather_cli.py --help
Usage: weather_cli.py [OPTIONS] COMMAND [ARGS]...

  Simple weather information tool

Options:
  --help  Show this message and exit.

Commands:
  current   Get current weather for a city.
  forecast  Get weather forecast for upcoming days.

$ python weather_cli.py current London
Weather in London: Sunny, 72F

$ python weather_cli.py forecast Paris --days 3
Forecast for Paris (3 days):
  Day 1: Partly cloudy
  Day 2: Partly cloudy
  Day 3: Partly cloudy

Notice how Typer automatically converted the docstrings into help text, made days optional because it has a default value, and even inferred that it should be passed as --days flag. This is zero-configuration development.

Adding Arguments and Options

CLI parameters come in two flavors: arguments (positional, required) and options (named, optional with defaults). Understanding this distinction helps design intuitive CLIs. An argument like filename is positional and required. An option like --output is named and typically optional.

Typer infers the type from your function signature, but sometimes you need more control. Use typer.Argument() and typer.Option() to customize behavior:

# file_processor.py
import typer
from pathlib import Path

app = typer.Typer()

@app.command()
def process(
    input_file: Path = typer.Argument(..., help="File to process"),
    output_file: Path = typer.Option(None, help="Output file path"),
    verbose: bool = typer.Option(False, "-v", "--verbose", help="Verbose output"),
    count: int = typer.Option(1, "-c", "--count", help="Number of iterations")
):
    """Process a file with various options."""
    input_text = input_file.read_text()
    typer.echo(f"Read {len(input_text)} characters from {input_file}")

    if verbose:
        typer.echo(f"Verbose: Processing with count={count}")

    if output_file:
        output_file.write_text(input_text.upper())
        typer.echo(f"Wrote output to {output_file}")

if __name__ == "__main__":
    app()

Output:

$ python file_processor.py --help
Usage: file_processor.py [OPTIONS] INPUT_FILE

  Process a file with various options.

Options:
  --output-file PATH  Output file path
  -v, --verbose       Verbose output
  -c, --count INTEGER  Number of iterations
  --help              Show this message and exit.

Arguments:
  INPUT_FILE  File to process  [required]

$ echo "hello world" > input.txt
$ python file_processor.py input.txt --output-file output.txt -v --count 2
Read 11 characters from input.txt
Verbose: Processing with count=2
Wrote output to output.txt

The ... (Ellipsis) in typer.Argument(...) indicates a required argument. The typer.Option() call lets you specify default values, short flags (-v), and long flags (--verbose) simultaneously. Typer automatically converts hyphens to underscores in flag names, so --output-file maps to the output_file parameter.

Type Annotations for Validation

One of Typer’s superpowers is automatic validation through type hints. When you declare a parameter as int, Typer ensures the user provides an integer. If they don’t, Typer shows a helpful error message instead of crashing with a cryptic traceback.

# calculator.py
import typer

app = typer.Typer()

@app.command()
def add(a: int, b: int):
    """Add two integers."""
    typer.echo(f"{a} + {b} = {a + b}")

@app.command()
def greet(name: str, age: int = 25):
    """Greet someone with their age."""
    typer.echo(f"Hello, {name}! You are {age} years old.")

@app.command()
def enable_feature(feature_name: str, enabled: bool = True):
    """Toggle a feature on or off."""
    status = "enabled" if enabled else "disabled"
    typer.echo(f"Feature '{feature_name}' is {status}")

if __name__ == "__main__":
    app()

Output:

$ python calculator.py add 5 3
5 + 3 = 8

$ python calculator.py add five 3
Error: Invalid value for 'A': 'five' is not a valid integer.

$ python calculator.py greet Alice 30
Hello, Alice! You are 30 years old.

$ python calculator.py enable_feature logging --no-enabled
Feature 'logging' is disabled

Notice how passing a string where an integer is expected produces a clear error message. Typer validates at the CLI layer, not in your code. Boolean flags get special treatment: you can pass --enabled/--no-enabled or just toggle the default. This is powerful validation without writing a single if-statement for type checking.

Multiple Commands with app.command()

Professional CLI applications often have subcommands. Git is the classic example: git commit, git push, and git pull are all subcommands. Typer makes this structure effortless. Every function decorated with @app.command() becomes a subcommand automatically.

# database_cli.py
import typer

app = typer.Typer()

@app.command()
def migrate(version: str = typer.Option("latest")):
    """Run database migrations."""
    typer.echo(f"Migrating to version: {version}")

@app.command()
def backup(database: str = typer.Argument("main"), output: str = typer.Option("backup.sql")):
    """Create a database backup."""
    typer.echo(f"Backing up '{database}' to {output}")

@app.command()
def restore(backup_file: str = typer.Argument(...)):
    """Restore database from backup."""
    typer.echo(f"Restoring from {backup_file}")

@app.command()
def status():
    """Show database status."""
    typer.echo("Database Status: OK")
    typer.echo("Tables: 42")
    typer.echo("Size: 2.3 GB")

if __name__ == "__main__":
    app()

Output:

$ python database_cli.py --help
Usage: database_cli.py [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  backup    Create a database backup.
  migrate   Run database migrations.
  restore   Restore database from backup.
  status    Show database status.

$ python database_cli.py status
Database Status: OK
Tables: 42
Size: 2.3 GB

$ python database_cli.py backup mydb --output backup_2026.sql
Backing up 'mydb' to backup_2026.sql

This structure scales beautifully. As your application grows, you can organize commands in separate modules and import them, keeping the codebase maintainable.

Hierarchical command tree structure with connected nodes
Subcommands scale from simple to enterprise-grade tools without refactoring.

Interactive Prompts and Confirmations

Sometimes you need to ask the user for input during execution, not at the command line. Typer provides interactive prompts for this scenario. Use typer.prompt() for collecting input and typer.confirm() for yes/no questions.

# interactive_app.py
import typer

app = typer.Typer()

@app.command()
def create_user():
    """Interactively create a new user."""
    username = typer.prompt("Enter username")
    email = typer.prompt("Enter email")
    password = typer.prompt("Enter password", hide_input=True)

    typer.echo(f"User '{username}' created successfully")

@app.command()
def delete_file(filename: str):
    """Delete a file with confirmation."""
    if typer.confirm(f"Delete '{filename}'?"):
        typer.echo(f"Deleted {filename}")
    else:
        typer.echo("Cancelled")

@app.command()
def setup():
    """Run interactive setup wizard."""
    project_name = typer.prompt("Project name")
    author = typer.prompt("Author name")
    use_git = typer.confirm("Initialize Git repository?")

    typer.echo(f"Setting up project '{project_name}'...")
    if use_git:
        typer.echo("Initialized Git repository")
    typer.echo(f"Project ready! ({author})")

if __name__ == "__main__":
    app()

Output:

$ python interactive_app.py create_user
Enter username: alice
Enter email: alice@example.com
Enter password:
User 'alice' created successfully

$ python interactive_app.py delete_file data.csv
Delete 'data.csv'? [y/N]: y
Deleted data.csv

$ python interactive_app.py setup
Project name: MyApp
Author name: Bob Smith
Initialize Git repository? [y/N]: y
Setting up project 'MyApp'...
Initialized Git repository
Project ready! (Bob Smith)

The hide_input=True parameter masks password input, preventing shoulder surfers from seeing sensitive data. typer.confirm() accepts yes/no responses flexibly, handling “y”, “yes”, “n”, “no” and returning a boolean. This creates seamless user experiences without managing stdin directly.

Rich Output with Colors

Boring terminals are outdated. The Rich library (included with Typer) enables beautiful colored output, tables, and formatted text. This transforms CLIs from utilitarian to delightful.

# styled_output.py
import typer
from rich.console import Console
from rich.table import Table
from rich import print as rprint

app = typer.Typer()
console = Console()

@app.command()
def colors():
    """Display colored text."""
    rprint("[bold red]Error:[/bold red] Something went wrong!")
    rprint("[green]Success:[/green] Operation completed")
    rprint("[cyan]Info:[/cyan] Current status is normal")

@app.command()
def show_status():
    """Display status with a formatted table."""
    table = Table(title="System Status")
    table.add_column("Component", style="cyan")
    table.add_column("Status", style="magenta")
    table.add_column("Load", style="green")

    table.add_row("CPU", "OK", "45%")
    table.add_row("Memory", "OK", "62%")
    table.add_row("Disk", "Warning", "88%")

    console.print(table)

@app.command()
def progress_demo():
    """Show progress with real-time updates."""
    with console.status("[bold green]Processing...") as status:
        import time
        for i in range(5):
            time.sleep(0.2)
            status.update(f"[bold green]Processing step {i+1}/5...")
    console.print("[bold green]Done!")

if __name__ == "__main__":
    app()

Output:

$ python styled_output.py colors
Error: Something went wrong!
Success: Operation completed
Info: Current status is normal

$ python styled_output.py show_status
System Status
Component    Status      Load
CPU          OK          45%
Memory       OK          62%
Disk         Warning     88%

$ python styled_output.py progress_demo
Processing step 5/5...
Done!

Rich markup is simple: [bold red]text[/bold red] applies bold red styling. You can create tables, progress bars, panels, and more. This visual feedback keeps users informed and engaged, especially important for long-running operations.

Error Handling in CLI Apps

Proper error handling separates production-ready CLIs from toy scripts. Typer provides typer.Exit() to terminate with a specific exit code, and the rich.console.Console class has methods for displaying errors elegantly.

# error_handling.py
import typer
from pathlib import Path
from rich.console import Console

app = typer.Typer()
console = Console()

@app.command()
def read_file(filename: str):
    """Read and display a file."""
    try:
        file_path = Path(filename)
        if not file_path.exists():
            console.print(f"[red]Error:[/red] File '{filename}' not found", style="bold")
            raise typer.Exit(code=1)

        content = file_path.read_text()
        console.print(f"[green]Success:[/green] Read {len(content)} characters")
        print(content)
    except Exception as e:
        console.print(f"[red]Error:[/red] {str(e)}", style="bold")
        raise typer.Exit(code=2)

@app.command()
def process(input_file: str, output_file: str = "output.txt"):
    """Process a file with error checking."""
    input_path = Path(input_file)
    output_path = Path(output_file)

    if not input_path.exists():
        console.print(f"[red]Input Error:[/red] {input_file} not found", style="bold red")
        raise typer.Exit(code=1)

    if output_path.exists():
        if not typer.confirm(f"Overwrite {output_file}?"):
            console.print("[yellow]Cancelled[/yellow]")
            raise typer.Exit(code=0)

    try:
        data = input_path.read_text()
        output_path.write_text(data.upper())
        console.print(f"[green]Success:[/green] Processed {input_file} -> {output_file}")
    except IOError as e:
        console.print(f"[red]IO Error:[/red] {str(e)}", style="bold red")
        raise typer.Exit(code=3)

if __name__ == "__main__":
    app()

Output:

$ python error_handling.py read_file missing.txt
Error: File 'missing.txt' not found

$ echo $?
1

$ python error_handling.py read_file data.txt
Success: Read 42 characters
[file contents here]

$ python error_handling.py process input.txt --output-file output.txt
Success: Processed input.txt -> output.txt

Exit codes matter for automation and scripting. Return 0 for success, non-zero for failure. This allows shell scripts and other tools to detect success and fail fast. Always validate user input early and fail fast with clear messages.

Comparing dark broken terminal with bright colorful terminal
Rich transforms cryptic errors into developer-friendly feedback.

Real-Life Example: Smart File Organizer

Now let’s build a complete, production-ready CLI tool: a smart file organizer that sorts files in a directory by their extension. This example combines everything we’ve learned: multiple commands, type validation, interactive prompts, error handling, and rich output.

# file_organizer.py
import typer
from pathlib import Path
from rich.console import Console
from rich.table import Table
from collections import defaultdict
import shutil

app = typer.Typer(help="Smart file organizer with validation and safety checks")
console = Console()

@app.command()
def organize(
    directory: Path = typer.Argument(".", help="Directory to organize"),
    dry_run: bool = typer.Option(True, help="Preview changes without executing"),
    create_folders: bool = typer.Option(True, help="Create extension folders"),
):
    """Organize files in a directory by extension."""
    if not directory.exists():
        console.print(f"[red]Error:[/red] Directory '{directory}' not found", style="bold")
        raise typer.Exit(code=1)

    if not directory.is_dir():
        console.print(f"[red]Error:[/red] '{directory}' is not a directory", style="bold")
        raise typer.Exit(code=1)

    # Group files by extension
    files_by_ext = defaultdict(list)
    for file in directory.iterdir():
        if file.is_file():
            ext = file.suffix or "[no-extension]"
            files_by_ext[ext].append(file)

    if not files_by_ext:
        console.print("[yellow]No files found to organize[/yellow]")
        raise typer.Exit(code=0)

    # Display summary
    table = Table(title="Files to Organize")
    table.add_column("Extension", style="cyan")
    table.add_column("Count", style="green")

    for ext, files in sorted(files_by_ext.items()):
        table.add_row(ext, str(len(files)))

    console.print(table)

    if dry_run:
        console.print("[yellow]Dry-run mode: No changes will be made[/yellow]")
        return

    if not typer.confirm("Execute organization?"):
        console.print("[yellow]Cancelled[/yellow]")
        raise typer.Exit(code=0)

    # Move files
    moved_count = 0
    for ext, files in files_by_ext.items():
        if create_folders and ext != "[no-extension]":
            folder = directory / ext.lstrip(".")
            folder.mkdir(exist_ok=True)

            for file in files:
                try:
                    shutil.move(str(file), str(folder / file.name))
                    moved_count += 1
                except Exception as e:
                    console.print(f"[red]Failed to move {file.name}:[/red] {str(e)}")

    console.print(f"[green]Success:[/green] Moved {moved_count} files")

@app.command()
def analyze(directory: Path = typer.Argument(".", help="Directory to analyze")):
    """Analyze file distribution in a directory."""
    if not directory.exists() or not directory.is_dir():
        console.print(f"[red]Error:[/red] Invalid directory '{directory}'", style="bold")
        raise typer.Exit(code=1)

    total_size = 0
    files_by_ext = defaultdict(int)

    for file in directory.rglob("*"):
        if file.is_file():
            ext = file.suffix or "[no-extension]"
            files_by_ext[ext] += 1
            total_size += file.stat().st_size

    table = Table(title=f"Analysis of {directory}")
    table.add_column("Extension", style="cyan")
    table.add_column("Files", style="green")

    for ext in sorted(files_by_ext.keys()):
        table.add_row(ext, str(files_by_ext[ext]))

    console.print(table)
    console.print(f"Total files: {sum(files_by_ext.values())}")
    console.print(f"Total size: {total_size / (1024*1024):.2f} MB")

if __name__ == "__main__":
    app()

Output:

$ python file_organizer.py organize ./test_dir
Files to Organize
Extension    Count
.txt         5
.pdf         3
.jpg         7
.py          2
Dry-run mode: No changes will be made

$ python file_organizer.py organize ./test_dir --no-dry-run
Files to Organize
Extension    Count
.txt         5
.pdf         3
.jpg         7
.py          2
Execute organization? [y/N]: y
Success: Moved 17 files

$ python file_organizer.py analyze ./test_dir
Analysis of ./test_dir
Extension    Files
.jpg         7
.pdf         3
.py          2
.txt         5
Total files: 17
Total size: 45.32 MB

This file organizer demonstrates production best practices: it validates input, provides dry-run mode for safety, uses tables for clarity, handles errors gracefully, and offers multiple commands for different use cases. You could package this as a standalone tool and distribute it via pip.

Frequently Asked Questions

How do I package a Typer app as a standalone tool?

Use setuptools or Poetry to create a package with an entry point. In your pyproject.toml, add:

[project.scripts]
my-cli = "my_module:app"

Then install with pip install -e .. Your app becomes available as a system command: my-cli --help.

Can Typer generate shell completion scripts?

Yes! Typer apps automatically support bash, zsh, and fish completion through the shellingham library. Users can run python my_cli.py --install-completion to set up completions for their shell.

How should I test Typer applications?

Use the CliRunner from Click (which Typer uses under the hood). Testing example:

from typer.testing import CliRunner

runner = CliRunner()
result = runner.invoke(app, ["add", "5", "3"])
assert result.exit_code == 0
assert "8" in result.output

What about complex types like lists or JSON objects?

Use Python’s built-in types directly. Typer handles List[str], List[int], and other generic types intelligently. For JSON, accept a string and parse it with json.loads() in your function.

How do I use environment variables in a Typer app?

Use typer.Option() with the envvar parameter: api_key: str = typer.Option(..., envvar="API_KEY"). Typer will check the environment variable if the CLI argument isn’t provided.

Can I create command groups or nested subcommands?

Yes! Create separate Typer instances and add them as command groups:

db_app = typer.Typer()
@db_app.command()
def migrate(): pass

app = typer.Typer()
app.add_typer(db_app, name="db")

# Usage: python cli.py db migrate

Conclusion

Typer brings modern Python practices to CLI development. By leveraging type hints and sensible defaults, it eliminates boilerplate while maintaining power and flexibility. Whether you’re building internal tools, developer utilities, or the next popular open-source CLI, Typer gives you a solid foundation.

The journey from simple functions to professional CLI applications is smooth with Typer. Start with a basic command, add features incrementally, and scale to complex multi-command applications without refactoring. For deeper learning, explore the official Typer documentation and examine real-world projects using Typer on GitHub.

Your CLI adventure awaits. Happy building!

Deepen your command-line expertise with these related tutorials:

How To Use SQLAlchemy 2.0 ORM with Python

How To Use SQLAlchemy 2.0 ORM with Python

Last Updated: June 01, 2026

Intermediate

SQLAlchemy is the gold standard for Object-Relational Mapping in Python. Version 2.0 represents a major evolution, introducing a more intuitive API that emphasizes explicit, modern patterns while maintaining backward compatibility. Whether you’re building a small Flask application or a complex data management system, SQLAlchemy 2.0 provides the tools to interact with databases using Python objects instead of raw SQL strings.

The ORM (Object-Relational Mapping) layer in SQLAlchemy 2.0 allows you to define database tables as Python classes, called models. Once you define a model, you can perform all database operations–creating records, querying data, updating rows, and deleting entries–using Pythonic syntax. The new select() construct and DeclarativeBase provide clearer, more expressive patterns than earlier versions.

In this tutorial, we’ll explore the key features of SQLAlchemy 2.0 ORM: how to define models, manage database sessions, perform CRUD operations, query data with the new select() API, establish relationships between tables, handle transactions, and build a real-world example. By the end, you’ll understand how to leverage SQLAlchemy 2.0 to create robust, maintainable database-driven applications.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Quick Example: 20 Lines of SQLAlchemy 2.0

Let’s start with a complete, working example to see SQLAlchemy 2.0 in action:

# quick_example.py
from sqlalchemy import create_engine, String
from sqlalchemy.orm import DeclarativeBase, Session, Mapped, mapped_column

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = 'users'
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column(String(50))

engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)

with Session(engine) as session:
    session.add(User(name='Alice'))
    session.add(User(name='Bob'))
    session.commit()

with Session(engine) as session:
    from sqlalchemy import select
    users = session.scalars(select(User)).all()
    for user in users:
        print(f'{user.id}: {user.name}')

Output:

1: Alice
2: Bob

This example demonstrates the core workflow: define a model inheriting from DeclarativeBase, create an engine, manage a session, insert records, and query them using select(). Notice the type hints (Mapped[str]) and the modern syntax–this is SQLAlchemy 2.0 style.

What Is SQLAlchemy ORM?

SQLAlchemy provides multiple ways to interact with databases. The ORM layer sits at the highest abstraction level, letting you work with Python objects. Here’s how it compares to alternatives:

Approach How It Works Pros Cons
Raw SQL Write SQL strings directly in Python Maximum control, direct database access Error-prone, requires manual parameter binding, not Pythonic
SQLAlchemy Core Use SQL expression language to build queries programmatically Type-safe, database-agnostic, composable Still working with table/column constructs, not Python objects
SQLAlchemy ORM Map database tables to Python classes, query objects directly Pythonic, intuitive, supports relationships and complex queries, automatic change tracking Slightly more overhead, must understand session lifecycle

SQLAlchemy 2.0’s ORM is the most productive choice for most applications because it combines clarity with power. You define your data model once, and the ORM handles the translation to SQL behind the scenes.

Installing SQLAlchemy and Setting Up

Install SQLAlchemy using pip:

# shell
pip install sqlalchemy

Output:

Successfully installed sqlalchemy-2.0.x

Verify the installation:

# check_version.py
import sqlalchemy
print(f'SQLAlchemy version: {sqlalchemy.__version__}')

Output:

SQLAlchemy version: 2.0.x

For this tutorial, we’ll use SQLite in-memory databases (specified as sqlite:///:memory:), which requires no external setup. For production use with PostgreSQL, MySQL, or other databases, install the appropriate driver (e.g., pip install psycopg2-binary for PostgreSQL).

Defining Models with DeclarativeBase

In SQLAlchemy 2.0, you define models by creating a class that inherits from DeclarativeBase. This base class automatically handles the mapping between your Python class and the database table.

Creating the DeclarativeBase

# models_setup.py
from sqlalchemy.orm import DeclarativeBase

class Base(DeclarativeBase):
    pass

Output:

(No output - this defines the base class)

The Base class is the foundation for all your models. It tracks metadata (table definitions) and provides utilities for creating tables.

Defining a Model with Columns

Use Mapped and mapped_column() to define model attributes in SQLAlchemy 2.0:

# product_model.py
from sqlalchemy import String, Float, Integer
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column

class Base(DeclarativeBase):
    pass

class Product(Base):
    __tablename__ = 'products'

    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column(String(100), nullable=False)
    price: Mapped[float] = mapped_column(Float, default=0.0)
    stock: Mapped[int] = mapped_column(Integer, default=0)

Output:

(No output - this defines the model structure)

Key points about this model:

  • __tablename__ specifies the database table name
  • Mapped[type] is a type hint that declares the Python type of the column
  • mapped_column() specifies database-level constraints (primary key, nullability, defaults)
  • primary_key=True makes id the primary key with auto-increment behavior
  • nullable=False ensures the name field cannot be NULL
  • default=0.0 provides a default value for new records

Common Column Types

# column_types_example.py
from sqlalchemy import String, Integer, Float, Boolean, DateTime, Text, Date
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from datetime import datetime, date

class Base(DeclarativeBase):
    pass

class Article(Base):
    __tablename__ = 'articles'

    id: Mapped[int] = mapped_column(primary_key=True)
    title: Mapped[str] = mapped_column(String(255))
    content: Mapped[str] = mapped_column(Text)
    is_published: Mapped[bool] = mapped_column(Boolean, default=False)
    rating: Mapped[float] = mapped_column(Float)
    views: Mapped[int] = mapped_column(Integer, default=0)
    created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow)
    published_date: Mapped[date] = mapped_column(Date, nullable=True)

Output:

(No output - demonstrates various column types)
Character drawing blueprints of connected blocks representing SQLAlchemy ORM models
SQLAlchemy ORM — Python objects in, SQL magic out.

Creating Tables and the Engine

The SQLAlchemy engine is your gateway to the database. It manages connections and executes SQL. To create tables, you call metadata.create_all():

# create_tables.py
from sqlalchemy import create_engine
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
from sqlalchemy import String

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = 'users'
    id: Mapped[int] = mapped_column(primary_key=True)
    email: Mapped[str] = mapped_column(String(100), unique=True)

# Create an in-memory SQLite database
engine = create_engine('sqlite:///:memory:', echo=False)

# Create all tables defined in metadata
Base.metadata.create_all(engine)

print('Tables created successfully!')

Output:

Tables created successfully!

The connection string format is dialect+driver://user:password@host:port/database. Examples:

  • sqlite:///:memory: – In-memory SQLite (perfect for testing)
  • sqlite:///app.db – File-based SQLite
  • postgresql://user:pass@localhost/dbname – PostgreSQL
  • mysql+pymysql://user:pass@localhost/dbname – MySQL

Sessions and Basic CRUD Operations

A Session is a context manager that tracks changes to your objects and coordinates with the database. CRUD stands for Create, Read, Update, Delete–the fundamental database operations.

Create (Insert) Records

# create_records.py
from sqlalchemy import create_engine, String
from sqlalchemy.orm import DeclarativeBase, Session, Mapped, mapped_column

class Base(DeclarativeBase):
    pass

class Book(Base):
    __tablename__ = 'books'
    id: Mapped[int] = mapped_column(primary_key=True)
    title: Mapped[str] = mapped_column(String(100))
    author: Mapped[str] = mapped_column(String(100))

engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)

# Create and insert records
with Session(engine) as session:
    book1 = Book(title='Python Basics', author='Alice Johnson')
    book2 = Book(title='Web Dev with Django', author='Bob Smith')

    session.add(book1)
    session.add(book2)
    session.commit()

    print(f'Created book with ID: {book1.id}')
    print(f'Created book with ID: {book2.id}')

Output:

Created book with ID: 1
Created book with ID: 2

When you commit, SQLAlchemy assigns primary keys (IDs) to new objects. The session tracks the objects and only issues SQL when you call commit().

Read (Query) Records

# read_records.py
from sqlalchemy import create_engine, String, select
from sqlalchemy.orm import DeclarativeBase, Session, Mapped, mapped_column

class Base(DeclarativeBase):
    pass

class Book(Base):
    __tablename__ = 'books'
    id: Mapped[int] = mapped_column(primary_key=True)
    title: Mapped[str] = mapped_column(String(100))
    author: Mapped[str] = mapped_column(String(100))

engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)

with Session(engine) as session:
    session.add(Book(title='Python Basics', author='Alice Johnson'))
    session.add(Book(title='Web Dev with Django', author='Bob Smith'))
    session.commit()

# Read records
with Session(engine) as session:
    stmt = select(Book)
    books = session.scalars(stmt).all()

    for book in books:
        print(f'{book.id}: {book.title} by {book.author}')

Output:

1: Python Basics by Alice Johnson
2: Web Dev with Django by Bob Smith

Update Records

# update_records.py
from sqlalchemy import create_engine, String, select
from sqlalchemy.orm import DeclarativeBase, Session, Mapped, mapped_column

class Base(DeclarativeBase):
    pass

class Book(Base):
    __tablename__ = 'books'
    id: Mapped[int] = mapped_column(primary_key=True)
    title: Mapped[str] = mapped_column(String(100))
    author: Mapped[str] = mapped_column(String(100))

engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)

with Session(engine) as session:
    session.add(Book(title='Python Basics', author='Alice Johnson'))
    session.commit()

# Update a record
with Session(engine) as session:
    stmt = select(Book).where(Book.title == 'Python Basics')
    book = session.scalars(stmt).first()

    if book:
        book.author = 'Alice J. Johnson'
        session.commit()
        print(f'Updated: {book.title} by {book.author}')

Output:

Updated: Python Basics by Alice J. Johnson

Delete Records

# delete_records.py
from sqlalchemy import create_engine, String, select
from sqlalchemy.orm import DeclarativeBase, Session, Mapped, mapped_column

class Base(DeclarativeBase):
    pass

class Book(Base):
    __tablename__ = 'books'
    id: Mapped[int] = mapped_column(primary_key=True)
    title: Mapped[str] = mapped_column(String(100))

engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)

with Session(engine) as session:
    session.add(Book(title='Old Book'))
    session.commit()

# Delete a record
with Session(engine) as session:
    stmt = select(Book).where(Book.title == 'Old Book')
    book = session.scalars(stmt).first()

    if book:
        session.delete(book)
        session.commit()
        print('Book deleted successfully')

Output:

Book deleted successfully
Character placing building blocks on grid representing CRUD operations
Create, read, update, delete — the four verbs every ORM speaks fluently.

Querying with select()

SQLAlchemy 2.0’s select() construct is the modern way to build queries. It’s more expressive than the legacy query() method and provides better IDE support through type hints.

Basic Selects

# basic_select.py
from sqlalchemy import create_engine, String, select
from sqlalchemy.orm import DeclarativeBase, Session, Mapped, mapped_column

class Base(DeclarativeBase):
    pass

class Student(Base):
    __tablename__ = 'students'
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column(String(100))
    grade: Mapped[int]

engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)

with Session(engine) as session:
    session.add(Student(name='Alice', grade=95))
    session.add(Student(name='Bob', grade=87))
    session.add(Student(name='Charlie', grade=92))
    session.commit()

# Select all
with Session(engine) as session:
    stmt = select(Student)
    all_students = session.scalars(stmt).all()
    print(f'Total students: {len(all_students)}')

    # Select first
    first = session.scalars(select(Student)).first()
    print(f'First student: {first.name}')

Output:

Total students: 3
First student: Alice

Filtering Results

# filtering.py
from sqlalchemy import create_engine, String, select
from sqlalchemy.orm import DeclarativeBase, Session, Mapped, mapped_column

class Base(DeclarativeBase):
    pass

class Student(Base):
    __tablename__ = 'students'
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column(String(100))
    grade: Mapped[int]

engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)

with Session(engine) as session:
    session.add_all([
        Student(name='Alice', grade=95),
        Student(name='Bob', grade=87),
        Student(name='Charlie', grade=92),
        Student(name='Diana', grade=88)
    ])
    session.commit()

with Session(engine) as session:
    # Equal comparison
    stmt = select(Student).where(Student.name == 'Alice')
    result = session.scalars(stmt).first()
    print(f'Found: {result.name} (Grade: {result.grade})')

    # Greater than
    stmt = select(Student).where(Student.grade > 90)
    high_performers = session.scalars(stmt).all()
    print(f'High performers: {[s.name for s in high_performers]}')

    # Like pattern
    stmt = select(Student).where(Student.name.like('D%'))
    result = session.scalars(stmt).first()
    print(f'Names starting with D: {result.name}')

Output:

Found: Alice (Grade: 95)
High performers: ['Alice', 'Charlie']
Names starting with D: Diana

Ordering and Limiting

# ordering_limiting.py
from sqlalchemy import create_engine, String, select, desc
from sqlalchemy.orm import DeclarativeBase, Session, Mapped, mapped_column

class Base(DeclarativeBase):
    pass

class Student(Base):
    __tablename__ = 'students'
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column(String(100))
    grade: Mapped[int]

engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)

with Session(engine) as session:
    session.add_all([
        Student(name='Alice', grade=95),
        Student(name='Bob', grade=87),
        Student(name='Charlie', grade=92),
        Student(name='Diana', grade=88)
    ])
    session.commit()

with Session(engine) as session:
    # Order ascending
    stmt = select(Student).order_by(Student.grade)
    lowest = session.scalars(stmt).first()
    print(f'Lowest grade: {lowest.name} ({lowest.grade})')

    # Order descending
    stmt = select(Student).order_by(desc(Student.grade))
    highest = session.scalars(stmt).first()
    print(f'Highest grade: {highest.name} ({highest.grade})')

    # Limit
    stmt = select(Student).order_by(desc(Student.grade)).limit(2)
    top_two = session.scalars(stmt).all()
    print(f'Top 2 students: {[s.name for s in top_two]}')

Output:

Lowest grade: Bob (87)
Highest grade: Alice (95)
Top 2 students: ['Alice', 'Charlie']

Joins Between Tables

# joins.py
from sqlalchemy import create_engine, String, select, ForeignKey
from sqlalchemy.orm import DeclarativeBase, Session, Mapped, mapped_column, relationship

class Base(DeclarativeBase):
    pass

class Department(Base):
    __tablename__ = 'departments'
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column(String(100))
    employees: Mapped[list['Employee']] = relationship(back_populates='department')

class Employee(Base):
    __tablename__ = 'employees'
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column(String(100))
    department_id: Mapped[int] = mapped_column(ForeignKey('departments.id'))
    department: Mapped[Department] = relationship(back_populates='employees')

engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)

with Session(engine) as session:
    dept_eng = Department(name='Engineering')
    dept_hr = Department(name='HR')

    session.add_all([
        Employee(name='Alice', department=dept_eng),
        Employee(name='Bob', department=dept_eng),
        Employee(name='Charlie', department=dept_hr)
    ])
    session.commit()

with Session(engine) as session:
    # Join departments and employees
    stmt = select(Employee).join(Department).where(Department.name == 'Engineering')
    eng_employees = session.scalars(stmt).all()
    print(f'Engineering employees: {[e.name for e in eng_employees]}')

Output:

Engineering employees: ['Alice', 'Bob']

Relationships Between Models

Relationships let you traverse from one model to another. SQLAlchemy handles the foreign key constraints and makes it easy to load related objects.

One-to-Many Relationships

# one_to_many.py
from sqlalchemy import create_engine, String, ForeignKey
from sqlalchemy.orm import DeclarativeBase, Session, Mapped, mapped_column, relationship

class Base(DeclarativeBase):
    pass

class Author(Base):
    __tablename__ = 'authors'
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column(String(100))
    books: Mapped[list['Book']] = relationship(back_populates='author', cascade='all, delete-orphan')

class Book(Base):
    __tablename__ = 'books'
    id: Mapped[int] = mapped_column(primary_key=True)
    title: Mapped[str] = mapped_column(String(100))
    author_id: Mapped[int] = mapped_column(ForeignKey('authors.id'))
    author: Mapped[Author] = relationship(back_populates='books')

engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)

with Session(engine) as session:
    author = Author(name='George Orwell')
    author.books = [
        Book(title='1984'),
        Book(title='Animal Farm')
    ]
    session.add(author)
    session.commit()

with Session(engine) as session:
    from sqlalchemy import select
    author = session.scalars(select(Author).where(Author.name == 'George Orwell')).first()
    print(f'Author: {author.name}')
    for book in author.books:
        print(f'  - {book.title}')

Output:

Author: George Orwell
  - 1984
  - Animal Farm

Many-to-Many Relationships

# many_to_many.py
from sqlalchemy import create_engine, String, ForeignKey, Table, Column
from sqlalchemy.orm import DeclarativeBase, Session, Mapped, mapped_column, relationship

class Base(DeclarativeBase):
    pass

# Association table for many-to-many
student_course = Table(
    'student_course',
    Base.metadata,
    Column('student_id', ForeignKey('students.id'), primary_key=True),
    Column('course_id', ForeignKey('courses.id'), primary_key=True)
)

class Student(Base):
    __tablename__ = 'students'
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column(String(100))
    courses: Mapped[list['Course']] = relationship(secondary=student_course, back_populates='students')

class Course(Base):
    __tablename__ = 'courses'
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column(String(100))
    students: Mapped[list[Student]] = relationship(secondary=student_course, back_populates='courses')

engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)

with Session(engine) as session:
    python = Course(name='Python 101')
    math = Course(name='Calculus I')

    alice = Student(name='Alice', courses=[python, math])
    bob = Student(name='Bob', courses=[python])

    session.add_all([alice, bob])
    session.commit()

with Session(engine) as session:
    from sqlalchemy import select
    student = session.scalars(select(Student).where(Student.name == 'Alice')).first()
    print(f'Alice is taking: {[c.name for c in student.courses]}')

Output:

Alice is taking: ['Python 101', 'Calculus I']
Character connecting blocks with chains representing database relationships
Foreign keys in Python land — relationship() does the joining for you.

Transactions and Error Handling

A transaction is a sequence of database operations that either all succeed or all fail. SQLAlchemy sessions handle transactions automatically, but you can control commit/rollback behavior explicitly.

Basic Commit and Rollback

# transactions.py
from sqlalchemy import create_engine, String
from sqlalchemy.orm import DeclarativeBase, Session, Mapped, mapped_column

class Base(DeclarativeBase):
    pass

class Account(Base):
    __tablename__ = 'accounts'
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column(String(100))
    balance: Mapped[float]

engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)

# Create initial accounts
with Session(engine) as session:
    session.add_all([
        Account(name='Alice', balance=1000.0),
        Account(name='Bob', balance=500.0)
    ])
    session.commit()

# Simulate a transfer with error handling
with Session(engine) as session:
    try:
        alice = session.query(Account).filter_by(name='Alice').first()
        bob = session.query(Account).filter_by(name='Bob').first()

        # Transfer 200 from Alice to Bob
        alice.balance -= 200
        bob.balance += 200

        session.commit()
        print(f'Transfer successful: Alice={alice.balance}, Bob={bob.balance}')
    except Exception as e:
        session.rollback()
        print(f'Transfer failed: {e}')

Output:

Transfer successful: Alice=800.0, Bob=700.0

Error Handling with Try-Except

# error_handling.py
from sqlalchemy import create_engine, String, exc
from sqlalchemy.orm import DeclarativeBase, Session, Mapped, mapped_column

class Base(DeclarativeBase):
    pass

class User(Base):
    __tablename__ = 'users'
    id: Mapped[int] = mapped_column(primary_key=True)
    email: Mapped[str] = mapped_column(String(100), unique=True)

engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)

with Session(engine) as session:
    session.add(User(email='alice@example.com'))
    session.commit()

# Try to add duplicate email
with Session(engine) as session:
    try:
        session.add(User(email='alice@example.com'))
        session.commit()
    except exc.IntegrityError as e:
        session.rollback()
        print('Error: Email already exists')
    except Exception as e:
        session.rollback()
        print(f'Unexpected error: {e}')

Output:

Error: Email already exists
Character at vault with combination dial representing database transactions
Transactions — commit when ready, rollback when not. No half measures.

Real-Life Example: Blog Database

Let’s build a complete blog system with Post, Author, and Tag models, demonstrating relationships, CRUD operations, and queries.

# blog_system.py
from sqlalchemy import create_engine, String, Text, ForeignKey, Table, Column, select, desc
from sqlalchemy.orm import DeclarativeBase, Session, Mapped, mapped_column, relationship
from datetime import datetime

class Base(DeclarativeBase):
    pass

# Association table for many-to-many relationship
post_tag = Table(
    'post_tag',
    Base.metadata,
    Column('post_id', ForeignKey('posts.id'), primary_key=True),
    Column('tag_id', ForeignKey('tags.id'), primary_key=True)
)

class Author(Base):
    __tablename__ = 'authors'
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column(String(100), nullable=False)
    email: Mapped[str] = mapped_column(String(100), unique=True)
    posts: Mapped[list['Post']] = relationship(back_populates='author', cascade='all, delete-orphan')

class Post(Base):
    __tablename__ = 'posts'
    id: Mapped[int] = mapped_column(primary_key=True)
    title: Mapped[str] = mapped_column(String(200), nullable=False)
    content: Mapped[str] = mapped_column(Text)
    created_at: Mapped[datetime] = mapped_column(default=datetime.utcnow)
    author_id: Mapped[int] = mapped_column(ForeignKey('authors.id'))
    author: Mapped[Author] = relationship(back_populates='posts')
    tags: Mapped[list['Tag']] = relationship(secondary=post_tag, back_populates='posts')

class Tag(Base):
    __tablename__ = 'tags'
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str] = mapped_column(String(50), unique=True)
    posts: Mapped[list[Post]] = relationship(secondary=post_tag, back_populates='tags')

# Setup
engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)

# Create blog data
with Session(engine) as session:
    author1 = Author(name='Alice', email='alice@blog.com')
    author2 = Author(name='Bob', email='bob@blog.com')

    python_tag = Tag(name='Python')
    web_tag = Tag(name='Web')

    post1 = Post(
        title='Getting Started with Python',
        content='Python is a great language...',
        author=author1,
        tags=[python_tag, web_tag]
    )
    post2 = Post(
        title='Advanced ORM Techniques',
        content='SQLAlchemy provides powerful ORM features...',
        author=author1,
        tags=[python_tag]
    )
    post3 = Post(
        title='Web Development Tips',
        content='Here are some web development best practices...',
        author=author2,
        tags=[web_tag]
    )

    session.add_all([author1, author2, python_tag, web_tag, post1, post2, post3])
    session.commit()

# Query examples
with Session(engine) as session:
    # Find all posts by an author
    stmt = select(Post).join(Author).where(Author.name == 'Alice').order_by(desc(Post.created_at))
    alice_posts = session.scalars(stmt).all()
    print(f'Posts by Alice: {len(alice_posts)}')
    for post in alice_posts:
        print(f'  - {post.title}')

    # Find all posts with a specific tag
    stmt = select(Post).join(Post.tags).where(Tag.name == 'Python')
    python_posts = session.scalars(stmt).all()
    print(f'\nPython posts: {len(python_posts)}')
    for post in python_posts:
        print(f'  - {post.title} by {post.author.name}')

    # Count total posts
    stmt = select(Post)
    total_posts = len(session.scalars(stmt).all())
    print(f'\nTotal blog posts: {total_posts}')

Output:

Posts by Alice: 2
  - Advanced ORM Techniques
  - Getting Started with Python

Python posts: 2
  - Getting Started with Python by Alice
  - Advanced ORM Techniques by Alice

Total blog posts: 3

This example showcases the full power of SQLAlchemy 2.0 ORM: defining multiple related models, using many-to-many relationships, performing complex queries with joins, and maintaining referential integrity through cascading deletes.

Frequently Asked Questions

What’s the difference between Mapped and traditional type hints?

Mapped[type] is SQLAlchemy 2.0’s way to combine Python type hints with ORM metadata. It tells SQLAlchemy about the column while also providing type information to your IDE and type checkers. The legacy approach used no type hints.

When should I use relationships versus manual joins?

Use relationships when you want to access related objects as Python attributes (e.g., author.posts). Use manual joins when you need more control over the query or want to fetch only specific columns. Relationships are more Pythonic and handle lazy loading by default.

What’s the difference between add() and add_all()?

session.add(obj) adds a single object. session.add_all([obj1, obj2]) adds multiple objects at once. Use add_all() for convenience when inserting several objects.

How do I handle database connection pooling?

SQLAlchemy’s engine manages connection pooling automatically. For production applications, configure pool settings when creating the engine: engine = create_engine('postgresql://...', pool_size=10, max_overflow=20).

Can I use SQLAlchemy ORM with async code?

Yes! SQLAlchemy 2.0 includes async support using AsyncSession and create_async_engine(). This is useful for high-concurrency web applications. However, the basic patterns remain the same.

What happens if I forget to commit()?

Changes are held in the session but not persisted to the database. When the session context exits (the with block ends), uncommitted changes are rolled back. Always call commit() to save changes.

How do I avoid the N+1 query problem?

The N+1 problem happens when loading a parent object triggers a separate query for each child. Use eager loading with selectinload() or joinedload() to fetch related objects in one query: select(Author).options(selectinload(Author.posts)).

Conclusion

SQLAlchemy 2.0 brings modern Python patterns to database programming. By using DeclarativeBase for model definition, select() for queries, and proper session management, you can build robust data-driven applications without writing a single SQL string. The ORM layer abstracts away database details while remaining transparent and powerful.

Key takeaways from this tutorial:

  • Models inherit from DeclarativeBase and use Mapped type hints
  • select() is the modern way to build type-safe queries
  • Sessions manage transactions and object tracking
  • Relationships make it natural to traverse related objects
  • Always handle errors and rollback on failure
  • Eager loading prevents common performance pitfalls

For next steps, explore SQLAlchemy’s advanced features like hybrid properties, custom types, and query optimizations. Consider integrating SQLAlchemy with frameworks like Flask or FastAPI for web development. As you grow more comfortable with the ORM, you’ll find that SQLAlchemy’s power and flexibility make it an excellent choice for any Python project requiring database interaction.

Now you have a complete, production-ready reference for SQLAlchemy 2.0 ORM. Use this guide to build, query, and maintain your database layer with confidence.