How To Use Python Itertools for Efficient Looping

How To Use Python Itertools for Efficient Looping

Last Updated: June 01, 2026

Intermediate

Python’s itertools module is a powerhouse for creating efficient iterators that save memory and speed up your data processing. Instead of building entire lists in memory, itertools generates values on the fly, making it perfect for working with large datasets or creating complex iteration patterns.

If you have ever needed to combine multiple lists, group data by a key, or generate all possible permutations of a set, itertools has a clean, optimized solution ready to go. These tools are implemented in C under the hood, so they run significantly faster than equivalent pure Python code.

In this tutorial, you will learn the most practical itertools functions through real examples. We will cover infinite iterators, combinatoric generators, data grouping, chaining, and filtering — everything you need to write more Pythonic and memory-efficient code.

Quick Answer

The itertools module provides fast, memory-efficient iterator building blocks. Key functions include chain() for combining iterables, groupby() for grouping data, product() for cartesian products, combinations() and permutations() for combinatorics, and islice() for slicing iterators. Import with from itertools import chain, groupby, product, combinations.

Pubs - Python How To Program

Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Quick Example

from itertools import chain, islice, count

# Chain multiple iterables together seamlessly
combined = chain([1, 2, 3], ['a', 'b'], [True, False])
print(list(combined))

# Take the first 5 even numbers from an infinite counter
evens = (x for x in count(0, 2))
first_five = list(islice(evens, 5))
print(first_five)
[1, 2, 3, ‘a’, ‘b’, True, False]
[0, 2, 4, 6, 8]

The chain() function combines three separate iterables into one seamless stream without creating a new list in memory. The islice() function safely takes a slice from an infinite iterator, something you cannot do with regular list slicing.

What Is the Itertools Module?

The itertools module is part of Python’s standard library and provides a collection of fast, memory-efficient tools for creating and working with iterators. The module is inspired by constructs from functional programming languages like APL, Haskell, and SML.

The key advantage of itertools is lazy evaluation. Instead of building a complete list in memory, each function produces values one at a time as they are requested. This means you can work with datasets much larger than your available RAM, process infinite sequences, and build complex data pipelines that remain efficient.

The functions in itertools fall into three categories: infinite iterators that produce values forever, finite iterators that process input sequences, and combinatoric generators that produce arrangements of elements. All of them are implemented in C for maximum performance.

itertools.chain, cycle, repeat. The infinite generators you didn't know you need
itertools.chain, cycle, repeat. The infinite generators you didn’t know you needed.

Infinite Iterators

count — Infinite Counter

The count() function generates an endless sequence of numbers starting from a given value with a specified step. You must always use it with something that limits the output, like islice() or a break in a loop.

from itertools import count, islice

# Count from 10 with step 5
counter = count(10, 5)
print(list(islice(counter, 6)))

# Useful for generating IDs
def id_generator(prefix="ID"):
    for num in count(1):
        yield f"{prefix}-{num:04d}"

ids = id_generator("ORD")
print([next(ids) for _ in range(4)])
[10, 15, 20, 25, 30, 35]
[‘ORD-0001’, ‘ORD-0002’, ‘ORD-0003’, ‘ORD-0004’]

cycle — Repeat a Sequence Forever

The cycle() function takes an iterable and repeats it infinitely. This is perfect for round-robin scheduling, alternating patterns, or rotating through a fixed set of options.

from itertools import cycle, islice

# Alternate between teams for task assignment
teams = cycle(["Alpha", "Beta", "Gamma"])
tasks = ["Deploy v2.1", "Fix login bug", "Update docs",
         "Database migration", "API refactor", "Write tests", "Code review"]

assignments = {task: next(teams) for task in tasks}
for task, team in assignments.items():
    print(f"  {team}: {task}")
Alpha: Deploy v2.1
Beta: Fix login bug
Gamma: Update docs
Alpha: Database migration
Beta: API refactor
Gamma: Write tests
Alpha: Code review

repeat — Repeat a Value

The repeat() function produces the same value over and over, either infinitely or a specified number of times. It is commonly used with map() or zip() to provide a constant value alongside changing data.

from itertools import repeat

# Create a list of default configurations
defaults = list(repeat({"enabled": True, "retries": 3}, 4))
print(defaults)

# Use with map for element-wise operations
import operator
bases = [2, 3, 4, 5]
squared = list(map(operator.pow, bases, repeat(2)))
print(f"Squared: {squared}")
cubed = list(map(operator.pow, bases, repeat(3)))
print(f"Cubed: {cubed}")
[{‘enabled’: True, ‘retries’: 3}, {‘enabled’: True, ‘retries’: 3}, {‘enabled’: True, ‘retries’: 3}, {‘enabled’: True, ‘retries’: 3}]
Squared: [4, 9, 16, 25]
Cubed: [8, 27, 64, 125]

Finite Iterators

chain — Combine Multiple Iterables

The chain() function links multiple iterables together into a single continuous stream. It processes the first iterable completely, then moves to the second, and so on — all without creating an intermediate list.

from itertools import chain

# Merge data from multiple sources
database_users = ["alice", "bob"]
api_users = ["charlie", "diana"]
file_users = ["eve"]

all_users = list(chain(database_users, api_users, file_users))
print(f"All users: {all_users}")

# chain.from_iterable flattens a list of lists
nested_data = [["python", "java"], ["rust", "go"], ["ruby"]]
flat = list(chain.from_iterable(nested_data))
print(f"Flattened: {flat}")
All users: [‘alice’, ‘bob’, ‘charlie’, ‘diana’, ‘eve’]
Flattened: [‘python’, ‘java’, ‘rust’, ‘go’, ‘ruby’]

groupby — Group Consecutive Elements

The groupby() function groups consecutive elements that share the same key. The data must be sorted by the grouping key first, or you will get unexpected results.

from itertools import groupby
from operator import itemgetter

# Group transactions by category
transactions = [
    {"category": "food", "amount": 25.50},
    {"category": "food", "amount": 12.00},
    {"category": "transport", "amount": 35.00},
    {"category": "transport", "amount": 15.50},
    {"category": "entertainment", "amount": 45.00},
    {"category": "food", "amount": 8.75},
]

# Sort first, then group
sorted_trans = sorted(transactions, key=itemgetter("category"))
for category, group in groupby(sorted_trans, key=itemgetter("category")):
    items = list(group)
    total = sum(t["amount"] for t in items)
    print(f"  {category}: {len(items)} transactions, ${total:.2f}")
entertainment: 1 transactions, $45.00
food: 3 transactions, $46.25
transport: 2 transactions, $50.50
Important: The data must be sorted by the same key you pass to groupby. If unsorted, groupby will create a new group every time the key changes, resulting in multiple groups for the same key value.

islice — Slice Any Iterator

The islice() function works like regular list slicing but on any iterator, including infinite ones and generators. Unlike list slicing, it does not support negative indices because iterators cannot go backwards.

from itertools import islice

# Slice a generator (can't use regular slicing)
def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

# Get fibonacci numbers 10 through 15
fib_slice = list(islice(fibonacci(), 10, 16))
print(f"Fibonacci 10-15: {fib_slice}")

# Read first 3 lines from a large dataset (simulated)
data_lines = (f"Row {i}: data_{i}" for i in range(1000000))
preview = list(islice(data_lines, 3))
print(f"Preview: {preview}")
Fibonacci 10-15: [55, 89, 144, 233, 377, 610]
Preview: [‘Row 0: data_0’, ‘Row 1: data_1’, ‘Row 2: data_2’]

Lazy iteration. Memory stays constant, time stays linear.
Lazy iteration. Memory stays constant, time stays linear.

Combinatoric Iterators

product — Cartesian Product

The product() function computes the cartesian product of input iterables, equivalent to nested for loops. This is perfect for generating all combinations of options.

from itertools import product

# Generate all t-shirt variants
sizes = ["S", "M", "L"]
colors = ["Red", "Blue"]
styles = ["V-neck", "Crew"]

variants = list(product(sizes, colors, styles))
print(f"Total variants: {len(variants)}")
for v in variants[:6]:
    print(f"  {v[0]} {v[1]} {v[2]}")

# Generate grid coordinates
grid = list(product(range(3), range(3)))
print(f"\n3x3 grid: {grid}")
Total variants: 12
S Red V-neck
S Red Crew
S Blue V-neck
S Blue Crew
M Red V-neck
M Red Crew

3×3 grid: [(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2, 2)]

combinations and permutations

The combinations() function generates all unique groups of a given size where order does not matter. The permutations() function generates all arrangements where order does matter.

from itertools import combinations, permutations

# Pick 2-person teams from 4 candidates
candidates = ["Alice", "Bob", "Charlie", "Diana"]

teams = list(combinations(candidates, 2))
print(f"Possible teams ({len(teams)}):")
for team in teams:
    print(f"  {team[0]} & {team[1]}")

# Order matters for race finish positions
runners = ["A", "B", "C"]
finishes = list(permutations(runners))
print(f"\nPossible race finishes ({len(finishes)}):")
for f in finishes:
    print(f"  1st: {f[0]}, 2nd: {f[1]}, 3rd: {f[2]}")
Possible teams (6):
Alice & Bob
Alice & Charlie
Alice & Diana
Bob & Charlie
Bob & Diana
Charlie & Diana

Possible race finishes (6):
1st: A, 2nd: B, 3rd: C
1st: A, 2nd: C, 3rd: B
1st: B, 2nd: A, 3rd: C
1st: B, 2nd: C, 3rd: A
1st: C, 2nd: A, 3rd: B
1st: C, 2nd: B, 3rd: A

Real-Life Project: Building a Data Pipeline with Itertools

Let us build a practical data processing pipeline that uses multiple itertools functions to efficiently process log data. This pipeline chains, filters, groups, and summarizes data while keeping memory usage minimal.

from itertools import chain, groupby, islice, accumulate
from operator import itemgetter
from collections import Counter
import operator

# Simulated log data from multiple servers
server1_logs = [
    {"timestamp": "2024-01-15 10:01", "level": "INFO", "service": "auth", "message": "User login"},
    {"timestamp": "2024-01-15 10:02", "level": "ERROR", "service": "auth", "message": "Invalid token"},
    {"timestamp": "2024-01-15 10:05", "level": "INFO", "service": "api", "message": "Request processed"},
    {"timestamp": "2024-01-15 10:07", "level": "WARNING", "service": "db", "message": "Slow query"},
]

server2_logs = [
    {"timestamp": "2024-01-15 10:01", "level": "INFO", "service": "api", "message": "Health check"},
    {"timestamp": "2024-01-15 10:03", "level": "ERROR", "service": "api", "message": "Timeout"},
    {"timestamp": "2024-01-15 10:04", "level": "ERROR", "service": "db", "message": "Connection lost"},
    {"timestamp": "2024-01-15 10:06", "level": "INFO", "service": "auth", "message": "Token refreshed"},
]

# Step 1: Chain all logs together
all_logs = list(chain(server1_logs, server2_logs))
print(f"Total log entries: {len(all_logs)}")

# Step 2: Sort and group by service
sorted_by_service = sorted(all_logs, key=itemgetter("service"))
print("\nLogs by service:")
for service, logs in groupby(sorted_by_service, key=itemgetter("service")):
    log_list = list(logs)
    error_count = sum(1 for l in log_list if l["level"] == "ERROR")
    print(f"  {service}: {len(log_list)} entries ({error_count} errors)")

# Step 3: Sort and group by log level
sorted_by_level = sorted(all_logs, key=itemgetter("level"))
print("\nLogs by level:")
for level, logs in groupby(sorted_by_level, key=itemgetter("level")):
    count = len(list(logs))
    print(f"  {level}: {count}")

# Step 4: Running error count using accumulate
error_flags = [1 if log["level"] == "ERROR" else 0 for log in all_logs]
running_errors = list(accumulate(error_flags, operator.add))
print(f"\nRunning error count: {running_errors}")
print(f"Total errors: {running_errors[-1]}")

# Step 5: Get the latest 3 entries
latest = list(islice(sorted(all_logs, key=itemgetter("timestamp"), reverse=True), 3))
print("\nLatest 3 entries:")
for entry in latest:
    print(f"  [{entry['level']}] {entry['timestamp']} - {entry['service']}: {entry['message']}")
Total log entries: 8

Logs by service:
api: 3 entries (1 errors)
auth: 3 entries (1 errors)
db: 2 entries (1 errors)

Logs by level:
ERROR: 3
INFO: 4
WARNING: 1

Running error count: [0, 1, 1, 1, 1, 2, 3, 1]
Total errors: 3

Latest 3 entries:
[WARNING] 2024-01-15 10:07 – db: Slow query
[INFO] 2024-01-15 10:06 – auth: Token refreshed
[INFO] 2024-01-15 10:05 – api: Request processed

groupby: structure your iterator by neighboring keys.
groupby: structure your iterator by neighboring keys.

Common Pitfalls and Troubleshooting

Problem Cause Solution
groupby returns unexpected groups Data not sorted by grouping key Always sort by the same key before calling groupby
Iterator exhausted after first use Iterators can only be consumed once Convert to list if you need multiple passes, or use tee()
Memory error with product() Cartesian product of large sets creates huge output Use islice() to limit output, or process items one at a time
Infinite loop with count() or cycle() No termination condition Always pair infinite iterators with islice(), takewhile(), or break
accumulate gives wrong type Initial value type mismatch Pass an explicit initial value matching your expected type
combinations_with_replacement unexpected Confused with regular combinations Use combinations() for no repeats, combinations_with_replacement() when repeats are allowed

Frequently Asked Questions

What is the difference between itertools.chain and list concatenation?

chain() creates a lazy iterator that processes elements one at a time without creating a new list in memory. List concatenation with + creates an entirely new list containing all elements from both sources. For large datasets, chain is significantly more memory-efficient because it never holds more than one element at a time.

Can I use itertools with pandas DataFrames?

Yes, but with some caveats. Itertools functions work with any iterable, so you can use them on DataFrame columns, rows from iterrows(), or index values. However, pandas has its own optimized methods like groupby, merge, and concat that are usually faster for DataFrame operations. Use itertools when working with pure Python iterables or when pandas does not have an equivalent function.

How do I restart an exhausted iterator?

You cannot restart a consumed iterator. Instead, use itertools.tee() to create independent copies before consuming, store the data in a list if it fits in memory, or recreate the iterator from the original source. For generators, call the generator function again to get a fresh iterator.

When should I use product() versus nested for loops?

product() is cleaner and more readable than nested loops when you need all combinations of multiple iterables. It also makes it easy to dynamically change the number of nested dimensions. Use nested for loops when you need complex logic between iterations, early exits, or when only some combinations should be processed.

Is itertools faster than list comprehensions?

For simple operations, list comprehensions and itertools have similar speed. The main advantage of itertools is memory efficiency rather than raw speed. When processing millions of items, itertools avoids building large intermediate lists, which prevents memory issues and can actually be faster due to reduced memory allocation overhead. For small datasets, the difference is negligible.

Conclusion

The itertools module transforms how you handle iteration in Python. You learned how infinite iterators like count() and cycle() create endless streams, how chain() and groupby() organize data efficiently, how combinatoric tools like product() and combinations() generate arrangements, and how islice() safely limits output from any iterator.

The data pipeline example showed how these tools compose naturally to build efficient processing chains. Start by replacing list concatenation with chain() and nested loops with product() in your existing code — those two changes will immediately make your code more readable and memory-efficient. As you get comfortable, explore groupby() and accumulate() for more advanced data processing patterns.

Related Articles

Frequently Asked Questions

When should I reach for itertools instead of writing a loop?

When you’re chaining, grouping, taking, dropping, or otherwise transforming an iterable. itertools.chain flattens iterables of iterables; islice gives you Python’s equivalent of array[2:10] for any iterator; groupby clusters consecutive equal elements; tee duplicates one iterator into N. The wins are memory (no intermediate lists) and clarity (the itertools name says what’s happening).

What’s the difference between zip() and itertools.zip_longest()?

zip() stops at the shortest input. zip_longest() runs until the longest is exhausted, padding the missing values with fillvalue (default None). Use zip_longest when you’re aligning lists of slightly different lengths and don’t want silent truncation. Use zip() when length mismatch is a bug you want to surface.

Are itertools functions faster than equivalent generator expressions?

Yes, usually by 1.5-3x — itertools is implemented in C while comprehensions and generator expressions are Python bytecode. For loops over millions of items the difference becomes noticeable. For loops over hundreds the readability of the comprehension matters more than the speed.

How do I get a sliding window of N items from an iterable?

Use itertools.pairwise() for N=2 (Python 3.10+). For arbitrary N, the recipe is: from itertools import islice, tee — windows = zip(*(islice(t, i, None) for i, t in enumerate(tee(iterable, N)))). Polars and pandas have native rolling-window operations that are faster if you’re already in dataframe land.

Can itertools handle infinite sequences safely?

Yes — that’s one of its strengths. count(start, step) generates an infinite arithmetic progression, cycle(iterable) loops forever, repeat(value, times) repeats. Combine with takewhile or islice to bound them. The danger is calling list() on an infinite iterator — Python will happily exhaust your RAM trying to materialise it.

How To Use Python Functools for Higher-Order Functions

How To Use Python Functools for Higher-Order Functions

Last Updated: June 01, 2026

Intermediate

Python’s functools module is a treasure chest of higher-order functions that transform how you write and compose functions. Whether you need to cache expensive computations, create partial function applications, or build powerful decorators, functools provides elegant solutions that make your code cleaner and faster.

If you have ever written the same wrapper logic around multiple functions, or wished you could “freeze” some arguments into a function call, functools is exactly what you need. It sits in the standard library, so there is nothing extra to install — just import and go.

In this tutorial, you will master the most practical functools tools with hands-on examples you can use in real projects today. We will cover caching, partial application, function composition, comparison helpers, and more.

Quick Answer

The functools module provides higher-order functions for working with callable objects. The most commonly used tools are @lru_cache for memoization, partial() for freezing function arguments, reduce() for cumulative operations, and @wraps for building proper decorators. Import with from functools import lru_cache, partial, reduce, wraps.

Pubs - Python How To Program

Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Quick Example

from functools import lru_cache

@lru_cache(maxsize=128)
def fibonacci(n):
    if n < 2:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

print(fibonacci(50))
print(fibonacci.cache_info())
12586269025
CacheInfo(hits=48, misses=51, maxsize=128, currsize=51)

Without the cache, computing fibonacci(50) would take an impossibly long time due to exponential recursive calls. With @lru_cache, it returns instantly because each unique input is computed only once.

What Is the Functools Module?

The functools module is part of Python's standard library and provides functions that act on or return other functions. The name stands for "function tools," and every utility in the module helps you work with callables more effectively.

Higher-order functions are functions that take other functions as arguments or return functions as results. This is a core concept in functional programming, and functools brings these ideas into Python in a practical, Pythonic way. You do not need to adopt a fully functional style — you can sprinkle these tools into your existing object-oriented or procedural code wherever they help.

The module has been part of Python since version 2.5 and has gained powerful additions over the years. Python 3.8 added cached_property, and Python 3.9 improved cache as a simpler alias for unbounded LRU caching.

@<a href=lru_cache: when the same call returns the same result, remember it." style="max-width:100%;height:auto;border-radius:8px;" />
@lru_cache: when the same call returns the same result, remember it.

Core Functools Tools

lru_cache — Automatic Memoization

The @lru_cache decorator caches function results based on the arguments passed. LRU stands for "Least Recently Used," meaning when the cache reaches its maximum size, the oldest unused entries get evicted first.

from functools import lru_cache
import time

@lru_cache(maxsize=256)
def expensive_lookup(user_id):
    # Simulate a slow database query
    time.sleep(0.5)
    return {"id": user_id, "name": f"User_{user_id}", "active": True}

# First call takes 0.5 seconds
start = time.time()
result = expensive_lookup(42)
print(f"First call: {time.time() - start:.3f}s -> {result}")

# Second call is instant (cached)
start = time.time()
result = expensive_lookup(42)
print(f"Cached call: {time.time() - start:.6f}s -> {result}")

# Check cache statistics
print(expensive_lookup.cache_info())

# Clear the cache when needed
expensive_lookup.cache_clear()
First call: 0.501s -> {'id': 42, 'name': 'User_42', 'active': True}
Cached call: 0.000002s -> {'id': 42, 'name': 'User_42', 'active': True}
CacheInfo(hits=1, misses=1, maxsize=256, currsize=1)
Important: All arguments to a cached function must be hashable (strings, numbers, tuples). You cannot cache functions that accept lists or dictionaries as arguments. Convert them to tuples or frozensets first.

cache — Unbounded Memoization

Python 3.9 introduced @cache as a simpler alternative to @lru_cache(maxsize=None). It caches every unique call forever, which is perfect when you know the set of possible inputs is bounded.

from functools import cache

@cache
def factorial(n):
    if n <= 1:
        return 1
    return n * factorial(n - 1)

print(factorial(10))
print(factorial(20))  # Reuses cached results for 1-10
print(factorial.cache_info())
3628800
2432902008176640000
CacheInfo(hits=10, misses=20, maxsize=None, currsize=20)

partial — Freeze Function Arguments

The partial() function creates a new callable with some arguments pre-filled. This is incredibly useful when you need to pass a function somewhere that expects fewer arguments than your function takes.

from functools import partial

def power(base, exponent):
    return base ** exponent

# Create specialized functions
square = partial(power, exponent=2)
cube = partial(power, exponent=3)

print(square(5))
print(cube(4))

# Practical example: configuring a logger
def log_message(level, component, message):
    print(f"[{level}] {component}: {message}")

# Create component-specific loggers
auth_log = partial(log_message, component="AUTH")
db_log = partial(log_message, component="DATABASE")

auth_log("INFO", message="User logged in")
auth_log("WARNING", message="Failed login attempt")
db_log("ERROR", message="Connection timeout")
25
64
[INFO] AUTH: User logged in
[WARNING] AUTH: Failed login attempt
[ERROR] DATABASE: Connection timeout

reduce — Cumulative Operations

The reduce() function applies a two-argument function cumulatively to the items of a sequence, reducing it to a single value. It processes items left to right, carrying the result forward at each step.

from functools import reduce

# Sum of all numbers (same as built-in sum)
numbers = [1, 2, 3, 4, 5]
total = reduce(lambda a, b: a + b, numbers)
print(f"Sum: {total}")

# Find maximum value (same as built-in max)
largest = reduce(lambda a, b: a if a > b else b, numbers)
print(f"Max: {largest}")

# Flatten nested lists
nested = [[1, 2], [3, 4], [5, 6]]
flat = reduce(lambda a, b: a + b, nested)
print(f"Flattened: {flat}")

# Build a dictionary from pairs
pairs = [("name", "Alice"), ("age", 30), ("city", "Melbourne")]
result = reduce(lambda d, pair: {**d, pair[0]: pair[1]}, pairs, {})
print(f"Dict: {result}")
Sum: 15
Max: 5
Flattened: [1, 2, 3, 4, 5, 6]
Dict: {'name': 'Alice', 'age': 30, 'city': 'Melbourne'}
Tip: While reduce is powerful, Python's built-in functions like sum(), max(), min(), and any() are more readable for common cases. Use reduce when you need a custom accumulation pattern that does not have a built-in equivalent.

wraps — Build Proper Decorators

When you write a decorator, the wrapper function replaces the original function's metadata (name, docstring, signature). The @wraps decorator preserves this metadata, which is essential for debugging and documentation tools.

from functools import wraps
import time

def timing_decorator(func):
    @wraps(func)  # Preserves func's __name__, __doc__, etc.
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        elapsed = time.perf_counter() - start
        print(f"{func.__name__} took {elapsed:.4f}s")
        return result
    return wrapper

@timing_decorator
def process_data(items):
    """Process a list of items and return the total."""
    return sum(items)

# The function metadata is preserved
print(f"Name: {process_data.__name__}")
print(f"Doc: {process_data.__doc__}")
print(f"Result: {process_data(range(1000000))}")
Name: process_data
Doc: Process a list of items and return the total.
process_data took 0.0234s
Result: 499999500000

cached_property — One-Time Computed Attributes

The @cached_property decorator turns a method into a property that is computed once and then cached as a normal attribute. This is perfect for expensive calculations that do not change after the object is created.

from functools import cached_property
import statistics

class DataAnalysis:
    def __init__(self, data):
        self._data = list(data)

    @cached_property
    def mean(self):
        print("Computing mean...")
        return statistics.mean(self._data)

    @cached_property
    def std_dev(self):
        print("Computing standard deviation...")
        return statistics.stdev(self._data)

    @cached_property
    def summary(self):
        print("Building summary...")
        return {
            "count": len(self._data),
            "mean": self.mean,
            "std_dev": self.std_dev,
            "min": min(self._data),
            "max": max(self._data)
        }

analysis = DataAnalysis(range(1, 10001))
print(analysis.mean)      # Computes and caches
print(analysis.mean)      # Returns cached value (no "Computing..." message)
print(analysis.summary)   # Triggers mean cache hit, computes std_dev
Computing mean...
5000.5
5000.5
Building summary...
Computing standard deviation...
{'count': 10000, 'mean': 5000.5, 'std_dev': 2886.896, 'min': 1, 'max': 10000}

total_ordering — Complete Comparison Methods

The @total_ordering class decorator lets you define just __eq__ and one ordering method (__lt__, __le__, __gt__, or __ge__), and it fills in the rest automatically.

from functools import total_ordering

@total_ordering
class Version:
    def __init__(self, major, minor, patch):
        self.major = major
        self.minor = minor
        self.patch = patch

    def __eq__(self, other):
        return (self.major, self.minor, self.patch) == \
               (other.major, other.minor, other.patch)

    def __lt__(self, other):
        return (self.major, self.minor, self.patch) < \
               (other.major, other.minor, other.patch)

    def __repr__(self):
        return f"Version({self.major}.{self.minor}.{self.patch})"

versions = [Version(2, 1, 0), Version(1, 9, 5), Version(2, 0, 1), Version(1, 9, 5)]
print(sorted(versions))
print(Version(2, 0, 0) >= Version(1, 9, 9))
print(Version(1, 0, 0) <= Version(1, 0, 0))
[Version(1.9.5), Version(1.9.5), Version(2.0.1), Version(2.1.0)]
True
True

Real-Life Project: Building a Plugin System with Functools

Let us build a practical plugin system for a web application that uses several functools features together. This system registers handler functions, caches their results, and supports partial configuration.

from functools import wraps, partial, lru_cache, reduce
from collections import defaultdict
import time
import json

class PluginRegistry:
    """A plugin system using functools for caching and composition."""

    def __init__(self):
        self._plugins = defaultdict(list)
        self._middleware = []

    def register(self, event_type, priority=0):
        """Decorator to register a handler for an event type."""
        def decorator(func):
            @wraps(func)
            def wrapper(*args, **kwargs):
                return func(*args, **kwargs)
            wrapper._priority = priority
            self._plugins[event_type].append(wrapper)
            self._plugins[event_type].sort(
                key=lambda f: f._priority, reverse=True
            )
            return wrapper
        return decorator

    def add_middleware(self, middleware_func):
        """Add a middleware function that processes all events."""
        self._middleware.append(middleware_func)

    @lru_cache(maxsize=64)
    def get_handlers(self, event_type):
        """Get cached tuple of handlers for an event type."""
        return tuple(self._plugins.get(event_type, []))

    def emit(self, event_type, data):
        """Emit an event through middleware then to handlers."""
        # Apply middleware chain using reduce
        processed = reduce(
            lambda d, mw: mw(event_type, d),
            self._middleware,
            data
        )

        handlers = self._plugins.get(event_type, [])
        results = []
        for handler in handlers:
            result = handler(processed)
            if result is not None:
                results.append(result)
        return results


# Create registry and register plugins
registry = PluginRegistry()

# Middleware: add timestamp to all events
def timestamp_middleware(event_type, data):
    return {**data, "timestamp": time.time()}

# Middleware: log all events
def logging_middleware(event_type, data):
    print(f"  [LOG] Event '{event_type}' with keys: {list(data.keys())}")
    return data

registry.add_middleware(timestamp_middleware)
registry.add_middleware(logging_middleware)

@registry.register("user.login", priority=10)
def validate_login(data):
    """Check if the user credentials are valid."""
    if data.get("username") and data.get("password"):
        return {"status": "validated", "user": data["username"]}
    return {"status": "invalid"}

@registry.register("user.login", priority=5)
def record_login(data):
    """Record the login attempt for analytics."""
    return {"recorded": True, "user": data.get("username")}

@registry.register("data.transform", priority=10)
def normalize_data(data):
    """Normalize string fields to lowercase."""
    return {
        k: v.lower() if isinstance(v, str) else v
        for k, v in data.items()
    }

# Using partial to create pre-configured event emitters
emit_login = partial(registry.emit, "user.login")
emit_transform = partial(registry.emit, "data.transform")

# Emit events
print("Login event:")
results = emit_login({"username": "alice", "password": "secret123"})
for r in results:
    print(f"  Result: {r}")

print("\nTransform event:")
results = emit_transform({"Name": "BOB", "City": "MELBOURNE", "age": 25})
for r in results:
    print(f"  Result: {r}")
Login event:
[LOG] Event 'user.login' with keys: ['username', 'password', 'timestamp']
Result: {'status': 'validated', 'user': 'alice'}
Result: {'recorded': True, 'user': 'alice'}

Transform event:
[LOG] Event 'data.transform' with keys: ['Name', 'City', 'age', 'timestamp']
Result: {'Name': 'bob', 'City': 'melbourne', 'age': 25, 'timestamp': 1712745600.0}

partial(): freeze arguments and pass the function. Currying, almost.
partial(): freeze arguments and pass the function. Currying, almost.

Common Pitfalls and Troubleshooting

Problem Cause Solution
TypeError: unhashable type with lru_cache Passing a list or dict as argument to cached function Convert to tuple or frozenset before passing
Memory growing with @cache Unbounded cache stores every unique call Use @lru_cache(maxsize=N) to limit cache size
cached_property not updating Value computed once and stored as attribute Delete the attribute with del obj.prop to force recompute
Decorated function loses metadata Missing @wraps in decorator Add @wraps(func) to every decorator wrapper
reduce gives unexpected result Missing initial value argument Pass initializer as third argument to reduce()
partial kwargs overridden Caller passes same keyword argument Document which args are frozen or use positional args

Frequently Asked Questions

What is the difference between @cache and @lru_cache?

@cache is equivalent to @lru_cache(maxsize=None). It stores every unique call result forever without evicting old entries. Use @lru_cache(maxsize=N) when you want to limit memory usage by keeping only the N most recent unique results. For most applications, @lru_cache with a reasonable maxsize is the safer choice.

Can I use lru_cache on class methods?

Yes, but with a caveat. The self parameter becomes part of the cache key, meaning each instance gets its own cache entries. For instance-level caching, consider @cached_property instead. For class-level caching, use a module-level function or a custom descriptor.

Is functools.reduce the same as a for loop?

Functionally, yes — reduce performs the same cumulative operation you could write with a for loop. However, reduce expresses the intent more declaratively. Use reduce when the accumulation pattern is clear and concise. If the logic is complex or needs multiple lines, a regular for loop is more readable and Pythonic.

How do I clear a cached_property value?

Delete the attribute using del instance.property_name. The next access will recompute and re-cache the value. This works because cached_property stores the result as a regular instance attribute that shadows the descriptor.

When should I use partial instead of a lambda?

Use partial() when you want to freeze arguments of an existing function — it is more readable, preserves the original function's metadata, and works better with pickling. Use a lambda when you need a quick inline expression or when the logic goes beyond simple argument freezing. In general, partial is preferred for configuration-style currying.

@wraps: keep the metadata when you wrap a function. Use it always.
@wraps: keep the metadata when you wrap a function. Use it always.

Conclusion

The functools module gives you powerful tools that make Python functions more flexible and efficient. You learned how @lru_cache and @cache can dramatically speed up expensive or recursive functions, how partial() creates specialized versions of general functions, how reduce() handles cumulative operations, and how @wraps keeps your decorators well-behaved.

These tools work beautifully together, as the plugin system example showed. Start by adding @lru_cache to your most expensive functions and @wraps to your decorators — those two changes alone will improve most Python projects. From there, explore partial() and cached_property as your needs grow.

partial(): Pre-Fill Arguments

from functools import partial

def power(base, exponent):
    return base ** exponent

square = partial(power, exponent=2)
cube = partial(power, exponent=3)

print(square(5))   # 25
print(cube(3))     # 27

# Useful for callbacks and higher-order functions
import threading
def task(user_id, action):
    print("Running:", action, "for", user_id)

threading.Timer(5, partial(task, "user123", "cleanup")).start()

partial freezes some arguments; the resulting callable accepts the rest. Perfect for adapting a function's signature to a callback that expects fewer arguments.

@lru_cache: Memoization for Free

from functools import lru_cache

@lru_cache(maxsize=1000)
def expensive(query):
    return slow_database_lookup(query)

# Cache stats
print(expensive.cache_info())     # CacheInfo(hits=12, misses=4, maxsize=1000, currsize=4)
expensive.cache_clear()

@lru_cache turns any pure function into a memoized one. Calls with the same arguments return the cached result. maxsize=None for unbounded; finite values evict least-recently-used entries.

@cache: lru_cache With Unbounded Memory

from functools import cache

@cache
def fibonacci(n):
    if n < 2:
        return n
    return fibonacci(n-1) + fibonacci(n-2)

print(fibonacci(100))   # instant, even though recursive

@cache (Python 3.9+) is shorthand for @lru_cache(maxsize=None) — slightly faster, simpler API.

reduce(): Fold a Sequence

from functools import reduce

# Sum of a list (just use sum())
total = reduce(lambda acc, x: acc + x, [1, 2, 3, 4, 5])   # 15

# Product
import operator
product = reduce(operator.mul, [1, 2, 3, 4, 5])   # 120

# Max
maximum = reduce(lambda a, b: a if a > b else b, [5, 3, 8, 1])   # 8

# Build a dict from key-value pairs
pairs = [("a", 1), ("b", 2), ("c", 3)]
d = reduce(lambda acc, kv: {**acc, kv[0]: kv[1]}, pairs, {})

reduce is powerful but often overkill. For common operations (sum, max, min) Python has builtins. Reach for reduce only when the operation doesn't have a standard equivalent.

@wraps: Preserve Decorated Function Metadata

from functools import wraps
import time

def timed(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        t0 = time.perf_counter()
        result = func(*args, **kwargs)
        print(f"{func.__name__} took {time.perf_counter()-t0:.4f}s")
        return result
    return wrapper

@timed
def slow_function(x):
    """Does something slow."""
    time.sleep(0.1)
    return x

slow_function(5)
print(slow_function.__name__)   # 'slow_function' (not 'wrapper')
print(slow_function.__doc__)    # 'Does something slow.' (preserved)

Without @wraps, your decorated function's __name__ and __doc__ become 'wrapper' and None — breaks help(), Sphinx, inspect. Always use @wraps on decorator inner functions.

singledispatch: Type-Based Function Overloading

from functools import singledispatch

@singledispatch
def render(item):
    raise NotImplementedError(f"No renderer for {type(item)}")

@render.register
def _(item: int):
    return f"Integer: {item}"

@render.register
def _(item: str):
    return f"String: {item!r}"

@render.register
def _(item: list):
    return "List of " + str(len(item)) + " items"

print(render(42))         # 'Integer: 42'
print(render("hello"))    # "String: 'hello'"
print(render([1, 2, 3]))  # 'List of 3 items'

Python's answer to method overloading. The right implementation runs based on the FIRST argument's type. Useful for serializers, formatters, anything that's polymorphic.

Common Pitfalls

  • lru_cache on mutable arguments. Arguments must be hashable. Lists, dicts, sets — won't cache. Convert to tuple/frozenset/string before calling.
  • Method caching memory leak. @lru_cache on a method captures self in the cache key, preventing instance garbage collection. Use cached_property for instance methods.
  • Decorator without @wraps. Loses function metadata. Always use @wraps inside decorator definitions.
  • Mutating cached return. @cache returns a reference, not a copy. Modifying the result corrupts the cache. Make defensive copies if the caller might mutate.
  • singledispatch on positional args. Only dispatches on the FIRST argument. For multi-argument dispatch, use plum or multipledispatch.

FAQ

Q: lru_cache or cachetools?
A: lru_cache for simple cases (one function, in-memory, no TTL). cachetools when you need TTLs, custom eviction policies, or thread safety beyond stdlib.

Q: Will lru_cache leak memory?
A: With finite maxsize, no — evicts LRU entries. With maxsize=None (or @cache), can grow unbounded. Monitor with cache_info().

Q: Use partial vs lambda?
A: partial when you want a picklable, named-keyword version (useful for multiprocessing). lambda when it's a one-liner used immediately.

Q: reduce vs explicit loop?
A: Loop is usually clearer. reduce shines for chain operations where the loop body is short and the accumulator is the whole story.

Q: cached_property when?
A: For per-instance memoization of expensive properties — first access computes, subsequent accesses return cached. Cleanly garbage-collects with the instance.

Wrapping Up

functools is a toolbox of higher-order primitives. partial for argument freezing; lru_cache/cache for memoization; wraps for decorator hygiene; singledispatch for type-based polymorphism; reduce for folds. Master these and your Python becomes more expressive without becoming clever-for-clever's-sake.

Related Articles

How To Use Python Contextlib for Resource Management

How To Use Python Contextlib for Resource Management

Last Updated: June 01, 2026

Intermediate

Every Python developer learns to use with open('file.txt') as f early on, but few explore the full power of context managers beyond file handling. If you have ever needed to manage database connections, acquire locks, temporarily change settings, or ensure cleanup code always runs, the contextlib module is your toolkit. It turns complex resource management into clean, readable code.

The contextlib module is part of Python’s standard library, so there is nothing to install. It provides decorators and utilities that let you create context managers without writing a full class with __enter__ and __exit__ methods. The most powerful tool is @contextmanager, which turns a simple generator function into a fully functional context manager.

In this tutorial, you will learn how to create custom context managers with @contextmanager, manage multiple resources with ExitStack, suppress specific exceptions cleanly, redirect output streams, and build reusable resource management patterns. By the end, you will write context managers as naturally as you write regular functions.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Custom Context Manager: Quick Example

Here is how to create a timing context manager that measures how long a block of code takes to run:

# quick_timer.py
from contextlib import contextmanager
import time

@contextmanager
def timer(label="Block"):
    start = time.perf_counter()
    yield
    elapsed = time.perf_counter() - start
    print(f"{label} took {elapsed:.4f} seconds")

# Use it
with timer("Data processing"):
    total = sum(range(1_000_000))
    print(f"Sum: {total}")

Output:

Sum: 499999500000
Data processing took 0.0312 seconds

The @contextmanager decorator transforms the generator function into a context manager. Everything before yield runs on entry (like __enter__), and everything after yield runs on exit (like __exit__). The yield itself is where your with block executes.

What is contextlib and Why Use It?

Context managers in Python follow the protocol defined by the __enter__ and __exit__ magic methods. Any object with these two methods can be used with the with statement. The contextlib module provides shortcuts that save you from writing boilerplate class definitions for simple use cases.

Without contextlib, creating a context manager requires a full class with two methods. With it, you can do the same thing in a few lines using a generator function. This matters because context managers are everywhere in professional Python code — they manage database transactions, HTTP sessions, temporary files, thread locks, and any resource that needs guaranteed cleanup.

ApproachLines of CodeBest For
Class with __enter__/__exit__10-20 linesComplex state management, reusable libraries
@contextmanager decorator5-10 linesSimple setup/teardown, one-off utilities
ExitStack3-5 linesDynamic number of resources, conditional cleanup

The rule of thumb: if your context manager is straightforward setup-yield-cleanup, use @contextmanager. If it needs to manage dynamic or conditional resources, reach for ExitStack.

with-blocks: clean up always happens, even when things blow up.
with-blocks: clean up always happens, even when things blow up.

The @contextmanager Decorator

The @contextmanager decorator is the workhorse of the module. It takes a generator that yields exactly once and turns it into a context manager. Here is a practical example that temporarily changes the working directory:

# change_dir.py
from contextlib import contextmanager
import os

@contextmanager
def change_directory(path):
    """Temporarily change the working directory."""
    original = os.getcwd()
    try:
        os.chdir(path)
        yield path
    finally:
        os.chdir(original)

# Usage
print(f"Before: {os.getcwd()}")
with change_directory("/tmp") as new_dir:
    print(f"Inside: {os.getcwd()}")
    print(f"Yielded: {new_dir}")
print(f"After: {os.getcwd()}")

Output:

Before: /home/user/project
Inside: /tmp
Yielded: /tmp
After: /home/user/project

The try/finally block inside the generator ensures cleanup happens even if an exception occurs within the with block. Whatever you pass to yield becomes the value bound by as in the with statement.

Handling Errors in Context Managers

When an exception occurs inside a with block, it gets thrown into the generator at the yield point. You can catch it, handle it, or let it propagate:

# error_handling.py
from contextlib import contextmanager

@contextmanager
def safe_operation(name):
    """Context manager that logs errors without suppressing them."""
    print(f"Starting: {name}")
    try:
        yield
    except Exception as e:
        print(f"Error in {name}: {type(e).__name__}: {e}")
        raise  # Re-raise to let caller handle it
    finally:
        print(f"Finished: {name}")

# Normal usage
with safe_operation("calculation"):
    result = 42 / 2
    print(f"Result: {result}")

print("---")

# With an error
try:
    with safe_operation("bad calculation"):
        result = 42 / 0
except ZeroDivisionError:
    print("Caught the error outside")

Output:

Starting: calculation
Result: 21.0
Finished: calculation
---
Starting: bad calculation
Error in bad calculation: ZeroDivisionError: division by zero
Finished: bad calculation
Caught the error outside

The finally block guarantees cleanup runs whether the operation succeeds or fails. If you want to suppress the exception (prevent it from propagating), do not re-raise it — but be careful, as silently swallowing exceptions makes debugging difficult.

Managing Multiple Resources with ExitStack

When you need to manage a variable number of resources — like opening multiple files determined at runtime — ExitStack is the right tool:

# exitstack_demo.py
from contextlib import ExitStack
import tempfile
import os

def process_multiple_files(filenames):
    """Open and process multiple files safely."""
    with ExitStack() as stack:
        # Open all files -- ExitStack closes them all on exit
        files = [
            stack.enter_context(open(f, 'w'))
            for f in filenames
        ]
        
        # Write to each file
        for i, f in enumerate(files):
            f.write(f"Content for file {i}\n")
            print(f"Wrote to {filenames[i]}")
        
        print(f"All {len(files)} files open simultaneously")
    # All files are closed here, even if an error occurred
    print("All files closed")

# Create temp files for demo
temp_dir = tempfile.mkdtemp()
filenames = [os.path.join(temp_dir, f"file_{i}.txt") for i in range(3)]
process_multiple_files(filenames)

# Verify files are written and closed
for f in filenames:
    print(f"{os.path.basename(f)}: {open(f).read().strip()}")

Output:

Wrote to /tmp/tmpXXXXXX/file_0.txt
Wrote to /tmp/tmpXXXXXX/file_1.txt
Wrote to /tmp/tmpXXXXXX/file_2.txt
All 3 files open simultaneously
All files closed
file_0.txt: Content for file 0
file_1.txt: Content for file 1
file_2.txt: Content for file 2

The key advantage of ExitStack is that it handles cleanup for all registered resources, even if opening a later resource fails. Without it, you would need deeply nested with statements or manual cleanup logic.

@contextmanager: a decorator that turns a generator into a with-block.
@contextmanager: a decorator that turns a generator into a with-block.

Suppressing Exceptions with suppress()

Sometimes you want to ignore specific exceptions cleanly. Instead of writing try/except: pass, use contextlib.suppress():

# suppress_demo.py
from contextlib import suppress
import os

# Without suppress (verbose)
try:
    os.remove("nonexistent_file.txt")
except FileNotFoundError:
    pass

# With suppress (clean)
with suppress(FileNotFoundError):
    os.remove("nonexistent_file.txt")

# Multiple exception types
with suppress(FileNotFoundError, PermissionError):
    os.remove("/protected/file.txt")

print("Cleanup complete -- no crashes")

Output:

Cleanup complete -- no crashes

Use suppress() when you genuinely do not care about the exception — like deleting a file that might not exist, or disconnecting a client that might already be disconnected. Do not use it to hide errors you should be handling.

Redirecting Output with redirect_stdout

The redirect_stdout and redirect_stderr context managers temporarily redirect output streams. This is useful for capturing output from third-party libraries or silencing noisy functions:

# redirect_demo.py
from contextlib import redirect_stdout, redirect_stderr
import io

# Capture stdout to a string
buffer = io.StringIO()
with redirect_stdout(buffer):
    print("This goes to the buffer")
    print("So does this")

captured = buffer.getvalue()
print(f"Captured {len(captured)} characters:")
print(repr(captured))

# Silence stderr
with redirect_stderr(io.StringIO()):
    import warnings
    warnings.warn("This warning is silenced")

Output:

Captured 39 characters:
'This goes to the buffer\nSo does this\n'

This pattern is especially useful in testing, where you need to verify what a function prints without modifying the function itself.

ExitStack: stack a dozen context managers without nested ifs.
ExitStack: stack a dozen context managers without nested ifs.

Real-Life Example: Database Transaction Manager

Let us build a practical transaction manager that handles database connections, commits on success, and rolls back on failure — all with proper resource cleanup:

# transaction_manager.py
from contextlib import contextmanager, ExitStack
import sqlite3
import os
import tempfile

@contextmanager
def database_connection(db_path):
    """Manage a database connection lifecycle."""
    conn = sqlite3.connect(db_path)
    try:
        yield conn
    finally:
        conn.close()

@contextmanager
def transaction(conn):
    """Manage a database transaction with auto-commit/rollback."""
    cursor = conn.cursor()
    try:
        yield cursor
        conn.commit()
        print("Transaction committed")
    except Exception as e:
        conn.rollback()
        print(f"Transaction rolled back: {e}")
        raise

@contextmanager
def managed_database(db_path):
    """Complete database session with connection and transaction."""
    with ExitStack() as stack:
        conn = stack.enter_context(database_connection(db_path))
        cursor = stack.enter_context(transaction(conn))
        yield cursor

# Demo
db_path = os.path.join(tempfile.mkdtemp(), "demo.db")

# Successful transaction
with managed_database(db_path) as cursor:
    cursor.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT, email TEXT)")
    cursor.execute("INSERT INTO users (name, email) VALUES (?, ?)", ("Alice", "alice@example.com"))
    cursor.execute("INSERT INTO users (name, email) VALUES (?, ?)", ("Bob", "bob@example.com"))

# Read back
with managed_database(db_path) as cursor:
    cursor.execute("SELECT * FROM users")
    for row in cursor.fetchall():
        print(f"  User: {row}")

# Failed transaction (rollback)
print("\nAttempting bad insert:")
try:
    with managed_database(db_path) as cursor:
        cursor.execute("INSERT INTO users (name, email) VALUES (?, ?)", ("Charlie", "charlie@example.com"))
        raise ValueError("Simulated error -- transaction should rollback")
except ValueError:
    pass

# Verify rollback worked
with managed_database(db_path) as cursor:
    cursor.execute("SELECT COUNT(*) FROM users")
    count = cursor.fetchone()[0]
    print(f"Users after rollback: {count} (Charlie was NOT added)")

Output:

Transaction committed
Transaction committed
  User: (1, 'Alice', 'alice@example.com')
  User: (2, 'Bob', 'bob@example.com')

Attempting bad insert:
Transaction rolled back: Simulated error -- transaction should rollback
Transaction committed
Users after rollback: 2 (Charlie was NOT added)

This example composes three context managers: database_connection handles the connection lifecycle, transaction handles commit/rollback logic, and managed_database combines both using ExitStack. The composition pattern keeps each context manager focused on a single responsibility while providing a convenient combined interface.

Frequently Asked Questions

When should I use a class-based context manager instead of @contextmanager?

Use a class when you need to store state between __enter__ and __exit__, when the context manager will be reused as a library component, or when you need the __exit__ method’s exception arguments to make decisions about exception handling. Use @contextmanager for simple setup-teardown patterns where a generator function is more readable.

Does contextlib work with async code?

Yes. Python 3.7+ includes contextlib.asynccontextmanager for async generators, and AsyncExitStack for managing async resources. The patterns are identical — just use async with instead of with and async def instead of def.

Can I nest context managers?

Yes, and there are multiple ways: nested with statements, comma-separated managers in a single with statement (with A() as a, B() as b:), or ExitStack for dynamic nesting. The comma syntax is preferred for a fixed number of managers; ExitStack is preferred when the number is determined at runtime.

Can I reuse a context manager instance?

It depends. @contextmanager-based managers are single-use — calling __enter__ twice on the same instance raises a RuntimeError. Class-based managers can be made reentrant by resetting state in __enter__. If you need a reentrant version, create a new instance each time or use a factory function.

What is contextlib.closing() for?

The closing() wrapper calls .close() on an object when the with block exits. Use it for objects that have a close() method but do not implement the context manager protocol natively — like urllib.request.urlopen() in older Python versions or custom connection objects.

Conclusion

The contextlib module transforms resource management from boilerplate-heavy class definitions into clean, expressive patterns. We covered @contextmanager for generator-based managers, ExitStack for dynamic resource composition, suppress() for clean exception ignoring, and redirect_stdout for output capture. The transaction manager example showed how these tools compose into production-ready patterns.

Try extending the database transaction manager with connection pooling, nested savepoints, or retry logic. For the complete API reference, see the official contextlib documentation.

How To Build a WebSocket Server with Python

How To Build a WebSocket Server with Python

Last Updated: June 01, 2026

Intermediate

If you have ever built a chat application, a live dashboard, or a multiplayer game, you know that regular HTTP requests fall short when you need real-time, bidirectional communication. Polling the server every few seconds wastes bandwidth and adds latency. WebSockets solve this by keeping a persistent connection open between client and server, allowing both sides to send messages at any time.

Python’s websockets library makes building WebSocket servers and clients remarkably straightforward. It is built on top of asyncio, so it integrates naturally with Python’s async ecosystem. You only need to install one package — pip install websockets — and you are ready to go.

In this tutorial, you will learn how to create a WebSocket server, build a client that connects to it, implement broadcast messaging for chat-style applications, handle connection lifecycle events, and add basic authentication. By the end, you will have a working real-time chat server that you can extend for your own projects.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

WebSocket Echo Server: Quick Example

Part of the Python Web Frameworks Hub. See the full hub for related Python tutorials.

Here is the simplest possible WebSocket server — it echoes back whatever the client sends:

# echo_server.py
import asyncio
import websockets

async def echo(websocket):
    async for message in websocket:
        print(f"Received: {message}")
        await websocket.send(f"Echo: {message}")

async def main():
    async with websockets.serve(echo, "localhost", 8765):
        print("Echo server running on ws://localhost:8765")
        await asyncio.Future()  # Run forever

asyncio.run(main())

Output (server):

Echo server running on ws://localhost:8765
Received: Hello WebSocket!
Received: How are you?

The websockets.serve() function starts a server that calls the echo handler for every new connection. The async for message in websocket pattern automatically handles the connection lifecycle — it reads messages until the client disconnects, then exits cleanly. We will build on this pattern throughout the tutorial.

What Are WebSockets and Why Use Them?

WebSockets are a communication protocol that provides full-duplex (two-way) communication channels over a single TCP connection. Unlike HTTP, where the client must initiate every request, WebSockets allow either side to send data at any time after the initial handshake.

The WebSocket protocol starts with an HTTP upgrade request. The client sends a regular HTTP request with an Upgrade: websocket header, and if the server agrees, the connection is upgraded from HTTP to WebSocket. From that point on, both sides communicate using lightweight WebSocket frames instead of full HTTP requests.

FeatureHTTPWebSocket
DirectionClient to server onlyBidirectional
ConnectionNew connection per requestPersistent single connection
OverheadHeaders on every requestMinimal frame overhead (2-14 bytes)
LatencyRound-trip per requestNear real-time
Use caseREST APIs, page loadsChat, live feeds, gaming

Use WebSockets when you need low-latency, real-time updates. Use HTTP when you need stateless request-response patterns (like REST APIs).

WebSockets: HTTP grew tired of one-way conversations.
WebSockets: HTTP grew tired of one-way conversations.

Setting Up the websockets Library

Install the library with pip:

# install.sh
pip install websockets

Output:

Successfully installed websockets-13.1

The websockets library requires Python 3.8 or later. It has no dependencies beyond the standard library’s asyncio module, keeping your dependency tree clean.

Building a WebSocket Client

To test our servers, we need a client. Here is a simple interactive client that sends messages and prints responses:

# client.py
import asyncio
import websockets

async def chat_client():
    uri = "ws://localhost:8765"
    async with websockets.connect(uri) as websocket:
        print("Connected to server. Type messages (Ctrl+C to quit):")
        
        # Send a greeting
        await websocket.send("Hello from Python client!")
        response = await websocket.recv()
        print(f"Server says: {response}")
        
        # Interactive loop
        while True:
            message = input("> ")
            await websocket.send(message)
            response = await websocket.recv()
            print(f"Server says: {response}")

asyncio.run(chat_client())

Output:

Connected to server. Type messages (Ctrl+C to quit):
Server says: Echo: Hello from Python client!
> Testing 123
Server says: Echo: Testing 123

The websockets.connect() context manager handles the connection lifecycle automatically. When you exit the async with block (or the program ends), the connection closes cleanly with a proper WebSocket close handshake.

Push notifications, live dashboards, chat. WebSockets does it all.
Push notifications, live dashboards, chat. WebSockets does it all.

Building a Broadcast Chat Server

A real chat server needs to broadcast messages from one client to all connected clients. This requires tracking active connections in a set:

# chat_server.py
import asyncio
import websockets
import json
from datetime import datetime

connected_clients = set()

async def broadcast(message, sender=None):
    """Send a message to all connected clients except the sender."""
    disconnected = set()
    for client in connected_clients:
        if client != sender:
            try:
                await client.send(message)
            except websockets.ConnectionClosed:
                disconnected.add(client)
    # Clean up disconnected clients
    connected_clients -= disconnected

async def chat_handler(websocket):
    """Handle a single client connection."""
    # Register client
    connected_clients.add(websocket)
    client_id = f"User-{id(websocket) % 1000}"
    print(f"{client_id} connected. Total clients: {len(connected_clients)}")
    
    # Notify others
    join_msg = json.dumps({
        "type": "system",
        "message": f"{client_id} joined the chat",
        "timestamp": datetime.now().isoformat()
    })
    await broadcast(join_msg, sender=websocket)
    
    try:
        async for message in websocket:
            # Wrap message with metadata
            chat_msg = json.dumps({
                "type": "chat",
                "sender": client_id,
                "message": message,
                "timestamp": datetime.now().isoformat()
            })
            print(f"{client_id}: {message}")
            await broadcast(chat_msg)
    except websockets.ConnectionClosed:
        pass
    finally:
        # Unregister client
        connected_clients.discard(websocket)
        leave_msg = json.dumps({
            "type": "system",
            "message": f"{client_id} left the chat",
            "timestamp": datetime.now().isoformat()
        })
        await broadcast(leave_msg)
        print(f"{client_id} disconnected. Total clients: {len(connected_clients)}")

async def main():
    async with websockets.serve(chat_handler, "localhost", 8765):
        print("Chat server running on ws://localhost:8765")
        await asyncio.Future()

asyncio.run(main())

Output (server with two clients):

Chat server running on ws://localhost:8765
User-142 connected. Total clients: 1
User-857 connected. Total clients: 2
User-142: Hello everyone!
User-857: Hey there!
User-142 disconnected. Total clients: 1

The connected_clients set tracks all active connections. When a client sends a message, broadcast() forwards it to every other connected client. The try/finally block ensures we clean up the client set even if the connection drops unexpectedly.

Handling Connection Errors Gracefully

Real-world WebSocket connections drop unexpectedly due to network issues, client crashes, or timeouts. Robust error handling is essential:

# robust_server.py
import asyncio
import websockets
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("websocket-server")

async def robust_handler(websocket):
    """Handler with comprehensive error management."""
    remote = websocket.remote_address
    logger.info(f"New connection from {remote}")
    
    try:
        # Set a ping interval to detect dead connections
        async for message in websocket:
            if len(message) > 10000:
                await websocket.send("Error: Message too long (max 10000 chars)")
                continue
            
            await websocket.send(f"Processed: {message}")
            
    except websockets.ConnectionClosedError as e:
        logger.warning(f"Connection closed with error: {e.code} {e.reason}")
    except websockets.ConnectionClosedOK:
        logger.info(f"Connection closed normally from {remote}")
    except Exception as e:
        logger.error(f"Unexpected error: {e}")
    finally:
        logger.info(f"Cleanup complete for {remote}")

async def main():
    async with websockets.serve(
        robust_handler,
        "localhost",
        8765,
        ping_interval=20,     # Send ping every 20 seconds
        ping_timeout=10,      # Wait 10 seconds for pong
        max_size=2**20,       # Max message size: 1MB
        close_timeout=5       # Wait 5 seconds for close handshake
    ):
        logger.info("Robust server running on ws://localhost:8765")
        await asyncio.Future()

asyncio.run(main())

Output:

INFO:websocket-server:Robust server running on ws://localhost:8765
INFO:websocket-server:New connection from ('127.0.0.1', 54321)
WARNING:websocket-server:Connection closed with error: 1006 
INFO:websocket-server:Cleanup complete for ('127.0.0.1', 54321)

The ping_interval and ping_timeout parameters enable automatic keepalive detection. If a client stops responding to pings within the timeout, the server closes the connection and triggers cleanup. The max_size parameter prevents clients from sending oversized messages that could exhaust server memory.

Heartbeats: prove the socket is alive, not just open.
Heartbeats: prove the socket is alive, not just open.

Adding Basic Authentication

You can authenticate WebSocket connections using query parameters, headers, or an initial authentication message. Here is a token-based approach:

# auth_server.py
import asyncio
import websockets
import json
import secrets

# Valid tokens (in production, use a database)
VALID_TOKENS = {
    "demo-token-abc123": "alice",
    "demo-token-def456": "bob"
}

async def authenticate(websocket):
    """Authenticate the client using the first message."""
    try:
        auth_msg = await asyncio.wait_for(websocket.recv(), timeout=5.0)
        data = json.loads(auth_msg)
        token = data.get("token", "")
        
        for valid_token, username in VALID_TOKENS.items():
            if secrets.compare_digest(token, valid_token):
                await websocket.send(json.dumps({
                    "type": "auth", 
                    "status": "ok",
                    "username": username
                }))
                return username
        
        await websocket.send(json.dumps({
            "type": "auth",
            "status": "error", 
            "message": "Invalid token"
        }))
        return None
        
    except asyncio.TimeoutError:
        await websocket.send(json.dumps({
            "type": "auth",
            "status": "error",
            "message": "Authentication timeout"
        }))
        return None

async def secure_handler(websocket):
    """Handler that requires authentication."""
    username = await authenticate(websocket)
    if not username:
        await websocket.close(1008, "Authentication failed")
        return
    
    print(f"{username} authenticated successfully")
    
    async for message in websocket:
        response = f"[{username}] {message}"
        await websocket.send(response)

async def main():
    async with websockets.serve(secure_handler, "localhost", 8765):
        print("Secure server running on ws://localhost:8765")
        await asyncio.Future()

asyncio.run(main())

Output:

Secure server running on ws://localhost:8765
alice authenticated successfully

The server expects the first message to contain a JSON object with a token field. It uses secrets.compare_digest() for timing-safe comparison and gives the client 5 seconds to authenticate before timing out. If authentication fails, the connection is closed with WebSocket status code 1008 (Policy Violation).

Real-Life Example: Live Notification System

Let us build a notification system where a server pushes real-time alerts to connected clients based on their subscribed topics:

# notification_server.py
import asyncio
import websockets
import json
from datetime import datetime
from collections import defaultdict

class NotificationServer:
    """Real-time notification server with topic subscriptions."""
    
    def __init__(self):
        self.subscribers = defaultdict(set)  # topic -> set of websockets
        self.clients = {}  # websocket -> client info
    
    async def handle_subscribe(self, websocket, topics):
        """Subscribe a client to one or more topics."""
        for topic in topics:
            self.subscribers[topic].add(websocket)
        self.clients[websocket]["topics"] = topics
        await websocket.send(json.dumps({
            "type": "subscribed",
            "topics": topics
        }))
    
    async def notify(self, topic, message):
        """Send a notification to all subscribers of a topic."""
        if topic not in self.subscribers:
            return 0
        
        notification = json.dumps({
            "type": "notification",
            "topic": topic,
            "message": message,
            "timestamp": datetime.now().isoformat()
        })
        
        sent = 0
        disconnected = set()
        for client in self.subscribers[topic]:
            try:
                await client.send(notification)
                sent += 1
            except websockets.ConnectionClosed:
                disconnected.add(client)
        
        # Cleanup
        self.subscribers[topic] -= disconnected
        return sent
    
    async def handler(self, websocket):
        """Main connection handler."""
        self.clients[websocket] = {"topics": [], "connected": datetime.now()}
        
        try:
            async for raw_message in websocket:
                data = json.loads(raw_message)
                action = data.get("action")
                
                if action == "subscribe":
                    await self.handle_subscribe(
                        websocket, data.get("topics", [])
                    )
                elif action == "publish":
                    topic = data.get("topic")
                    message = data.get("message")
                    count = await self.notify(topic, message)
                    await websocket.send(json.dumps({
                        "type": "published",
                        "topic": topic,
                        "recipients": count
                    }))
        except websockets.ConnectionClosed:
            pass
        finally:
            # Remove from all subscriptions
            for topic_subs in self.subscribers.values():
                topic_subs.discard(websocket)
            self.clients.pop(websocket, None)

server = NotificationServer()

async def main():
    async with websockets.serve(server.handler, "localhost", 8765):
        print("Notification server running on ws://localhost:8765")
        
        # Simulate periodic system notifications
        async def system_alerts():
            while True:
                await asyncio.sleep(30)
                await server.notify("system", "Heartbeat check - all systems normal")
        
        await asyncio.gather(
            asyncio.Future(),  # Run forever
            system_alerts()
        )

asyncio.run(main())

Output:

Notification server running on ws://localhost:8765

This server supports topic-based subscriptions. Clients subscribe to topics like “alerts”, “updates”, or “system”, and only receive notifications for their subscribed topics. The notify() method handles broadcasting to subscribers and automatically cleans up disconnected clients. You could extend this with persistent message queues, delivery confirmation, or priority levels.

Frequently Asked Questions

How many concurrent connections can a Python WebSocket server handle?

A single Python process using asyncio can typically handle 10,000-50,000 concurrent WebSocket connections, depending on message frequency and server resources. The websockets library is efficient with memory, using roughly 10KB per connection. For higher scale, use multiple processes behind a load balancer like Nginx.

How do I add TLS/SSL encryption (wss://)?

Pass an ssl context to websockets.serve(): ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER), then load your certificate with ssl_context.load_cert_chain(certfile, keyfile). In production, use a reverse proxy like Nginx to handle TLS termination instead.

How do I handle client reconnection?

The websockets library does not auto-reconnect. On the client side, wrap your connection in a retry loop with exponential backoff. The websockets.connect() context manager closes cleanly, so you can simply reconnect in a while True loop with a try/except block catching ConnectionClosed errors.

Can I send binary data over WebSockets?

Yes. The websockets library automatically detects whether you send str (text frames) or bytes (binary frames). Use await websocket.send(b"\x00\x01\x02") for binary data. This is useful for sending images, audio streams, or protobuf-encoded messages.

Can I use WebSockets with Django or Flask?

Django supports WebSockets through Django Channels, which adds an ASGI layer. Flask does not natively support WebSockets, but you can use Flask-SocketIO or run a separate websockets server alongside Flask. For new projects needing WebSocket support, consider FastAPI, which has native WebSocket support built on Starlette.

Conclusion

You now have a solid foundation for building WebSocket applications in Python. We covered creating echo servers, building broadcast chat systems, handling connection errors gracefully, adding authentication, and implementing topic-based notification systems. The websockets library’s async-first design makes it easy to handle thousands of concurrent connections with clean, readable code.

Try extending the notification server with features like message persistence, delivery acknowledgments, or a web-based admin dashboard. For the complete API reference, see the official websockets documentation.

Bare websockets Library Server

The websockets library is the canonical pure-Python implementation — modern async, type-hinted, RFC-compliant. The minimal echo server is 8 lines:

# pip install websockets

import asyncio
from websockets.asyncio.server import serve

async def echo(ws):
    async for message in ws:
        await ws.send(f"Echo: {message}")

async def main():
    async with serve(echo, "0.0.0.0", 8765) as server:
        await server.serve_forever()

asyncio.run(main())

Connect from anywhere — browser JavaScript (new WebSocket("ws://localhost:8765")), wscat, or another Python client via websockets.connect. The handler runs once per connection; async for message reads incoming frames until the client disconnects.

Broadcasting to Multiple Clients

For chat rooms or live dashboards, track all connected clients in a set, broadcast each message to all of them:

import asyncio
from websockets.asyncio.server import serve
import json

clients = set()

async def handler(ws):
    clients.add(ws)
    try:
        async for raw in ws:
            payload = json.loads(raw)
            outbound = json.dumps({"user": payload["user"], "text": payload["text"]})
            # Send to everyone, ignore disconnected clients
            await asyncio.gather(
                *[c.send(outbound) for c in clients],
                return_exceptions=True,
            )
    finally:
        clients.remove(ws)

async def main():
    async with serve(handler, "0.0.0.0", 8765):
        await asyncio.Future()  # run forever

asyncio.run(main())

Auth, Subprotocols, and Path Routing

Production WebSocket servers need auth and routing. Use the process_request hook for HTTP-level checks before upgrading to WebSocket:

from websockets.asyncio.server import serve
from http import HTTPStatus

async def auth_handler(connection, request):
    # Reject before upgrade
    token = request.headers.get("Authorization", "").removeprefix("Bearer ")
    if not is_valid(token):
        return connection.respond(HTTPStatus.UNAUTHORIZED, "Bad token\n")
    # Attach user info for the handler
    connection.user_id = get_user_id_from_token(token)

async def handler(ws):
    user_id = ws.user_id
    async for msg in ws:
        # use user_id in messages
        pass

async def main():
    async with serve(handler, "0.0.0.0", 8765, process_request=auth_handler):
        await asyncio.Future()

WebSockets in FastAPI

If you already have a FastAPI app, FastAPI ships its own WebSocket support — no second framework needed:

from fastapi import FastAPI, WebSocket, WebSocketDisconnect

app = FastAPI()
active = []

@app.websocket("/ws")
async def ws_endpoint(websocket: WebSocket):
    await websocket.accept()
    active.append(websocket)
    try:
        while True:
            data = await websocket.receive_text()
            for client in active:
                await client.send_text(f"[broadcast] {data}")
    except WebSocketDisconnect:
        active.remove(websocket)

Heartbeats and Reconnection

WebSockets across the public internet drop silently — proxies and NAT timeouts kill idle connections after 30-120 seconds. Send pings to keep them alive:

async with serve(handler, "0.0.0.0", 8765, ping_interval=20, ping_timeout=10):
    ...

For clients, implement exponential-backoff reconnect on the JS side — bare browsers don’t auto-reconnect.

Common Pitfalls

  • Blocking inside the handler. A synchronous time.sleep(5) blocks the entire event loop. Use await asyncio.sleep(5). Long CPU work belongs in asyncio.to_thread.
  • Forgetting to remove disconnected clients. Without cleanup, broadcasting tries to send on a dead socket — exception, server crashes. Use try/finally with remove.
  • Treating each message as JSON without try/except. A malformed message kills the handler. Wrap json.loads in a try/except and send an error message back.
  • Holding references in the broadcast list. Long-lived sets of WebSocket objects can leak memory if a handler exits without removing itself. Always remove in finally.
  • Skipping wss:// in production. Plain ws:// sends data unencrypted. Always terminate TLS at your reverse proxy and use wss://.

FAQ

Q: websockets library or FastAPI WebSockets?
A: FastAPI if you already have a FastAPI app. The websockets library when you want a dedicated, low-overhead WebSocket-only server.

Q: How many concurrent connections can one server handle?
A: Thousands on a single Python process, tens of thousands with uvloop. Beyond that, scale horizontally with a load balancer that supports sticky sessions or use a pub/sub layer (Redis) for cross-instance broadcasts.

Q: How do I scale broadcasts across multiple server instances?
A: Use Redis pub/sub as the message bus. Each instance subscribes to a channel; broadcasts go through Redis; every connected client gets the message regardless of which instance they’re on.

Q: Polling vs WebSockets vs SSE?
A: WebSockets for bidirectional real-time (chat, multiplayer games). Server-Sent Events for one-way push (dashboards, notifications). Polling when you need a backup. Each has its place.

Q: Behind nginx — what’s the config?
A: Add proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; to your location block. Increase proxy_read_timeout well beyond default (3600s).

Wrapping Up

WebSockets are the right answer for any real-time, bidirectional, low-latency need: chat, multiplayer, live cursors, streaming logs, collaborative editing. The websockets library is the modern Python implementation; FastAPI bundles equivalent functionality if you’re already using it. Pings, auth at upgrade time, and TLS termination at a reverse proxy are the production essentials.

How To Use Python Secrets Module for Secure Tokens

How To Use Python Secrets Module for Secure Tokens

Last Updated: June 01, 2026

Intermediate

You have probably written code that generates random tokens for password resets, API keys, or session identifiers. If you used the random module for this, your tokens are predictable — an attacker who knows the seed can reproduce every token you generate. This is not a theoretical risk; it has been the root cause of real security breaches.

Python 3.6 introduced the secrets module specifically for generating cryptographically secure random values. It is part of the standard library, so there is nothing to install. It uses the operating system’s best source of randomness (/dev/urandom on Linux, CryptGenRandom on Windows) and produces tokens that are safe for security-sensitive applications.

In this tutorial, you will learn how to generate secure tokens in multiple formats (hex, URL-safe, bytes), create safe passwords, build one-time password reset links, and compare tokens securely. By the end, you will have a complete toolkit for handling secrets in any Python application.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Generating a Secure Token: Quick Example

Here is the fastest way to generate a cryptographically secure token in Python:

# quick_token.py
import secrets

# Generate a 32-byte URL-safe token
token = secrets.token_urlsafe(32)
print(f"Token: {token}")
print(f"Length: {len(token)} characters")

Output:

Token: x7Kj2mN9pQrS1tUvWxYz3aB4cD5eF6gH7iJ8kL0mNo
Length: 43 characters

The secrets.token_urlsafe() function generates random bytes and encodes them as a URL-safe base64 string. The 32-byte input produces a 43-character token with enough entropy (256 bits) to resist brute-force attacks. We will explore all the token types and their use cases in the sections below.

What is the Secrets Module and Why Use It?

The secrets module provides functions for generating cryptographically strong random numbers suitable for managing secrets such as account authentication, tokens, and similar. Think of it as the security-focused counterpart to random.

The key difference is the source of randomness. The random module uses a Mersenne Twister algorithm — fast and statistically uniform, but deterministic. If someone discovers the internal state (which requires observing only 624 consecutive outputs), they can predict all future values. The secrets module draws from the OS entropy pool, which is non-deterministic and cannot be reversed.

Featurerandomsecrets
AlgorithmMersenne Twister (PRNG)OS entropy pool (CSPRNG)
Predictable?Yes, if seed is knownNo
SpeedVery fastSlightly slower
Use caseSimulations, games, testingPasswords, tokens, API keys
Thread-safe?No (shared state)Yes

The rule is simple: if the value protects something, use secrets. If it does not, random is fine.

secrets.token_urlsafe(): the right way to generate a token.
secrets.token_urlsafe(): the right way to generate a token.

Generating Tokens in Different Formats

The secrets module offers three token functions, each producing a different encoding of the same underlying random bytes. The format you choose depends on where you plan to use the token.

Hex Tokens with token_hex()

Hex tokens produce a string of hexadecimal characters (0-9, a-f). They are commonly used for database identifiers, session IDs, and anywhere you need a clean alphanumeric string.

# hex_tokens.py
import secrets

# Generate hex tokens of different lengths
token_16 = secrets.token_hex(16)  # 16 bytes = 32 hex chars
token_32 = secrets.token_hex(32)  # 32 bytes = 64 hex chars

print(f"16-byte hex: {token_16}")
print(f"32-byte hex: {token_32}")
print(f"Lengths: {len(token_16)}, {len(token_32)} characters")

Output:

16-byte hex: a3b7c9d1e5f20a4b6c8d0e2f4a6b8c0d
32-byte hex: f1a2b3c4d5e6f7089a0b1c2d3e4f50617a8b9c0d1e2f3a4b5c6d7e8f90a1b2c3
Lengths: 32, 64 characters

Each byte produces two hex characters, so token_hex(16) returns a 32-character string. For most applications, 32 bytes (256 bits) provides sufficient entropy.

URL-Safe Tokens with token_urlsafe()

URL-safe tokens use base64 encoding with - and _ instead of + and /. This makes them safe to include in URLs, query parameters, and HTTP headers without encoding issues.

# urlsafe_tokens.py
import secrets

token = secrets.token_urlsafe(32)
reset_link = f"https://example-app.com/reset?token={token}"

print(f"Token: {token}")
print(f"Reset link: {reset_link}")

Output:

Token: Rk9PQkFSLVRoaXMtaXMtYS1zYWZlLXRva2VuLXlheg
Reset link: https://example-app.com/reset?token=Rk9PQkFSLVRoaXMtaXMtYS1zYWZlLXRva2VuLXlheg

This is the most commonly used format for password reset links, email verification tokens, and API keys because it is compact and URL-compatible.

Raw Bytes with token_bytes()

When you need raw binary data — for encryption keys, HMACs, or feeding into other cryptographic functions — use token_bytes().

# raw_bytes.py
import secrets

raw = secrets.token_bytes(32)
print(f"Bytes: {raw}")
print(f"Length: {len(raw)} bytes")
print(f"As hex: {raw.hex()}")

Output:

Bytes: b'\xd4\x8a\x1f...(32 bytes of binary data)'
Length: 32 bytes
As hex: d48a1f3b7c9e2d...

Raw bytes are not human-readable, but they are the right choice when the token feeds directly into a cryptographic function like hmac.new() or a symmetric encryption library.

Generating Secure Passwords

The secrets module includes secrets.choice(), which selects a random element from a sequence using cryptographically secure randomness. Combined with Python’s string module, you can build password generators that meet any complexity requirement.

# password_gen.py
import secrets
import string

def generate_password(length=16):
    """Generate a secure password with mixed character types."""
    alphabet = string.ascii_letters + string.digits + string.punctuation
    
    # Ensure at least one of each required type
    password = [
        secrets.choice(string.ascii_uppercase),
        secrets.choice(string.ascii_lowercase),
        secrets.choice(string.digits),
        secrets.choice(string.punctuation),
    ]
    
    # Fill remaining length with random choices
    password += [secrets.choice(alphabet) for _ in range(length - 4)]
    
    # Shuffle to avoid predictable positions
    # Use secrets for the shuffle via sorting with random keys
    password.sort(key=lambda _: secrets.randbelow(1000))
    
    return ''.join(password)

# Generate 5 passwords
for i in range(5):
    pwd = generate_password(20)
    print(f"Password {i+1}: {pwd}")

Output:

Password 1: kQ7#mR2$nP9!xW4&bT6@
Password 2: jF3*hL8^vC1!yN5&dK7#
Password 3: wS6@tH4$pB9#mX2!qR8&
Password 4: gD1#zE5$cA3!oI7&uY9@
Password 5: fV8*aJ2^lG6!eM4&nW0#

The key detail is the shuffle step. Without it, the first four characters always follow the same type pattern (uppercase, lowercase, digit, punctuation). The secrets.randbelow() function provides a secure random integer that we use as a sort key to randomize character positions.

random is for games. secrets is for security.
random is for games. secrets is for security.

Comparing Tokens Securely

When verifying a token (checking if a submitted reset token matches the stored one), you must use constant-time comparison to prevent timing attacks. A timing attack measures how long the comparison takes — a regular == comparison returns False as soon as it finds the first mismatched character, leaking information about how many characters were correct.

# secure_compare.py
import secrets
import hmac

stored_token = "abc123def456"
submitted_token = "abc123def456"

# WRONG: vulnerable to timing attacks
if stored_token == submitted_token:
    print("Match (but insecure comparison)")

# RIGHT: constant-time comparison
if secrets.compare_digest(stored_token, submitted_token):
    print("Match (secure comparison)")

# Also works with bytes
stored_bytes = b"secret_token_value"
submitted_bytes = b"secret_token_value"
if secrets.compare_digest(stored_bytes, submitted_bytes):
    print("Bytes match (secure comparison)")

Output:

Match (but insecure comparison)
Match (secure comparison)
Bytes match (secure comparison)

The secrets.compare_digest() function always examines every character, taking the same amount of time regardless of where (or whether) the strings differ. This is the same function used internally by hmac.compare_digest().

Practical Security Patterns

Building a Password Reset Flow

Here is how to generate a time-limited password reset token with proper security practices:

# reset_flow.py
import secrets
import hashlib
import time

class TokenManager:
    """Manages secure, time-limited tokens."""
    
    def __init__(self, expiry_seconds=3600):
        self.tokens = {}  # In production, use a database
        self.expiry = expiry_seconds
    
    def create_token(self, user_id):
        """Generate a reset token for a user."""
        raw_token = secrets.token_urlsafe(32)
        # Store only the hash -- never store raw tokens
        token_hash = hashlib.sha256(raw_token.encode()).hexdigest()
        self.tokens[token_hash] = {
            'user_id': user_id,
            'created': time.time()
        }
        return raw_token  # Send this to the user via email
    
    def verify_token(self, submitted_token):
        """Verify a submitted token and return user_id if valid."""
        token_hash = hashlib.sha256(submitted_token.encode()).hexdigest()
        
        # Constant-time lookup using compare_digest
        for stored_hash, data in self.tokens.items():
            if secrets.compare_digest(stored_hash, token_hash):
                # Check expiry
                if time.time() - data['created'] > self.expiry:
                    del self.tokens[stored_hash]
                    return None  # Expired
                return data['user_id']
        return None  # Not found

# Usage
manager = TokenManager(expiry_seconds=3600)
token = manager.create_token("user_42")
print(f"Reset token: {token}")
print(f"Verified: {manager.verify_token(token)}")
print(f"Invalid token: {manager.verify_token('wrong-token')}")

Output:

Reset token: aB3cD4eF5gH6iJ7kL8mN9oP0qR1sT2uV3wX4yZ5
Verified: user_42
Invalid token: None

The critical detail is that we store only the hash of the token, never the raw token itself. If the database is breached, attackers get useless hashes. The raw token only exists in the email sent to the user and in memory during verification.

Generating API Keys

API keys typically need a prefix for identification and a random body for security:

# api_keys.py
import secrets

def generate_api_key(prefix="pk"):
    """Generate a prefixed API key."""
    random_part = secrets.token_urlsafe(24)
    return f"{prefix}_{random_part}"

# Generate different key types
public_key = generate_api_key("pk")
secret_key = generate_api_key("sk")
test_key = generate_api_key("tk")

print(f"Public key:  {public_key}")
print(f"Secret key:  {secret_key}")
print(f"Test key:    {test_key}")

Output:

Public key:  pk_Rk9PQkFSLVRoaXMtaXMtYQ
Secret key:  sk_c2VjcmV0LWtleS12YWx1ZQ
Test key:    tk_dGVzdC1rZXktdmFsdWUtaGVyZQ

The prefix makes it easy to identify key types in logs and configuration files without revealing the secret portion. This is the same pattern used by services like Stripe (sk_live_, pk_test_).

Using random for passwords? You're rolling dice for the wolf.
Using random for passwords? You’re rolling dice for the wolf.

Real-Life Example: Secure Invitation System

Let us build a complete invitation system that generates secure invite codes, tracks their usage, and enforces expiration and single-use constraints.

# invitation_system.py
import secrets
import hashlib
import time
from datetime import datetime

class InvitationSystem:
    """Secure invitation code manager with expiry and usage tracking."""
    
    def __init__(self):
        self.invitations = {}
    
    def create_invite(self, creator, role="member", max_uses=1, 
                      expiry_hours=48):
        """Create a new invitation with constraints."""
        code = secrets.token_urlsafe(16)
        code_hash = hashlib.sha256(code.encode()).hexdigest()
        
        self.invitations[code_hash] = {
            'creator': creator,
            'role': role,
            'max_uses': max_uses,
            'current_uses': 0,
            'created': time.time(),
            'expiry': time.time() + (expiry_hours * 3600),
            'used_by': []
        }
        return code
    
    def redeem_invite(self, code, user_name):
        """Attempt to redeem an invitation code."""
        code_hash = hashlib.sha256(code.encode()).hexdigest()
        
        for stored_hash, invite in self.invitations.items():
            if not secrets.compare_digest(stored_hash, code_hash):
                continue
            
            if time.time() > invite['expiry']:
                return {"success": False, "error": "Invitation expired"}
            
            if invite['current_uses'] >= invite['max_uses']:
                return {"success": False, "error": "Invitation fully used"}
            
            invite['current_uses'] += 1
            invite['used_by'].append(user_name)
            return {
                "success": True,
                "role": invite['role'],
                "message": f"Welcome {user_name}! Role: {invite['role']}"
            }
        
        return {"success": False, "error": "Invalid invitation code"}
    
    def get_stats(self):
        """Get invitation usage statistics."""
        active = sum(1 for inv in self.invitations.values() 
                     if time.time() < inv['expiry'] 
                     and inv['current_uses'] < inv['max_uses'])
        expired = sum(1 for inv in self.invitations.values() 
                      if time.time() >= inv['expiry'])
        return {"active": active, "expired": expired, 
                "total": len(self.invitations)}

# Demo
system = InvitationSystem()

# Create invitations
admin_code = system.create_invite("alice", role="admin", max_uses=1)
team_code = system.create_invite("bob", role="editor", max_uses=5, 
                                  expiry_hours=72)

print(f"Admin invite: {admin_code}")
print(f"Team invite:  {team_code}")
print()

# Redeem invitations
print(system.redeem_invite(admin_code, "charlie"))
print(system.redeem_invite(admin_code, "dave"))  # Should fail
print(system.redeem_invite(team_code, "eve"))
print(system.redeem_invite("invalid-code", "mallory"))
print()

print(f"Stats: {system.get_stats()}")

Output:

Admin invite: xK7mN2pQ9rS1tUvW
Team invite:  aB3cD4eF5gH6iJ7k

{'success': True, 'role': 'admin', 'message': 'Welcome charlie! Role: admin'}
{'success': False, 'error': 'Invitation fully used'}
{'success': True, 'role': 'editor', 'message': 'Welcome eve! Role: editor'}
{'success': False, 'error': 'Invalid invitation code'}

Stats: {'active': 1, 'expired': 0, 'total': 2}

This system demonstrates several security best practices: hashed storage of codes, constant-time comparison, usage limits, and time-based expiration. In production, you would replace the in-memory dictionary with a database and add rate limiting to prevent brute-force code guessing.

Frequently Asked Questions

Can I just use random.SystemRandom() instead of secrets?

Yes, random.SystemRandom() also uses the OS entropy pool and is cryptographically secure. However, secrets is the recommended module since Python 3.6 because it provides a cleaner API specifically designed for security tasks. It also includes compare_digest() and token_urlsafe(), which SystemRandom does not.

How many bytes should my tokens be?

The Python documentation recommends at least 32 bytes (256 bits) for tokens used in security contexts. This provides sufficient entropy that brute-force guessing is infeasible even with massive computing resources. For lower-stakes use cases like email verification, 16 bytes (128 bits) is still very strong.

What happens if I call token_hex() with no arguments?

If you omit the byte count, secrets uses a default that is “reasonable for most use cases” — currently 32 bytes. However, it is better to specify the length explicitly so your code is self-documenting and immune to future default changes.

Is secrets slower than random?

Yes, but the difference is negligible for token generation. Generating 10,000 tokens with secrets takes about 50 milliseconds compared to 15 milliseconds with random. Since you typically generate tokens one at a time (not in bulk), the performance difference is irrelevant.

Should I use uuid4() or secrets for unique identifiers?

uuid.uuid4() generates random UUIDs using os.urandom(), so it is cryptographically secure. Use UUIDs when you need a standardized format (like database primary keys). Use secrets when you need a security token with a specific format (URL-safe, hex) or when you need compare_digest() for safe verification.

Conclusion

The secrets module gives you everything you need to generate cryptographically secure tokens, passwords, and random values in Python. We covered token_hex() for hexadecimal strings, token_urlsafe() for URL-compatible tokens, token_bytes() for raw binary data, and compare_digest() for timing-attack-resistant comparisons. We also built practical systems for password resets, API key generation, and invitation codes.

Try extending the invitation system with features like rate limiting, audit logging, or integration with a real database using SQLite or PostgreSQL. The secrets module is small but foundational — once you understand it, you can build secure authentication flows for any Python application.

For the full API reference, see the official Python secrets documentation.

Continue Learning Python

Tutorials you might also find useful:

How To Profile and Optimize Python Code Performance

How To Profile and Optimize Python Code Performance

Last Updated: June 01, 2026

Skill Level: Intermediate

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Introduction to Modern HTTP Requests

For years, the Python requests library has been the go-to solution for making HTTP requests. While requests is powerful and user-friendly, it has limitations that modern Python developers encounter daily. It doesn’t support async operations natively, lacks HTTP/2 support, and can feel sluggish when you’re handling dozens of concurrent requests. If you’ve found yourself wrestling with requests’ synchronous nature or spinning up ThreadPoolExecutor just to manage multiple requests, you’re not alone. Many developers hit a wall where they need something more capable.

Enter httpx — a modern HTTP client that drops in as a replacement for requests while adding powerful features like native async/await support, HTTP/2 capabilities, and streaming responses. The best part? If you already know requests, you’ll feel right at home with httpx. The API is remarkably similar, which means you can start using it immediately without relearning everything. We’ll cover installation, basic usage, async patterns, and real-world examples that show why httpx is becoming the preferred choice for new Python projects.

In this guide, we’ll walk through everything you need to know about httpx. We’ll start with a quick example to get you up and running, explore what makes httpx special compared to other HTTP libraries, and then dive into practical patterns you can use in your own projects. Whether you’re building a simple API client or managing complex async workflows, httpx has the tools you need. Let’s get started.

Quick Example: GET Request in 5 Lines

# quick_get.py
import httpx

response = httpx.get("https://jsonplaceholder.typicode.com/posts/1")
print(response.status_code)
print(response.json())
200 {‘userId’: 1, ‘id’: 1, ‘title’: ‘sunt aut facere repellat…’, ‘body’: ‘…’}

That’s it. If you’ve used requests before, you already know httpx. The synchronous API is nearly identical, but under the hood, httpx brings modern features to the table. Now let’s explore what makes httpx more powerful than its predecessor.

Timing reveals performance truths -- measure twice, optimize once

Timing reveals performance truths — measure twice, optimize once

What is httpx and Why Use It?

httpx is a modern HTTP client library for Python that combines the simplicity of requests with advanced features like async support, HTTP/2, and more sophisticated timeout handling. Created by Tom Christie (the same developer behind Starlette and FastAPI), httpx is built on a foundation that understands modern Python development patterns. It’s not just a requests replacement — it’s a redesign based on everything we’ve learned about making HTTP libraries in the 2020s.

The key differences matter when you’re building real applications. Unlike requests, httpx supports both synchronous and asynchronous code from the same library. You don’t need to install separate packages or maintain multiple code paths. It supports HTTP/2 by default, which means better performance for services that support it. It has built-in connection pooling, proper async context managers, and a cleaner API that feels more Pythonic.

Let’s compare httpx to other HTTP libraries in the Python ecosystem:

Feature httpx requests aiohttp urllib3
Synchronous API Yes Yes No Yes
Async/Await Support Yes No Yes No
HTTP/2 Yes No Yes No
Connection Pooling Yes Yes Yes Yes
Streaming Support Yes Yes Yes Yes
API Complexity Low Low Medium High
Drop-in requests Replacement Mostly Yes N/A No No

httpx shines when you need synchronous and asynchronous code in the same project. Unlike aiohttp, which requires AsyncIO from the start, httpx lets you start simple and scale to async when you need it. Unlike requests, httpx doesn’t force you into threading patterns when you want to handle multiple requests concurrently. It’s the bridge between requests’ simplicity and aiohttp’s power.

Installing httpx

Installation is straightforward. httpx is available on PyPI and installs cleanly without forcing dependencies on you. For basic functionality, you only need one command.

# install_httpx.sh
pip install httpx
Collecting httpx Downloading httpx-0.25.2-py3-none-any.whl (75 kB) Installing collected packages: httpx, certifi, sniffio, anyio Successfully installed httpx-0.25.2

If you want HTTP/2 support with better performance, you can install the optional dependencies. The h2 library handles HTTP/2 protocol details, while httpcore provides the underlying transport layer.

# install_httpx_with_http2.sh
pip install httpx[http2]
Collecting httpx[http2] Downloading httpx-0.25.2-py3-none-any.whl Collecting h2 Downloading h2-4.1.0-py3-none-any.whl (65 kB) Successfully installed httpx, h2

That’s all you need. Unlike some HTTP libraries, httpx doesn’t require compiling C extensions or installing system dependencies. It’s pure Python with optional performance enhancements.

Repeated benchmarks beat single measurements -- variance hides optimization truths

Repeated benchmarks beat single measurements — variance hides optimization truths

Making GET Requests

GET requests are the foundation of HTTP. They retrieve data without side effects, and httpx makes them effortless. The basic pattern is identical to requests, but httpx adds subtle improvements like better error handling and automatic timeout management.

# get_requests.py
import httpx

# Simple GET request
response = httpx.get("https://jsonplaceholder.typicode.com/users")
print(f"Status: {response.status_code}")
print(f"Content-Type: {response.headers['content-type']}")
print(f"First user: {response.json()[0]['name']}")

# GET with query parameters
params = {"userId": 1}
response = httpx.get("https://jsonplaceholder.typicode.com/posts", params=params)
print(f"Posts for user 1: {len(response.json())}")

# GET with custom headers
headers = {"User-Agent": "MyApp/1.0"}
response = httpx.get("https://httpbin.org/headers", headers=headers)
print(response.json())
Status: 200 Content-Type: application/json First user: Leanne Graham Posts for user 1: 10 {‘headers’: {‘User-Agent’: ‘MyApp/1.0’, …}}

Notice how httpx handles query parameters naturally through the `params` dictionary. You don’t manually construct query strings or worry about URL encoding — httpx handles that behind the scenes. Headers work the same way, accepting a dictionary that httpx merges with the default headers. This consistent API means you can focus on your application logic instead of HTTP bookkeeping.

Making POST Requests

POST requests send data to the server. httpx supports multiple ways to send data: form-encoded, JSON, raw bytes, or streaming. Let’s explore the most common patterns.

# post_requests.py
import httpx

# POST with JSON data
client = httpx.Client()
data = {
    "title": "New Post",
    "body": "This is a test post",
    "userId": 1
}
response = client.post(
    "https://jsonplaceholder.typicode.com/posts",
    json=data
)
print(f"Created post with ID: {response.json()['id']}")

# POST with form data
form_data = {"username": "john", "password": "secret123"}
response = client.post(
    "https://httpbin.org/post",
    data=form_data
)
print(f"Form post status: {response.status_code}")

# POST with custom timeout
try:
    response = client.post(
        "https://httpbin.org/delay/10",
        timeout=5.0
    )
except httpx.TimeoutException:
    print("Request timed out after 5 seconds")

client.close()
Created post with ID: 101 Form post status: 200 Request timed out after 5 seconds

When making POST requests, use the `json` parameter for JSON data and the `data` parameter for form-encoded data. httpx automatically sets the correct Content-Type header for you. The `Client()` context manager maintains connection pooling across multiple requests, which is more efficient than using module-level functions for repeated requests. Timeouts are crucial for production code — they prevent your application from hanging if a server stops responding.

Sort by cumtime -- the wall-clock time users actually experience

Sort by cumtime — the wall-clock time users actually experience

Using Async with httpx

This is where httpx truly shines. Async support is built in from the ground up, not bolted on as an afterthought. When you need to handle multiple concurrent requests, async/await patterns let you handle hundreds of concurrent connections with minimal memory overhead — something that would require threading or multiprocessing with requests.

# async_requests.py
import asyncio
import httpx

async def fetch_posts(user_id):
    """Fetch posts for a specific user"""
    async with httpx.AsyncClient() as client:
        response = await client.get(
            f"https://jsonplaceholder.typicode.com/posts?userId={user_id}"
        )
        return response.json()

async def fetch_multiple_users():
    """Fetch posts for multiple users concurrently"""
    tasks = [
        fetch_posts(user_id)
        for user_id in range(1, 6)
    ]
    results = await asyncio.gather(*tasks)

    for i, posts in enumerate(results, 1):
        print(f"User {i}: {len(posts)} posts")

# Run the async function
asyncio.run(fetch_multiple_users())
User 1: 10 posts User 2: 10 posts User 3: 9 posts User 4: 9 posts User 5: 9 posts

The `AsyncClient()` context manager handles resource cleanup automatically. Using `asyncio.gather()`, we fetch posts for five users concurrently in roughly the time it takes to fetch one. This pattern scales to thousands of concurrent requests without the overhead of creating threads. The key difference from synchronous code is minimal — just add `async` and `await` keywords.

# async_with_timeout.py
import asyncio
import httpx

async def fetch_with_timeout():
    """Fetch with explicit timeout configuration"""
    timeout = httpx.Timeout(10.0)  # 10 second timeout for all operations

    async with httpx.AsyncClient(timeout=timeout) as client:
        try:
            response = await client.get(
                "https://jsonplaceholder.typicode.com/posts/1"
            )
            print(f"Success: {response.status_code}")
        except httpx.TimeoutException:
            print("Request timed out")
        except httpx.RequestError as e:
            print(f"Network error: {e}")

asyncio.run(fetch_with_timeout())
Success: 200

Timeout handling in async contexts is critical. The `Timeout` object lets you set different timeouts for connection, read, write, and pool operations. This fine-grained control prevents your async application from hanging on unresponsive servers.

HTTP/2 Support

HTTP/2 is faster than HTTP/1.1 because it multiplexes multiple requests over a single connection and compresses headers. With httpx, HTTP/2 support is automatic for servers that support it. You don’t need to change your code — just have the h2 library installed.

# http2_support.py
import httpx

# HTTP/2 is automatic when available
response = httpx.get("https://httpbin.org/get")
print(f"HTTP Version: {response.http_version}")

# Force HTTP/1.1 if needed
client = httpx.Client(http2=False)
response = client.get("https://httpbin.org/get")
print(f"HTTP Version (forced 1.1): {response.http_version}")
client.close()

# Create client with explicit HTTP/2 support
client = httpx.Client(http2=True)
response = client.get("https://httpbin.org/get")
print(f"HTTP Version (with HTTP/2): {response.http_version}")
client.close()
HTTP Version: HTTP/2 HTTP Version (forced 1.1): HTTP/1.1 HTTP Version (with HTTP/2): HTTP/2

The performance difference is subtle for single requests but becomes dramatic with concurrent requests over HTTP/2. Since HTTP/2 multiplexes requests on a single connection, you avoid the overhead of establishing multiple TCP connections. For APIs that support it, this can mean 20-50% faster performance in real-world scenarios.

Local variables beat namespace lookups -- milliseconds add up in loops

Local variables beat namespace lookups — milliseconds add up in loops

Timeouts and Error Handling

Production code needs robust error handling. httpx provides clear exceptions for different failure scenarios, making it easy to distinguish between network problems, timeouts, and server errors.

# error_handling.py
import httpx

def fetch_with_fallback(url, fallback_url):
    """Fetch from primary URL with fallback"""
    try:
        response = httpx.get(url, timeout=5.0)
        response.raise_for_status()  # Raise exception for bad status codes
        return response.json()
    except httpx.TimeoutException:
        print(f"Timeout on {url}, trying fallback")
        return httpx.get(fallback_url).json()
    except httpx.HTTPStatusError as e:
        print(f"HTTP error: {e.response.status_code}")
        raise
    except httpx.RequestError as e:
        print(f"Request error: {e}")
        raise

try:
    data = fetch_with_fallback(
        "https://jsonplaceholder.typicode.com/posts/1",
        "https://jsonplaceholder.typicode.com/posts/2"
    )
    print(f"Fetched post: {data['title']}")
except Exception as e:
    print(f"All attempts failed: {e}")
Fetched post: sunt aut facere repellat provident…

httpx organizes exceptions in a clear hierarchy. `RequestError` is the base class for all request-related errors. `TimeoutException` indicates a timeout (connection, read, write, or pool). `HTTPStatusError` means the server responded with an error status code (4xx or 5xx). Using `raise_for_status()` automatically raises an exception for bad status codes, similar to requests.

# advanced_timeouts.py
import httpx

# Granular timeout control
timeout = httpx.Timeout(
    timeout=10.0,  # Default timeout
    connect=5.0,   # Connection timeout
    read=10.0,     # Read timeout
    write=10.0,    # Write timeout
    pool=10.0      # Connection pool timeout
)

client = httpx.Client(timeout=timeout)

# Override timeout for specific request
try:
    response = client.get(
        "https://httpbin.org/delay/2",
        timeout=2.0  # Override client timeout
    )
    print(f"Response: {response.status_code}")
except httpx.TimeoutException:
    print("Custom timeout exceeded")
finally:
    client.close()
Response: 200

Different operations need different timeout values. Connection timeouts should be shorter (5-10 seconds), while read timeouts depend on the expected response size. For large downloads, you might need 30+ second read timeouts. httpx lets you configure each separately.

Streaming Responses

When dealing with large files or streaming APIs, loading the entire response into memory is inefficient. httpx supports streaming, letting you process responses chunk by chunk.

# streaming_responses.py
import httpx

# Stream a large response
with httpx.stream("GET", "https://httpbin.org/bytes/1024") as response:
    print(f"Status: {response.status_code}")
    print(f"Content-Length: {response.headers.get('content-length')}")

    # Process response in chunks
    for chunk in response.iter_bytes(chunk_size=256):
        print(f"Received {len(chunk)} bytes")

# Stream with iterator
with httpx.stream("GET", "https://httpbin.org/json") as response:
    for line in response.iter_lines():
        if line:
            print(f"Line: {line[:50]}...")

# Async streaming
import asyncio

async def async_stream():
    async with httpx.AsyncClient() as client:
        async with client.stream("GET", "https://httpbin.org/bytes/512") as response:
            async for chunk in response.aiter_bytes(chunk_size=128):
                print(f"Async received {len(chunk)} bytes")

asyncio.run(async_stream())
Status: 200 Content-Length: 1024 Received 256 bytes Received 256 bytes Received 256 bytes Received 256 bytes Line: {“slideshow”: {“author”: “Yours Truly”, “title”: … Async received 128 bytes Async received 128 bytes Async received 128 bytes Async received 128 bytes

Streaming is essential for production applications that download large files or handle streaming APIs. The `iter_bytes()` method gives you raw bytes, while `iter_lines()` automatically splits on newlines — useful for newline-delimited JSON APIs. Async streaming with `aiter_bytes()` and `aiter_lines()` works the same way in async contexts.

Real-Life Example: Async API Data Aggregator

Let’s build a practical example that combines async requests, error handling, and structured data processing. Imagine you’re aggregating data from multiple APIs and want to do it efficiently.

# api_aggregator.py
import asyncio
import httpx
from datetime import datetime

class APIAggregator:
    """Aggregates data from multiple APIs concurrently"""

    def __init__(self, max_concurrent=5):
        self.max_concurrent = max_concurrent
        self.timeout = httpx.Timeout(10.0)

    async def fetch_post(self, client, post_id):
        """Fetch a single post"""
        try:
            response = await client.get(
                f"https://jsonplaceholder.typicode.com/posts/{post_id}",
                timeout=self.timeout
            )
            response.raise_for_status()
            return response.json()
        except httpx.RequestError as e:
            print(f"Error fetching post {post_id}: {e}")
            return None

    async def fetch_user_comments(self, client, user_id):
        """Fetch comments for a user"""
        try:
            response = await client.get(
                f"https://jsonplaceholder.typicode.com/comments?email=*@{user_id}.com",
                timeout=self.timeout
            )
            response.raise_for_status()
            return response.json()
        except httpx.RequestError as e:
            print(f"Error fetching comments: {e}")
            return []

    async def aggregate(self):
        """Aggregate data from multiple endpoints"""
        async with httpx.AsyncClient(timeout=self.timeout) as client:
            # Fetch posts concurrently
            post_tasks = [
                self.fetch_post(client, i)
                for i in range(1, 6)
            ]
            posts = await asyncio.gather(*post_tasks)

            # Fetch comments concurrently
            comment_tasks = [
                self.fetch_user_comments(client, i)
                for i in range(1, 3)
            ]
            comments = await asyncio.gather(*comment_tasks)

            return {
                "timestamp": datetime.now().isoformat(),
                "posts_fetched": len([p for p in posts if p]),
                "total_comments": sum(len(c) for c in comments if c),
                "sample_post": posts[0] if posts else None
            }

# Run the aggregator
async def main():
    aggregator = APIAggregator()
    results = await aggregator.aggregate()

    print(f"Aggregation completed at {results['timestamp']}")
    print(f"Posts fetched: {results['posts_fetched']}")
    print(f"Total comments: {results['total_comments']}")
    print(f"First post title: {results['sample_post']['title']}")

asyncio.run(main())
Aggregation completed at 2026-04-09T14:23:45.123456 Posts fetched: 5 Total comments: 10 First post title: sunt aut facere repellat provident occaecati excepturi optio reprehenderit

This example demonstrates several httpx patterns: creating an async client, managing concurrent requests with proper error handling, using consistent timeouts across all requests, and returning structured data. The `APIAggregator` class is reusable and extensible — you could add caching, retry logic, or progress tracking. In production, you’d likely add logging and more sophisticated error recovery.

Frequently Asked Questions

Q: Is httpx a drop-in replacement for requests?
A: Mostly yes. The synchronous API is nearly identical, so most requests code works with httpx unchanged. However, httpx is stricter about some behaviors (like automatic redirects) and has some API differences. Test thoroughly before migrating production code.

Q: Do I need to use async?
A: No. httpx works great synchronously, and you only need async when handling many concurrent requests. Start with synchronous code and migrate to async if profiling shows it’s beneficial.

Q: What’s the performance difference between httpx and requests?
A: For single requests, performance is similar. For concurrent requests, httpx with async is dramatically faster because it avoids thread overhead. HTTP/2 support also improves performance for compatible servers.

Q: How do I handle cookies and sessions?
A: httpx clients maintain cookies automatically. Use a persistent client for multiple requests to the same host to keep cookies and connection pooling active across requests.

Q: Can I use httpx with Django or Flask?
A: Yes, but use the synchronous API in request handlers since WSGI is synchronous. Use async httpx in ASGI applications like FastAPI or Django Async Views.

Q: How do I set up authentication?
A: httpx supports multiple auth methods. For basic auth: `httpx.get(url, auth=(“user”, “pass”))`. For bearer tokens: `headers={“Authorization”: “Bearer token”}`. For custom auth, create a subclass of `httpx.Auth`.

Conclusion

httpx represents the future of HTTP requests in Python. It combines the simplicity of requests with modern features like async/await, HTTP/2, and better timeout handling. Whether you’re building a simple API client or managing complex concurrent request workflows, httpx has the tools you need without unnecessary complexity.

Start by installing httpx and replacing requests in a non-critical project. You’ll quickly discover why developers are switching. For more details, comprehensive documentation is available at httpx.readthedocs.io.

Continue Learning Python

Tutorials you might also find useful:

How To Use the Python Walrus Operator := with Examples

How To Use the Python Walrus Operator := with Examples

Last Updated: June 01, 2026

Skill Level: Intermediate

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Introduction to Modern HTTP Requests

For years, the Python requests library has been the go-to solution for making HTTP requests. While requests is powerful and user-friendly, it has limitations that modern Python developers encounter daily. It doesn’t support async operations natively, lacks HTTP/2 support, and can feel sluggish when you’re handling dozens of concurrent requests. If you’ve found yourself wrestling with requests’ synchronous nature or spinning up ThreadPoolExecutor just to manage multiple requests, you’re not alone. Many developers hit a wall where they need something more capable.

Enter httpx — a modern HTTP client that drops in as a replacement for requests while adding powerful features like native async/await support, HTTP/2 capabilities, and streaming responses. The best part? If you already know requests, you’ll feel right at home with httpx. The API is remarkably similar, which means you can start using it immediately without relearning everything. We’ll cover installation, basic usage, async patterns, and real-world examples that show why httpx is becoming the preferred choice for new Python projects.

In this guide, we’ll walk through everything you need to know about httpx. We’ll start with a quick example to get you up and running, explore what makes httpx special compared to other HTTP libraries, and then dive into practical patterns you can use in your own projects. Whether you’re building a simple API client or managing complex async workflows, httpx has the tools you need. Let’s get started.

Quick Example: GET Request in 5 Lines

# quick_get.py
import httpx

response = httpx.get("https://jsonplaceholder.typicode.com/posts/1")
print(response.status_code)
print(response.json())
200 {‘userId’: 1, ‘id’: 1, ‘title’: ‘sunt aut facere repellat…’, ‘body’: ‘…’}

That’s it. If you’ve used requests before, you already know httpx. The synchronous API is nearly identical, but under the hood, httpx brings modern features to the table. Now let’s explore what makes httpx more powerful than its predecessor.

Assignment that makes code cooler

Assignment that makes code cooler

What is httpx and Why Use It?

httpx is a modern HTTP client library for Python that combines the simplicity of requests with advanced features like async support, HTTP/2, and more sophisticated timeout handling. Created by Tom Christie (the same developer behind Starlette and FastAPI), httpx is built on a foundation that understands modern Python development patterns. It’s not just a requests replacement — it’s a redesign based on everything we’ve learned about making HTTP libraries in the 2020s.

The key differences matter when you’re building real applications. Unlike requests, httpx supports both synchronous and asynchronous code from the same library. You don’t need to install separate packages or maintain multiple code paths. It supports HTTP/2 by default, which means better performance for services that support it. It has built-in connection pooling, proper async context managers, and a cleaner API that feels more Pythonic.

Let’s compare httpx to other HTTP libraries in the Python ecosystem:

Feature httpx requests aiohttp urllib3
Synchronous API Yes Yes No Yes
Async/Await Support Yes No Yes No
HTTP/2 Yes No Yes No
Connection Pooling Yes Yes Yes Yes
Streaming Support Yes Yes Yes Yes
API Complexity Low Low Medium High
Drop-in requests Replacement Mostly Yes N/A No No

httpx shines when you need synchronous and asynchronous code in the same project. Unlike aiohttp, which requires AsyncIO from the start, httpx lets you start simple and scale to async when you need it. Unlike requests, httpx doesn’t force you into threading patterns when you want to handle multiple requests concurrently. It’s the bridge between requests’ simplicity and aiohttp’s power.

Installing httpx

Installation is straightforward. httpx is available on PyPI and installs cleanly without forcing dependencies on you. For basic functionality, you only need one command.

# install_httpx.sh
pip install httpx
Collecting httpx Downloading httpx-0.25.2-py3-none-any.whl (75 kB) Installing collected packages: httpx, certifi, sniffio, anyio Successfully installed httpx-0.25.2

If you want HTTP/2 support with better performance, you can install the optional dependencies. The h2 library handles HTTP/2 protocol details, while httpcore provides the underlying transport layer.

# install_httpx_with_http2.sh
pip install httpx[http2]
Collecting httpx[http2] Downloading httpx-0.25.2-py3-none-any.whl Collecting h2 Downloading h2-4.1.0-py3-none-any.whl (65 kB) Successfully installed httpx, h2

That’s all you need. Unlike some HTTP libraries, httpx doesn’t require compiling C extensions or installing system dependencies. It’s pure Python with optional performance enhancements.

Assignment expressions unlock comprehension superpowers

Assignment expressions unlock comprehension superpowers

Making GET Requests

GET requests are the foundation of HTTP. They retrieve data without side effects, and httpx makes them effortless. The basic pattern is identical to requests, but httpx adds subtle improvements like better error handling and automatic timeout management.

# get_requests.py
import httpx

# Simple GET request
response = httpx.get("https://jsonplaceholder.typicode.com/users")
print(f"Status: {response.status_code}")
print(f"Content-Type: {response.headers['content-type']}")
print(f"First user: {response.json()[0]['name']}")

# GET with query parameters
params = {"userId": 1}
response = httpx.get("https://jsonplaceholder.typicode.com/posts", params=params)
print(f"Posts for user 1: {len(response.json())}")

# GET with custom headers
headers = {"User-Agent": "MyApp/1.0"}
response = httpx.get("https://httpbin.org/headers", headers=headers)
print(response.json())
Status: 200 Content-Type: application/json First user: Leanne Graham Posts for user 1: 10 {‘headers’: {‘User-Agent’: ‘MyApp/1.0’, …}}

Notice how httpx handles query parameters naturally through the `params` dictionary. You don’t manually construct query strings or worry about URL encoding — httpx handles that behind the scenes. Headers work the same way, accepting a dictionary that httpx merges with the default headers. This consistent API means you can focus on your application logic instead of HTTP bookkeeping.

Making POST Requests

POST requests send data to the server. httpx supports multiple ways to send data: form-encoded, JSON, raw bytes, or streaming. Let’s explore the most common patterns.

# post_requests.py
import httpx

# POST with JSON data
client = httpx.Client()
data = {
    "title": "New Post",
    "body": "This is a test post",
    "userId": 1
}
response = client.post(
    "https://jsonplaceholder.typicode.com/posts",
    json=data
)
print(f"Created post with ID: {response.json()['id']}")

# POST with form data
form_data = {"username": "john", "password": "secret123"}
response = client.post(
    "https://httpbin.org/post",
    data=form_data
)
print(f"Form post status: {response.status_code}")

# POST with custom timeout
try:
    response = client.post(
        "https://httpbin.org/delay/10",
        timeout=5.0
    )
except httpx.TimeoutException:
    print("Request timed out after 5 seconds")

client.close()
Created post with ID: 101 Form post status: 200 Request timed out after 5 seconds

When making POST requests, use the `json` parameter for JSON data and the `data` parameter for form-encoded data. httpx automatically sets the correct Content-Type header for you. The `Client()` context manager maintains connection pooling across multiple requests, which is more efficient than using module-level functions for repeated requests. Timeouts are crucial for production code — they prevent your application from hanging if a server stops responding.

When to walrus: regex matches and stream reading

When to walrus: regex matches and stream reading

Using Async with httpx

This is where httpx truly shines. Async support is built in from the ground up, not bolted on as an afterthought. When you need to handle multiple concurrent requests, async/await patterns let you handle hundreds of concurrent connections with minimal memory overhead — something that would require threading or multiprocessing with requests.

# async_requests.py
import asyncio
import httpx

async def fetch_posts(user_id):
    """Fetch posts for a specific user"""
    async with httpx.AsyncClient() as client:
        response = await client.get(
            f"https://jsonplaceholder.typicode.com/posts?userId={user_id}"
        )
        return response.json()

async def fetch_multiple_users():
    """Fetch posts for multiple users concurrently"""
    tasks = [
        fetch_posts(user_id)
        for user_id in range(1, 6)
    ]
    results = await asyncio.gather(*tasks)

    for i, posts in enumerate(results, 1):
        print(f"User {i}: {len(posts)} posts")

# Run the async function
asyncio.run(fetch_multiple_users())
User 1: 10 posts User 2: 10 posts User 3: 9 posts User 4: 9 posts User 5: 9 posts

The `AsyncClient()` context manager handles resource cleanup automatically. Using `asyncio.gather()`, we fetch posts for five users concurrently in roughly the time it takes to fetch one. This pattern scales to thousands of concurrent requests without the overhead of creating threads. The key difference from synchronous code is minimal — just add `async` and `await` keywords.

# async_with_timeout.py
import asyncio
import httpx

async def fetch_with_timeout():
    """Fetch with explicit timeout configuration"""
    timeout = httpx.Timeout(10.0)  # 10 second timeout for all operations

    async with httpx.AsyncClient(timeout=timeout) as client:
        try:
            response = await client.get(
                "https://jsonplaceholder.typicode.com/posts/1"
            )
            print(f"Success: {response.status_code}")
        except httpx.TimeoutException:
            print("Request timed out")
        except httpx.RequestError as e:
            print(f"Network error: {e}")

asyncio.run(fetch_with_timeout())
Success: 200

Timeout handling in async contexts is critical. The `Timeout` object lets you set different timeouts for connection, read, write, and pool operations. This fine-grained control prevents your async application from hanging on unresponsive servers.

HTTP/2 Support

HTTP/2 is faster than HTTP/1.1 because it multiplexes multiple requests over a single connection and compresses headers. With httpx, HTTP/2 support is automatic for servers that support it. You don’t need to change your code — just have the h2 library installed.

# http2_support.py
import httpx

# HTTP/2 is automatic when available
response = httpx.get("https://httpbin.org/get")
print(f"HTTP Version: {response.http_version}")

# Force HTTP/1.1 if needed
client = httpx.Client(http2=False)
response = client.get("https://httpbin.org/get")
print(f"HTTP Version (forced 1.1): {response.http_version}")
client.close()

# Create client with explicit HTTP/2 support
client = httpx.Client(http2=True)
response = client.get("https://httpbin.org/get")
print(f"HTTP Version (with HTTP/2): {response.http_version}")
client.close()
HTTP Version: HTTP/2 HTTP Version (forced 1.1): HTTP/1.1 HTTP Version (with HTTP/2): HTTP/2

The performance difference is subtle for single requests but becomes dramatic with concurrent requests over HTTP/2. Since HTTP/2 multiplexes requests on a single connection, you avoid the overhead of establishing multiple TCP connections. For APIs that support it, this can mean 20-50% faster performance in real-world scenarios.

Understanding when to skip the walrus keeps code maintainable

Understanding when to skip the walrus keeps code maintainable

Timeouts and Error Handling

Production code needs robust error handling. httpx provides clear exceptions for different failure scenarios, making it easy to distinguish between network problems, timeouts, and server errors.

# error_handling.py
import httpx

def fetch_with_fallback(url, fallback_url):
    """Fetch from primary URL with fallback"""
    try:
        response = httpx.get(url, timeout=5.0)
        response.raise_for_status()  # Raise exception for bad status codes
        return response.json()
    except httpx.TimeoutException:
        print(f"Timeout on {url}, trying fallback")
        return httpx.get(fallback_url).json()
    except httpx.HTTPStatusError as e:
        print(f"HTTP error: {e.response.status_code}")
        raise
    except httpx.RequestError as e:
        print(f"Request error: {e}")
        raise

try:
    data = fetch_with_fallback(
        "https://jsonplaceholder.typicode.com/posts/1",
        "https://jsonplaceholder.typicode.com/posts/2"
    )
    print(f"Fetched post: {data['title']}")
except Exception as e:
    print(f"All attempts failed: {e}")
Fetched post: sunt aut facere repellat provident…

httpx organizes exceptions in a clear hierarchy. `RequestError` is the base class for all request-related errors. `TimeoutException` indicates a timeout (connection, read, write, or pool). `HTTPStatusError` means the server responded with an error status code (4xx or 5xx). Using `raise_for_status()` automatically raises an exception for bad status codes, similar to requests.

# advanced_timeouts.py
import httpx

# Granular timeout control
timeout = httpx.Timeout(
    timeout=10.0,  # Default timeout
    connect=5.0,   # Connection timeout
    read=10.0,     # Read timeout
    write=10.0,    # Write timeout
    pool=10.0      # Connection pool timeout
)

client = httpx.Client(timeout=timeout)

# Override timeout for specific request
try:
    response = client.get(
        "https://httpbin.org/delay/2",
        timeout=2.0  # Override client timeout
    )
    print(f"Response: {response.status_code}")
except httpx.TimeoutException:
    print("Custom timeout exceeded")
finally:
    client.close()
Response: 200

Different operations need different timeout values. Connection timeouts should be shorter (5-10 seconds), while read timeouts depend on the expected response size. For large downloads, you might need 30+ second read timeouts. httpx lets you configure each separately.

Streaming Responses

When dealing with large files or streaming APIs, loading the entire response into memory is inefficient. httpx supports streaming, letting you process responses chunk by chunk.

# streaming_responses.py
import httpx

# Stream a large response
with httpx.stream("GET", "https://httpbin.org/bytes/1024") as response:
    print(f"Status: {response.status_code}")
    print(f"Content-Length: {response.headers.get('content-length')}")

    # Process response in chunks
    for chunk in response.iter_bytes(chunk_size=256):
        print(f"Received {len(chunk)} bytes")

# Stream with iterator
with httpx.stream("GET", "https://httpbin.org/json") as response:
    for line in response.iter_lines():
        if line:
            print(f"Line: {line[:50]}...")

# Async streaming
import asyncio

async def async_stream():
    async with httpx.AsyncClient() as client:
        async with client.stream("GET", "https://httpbin.org/bytes/512") as response:
            async for chunk in response.aiter_bytes(chunk_size=128):
                print(f"Async received {len(chunk)} bytes")

asyncio.run(async_stream())
Status: 200 Content-Length: 1024 Received 256 bytes Received 256 bytes Received 256 bytes Received 256 bytes Line: {“slideshow”: {“author”: “Yours Truly”, “title”: … Async received 128 bytes Async received 128 bytes Async received 128 bytes Async received 128 bytes

Streaming is essential for production applications that download large files or handle streaming APIs. The `iter_bytes()` method gives you raw bytes, while `iter_lines()` automatically splits on newlines — useful for newline-delimited JSON APIs. Async streaming with `aiter_bytes()` and `aiter_lines()` works the same way in async contexts.

Real-Life Example: Async API Data Aggregator

Let’s build a practical example that combines async requests, error handling, and structured data processing. Imagine you’re aggregating data from multiple APIs and want to do it efficiently.

# api_aggregator.py
import asyncio
import httpx
from datetime import datetime

class APIAggregator:
    """Aggregates data from multiple APIs concurrently"""

    def __init__(self, max_concurrent=5):
        self.max_concurrent = max_concurrent
        self.timeout = httpx.Timeout(10.0)

    async def fetch_post(self, client, post_id):
        """Fetch a single post"""
        try:
            response = await client.get(
                f"https://jsonplaceholder.typicode.com/posts/{post_id}",
                timeout=self.timeout
            )
            response.raise_for_status()
            return response.json()
        except httpx.RequestError as e:
            print(f"Error fetching post {post_id}: {e}")
            return None

    async def fetch_user_comments(self, client, user_id):
        """Fetch comments for a user"""
        try:
            response = await client.get(
                f"https://jsonplaceholder.typicode.com/comments?email=*@{user_id}.com",
                timeout=self.timeout
            )
            response.raise_for_status()
            return response.json()
        except httpx.RequestError as e:
            print(f"Error fetching comments: {e}")
            return []

    async def aggregate(self):
        """Aggregate data from multiple endpoints"""
        async with httpx.AsyncClient(timeout=self.timeout) as client:
            # Fetch posts concurrently
            post_tasks = [
                self.fetch_post(client, i)
                for i in range(1, 6)
            ]
            posts = await asyncio.gather(*post_tasks)

            # Fetch comments concurrently
            comment_tasks = [
                self.fetch_user_comments(client, i)
                for i in range(1, 3)
            ]
            comments = await asyncio.gather(*comment_tasks)

            return {
                "timestamp": datetime.now().isoformat(),
                "posts_fetched": len([p for p in posts if p]),
                "total_comments": sum(len(c) for c in comments if c),
                "sample_post": posts[0] if posts else None
            }

# Run the aggregator
async def main():
    aggregator = APIAggregator()
    results = await aggregator.aggregate()

    print(f"Aggregation completed at {results['timestamp']}")
    print(f"Posts fetched: {results['posts_fetched']}")
    print(f"Total comments: {results['total_comments']}")
    print(f"First post title: {results['sample_post']['title']}")

asyncio.run(main())
Aggregation completed at 2026-04-09T14:23:45.123456 Posts fetched: 5 Total comments: 10 First post title: sunt aut facere repellat provident occaecati excepturi optio reprehenderit

This example demonstrates several httpx patterns: creating an async client, managing concurrent requests with proper error handling, using consistent timeouts across all requests, and returning structured data. The `APIAggregator` class is reusable and extensible — you could add caching, retry logic, or progress tracking. In production, you’d likely add logging and more sophisticated error recovery.

Frequently Asked Questions

Q: Is httpx a drop-in replacement for requests?
A: Mostly yes. The synchronous API is nearly identical, so most requests code works with httpx unchanged. However, httpx is stricter about some behaviors (like automatic redirects) and has some API differences. Test thoroughly before migrating production code.

Q: Do I need to use async?
A: No. httpx works great synchronously, and you only need async when handling many concurrent requests. Start with synchronous code and migrate to async if profiling shows it’s beneficial.

Q: What’s the performance difference between httpx and requests?
A: For single requests, performance is similar. For concurrent requests, httpx with async is dramatically faster because it avoids thread overhead. HTTP/2 support also improves performance for compatible servers.

Q: How do I handle cookies and sessions?
A: httpx clients maintain cookies automatically. Use a persistent client for multiple requests to the same host to keep cookies and connection pooling active across requests.

Q: Can I use httpx with Django or Flask?
A: Yes, but use the synchronous API in request handlers since WSGI is synchronous. Use async httpx in ASGI applications like FastAPI or Django Async Views.

Q: How do I set up authentication?
A: httpx supports multiple auth methods. For basic auth: `httpx.get(url, auth=(“user”, “pass”))`. For bearer tokens: `headers={“Authorization”: “Bearer token”}`. For custom auth, create a subclass of `httpx.Auth`.

Conclusion

httpx represents the future of HTTP requests in Python. It combines the simplicity of requests with modern features like async/await, HTTP/2, and better timeout handling. Whether you’re building a simple API client or managing complex concurrent request workflows, httpx has the tools you need without unnecessary complexity.

Start by installing httpx and replacing requests in a non-critical project. You’ll quickly discover why developers are switching. For more details, comprehensive documentation is available at httpx.readthedocs.io.

Continue Learning Python

Tutorials you might also find useful:

How To Handle Rate Limiting in Python API Calls

How To Handle Rate Limiting in Python API Calls

Last Updated: June 01, 2026

Skill Level: Intermediate

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Introduction to Modern HTTP Requests

For years, the Python requests library has been the go-to solution for making HTTP requests. While requests is powerful and user-friendly, it has limitations that modern Python developers encounter daily. It doesn’t support async operations natively, lacks HTTP/2 support, and can feel sluggish when you’re handling dozens of concurrent requests. If you’ve found yourself wrestling with requests’ synchronous nature or spinning up ThreadPoolExecutor just to manage multiple requests, you’re not alone. Many developers hit a wall where they need something more capable.

Enter httpx — a modern HTTP client that drops in as a replacement for requests while adding powerful features like native async/await support, HTTP/2 capabilities, and streaming responses. The best part? If you already know requests, you’ll feel right at home with httpx. The API is remarkably similar, which means you can start using it immediately without relearning everything. We’ll cover installation, basic usage, async patterns, and real-world examples that show why httpx is becoming the preferred choice for new Python projects.

In this guide, we’ll walk through everything you need to know about httpx. We’ll start with a quick example to get you up and running, explore what makes httpx special compared to other HTTP libraries, and then dive into practical patterns you can use in your own projects. Whether you’re building a simple API client or managing complex async workflows, httpx has the tools you need. Let’s get started.

Quick Example: GET Request in 5 Lines

# quick_get.py
import httpx

response = httpx.get("https://jsonplaceholder.typicode.com/posts/1")
print(response.status_code)
print(response.json())
200 {‘userId’: 1, ‘id’: 1, ‘title’: ‘sunt aut facere repellat…’, ‘body’: ‘…’}

That’s it. If you’ve used requests before, you already know httpx. The synchronous API is nearly identical, but under the hood, httpx brings modern features to the table. Now let’s explore what makes httpx more powerful than its predecessor.

When the 429 wave hits, read the headers

When the 429 wave hits, read the headers

What is httpx and Why Use It?

httpx is a modern HTTP client library for Python that combines the simplicity of requests with advanced features like async support, HTTP/2, and more sophisticated timeout handling. Created by Tom Christie (the same developer behind Starlette and FastAPI), httpx is built on a foundation that understands modern Python development patterns. It’s not just a requests replacement — it’s a redesign based on everything we’ve learned about making HTTP libraries in the 2020s.

The key differences matter when you’re building real applications. Unlike requests, httpx supports both synchronous and asynchronous code from the same library. You don’t need to install separate packages or maintain multiple code paths. It supports HTTP/2 by default, which means better performance for services that support it. It has built-in connection pooling, proper async context managers, and a cleaner API that feels more Pythonic.

Let’s compare httpx to other HTTP libraries in the Python ecosystem:

Feature httpx requests aiohttp urllib3
Synchronous API Yes Yes No Yes
Async/Await Support Yes No Yes No
HTTP/2 Yes No Yes No
Connection Pooling Yes Yes Yes Yes
Streaming Support Yes Yes Yes Yes
API Complexity Low Low Medium High
Drop-in requests Replacement Mostly Yes N/A No No

httpx shines when you need synchronous and asynchronous code in the same project. Unlike aiohttp, which requires AsyncIO from the start, httpx lets you start simple and scale to async when you need it. Unlike requests, httpx doesn’t force you into threading patterns when you want to handle multiple requests concurrently. It’s the bridge between requests’ simplicity and aiohttp’s power.

Installing httpx

Installation is straightforward. httpx is available on PyPI and installs cleanly without forcing dependencies on you. For basic functionality, you only need one command.

# install_httpx.sh
pip install httpx
Collecting httpx Downloading httpx-0.25.2-py3-none-any.whl (75 kB) Installing collected packages: httpx, certifi, sniffio, anyio Successfully installed httpx-0.25.2

If you want HTTP/2 support with better performance, you can install the optional dependencies. The h2 library handles HTTP/2 protocol details, while httpcore provides the underlying transport layer.

# install_httpx_with_http2.sh
pip install httpx[http2]
Collecting httpx[http2] Downloading httpx-0.25.2-py3-none-any.whl Collecting h2 Downloading h2-4.1.0-py3-none-any.whl (65 kB) Successfully installed httpx, h2

That’s all you need. Unlike some HTTP libraries, httpx doesn’t require compiling C extensions or installing system dependencies. It’s pure Python with optional performance enhancements.

Headers are the bread crumbs out of the rate limit forest

Headers are the bread crumbs out of the rate limit forest

Making GET Requests

GET requests are the foundation of HTTP. They retrieve data without side effects, and httpx makes them effortless. The basic pattern is identical to requests, but httpx adds subtle improvements like better error handling and automatic timeout management.

# get_requests.py
import httpx

# Simple GET request
response = httpx.get("https://jsonplaceholder.typicode.com/users")
print(f"Status: {response.status_code}")
print(f"Content-Type: {response.headers['content-type']}")
print(f"First user: {response.json()[0]['name']}")

# GET with query parameters
params = {"userId": 1}
response = httpx.get("https://jsonplaceholder.typicode.com/posts", params=params)
print(f"Posts for user 1: {len(response.json())}")

# GET with custom headers
headers = {"User-Agent": "MyApp/1.0"}
response = httpx.get("https://httpbin.org/headers", headers=headers)
print(response.json())
Status: 200 Content-Type: application/json First user: Leanne Graham Posts for user 1: 10 {‘headers’: {‘User-Agent’: ‘MyApp/1.0’, …}}

Notice how httpx handles query parameters naturally through the `params` dictionary. You don’t manually construct query strings or worry about URL encoding — httpx handles that behind the scenes. Headers work the same way, accepting a dictionary that httpx merges with the default headers. This consistent API means you can focus on your application logic instead of HTTP bookkeeping.

Making POST Requests

POST requests send data to the server. httpx supports multiple ways to send data: form-encoded, JSON, raw bytes, or streaming. Let’s explore the most common patterns.

# post_requests.py
import httpx

# POST with JSON data
client = httpx.Client()
data = {
    "title": "New Post",
    "body": "This is a test post",
    "userId": 1
}
response = client.post(
    "https://jsonplaceholder.typicode.com/posts",
    json=data
)
print(f"Created post with ID: {response.json()['id']}")

# POST with form data
form_data = {"username": "john", "password": "secret123"}
response = client.post(
    "https://httpbin.org/post",
    data=form_data
)
print(f"Form post status: {response.status_code}")

# POST with custom timeout
try:
    response = client.post(
        "https://httpbin.org/delay/10",
        timeout=5.0
    )
except httpx.TimeoutException:
    print("Request timed out after 5 seconds")

client.close()
Created post with ID: 101 Form post status: 200 Request timed out after 5 seconds

When making POST requests, use the `json` parameter for JSON data and the `data` parameter for form-encoded data. httpx automatically sets the correct Content-Type header for you. The `Client()` context manager maintains connection pooling across multiple requests, which is more efficient than using module-level functions for repeated requests. Timeouts are crucial for production code — they prevent your application from hanging if a server stops responding.

Exponential backoff: be patient, then exponentially more patient

Exponential backoff: be patient, then exponentially more patient

Using Async with httpx

This is where httpx truly shines. Async support is built in from the ground up, not bolted on as an afterthought. When you need to handle multiple concurrent requests, async/await patterns let you handle hundreds of concurrent connections with minimal memory overhead — something that would require threading or multiprocessing with requests.

# async_requests.py
import asyncio
import httpx

async def fetch_posts(user_id):
    """Fetch posts for a specific user"""
    async with httpx.AsyncClient() as client:
        response = await client.get(
            f"https://jsonplaceholder.typicode.com/posts?userId={user_id}"
        )
        return response.json()

async def fetch_multiple_users():
    """Fetch posts for multiple users concurrently"""
    tasks = [
        fetch_posts(user_id)
        for user_id in range(1, 6)
    ]
    results = await asyncio.gather(*tasks)

    for i, posts in enumerate(results, 1):
        print(f"User {i}: {len(posts)} posts")

# Run the async function
asyncio.run(fetch_multiple_users())
User 1: 10 posts User 2: 10 posts User 3: 9 posts User 4: 9 posts User 5: 9 posts

The `AsyncClient()` context manager handles resource cleanup automatically. Using `asyncio.gather()`, we fetch posts for five users concurrently in roughly the time it takes to fetch one. This pattern scales to thousands of concurrent requests without the overhead of creating threads. The key difference from synchronous code is minimal — just add `async` and `await` keywords.

# async_with_timeout.py
import asyncio
import httpx

async def fetch_with_timeout():
    """Fetch with explicit timeout configuration"""
    timeout = httpx.Timeout(10.0)  # 10 second timeout for all operations

    async with httpx.AsyncClient(timeout=timeout) as client:
        try:
            response = await client.get(
                "https://jsonplaceholder.typicode.com/posts/1"
            )
            print(f"Success: {response.status_code}")
        except httpx.TimeoutException:
            print("Request timed out")
        except httpx.RequestError as e:
            print(f"Network error: {e}")

asyncio.run(fetch_with_timeout())
Success: 200

Timeout handling in async contexts is critical. The `Timeout` object lets you set different timeouts for connection, read, write, and pool operations. This fine-grained control prevents your async application from hanging on unresponsive servers.

HTTP/2 Support

HTTP/2 is faster than HTTP/1.1 because it multiplexes multiple requests over a single connection and compresses headers. With httpx, HTTP/2 support is automatic for servers that support it. You don’t need to change your code — just have the h2 library installed.

# http2_support.py
import httpx

# HTTP/2 is automatic when available
response = httpx.get("https://httpbin.org/get")
print(f"HTTP Version: {response.http_version}")

# Force HTTP/1.1 if needed
client = httpx.Client(http2=False)
response = client.get("https://httpbin.org/get")
print(f"HTTP Version (forced 1.1): {response.http_version}")
client.close()

# Create client with explicit HTTP/2 support
client = httpx.Client(http2=True)
response = client.get("https://httpbin.org/get")
print(f"HTTP Version (with HTTP/2): {response.http_version}")
client.close()
HTTP Version: HTTP/2 HTTP Version (forced 1.1): HTTP/1.1 HTTP Version (with HTTP/2): HTTP/2

The performance difference is subtle for single requests but becomes dramatic with concurrent requests over HTTP/2. Since HTTP/2 multiplexes requests on a single connection, you avoid the overhead of establishing multiple TCP connections. For APIs that support it, this can mean 20-50% faster performance in real-world scenarios.

Decorators beat manual retry loops every time

Decorators beat manual retry loops every time

Timeouts and Error Handling

Production code needs robust error handling. httpx provides clear exceptions for different failure scenarios, making it easy to distinguish between network problems, timeouts, and server errors.

# error_handling.py
import httpx

def fetch_with_fallback(url, fallback_url):
    """Fetch from primary URL with fallback"""
    try:
        response = httpx.get(url, timeout=5.0)
        response.raise_for_status()  # Raise exception for bad status codes
        return response.json()
    except httpx.TimeoutException:
        print(f"Timeout on {url}, trying fallback")
        return httpx.get(fallback_url).json()
    except httpx.HTTPStatusError as e:
        print(f"HTTP error: {e.response.status_code}")
        raise
    except httpx.RequestError as e:
        print(f"Request error: {e}")
        raise

try:
    data = fetch_with_fallback(
        "https://jsonplaceholder.typicode.com/posts/1",
        "https://jsonplaceholder.typicode.com/posts/2"
    )
    print(f"Fetched post: {data['title']}")
except Exception as e:
    print(f"All attempts failed: {e}")
Fetched post: sunt aut facere repellat provident…

httpx organizes exceptions in a clear hierarchy. `RequestError` is the base class for all request-related errors. `TimeoutException` indicates a timeout (connection, read, write, or pool). `HTTPStatusError` means the server responded with an error status code (4xx or 5xx). Using `raise_for_status()` automatically raises an exception for bad status codes, similar to requests.

# advanced_timeouts.py
import httpx

# Granular timeout control
timeout = httpx.Timeout(
    timeout=10.0,  # Default timeout
    connect=5.0,   # Connection timeout
    read=10.0,     # Read timeout
    write=10.0,    # Write timeout
    pool=10.0      # Connection pool timeout
)

client = httpx.Client(timeout=timeout)

# Override timeout for specific request
try:
    response = client.get(
        "https://httpbin.org/delay/2",
        timeout=2.0  # Override client timeout
    )
    print(f"Response: {response.status_code}")
except httpx.TimeoutException:
    print("Custom timeout exceeded")
finally:
    client.close()
Response: 200

Different operations need different timeout values. Connection timeouts should be shorter (5-10 seconds), while read timeouts depend on the expected response size. For large downloads, you might need 30+ second read timeouts. httpx lets you configure each separately.

Streaming Responses

When dealing with large files or streaming APIs, loading the entire response into memory is inefficient. httpx supports streaming, letting you process responses chunk by chunk.

# streaming_responses.py
import httpx

# Stream a large response
with httpx.stream("GET", "https://httpbin.org/bytes/1024") as response:
    print(f"Status: {response.status_code}")
    print(f"Content-Length: {response.headers.get('content-length')}")

    # Process response in chunks
    for chunk in response.iter_bytes(chunk_size=256):
        print(f"Received {len(chunk)} bytes")

# Stream with iterator
with httpx.stream("GET", "https://httpbin.org/json") as response:
    for line in response.iter_lines():
        if line:
            print(f"Line: {line[:50]}...")

# Async streaming
import asyncio

async def async_stream():
    async with httpx.AsyncClient() as client:
        async with client.stream("GET", "https://httpbin.org/bytes/512") as response:
            async for chunk in response.aiter_bytes(chunk_size=128):
                print(f"Async received {len(chunk)} bytes")

asyncio.run(async_stream())
Status: 200 Content-Length: 1024 Received 256 bytes Received 256 bytes Received 256 bytes Received 256 bytes Line: {“slideshow”: {“author”: “Yours Truly”, “title”: … Async received 128 bytes Async received 128 bytes Async received 128 bytes Async received 128 bytes

Streaming is essential for production applications that download large files or handle streaming APIs. The `iter_bytes()` method gives you raw bytes, while `iter_lines()` automatically splits on newlines — useful for newline-delimited JSON APIs. Async streaming with `aiter_bytes()` and `aiter_lines()` works the same way in async contexts.

Know your limits, respect the headers, fetch responsibly

Know your limits, respect the headers, fetch responsibly

Real-Life Example: Async API Data Aggregator

Let’s build a practical example that combines async requests, error handling, and structured data processing. Imagine you’re aggregating data from multiple APIs and want to do it efficiently.

# api_aggregator.py
import asyncio
import httpx
from datetime import datetime

class APIAggregator:
    """Aggregates data from multiple APIs concurrently"""

    def __init__(self, max_concurrent=5):
        self.max_concurrent = max_concurrent
        self.timeout = httpx.Timeout(10.0)

    async def fetch_post(self, client, post_id):
        """Fetch a single post"""
        try:
            response = await client.get(
                f"https://jsonplaceholder.typicode.com/posts/{post_id}",
                timeout=self.timeout
            )
            response.raise_for_status()
            return response.json()
        except httpx.RequestError as e:
            print(f"Error fetching post {post_id}: {e}")
            return None

    async def fetch_user_comments(self, client, user_id):
        """Fetch comments for a user"""
        try:
            response = await client.get(
                f"https://jsonplaceholder.typicode.com/comments?email=*@{user_id}.com",
                timeout=self.timeout
            )
            response.raise_for_status()
            return response.json()
        except httpx.RequestError as e:
            print(f"Error fetching comments: {e}")
            return []

    async def aggregate(self):
        """Aggregate data from multiple endpoints"""
        async with httpx.AsyncClient(timeout=self.timeout) as client:
            # Fetch posts concurrently
            post_tasks = [
                self.fetch_post(client, i)
                for i in range(1, 6)
            ]
            posts = await asyncio.gather(*post_tasks)

            # Fetch comments concurrently
            comment_tasks = [
                self.fetch_user_comments(client, i)
                for i in range(1, 3)
            ]
            comments = await asyncio.gather(*comment_tasks)

            return {
                "timestamp": datetime.now().isoformat(),
                "posts_fetched": len([p for p in posts if p]),
                "total_comments": sum(len(c) for c in comments if c),
                "sample_post": posts[0] if posts else None
            }

# Run the aggregator
async def main():
    aggregator = APIAggregator()
    results = await aggregator.aggregate()

    print(f"Aggregation completed at {results['timestamp']}")
    print(f"Posts fetched: {results['posts_fetched']}")
    print(f"Total comments: {results['total_comments']}")
    print(f"First post title: {results['sample_post']['title']}")

asyncio.run(main())
Aggregation completed at 2026-04-09T14:23:45.123456 Posts fetched: 5 Total comments: 10 First post title: sunt aut facere repellat provident occaecati excepturi optio reprehenderit

This example demonstrates several httpx patterns: creating an async client, managing concurrent requests with proper error handling, using consistent timeouts across all requests, and returning structured data. The `APIAggregator` class is reusable and extensible — you could add caching, retry logic, or progress tracking. In production, you’d likely add logging and more sophisticated error recovery.

Frequently Asked Questions

Q: Is httpx a drop-in replacement for requests?
A: Mostly yes. The synchronous API is nearly identical, so most requests code works with httpx unchanged. However, httpx is stricter about some behaviors (like automatic redirects) and has some API differences. Test thoroughly before migrating production code.

Q: Do I need to use async?
A: No. httpx works great synchronously, and you only need async when handling many concurrent requests. Start with synchronous code and migrate to async if profiling shows it’s beneficial.

Q: What’s the performance difference between httpx and requests?
A: For single requests, performance is similar. For concurrent requests, httpx with async is dramatically faster because it avoids thread overhead. HTTP/2 support also improves performance for compatible servers.

Q: How do I handle cookies and sessions?
A: httpx clients maintain cookies automatically. Use a persistent client for multiple requests to the same host to keep cookies and connection pooling active across requests.

Q: Can I use httpx with Django or Flask?
A: Yes, but use the synchronous API in request handlers since WSGI is synchronous. Use async httpx in ASGI applications like FastAPI or Django Async Views.

Q: How do I set up authentication?
A: httpx supports multiple auth methods. For basic auth: `httpx.get(url, auth=(“user”, “pass”))`. For bearer tokens: `headers={“Authorization”: “Bearer token”}`. For custom auth, create a subclass of `httpx.Auth`.

Conclusion

httpx represents the future of HTTP requests in Python. It combines the simplicity of requests with modern features like async/await, HTTP/2, and better timeout handling. Whether you’re building a simple API client or managing complex concurrent request workflows, httpx has the tools you need without unnecessary complexity.

Start by installing httpx and replacing requests in a non-critical project. You’ll quickly discover why developers are switching. For more details, comprehensive documentation is available at httpx.readthedocs.io.

How To Use Python httpx for Modern HTTP Requests

How To Use Python httpx for Modern HTTP Requests

Last Updated: June 01, 2026

Skill Level: Intermediate

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Introduction to Modern HTTP Requests

For years, the Python requests library has been the go-to solution for making HTTP requests. While requests is powerful and user-friendly, it has limitations that modern Python developers encounter daily. It doesn’t support async operations natively, lacks HTTP/2 support, and can feel sluggish when you’re handling dozens of concurrent requests. If you’ve found yourself wrestling with requests’ synchronous nature or spinning up ThreadPoolExecutor just to manage multiple requests, you’re not alone. Many developers hit a wall where they need something more capable.

Enter httpx — a modern HTTP client that drops in as a replacement for requests while adding powerful features like native async/await support, HTTP/2 capabilities, and streaming responses. The best part? If you already know requests, you’ll feel right at home with httpx. The API is remarkably similar, which means you can start using it immediately without relearning everything. We’ll cover installation, basic usage, async patterns, and real-world examples that show why httpx is becoming the preferred choice for new Python projects.

In this guide, we’ll walk through everything you need to know about httpx. We’ll start with a quick example to get you up and running, explore what makes httpx special compared to other HTTP libraries, and then dive into practical patterns you can use in your own projects. Whether you’re building a simple API client or managing complex async workflows, httpx has the tools you need. Let’s get started.

Quick Example: GET Request in 5 Lines

# quick_get.py
import httpx

response = httpx.get("https://jsonplaceholder.typicode.com/posts/1")
print(response.status_code)
print(response.json())
200 {‘userId’: 1, ‘id’: 1, ‘title’: ‘sunt aut facere repellat…’, ‘body’: ‘…’}

That’s it. If you’ve used requests before, you already know httpx. The synchronous API is nearly identical, but under the hood, httpx brings modern features to the table. Now let’s explore what makes httpx more powerful than its predecessor.

httpx handles synchronous and asynchronous requests from a single library

httpx handles synchronous and asynchronous requests from a single library

What is httpx and Why Use It?

httpx is a modern HTTP client library for Python that combines the simplicity of requests with advanced features like async support, HTTP/2, and more sophisticated timeout handling. Created by Tom Christie (the same developer behind Starlette and FastAPI), httpx is built on a foundation that understands modern Python development patterns. It’s not just a requests replacement — it’s a redesign based on everything we’ve learned about making HTTP libraries in the 2020s.

The key differences matter when you’re building real applications. Unlike requests, httpx supports both synchronous and asynchronous code from the same library. You don’t need to install separate packages or maintain multiple code paths. It supports HTTP/2 by default, which means better performance for services that support it. It has built-in connection pooling, proper async context managers, and a cleaner API that feels more Pythonic.

Let’s compare httpx to other HTTP libraries in the Python ecosystem:

Feature httpx requests aiohttp urllib3
Synchronous API Yes Yes No Yes
Async/Await Support Yes No Yes No
HTTP/2 Yes No Yes No
Connection Pooling Yes Yes Yes Yes
Streaming Support Yes Yes Yes Yes
API Complexity Low Low Medium High
Drop-in requests Replacement Mostly Yes N/A No No

httpx shines when you need synchronous and asynchronous code in the same project. Unlike aiohttp, which requires AsyncIO from the start, httpx lets you start simple and scale to async when you need it. Unlike requests, httpx doesn’t force you into threading patterns when you want to handle multiple requests concurrently. It’s the bridge between requests’ simplicity and aiohttp’s power.

Installing httpx

Installation is straightforward. httpx is available on PyPI and installs cleanly without forcing dependencies on you. For basic functionality, you only need one command.

# install_httpx.sh
pip install httpx
Collecting httpx Downloading httpx-0.25.2-py3-none-any.whl (75 kB) Installing collected packages: httpx, certifi, sniffio, anyio Successfully installed httpx-0.25.2

If you want HTTP/2 support with better performance, you can install the optional dependencies. The h2 library handles HTTP/2 protocol details, while httpcore provides the underlying transport layer.

# install_httpx_with_http2.sh
pip install httpx[http2]
Collecting httpx[http2] Downloading httpx-0.25.2-py3-none-any.whl Collecting h2 Downloading h2-4.1.0-py3-none-any.whl (65 kB) Successfully installed httpx, h2

That’s all you need. Unlike some HTTP libraries, httpx doesn’t require compiling C extensions or installing system dependencies. It’s pure Python with optional performance enhancements.

POST, GET, PUT, DELETE -- all methods work seamlessly with httpx

POST, GET, PUT, DELETE — all methods work seamlessly with httpx

Making GET Requests

GET requests are the foundation of HTTP. They retrieve data without side effects, and httpx makes them effortless. The basic pattern is identical to requests, but httpx adds subtle improvements like better error handling and automatic timeout management.

# get_requests.py
import httpx

# Simple GET request
response = httpx.get("https://jsonplaceholder.typicode.com/users")
print(f"Status: {response.status_code}")
print(f"Content-Type: {response.headers['content-type']}")
print(f"First user: {response.json()[0]['name']}")

# GET with query parameters
params = {"userId": 1}
response = httpx.get("https://jsonplaceholder.typicode.com/posts", params=params)
print(f"Posts for user 1: {len(response.json())}")

# GET with custom headers
headers = {"User-Agent": "MyApp/1.0"}
response = httpx.get("https://httpbin.org/headers", headers=headers)
print(response.json())
Status: 200 Content-Type: application/json First user: Leanne Graham Posts for user 1: 10 {‘headers’: {‘User-Agent’: ‘MyApp/1.0’, …}}

Notice how httpx handles query parameters naturally through the `params` dictionary. You don’t manually construct query strings or worry about URL encoding — httpx handles that behind the scenes. Headers work the same way, accepting a dictionary that httpx merges with the default headers. This consistent API means you can focus on your application logic instead of HTTP bookkeeping.

Making POST Requests

POST requests send data to the server. httpx supports multiple ways to send data: form-encoded, JSON, raw bytes, or streaming. Let’s explore the most common patterns.

# post_requests.py
import httpx

# POST with JSON data
client = httpx.Client()
data = {
    "title": "New Post",
    "body": "This is a test post",
    "userId": 1
}
response = client.post(
    "https://jsonplaceholder.typicode.com/posts",
    json=data
)
print(f"Created post with ID: {response.json()['id']}")

# POST with form data
form_data = {"username": "john", "password": "secret123"}
response = client.post(
    "https://httpbin.org/post",
    data=form_data
)
print(f"Form post status: {response.status_code}")

# POST with custom timeout
try:
    response = client.post(
        "https://httpbin.org/delay/10",
        timeout=5.0
    )
except httpx.TimeoutException:
    print("Request timed out after 5 seconds")

client.close()
Created post with ID: 101 Form post status: 200 Request timed out after 5 seconds

When making POST requests, use the `json` parameter for JSON data and the `data` parameter for form-encoded data. httpx automatically sets the correct Content-Type header for you. The `Client()` context manager maintains connection pooling across multiple requests, which is more efficient than using module-level functions for repeated requests. Timeouts are crucial for production code — they prevent your application from hanging if a server stops responding.

Connection pooling keeps your HTTP clients fast and memory-efficient

Connection pooling keeps your HTTP clients fast and memory-efficient

Using Async with httpx

This is where httpx truly shines. Async support is built in from the ground up, not bolted on as an afterthought. When you need to handle multiple concurrent requests, async/await patterns let you handle hundreds of concurrent connections with minimal memory overhead — something that would require threading or multiprocessing with requests.

# async_requests.py
import asyncio
import httpx

async def fetch_posts(user_id):
    """Fetch posts for a specific user"""
    async with httpx.AsyncClient() as client:
        response = await client.get(
            f"https://jsonplaceholder.typicode.com/posts?userId={user_id}"
        )
        return response.json()

async def fetch_multiple_users():
    """Fetch posts for multiple users concurrently"""
    tasks = [
        fetch_posts(user_id)
        for user_id in range(1, 6)
    ]
    results = await asyncio.gather(*tasks)

    for i, posts in enumerate(results, 1):
        print(f"User {i}: {len(posts)} posts")

# Run the async function
asyncio.run(fetch_multiple_users())
User 1: 10 posts User 2: 10 posts User 3: 9 posts User 4: 9 posts User 5: 9 posts

The `AsyncClient()` context manager handles resource cleanup automatically. Using `asyncio.gather()`, we fetch posts for five users concurrently in roughly the time it takes to fetch one. This pattern scales to thousands of concurrent requests without the overhead of creating threads. The key difference from synchronous code is minimal — just add `async` and `await` keywords.

# async_with_timeout.py
import asyncio
import httpx

async def fetch_with_timeout():
    """Fetch with explicit timeout configuration"""
    timeout = httpx.Timeout(10.0)  # 10 second timeout for all operations

    async with httpx.AsyncClient(timeout=timeout) as client:
        try:
            response = await client.get(
                "https://jsonplaceholder.typicode.com/posts/1"
            )
            print(f"Success: {response.status_code}")
        except httpx.TimeoutException:
            print("Request timed out")
        except httpx.RequestError as e:
            print(f"Network error: {e}")

asyncio.run(fetch_with_timeout())
Success: 200

Timeout handling in async contexts is critical. The `Timeout` object lets you set different timeouts for connection, read, write, and pool operations. This fine-grained control prevents your async application from hanging on unresponsive servers.

HTTP/2 Support

HTTP/2 is faster than HTTP/1.1 because it multiplexes multiple requests over a single connection and compresses headers. With httpx, HTTP/2 support is automatic for servers that support it. You don’t need to change your code — just have the h2 library installed.

# http2_support.py
import httpx

# HTTP/2 is automatic when available
response = httpx.get("https://httpbin.org/get")
print(f"HTTP Version: {response.http_version}")

# Force HTTP/1.1 if needed
client = httpx.Client(http2=False)
response = client.get("https://httpbin.org/get")
print(f"HTTP Version (forced 1.1): {response.http_version}")
client.close()

# Create client with explicit HTTP/2 support
client = httpx.Client(http2=True)
response = client.get("https://httpbin.org/get")
print(f"HTTP Version (with HTTP/2): {response.http_version}")
client.close()
HTTP Version: HTTP/2 HTTP Version (forced 1.1): HTTP/1.1 HTTP Version (with HTTP/2): HTTP/2

The performance difference is subtle for single requests but becomes dramatic with concurrent requests over HTTP/2. Since HTTP/2 multiplexes requests on a single connection, you avoid the overhead of establishing multiple TCP connections. For APIs that support it, this can mean 20-50% faster performance in real-world scenarios.

HTTP/2 multiplexing lets httpx handle multiple requests on one connection

HTTP/2 multiplexing lets httpx handle multiple requests on one connection

Timeouts and Error Handling

Production code needs robust error handling. httpx provides clear exceptions for different failure scenarios, making it easy to distinguish between network problems, timeouts, and server errors.

# error_handling.py
import httpx

def fetch_with_fallback(url, fallback_url):
    """Fetch from primary URL with fallback"""
    try:
        response = httpx.get(url, timeout=5.0)
        response.raise_for_status()  # Raise exception for bad status codes
        return response.json()
    except httpx.TimeoutException:
        print(f"Timeout on {url}, trying fallback")
        return httpx.get(fallback_url).json()
    except httpx.HTTPStatusError as e:
        print(f"HTTP error: {e.response.status_code}")
        raise
    except httpx.RequestError as e:
        print(f"Request error: {e}")
        raise

try:
    data = fetch_with_fallback(
        "https://jsonplaceholder.typicode.com/posts/1",
        "https://jsonplaceholder.typicode.com/posts/2"
    )
    print(f"Fetched post: {data['title']}")
except Exception as e:
    print(f"All attempts failed: {e}")
Fetched post: sunt aut facere repellat provident…

httpx organizes exceptions in a clear hierarchy. `RequestError` is the base class for all request-related errors. `TimeoutException` indicates a timeout (connection, read, write, or pool). `HTTPStatusError` means the server responded with an error status code (4xx or 5xx). Using `raise_for_status()` automatically raises an exception for bad status codes, similar to requests.

# advanced_timeouts.py
import httpx

# Granular timeout control
timeout = httpx.Timeout(
    timeout=10.0,  # Default timeout
    connect=5.0,   # Connection timeout
    read=10.0,     # Read timeout
    write=10.0,    # Write timeout
    pool=10.0      # Connection pool timeout
)

client = httpx.Client(timeout=timeout)

# Override timeout for specific request
try:
    response = client.get(
        "https://httpbin.org/delay/2",
        timeout=2.0  # Override client timeout
    )
    print(f"Response: {response.status_code}")
except httpx.TimeoutException:
    print("Custom timeout exceeded")
finally:
    client.close()
Response: 200

Different operations need different timeout values. Connection timeouts should be shorter (5-10 seconds), while read timeouts depend on the expected response size. For large downloads, you might need 30+ second read timeouts. httpx lets you configure each separately.

Streaming Responses

When dealing with large files or streaming APIs, loading the entire response into memory is inefficient. httpx supports streaming, letting you process responses chunk by chunk.

# streaming_responses.py
import httpx

# Stream a large response
with httpx.stream("GET", "https://httpbin.org/bytes/1024") as response:
    print(f"Status: {response.status_code}")
    print(f"Content-Length: {response.headers.get('content-length')}")

    # Process response in chunks
    for chunk in response.iter_bytes(chunk_size=256):
        print(f"Received {len(chunk)} bytes")

# Stream with iterator
with httpx.stream("GET", "https://httpbin.org/json") as response:
    for line in response.iter_lines():
        if line:
            print(f"Line: {line[:50]}...")

# Async streaming
import asyncio

async def async_stream():
    async with httpx.AsyncClient() as client:
        async with client.stream("GET", "https://httpbin.org/bytes/512") as response:
            async for chunk in response.aiter_bytes(chunk_size=128):
                print(f"Async received {len(chunk)} bytes")

asyncio.run(async_stream())
Status: 200 Content-Length: 1024 Received 256 bytes Received 256 bytes Received 256 bytes Received 256 bytes Line: {“slideshow”: {“author”: “Yours Truly”, “title”: … Async received 128 bytes Async received 128 bytes Async received 128 bytes Async received 128 bytes

Streaming is essential for production applications that download large files or handle streaming APIs. The `iter_bytes()` method gives you raw bytes, while `iter_lines()` automatically splits on newlines — useful for newline-delimited JSON APIs. Async streaming with `aiter_bytes()` and `aiter_lines()` works the same way in async contexts.

Streaming responses keep memory usage constant regardless of file size

Streaming responses keep memory usage constant regardless of file size

Real-Life Example: Async API Data Aggregator

Let’s build a practical example that combines async requests, error handling, and structured data processing. Imagine you’re aggregating data from multiple APIs and want to do it efficiently.

# api_aggregator.py
import asyncio
import httpx
from datetime import datetime

class APIAggregator:
    """Aggregates data from multiple APIs concurrently"""

    def __init__(self, max_concurrent=5):
        self.max_concurrent = max_concurrent
        self.timeout = httpx.Timeout(10.0)

    async def fetch_post(self, client, post_id):
        """Fetch a single post"""
        try:
            response = await client.get(
                f"https://jsonplaceholder.typicode.com/posts/{post_id}",
                timeout=self.timeout
            )
            response.raise_for_status()
            return response.json()
        except httpx.RequestError as e:
            print(f"Error fetching post {post_id}: {e}")
            return None

    async def fetch_user_comments(self, client, user_id):
        """Fetch comments for a user"""
        try:
            response = await client.get(
                f"https://jsonplaceholder.typicode.com/comments?email=*@{user_id}.com",
                timeout=self.timeout
            )
            response.raise_for_status()
            return response.json()
        except httpx.RequestError as e:
            print(f"Error fetching comments: {e}")
            return []

    async def aggregate(self):
        """Aggregate data from multiple endpoints"""
        async with httpx.AsyncClient(timeout=self.timeout) as client:
            # Fetch posts concurrently
            post_tasks = [
                self.fetch_post(client, i)
                for i in range(1, 6)
            ]
            posts = await asyncio.gather(*post_tasks)

            # Fetch comments concurrently
            comment_tasks = [
                self.fetch_user_comments(client, i)
                for i in range(1, 3)
            ]
            comments = await asyncio.gather(*comment_tasks)

            return {
                "timestamp": datetime.now().isoformat(),
                "posts_fetched": len([p for p in posts if p]),
                "total_comments": sum(len(c) for c in comments if c),
                "sample_post": posts[0] if posts else None
            }

# Run the aggregator
async def main():
    aggregator = APIAggregator()
    results = await aggregator.aggregate()

    print(f"Aggregation completed at {results['timestamp']}")
    print(f"Posts fetched: {results['posts_fetched']}")
    print(f"Total comments: {results['total_comments']}")
    print(f"First post title: {results['sample_post']['title']}")

asyncio.run(main())
Aggregation completed at 2026-04-09T14:23:45.123456 Posts fetched: 5 Total comments: 10 First post title: sunt aut facere repellat provident occaecati excepturi optio reprehenderit

This example demonstrates several httpx patterns: creating an async client, managing concurrent requests with proper error handling, using consistent timeouts across all requests, and returning structured data. The `APIAggregator` class is reusable and extensible — you could add caching, retry logic, or progress tracking. In production, you’d likely add logging and more sophisticated error recovery.

Frequently Asked Questions

Q: Is httpx a drop-in replacement for requests?
A: Mostly yes. The synchronous API is nearly identical, so most requests code works with httpx unchanged. However, httpx is stricter about some behaviors (like automatic redirects) and has some API differences. Test thoroughly before migrating production code.

Q: Do I need to use async?
A: No. httpx works great synchronously, and you only need async when handling many concurrent requests. Start with synchronous code and migrate to async if profiling shows it’s beneficial.

Q: What’s the performance difference between httpx and requests?
A: For single requests, performance is similar. For concurrent requests, httpx with async is dramatically faster because it avoids thread overhead. HTTP/2 support also improves performance for compatible servers.

Q: How do I handle cookies and sessions?
A: httpx clients maintain cookies automatically. Use a persistent client for multiple requests to the same host to keep cookies and connection pooling active across requests.

Q: Can I use httpx with Django or Flask?
A: Yes, but use the synchronous API in request handlers since WSGI is synchronous. Use async httpx in ASGI applications like FastAPI or Django Async Views.

Q: How do I set up authentication?
A: httpx supports multiple auth methods. For basic auth: `httpx.get(url, auth=(“user”, “pass”))`. For bearer tokens: `headers={“Authorization”: “Bearer token”}`. For custom auth, create a subclass of `httpx.Auth`.

Conclusion

httpx represents the future of HTTP requests in Python. It combines the simplicity of requests with modern features like async/await, HTTP/2, and better timeout handling. Whether you’re building a simple API client or managing complex concurrent request workflows, httpx has the tools you need without unnecessary complexity.

Start by installing httpx and replacing requests in a non-critical project. You’ll quickly discover why developers are switching. For more details, comprehensive documentation is available at httpx.readthedocs.io.

How To Hash Passwords Safely in Python with bcrypt

How To Hash Passwords Safely in Python with bcrypt

Last Updated: June 01, 2026

Skill Level: Intermediate

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Why Password Security Matters in Your Applications

Every year, millions of passwords are exposed in data breaches. When companies store passwords in plain text or with weak encryption methods, attackers who gain access to the database can immediately use those credentials against users. A single security oversight can compromise not just one user, but your entire application’s integrity and your reputation. The good news? Protecting passwords isn’t complicated — bcrypt makes it remarkably straightforward to implement enterprise-grade password security in your Python applications today.

Bcrypt is a purpose-built password hashing library that automatically handles the complexity of secure password storage. Unlike generic hashing functions, bcrypt incorporates salt generation, adaptive work factors, and built-in protections against common attacks. Whether you’re building a small project or scaling to millions of users, bcrypt gives you the same security guarantees with just a few lines of code.

In this guide, we’ll explore how bcrypt works from first principles, then move through practical implementations including user registration, password verification, and real-world patterns you can use immediately. By the end, you’ll understand why bcrypt is the industry standard and how to integrate it into your projects with confidence.

Quick Example: Hash and Verify in 10 Lines

Before diving into the theory, let’s see bcrypt in action. This example demonstrates the complete workflow — hashing a password and then verifying a user’s input against that stored hash. It’s this simple and this secure.

# quick_hash.py
import bcrypt

password = "MySecurePassword123!"
hashed = bcrypt.hashpw(password.encode(), bcrypt.gensalt())
print("Hashed:", hashed)

user_input = "MySecurePassword123!"
is_valid = bcrypt.checkpw(user_input.encode(), hashed)
print("Password match:", is_valid)
Hashed: b'$2b$12$abcdefghijklmnopqrstuvwxyzAbCdEfGhIjKlMnOpQrStUvWxYz'
Password match: True

That’s the essence of password hashing with bcrypt. The hashpw() function handles salt generation and hashing internally, while checkpw() verifies inputs without needing to decrypt anything. This simplicity masks sophisticated security mechanisms working behind the scenes.

What is Password Hashing and Why Use bcrypt?

Password hashing is a one-way cryptographic function that transforms a plaintext password into an irreversible string of characters. The database stores only the hash, not the original password. When a user logs in, the application hashes their input and compares it to the stored hash — if they match, the password is correct. If attackers compromise the database, they get hashes, not passwords.

Not all hashing functions are suitable for passwords. General-purpose algorithms like MD5, SHA1, and even SHA256 are too fast — an attacker with access to your password hashes can run billions of guesses per second using GPU clusters. Bcrypt solves this by intentionally being slow and computationally expensive, making brute-force attacks impractical. Here’s how different approaches compare:

Method Speed Salt Adaptive Security Level
Plaintext Instant None No Catastrophic
MD5 ~1M hashes/sec Optional No Broken
SHA256 ~100M hashes/sec Optional No Weak for passwords
Bcrypt ~5 hashes/sec (configurable) Automatic Yes Industry standard

Bcrypt’s adaptive cost factor is crucial — as hardware becomes faster, you can increase the work factor to keep password cracking slow. A hash that takes 0.2 seconds to compute today will still take 0.2 seconds tomorrow, even as computers improve. This forward-looking design is why bcrypt remains relevant decades after its creation.

Installing bcrypt

Bcrypt requires a simple pip installation. It’s a compiled C extension for performance, and the package handles all dependency management across operating systems. Most Python environments will install it without issues, though some systems may need development headers for the C compiler.

# install_bcrypt.py
import subprocess
import sys

# Install bcrypt
subprocess.check_call([sys.executable, "-m", "pip", "install", "bcrypt"])

# Verify installation
import bcrypt
print("Bcrypt version:", bcrypt.__version__)
print("Installation successful!")
Collecting bcrypt
  Downloading bcrypt-4.1.2-cp312-cp312-linux_x86_64.whl
Installing collected packages: bcrypt
Successfully installed bcrypt-4.1.2
Bcrypt version: 4.1.2
Installation successful!

Once installed, bcrypt is immediately available for import. We recommend adding bcrypt to your project’s requirements.txt or pyproject.toml file so it’s installed automatically for anyone who clones your repository. The package is maintained actively and released regularly with security updates, so keep it current with pip install --upgrade bcrypt periodically.

Investigating hash functions like a crime scene -- following the trail that only goes one way

Investigating hash functions like a crime scene — following the trail that only goes one way

Hashing a Password

The hashing process in bcrypt is deceptively simple from a user perspective, but implements several sophisticated techniques internally. When you call hashpw(), bcrypt generates a cryptographically random salt, applies the Blowfish cipher multiple times based on the cost factor, and returns a single string containing all the information needed to verify passwords later.

Here’s the standard pattern for creating password hashes when users register or change their password:

# hash_password_demo.py
import bcrypt

def hash_password(password: str) -> str:
    """
    Hash a plaintext password using bcrypt.

    Args:
        password: The user's plaintext password

    Returns:
        The bcrypt hash as a string (can be stored in database)
    """
    # Encode string to bytes (bcrypt requires bytes)
    password_bytes = password.encode('utf-8')

    # Generate salt and hash in one operation
    # Default cost factor is 12
    hashed = bcrypt.hashpw(password_bytes, bcrypt.gensalt())

    # Return as string for database storage
    return hashed.decode('utf-8')

# Example usage
user_password = "SecurePass123!"
stored_hash = hash_password(user_password)
print(f"Original: {user_password}")
print(f"Stored:   {stored_hash}")
print(f"Length:   {len(stored_hash)} characters")
Original: SecurePass123!
Stored:   $2b$12$QaNu.rBvNp8zXrLbSKc.MOaVfVL6qfKfhIU3SYvKzGdpLp4I8TS6O
Length:   60 characters

Every hash is exactly 60 characters long, making it database-friendly to store in a VARCHAR(60) column. The hash contains embedded information: the algorithm identifier ($2b$), the cost factor ($12$), and the salt. You never need to store these separately — everything is self-contained in that single string.

A critical consideration: passwords must be encoded to bytes before hashing. Python 3 strings are Unicode by default, but bcrypt operates on bytes. Always use .encode('utf-8') or specify the encoding explicitly. Different encodings will produce different hashes, which is why consistency matters — the verification step must use the same encoding.

Verifying a Password

When a user logs in, you don’t decrypt the stored hash or compare strings directly. Instead, you hash the incoming password and let bcrypt compare the hashes internally using timing-safe comparison. This prevents timing attacks where attackers measure response times to gain information about the hash.

Here’s how to verify passwords securely:

# verify_password_demo.py
import bcrypt

def verify_password(provided_password: str, stored_hash: str) -> bool:
    """
    Verify a plaintext password against a bcrypt hash.

    Args:
        provided_password: Password the user provided at login
        stored_hash: The bcrypt hash from the database

    Returns:
        True if the password matches, False otherwise
    """
    # Encode both to bytes
    provided_bytes = provided_password.encode('utf-8')
    stored_bytes = stored_hash.encode('utf-8')

    # checkpw handles timing-safe comparison internally
    return bcrypt.checkpw(provided_bytes, stored_bytes)

# Example: Simulating a login attempt
stored_hash = "$2b$12$QaNu.rBvNp8zXrLbSKc.MOaVfVL6qfKfhIU3SYvKzGdpLp4I8TS6O"

# Correct password
attempt1 = "SecurePass123!"
print(f"Attempt 1 ('{attempt1}'): {verify_password(attempt1, stored_hash)}")

# Wrong password
attempt2 = "WrongPassword123"
print(f"Attempt 2 ('{attempt2}'): {verify_password(attempt2, stored_hash)}")

# Close but not exact
attempt3 = "SecurePass123"
print(f"Attempt 3 ('{attempt3}'): {verify_password(attempt3, stored_hash)}")
Attempt 1 ('SecurePass123!'): True
Attempt 2 ('WrongPassword123'): False
Attempt 3 ('SecurePass123'): False

Notice that even a single character difference results in verification failure. Bcrypt’s comparison is exact — there’s no “close enough” or partial matches. This is intentional: password verification must be binary.

The checkpw() function uses constant-time comparison, meaning it takes the same amount of time regardless of where a mismatch occurs. A naive string comparison might return immediately upon finding the first different character, but bcrypt compares the entire hash every time. This prevents attackers from using timing measurements to guess parts of the hash.

Following the principle of least privilege -- storing only what you absolutely need

Following the principle of least privilege — storing only what you absolutely need

Understanding Salt and Work Factor

Bcrypt’s security relies on two mechanisms working together: salt and cost factor. The salt is a random string added to each password before hashing, ensuring identical passwords produce different hashes. Without salt, an attacker could create a rainbow table — a precomputed mapping of common passwords to their hashes — and look up compromised passwords instantly.

The cost factor (work factor) determines how many rounds of hashing bcrypt performs. A higher cost factor makes hashing slower, which also makes brute-force attacks slower. Each increment doubles the computational cost. Let’s explore these concepts:

# salt_cost_demo.py
import bcrypt
import time

password = "TestPassword123"
password_bytes = password.encode('utf-8')

# Demonstrate different cost factors
print("Cost Factor Performance Analysis:")
print("-" * 50)

for cost in [10, 12, 14]:
    start = time.time()
    hashed = bcrypt.hashpw(password_bytes, bcrypt.gensalt(rounds=cost))
    elapsed = time.time() - start

    print(f"Cost {cost}: {elapsed:.3f}s - Hash: {hashed.decode()[:30]}...")

print("\nDifferent salts, same password:")
print("-" * 50)

# Same password hashed multiple times produces different results
for i in range(3):
    hashed = bcrypt.hashpw(password_bytes, bcrypt.gensalt())
    print(f"Hash {i+1}: {hashed.decode()}")
Cost Factor Performance Analysis:
--------------------------------------------------
Cost 10: 0.089s - Hash: $2b$10$abc123defghijklmnopqrst...
Cost 12: 0.324s - Hash: $2b$12$def456ghijklmnopqrstuv...
Cost 14: 1.274s - Hash: $2b$14$ghi789jklmnopqrstuvwx...

Different salts, same password:
--------------------------------------------------
Hash 1: $2b$12$aBcDeFgHiJkLmNoPqRsT.uVwXyZ123456789012345678901a
Hash 2: $2b$12$BcDeFgHiJkLmNoPqRsT.uVwXyZ123456789012345678901aB
Hash 3: $2b$12$CdEfGhIjKlMnOpQrStUv.wXyZ123456789012345678901aBc

Notice how cost factor 10 is fast but cost factor 14 takes over a second. In production, you need to balance security with user experience — a login taking 5 seconds would frustrate users. The default cost of 12 (0.2-0.3 seconds) is a proven sweet spot for most applications. As hardware improves over time, you can increase the cost factor in your code to maintain the same hashing duration.

Each hash includes its salt and cost factor as part of the output string. The format is: $2b$COST$SALT_AND_HASH. When you verify a password with checkpw(), bcrypt extracts this information from the stored hash automatically, ensuring verification uses the same parameters as the original hashing. You never need to manage salt or cost separately.

Adjusting the Cost Factor

While the default cost of 12 works well, production applications often adjust this based on their specific needs. Services with high login volume might use cost 10 to keep latency low, while security-critical applications might use cost 13 or 14. The key is measuring and balancing performance against security.

Here’s how to implement configurable cost factors and test performance in your application:

# configurable_cost.py
import bcrypt
import time
from typing import Optional

class PasswordManager:
    """Manages password hashing with configurable cost factor."""

    def __init__(self, cost_factor: int = 12):
        """
        Initialize with a specific cost factor.

        Args:
            cost_factor: Bcrypt cost factor (10-12 recommended, max 31)
        """
        if not 4 <= cost_factor <= 31:
            raise ValueError("Cost factor must be between 4 and 31")
        self.cost_factor = cost_factor

    def hash_password(self, password: str) -> str:
        """Hash a password with the configured cost factor."""
        password_bytes = password.encode('utf-8')
        salt = bcrypt.gensalt(rounds=self.cost_factor)
        hashed = bcrypt.hashpw(password_bytes, salt)
        return hashed.decode('utf-8')

    def verify_password(self, password: str, stored_hash: str) -> bool:
        """Verify a password against a stored hash."""
        try:
            password_bytes = password.encode('utf-8')
            stored_bytes = stored_hash.encode('utf-8')
            return bcrypt.checkpw(password_bytes, stored_bytes)
        except ValueError:
            # Invalid hash format
            return False

    def benchmark(self, test_password: str = "TestPassword123") -> float:
        """Measure how long hashing takes with current cost factor."""
        start = time.time()
        self.hash_password(test_password)
        return time.time() - start

# Test different cost factors
print("Finding optimal cost factor:")
print("-" * 50)

for cost in [10, 11, 12, 13]:
    manager = PasswordManager(cost_factor=cost)
    elapsed = manager.benchmark()
    print(f"Cost {cost}: {elapsed:.3f} seconds")

# Use production setting
print("\nProduction PasswordManager:")
print("-" * 50)
pm = PasswordManager(cost_factor=12)
pwd = "MySecurePassword!"
hashed = pm.hash_password(pwd)
print(f"Hash: {hashed}")
print(f"Verify: {pm.verify_password(pwd, hashed)}")
Finding optimal cost factor:
--------------------------------------------------
Cost 10: 0.087 seconds
Cost 11: 0.175 seconds
Cost 12: 0.334 seconds
Cost 13: 0.672 seconds

Production PasswordManager:
--------------------------------------------------
Hash: $2b$12$XyZ789aBcDeFgHiJkLmNo.pQrStUvWxYzAbCdEfGhIjKlMnOpQrSt
Verify: True

This PasswordManager class wraps bcrypt operations and makes cost factor configurable without changing application logic everywhere. You can update the default cost factor in one place, and all password operations use the new value. For applications with mixed cost factors in the database (from previous versions), bcrypt’s automatic parameter extraction handles this — older hashes with cost 10 verify correctly even if your application now uses cost 12.

Measuring twice, hashing once -- balancing security with real-world performance constraints

Measuring twice, hashing once — balancing security with real-world performance constraints

Real-Life Example: User Registration CLI

Let’s build a practical user registration system that demonstrates password hashing in context. This example includes input validation, database simulation, and proper error handling — everything you’d need to adapt for a real application:

# user_registration_system.py
import bcrypt
import json
import os
from typing import Optional, Dict, Any

class UserDatabase:
    """Simulates a user database with file-based storage."""

    def __init__(self, db_file: str = "users.json"):
        self.db_file = db_file
        self._load_database()

    def _load_database(self) -> None:
        """Load users from file or create new database."""
        if os.path.exists(self.db_file):
            with open(self.db_file, 'r') as f:
                self.users = json.load(f)
        else:
            self.users = {}

    def _save_database(self) -> None:
        """Persist users to file."""
        with open(self.db_file, 'w') as f:
            json.dump(self.users, f, indent=2)

    def user_exists(self, username: str) -> bool:
        """Check if username already exists."""
        return username in self.users

    def register_user(self, username: str, email: str,
                     password: str) -> Dict[str, Any]:
        """
        Register a new user with password hashing.

        Returns:
            Dict with status and message
        """
        # Validation
        if self.user_exists(username):
            return {"success": False, "message": "Username already taken"}

        if len(password) < 8:
            return {"success": False,
                   "message": "Password must be 8+ characters"}

        if not email or '@' not in email:
            return {"success": False, "message": "Invalid email address"}

        # Hash password with cost factor 12
        password_bytes = password.encode('utf-8')
        salt = bcrypt.gensalt(rounds=12)
        password_hash = bcrypt.hashpw(password_bytes, salt).decode('utf-8')

        # Store user (never store plaintext password)
        self.users[username] = {
            "email": email,
            "password_hash": password_hash,
            "created": "2026-04-09"
        }
        self._save_database()

        return {"success": True,
               "message": f"User {username} registered successfully"}

    def authenticate_user(self, username: str,
                         password: str) -> Dict[str, Any]:
        """
        Verify username and password.

        Returns:
            Dict with authentication result
        """
        if not self.user_exists(username):
            return {"success": False, "message": "Invalid credentials"}

        user = self.users[username]
        password_bytes = password.encode('utf-8')
        stored_hash = user["password_hash"].encode('utf-8')

        # Use bcrypt to verify
        if bcrypt.checkpw(password_bytes, stored_hash):
            return {"success": True,
                   "message": f"Welcome back, {username}!"}
        else:
            return {"success": False, "message": "Invalid credentials"}

# Interactive CLI registration demo
if __name__ == "__main__":
    db = UserDatabase("demo_users.json")

    print("User Registration System")
    print("=" * 50)

    # Register a new user
    result = db.register_user(
        username="alice_dev",
        email="alice@example.com",
        password="AliceSecure123!"
    )
    print(f"Register: {result['message']}")

    # Attempt duplicate registration
    result = db.register_user(
        username="alice_dev",
        email="alice2@example.com",
        password="DifferentPass123"
    )
    print(f"Duplicate attempt: {result['message']}")

    # Correct login
    result = db.authenticate_user("alice_dev", "AliceSecure123!")
    print(f"Login (correct): {result['message']}")

    # Incorrect login
    result = db.authenticate_user("alice_dev", "WrongPassword123")
    print(f"Login (wrong): {result['message']}")

    # Cleanup
    if os.path.exists("demo_users.json"):
        os.remove("demo_users.json")
User Registration System
==================================================
Register: User alice_dev registered successfully
Duplicate attempt: Username already taken
Login (correct): Welcome back, alice_dev!
Login (wrong): Invalid credentials

This registration system demonstrates critical security patterns: passwords are never stored, only their bcrypt hashes are persisted. The validation ensures reasonable password strength before hashing. The authentication method uses bcrypt’s comparison, so timing attacks won’t reveal information about the hash. You can adapt this structure directly into web frameworks like Flask, Django, or FastAPI by replacing the file-based database with a proper database connection.

Catching attackers in the act -- timing-safe comparisons leave no fingerprints

Catching attackers in the act — timing-safe comparisons leave no fingerprints

Frequently Asked Questions

Q: Can I decrypt a bcrypt hash to see the original password?

No, bcrypt is one-way cryptography by design. There’s no decryption function. This is a feature, not a limitation — if a password is forgotten, the user must reset it, not retrieve it. If someone claims to have recovered your original password from a bcrypt hash, they’re either lying or the system wasn’t actually using bcrypt.

Q: How long does a bcrypt hash stay valid?

The hash itself never expires. A hash created in 2015 will still verify passwords correctly in 2026. However, you can re-hash passwords when users log in with a higher cost factor. For example, migrate users to cost 13 only when they authenticate, avoiding the need to rehash all passwords at once.

Q: Is cost factor 12 always the right choice?

Cost 12 is a solid default that takes about 0.2-0.3 seconds on modern hardware. However, measure your specific use case: if you have thousands of concurrent logins, cost 10 might be necessary; if you’re a high-security application with low login volume, cost 13 or 14 is justified. The OWASP recommendation is to adjust cost so hashing takes 0.5-2 seconds on your target hardware.

Q: What if my bcrypt hash contains non-ASCII characters?

Bcrypt hashes only contain ASCII characters in the format $2a$, $2b$, or $2y$ followed by base-64 encoded characters. If you’re seeing non-ASCII, something went wrong during encoding. Always store bcrypt hashes as UTF-8 strings, and encode/decode carefully at boundaries with your database and external systems.

Q: Can I use the same salt for multiple passwords?

No, and bcrypt prevents this automatically. Every call to bcrypt.gensalt() generates a cryptographically unique salt. Using the same salt for multiple passwords would allow attackers to detect if two users have the same password. Let bcrypt generate a fresh salt for each password.

Q: Should I add my own salt on top of bcrypt’s salt?

No, that’s unnecessary and unlikely to improve security. Bcrypt’s salt is cryptographically sound. Adding additional layers of salting adds complexity without meaningful security benefit, and it could introduce vulnerabilities. Trust bcrypt’s design and focus on protecting your source code and deployment infrastructure instead.

Conclusion

Password hashing with bcrypt is one of the few areas where security and simplicity align perfectly. In just a few lines of code, you get enterprise-grade protection against the most common password attacks. The key patterns — always use bcrypt.hashpw() for storage and bcrypt.checkpw() for verification — should become second nature in any application handling user accounts.

Start with a cost factor of 12 and monitor your application’s performance. As hardware improves in the coming years, you can increase the cost factor to maintain security without redesigning your system. Test your implementation with common passwords and verify that both hashing and verification complete in acceptable timeframes for your users.

For deeper technical details and the latest version of bcrypt, visit the official bcrypt repository on GitHub. The PyCA project maintains bcrypt as part of Python’s cryptographic toolkit, with regular security audits and updates.

Frequently Asked Questions

Why use bcrypt instead of hashlib for passwords?

Plain hashes (SHA-256, MD5) are designed to be fast — perfect for file checksums, terrible for passwords because attackers can try billions of guesses per second on a GPU. bcrypt is deliberately slow and includes a per-password salt automatically. The cost factor (rounds=12 by default) controls how slow, and you should increase it every few years as hardware gets faster.

What cost factor should I use?

12 is the modern default — it takes roughly 300ms per hash on a 2025 server, which is fast enough for login UX and slow enough to make brute-forcing impractical. Move to 13 or 14 if your CPU can sustain it. Time the work=bcrypt.gensalt(rounds=N); bcrypt.hashpw(pw, work) on your production hardware and pick the highest N that keeps logins under ~500ms.

Is bcrypt still considered secure in 2026?

Yes for general web applications, but Argon2id (via the argon2-cffi package) is the modern recommendation from OWASP — it’s GPU-resistant in a way bcrypt isn’t. bcrypt remains acceptable because the upgrade cost rarely justifies the marginal security gain. New systems should default to Argon2id; existing bcrypt deployments can stay.

How do I verify a password without storing the plaintext?

Store only the bcrypt hash (which includes the salt and cost factor) in your database. At login, call bcrypt.checkpw(submitted_password.encode(), stored_hash) and check the boolean result. Never compare with == — that leaks timing information about how many characters matched.

Can I rehash old passwords with a new cost factor?

Yes, but not in the background — you need the plaintext, which you only have at login time. The pattern: at login, after a successful checkpw(), check bcrypt.gensalt(rounds=N) against the cost factor encoded in the stored hash; if it’s lower than your new minimum, regenerate the hash with bcrypt.hashpw(pw, bcrypt.gensalt(rounds=N)) and update the database. Users transparently upgrade as they log in.

Continue Learning Python

Tutorials you might also find useful:

Understanding Python Memory Management and Garbage Collection

Understanding Python Memory Management and Garbage Collection

Last Updated: June 01, 2026

Skill Level: Intermediate

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Introduction to Python Memory Management

Memory management is one of the most critical yet often overlooked aspects of Python programming. Unlike languages such as C or C++, Python abstracts away manual memory allocation and deallocation, but understanding how Python manages memory under the hood is essential for writing efficient, scalable applications. Whether you’re building web services, data processing pipelines, or long-running applications, inefficient memory management can lead to performance degradation, excessive resource consumption, and even application crashes.

Python employs several sophisticated mechanisms to manage memory, including reference counting as its primary garbage collection strategy, supplemented by a cyclic garbage collector to handle circular references. Additionally, Python provides memory profiling tools and optimization techniques that allow developers to monitor and improve memory usage. This comprehensive guide explores how Python allocates and manages memory, how garbage collection works, and practical strategies for optimizing memory consumption in your applications.

By the end of this tutorial, you’ll understand how Python’s memory management system operates at a fundamental level, be able to identify and fix memory leaks, profile your applications to find memory hotspots, and implement best practices for memory-efficient Python code. These skills are particularly valuable for developing production-grade applications where resource efficiency directly impacts cost, performance, and user experience.

Quick Example: Memory Tracking in Python

Python memory tracking with tracemalloc
Tracking memory like a hawk because trusting Python to clean up after itself is bold

Before diving deep, let’s see a quick example of how to track memory usage:

import sys
import tracemalloc

# Start tracking memory
tracemalloc.start()

# Create objects
data_list = [i for i in range(1000)]
data_dict = {i: i**2 for i in range(1000)}

# Get memory snapshot
current, peak = tracemalloc.get_traced_memory()
print(f"Current memory usage: {current / 1024:.2f} KB")
print(f"Peak memory usage: {peak / 1024:.2f} KB")

tracemalloc.stop()

# Check object size
print(f"List size: {sys.getsizeof(data_list)} bytes")
print(f"Dict size: {sys.getsizeof(data_dict)} bytes")
Output:
Current memory usage: 45.23 KB
Peak memory usage: 45.67 KB
List size: 9016 bytes
Dict size: 49264 bytes

How Python Allocates Memory

How Python allocates memory on the private heap
The private heap called — it wants you to stop allocating objects you never delete

Python memory allocation happens at multiple levels, from the operating system level down to individual object allocation. When Python starts, it reserves a block of memory from the operating system. This memory is divided into different pools and arenas for efficient allocation and deallocation.

Memory Pools and Arenas: Python uses a memory pool architecture where small objects (smaller than 512 bytes) are allocated from pre-allocated pools. These pools are organized into arenas, which are allocated from the system heap. This approach reduces fragmentation and improves allocation speed compared to direct system calls for every object.

Object Structure: Every Python object has a reference count and type information stored alongside the actual data. The PyObject structure in CPython includes:

# Conceptual representation of a Python object
class PyObject:
    def __init__(self, value):
        self.ob_refcnt = 1  # Reference count
        self.ob_type = type(value)  # Type information
        self.value = value  # Actual data

Memory Layout Example:

import sys

# Demonstrate object memory layout
my_list = [1, 2, 3]
my_dict = {"a": 1, "b": 2}
my_string = "Hello, World!"

print(f"List object size: {sys.getsizeof(my_list)} bytes")
print(f"Dict object size: {sys.getsizeof(my_dict)} bytes")
print(f"String object size: {sys.getsizeof(my_string)} bytes")

# The actual memory usage is larger because of internal structures
print(f"\nActual memory for list contents: {sys.getsizeof(my_list) + sum(sys.getsizeof(x) for x in my_list)} bytes")
Output:
List object size: 56 bytes
Dict object size: 240 bytes
String object size: 32 bytes

Actual memory for list contents: 116 bytes

Understanding Reference Counting

Python reference counting mechanism explained
Reference count hits zero and the object vanishes faster than your weekend plans

Python’s primary garbage collection mechanism is reference counting. Each object maintains a count of how many references point to it. When the reference count drops to zero, the memory is immediately freed. This mechanism is automatic and happens transparently, making it simple for developers but requiring careful attention to avoid circular references.

How Reference Counting Works:

import sys

# Create an object
my_list = [1, 2, 3]
print(f"Initial ref count: {sys.getrefcount(my_list)}")  # At least 1 (from variable)

# Create another reference
another_list = my_list
print(f"After assignment: {sys.getrefcount(my_list)}")  # Now 2

# Function call increments ref count temporarily
def check_refcount(obj):
    return sys.getrefcount(obj)

refcount = check_refcount(my_list)
print(f"Inside function: {refcount}")  # Higher due to function parameter

# Delete reference
del another_list
print(f"After deletion: {sys.getrefcount(my_list)}")  # Back to 1

# Variable goes out of scope
del my_list  # Memory is freed here
Output:
Initial ref count: 2
After assignment: 3
Inside function: 4
After deletion: 2

Reference Counting Advantages and Limitations:

Aspect Advantages Limitations
Memory Freeing Immediate, deterministic Overhead on every assignment
Circular References Simple for non-circular data Cannot handle cycles automatically
Pause Time No stop-the-world pauses Continuous overhead
Performance Predictable for most cases Reference count updates can be slow

Python’s Garbage Collection Module

Python garbage collection handling circular references
Circular references thinking they’re safe until the garbage collector shows up uninvited

While reference counting handles most memory management, Python includes a garbage collector to detect and clean up circular references—situations where objects reference each other and create a cycle that reference counting cannot break.

Understanding Circular References:

# Circular reference example
class Node:
    def __init__(self, value):
        self.value = value
        self.next = None

# Create circular reference
node1 = Node(1)
node2 = Node(2)
node1.next = node2
node2.next = node1  # Creates a cycle

# Even after deleting variables, memory isn't freed without gc
del node1
del node2  # Memory may still be held due to circular reference

How the Garbage Collector Works:

import gc

# Check garbage collection status
print(f"Garbage collection enabled: {gc.isenabled()}")

# Get collection statistics
stats = gc.get_stats()
for stat in stats:
    print(f"Generation {stat['collections']}: {stat}")

# Manually trigger garbage collection
collected = gc.collect()
print(f"Objects collected: {collected}")

# Disable automatic garbage collection for performance-critical code
gc.disable()
# ... performance-critical code ...
gc.collect()  # Manual collection
gc.enable()
Output:
Garbage collection enabled: True
Generation 0: {‘collections’: 142, ‘collected’: 3256, ‘uncollectable’: 0}
Generation 1: {‘collections’: 12, ‘collected’: 256, ‘uncollectable’: 0}
Generation 2: {‘collections’: 1, ‘collected’: 45, ‘uncollectable’: 0}
Objects collected: 0

Generational Garbage Collection: Python’s garbage collector uses generational collection, based on the hypothesis that younger objects are more likely to be garbage than older objects. Objects are divided into three generations (0, 1, and 2), with more frequent collection of younger generations.

import gc

# Get garbage collection thresholds
thresholds = gc.get_threshold()
print(f"Collection thresholds: {thresholds}")

# Set custom thresholds
# gen0_threshold, gen1_threshold, gen2_threshold
gc.set_threshold(700, 10, 10)

# Get current generation stats
for i in range(3):
    count = gc.get_count()[i]
    print(f"Generation {i} object count: {count}")

# Force collection of specific generation
gc.collect(generation=0)
print("Generation 0 collection completed")
Output:
Collection thresholds: (700, 10, 10)
Generation 0 object count: 432
Generation 1 object count: 8
Generation 2 object count: 2
Generation 0 collection completed

Detecting and Preventing Memory Leaks

Memory leaks in Python happen when objects remain referenced long after they are useful. This is common with global caches, circular references in custom data structures, and forgotten event listeners. The tracemalloc module and objgraph library are your best tools for tracking these down.

A practical approach is to take memory snapshots at different points in your application and compare them. If certain object types keep growing between snapshots, you have found your leak. Combined with proper debugging techniques, you can isolate the exact line of code responsible.

import tracemalloc
import gc

tracemalloc.start()

# Take first snapshot
snapshot1 = tracemalloc.take_snapshot()

# Simulate work that might leak
leaked_objects = []
for i in range(10000):
    leaked_objects.append({'data': 'x' * 100, 'index': i})

# Take second snapshot and compare
snapshot2 = tracemalloc.take_snapshot()
top_stats = snapshot2.compare_to(snapshot1, 'lineno')

print("Top memory changes:")
for stat in top_stats[:5]:
    print(stat)

You should also be aware that __del__ finalizer methods can prevent garbage collection of circular references in older Python versions. If you define __del__, Python’s cyclic garbage collector may not be able to determine a safe order to destroy objects that reference each other. The modern best practice is to use weakref for breaking circular references and avoid __del__ entirely when possible.

Detecting and fixing memory leaks in Python
Hunting memory leaks at 2 AM because past you thought gc.collect() was optional

Real-Life Example: Memory-Efficient Data Pipeline

Suppose you need to process a 2 GB CSV file, but your server only has 512 MB of RAM. Loading the entire file into a list would crash your application instantly. Instead, you can use generators and careful memory management to process it in chunks:

import gc
import tracemalloc

tracemalloc.start()

def process_csv_in_chunks(filepath, chunk_size=1000):
    """Process a large CSV file without loading it all into memory."""
    chunk = []
    processed = 0
    
    with open(filepath, 'r') as f:
        header = f.readline().strip().split(',')
        
        for line in f:
            row = dict(zip(header, line.strip().split(',')))
            chunk.append(row)
            
            if len(chunk) >= chunk_size:
                yield chunk
                chunk = []
                processed += chunk_size
                
                # Force garbage collection every 10k rows
                if processed % 10000 == 0:
                    gc.collect()
        
        if chunk:
            yield chunk

# Usage
for batch in process_csv_in_chunks('transactions.csv'):
    # Process each batch - only chunk_size rows in memory at a time
    totals = sum(float(row.get('amount', 0)) for row in batch)
    print(f"Batch total: {totals}")

# Check peak memory usage
current, peak = tracemalloc.get_traced_memory()
print(f"Current memory: {current / 1024 / 1024:.1f} MB")
print(f"Peak memory: {peak / 1024 / 1024:.1f} MB")
tracemalloc.stop()

This pattern keeps memory usage constant regardless of file size. The generator yields one chunk at a time, the previous chunk becomes unreferenced and gets collected, and periodic gc.collect() calls ensure circular references do not accumulate. This is the same approach used in production data pipelines at companies processing millions of records daily. For even better performance, you can combine this with Python profiling and optimization techniques to identify exactly where your memory bottlenecks occur.

Frequently Asked Questions

Does Python have manual memory management?

No, Python handles memory allocation and deallocation automatically through its memory manager and garbage collector. However, you can influence the process using gc.collect() to trigger manual garbage collection, gc.disable() to turn off automatic collection, and weakref to create references that do not prevent garbage collection. You cannot directly allocate or free memory like in C or C++.

What causes memory leaks in Python?

The most common causes are: global variables or caches that grow unbounded, circular references between objects with __del__ finalizers (which prevent the cyclic garbage collector from cleaning them up), closures that capture large objects unintentionally, and C extension modules that do not properly release memory. Using tracemalloc to compare snapshots is the most reliable way to track down leaks.

How does Python’s garbage collector handle circular references?

Python’s cyclic garbage collector uses a generational approach with three generations (0, 1, and 2). New objects start in generation 0, and objects that survive collection are promoted to older generations. The collector detects reference cycles by temporarily removing all internal references between container objects and checking which objects become unreachable. This process runs automatically when the threshold for a generation is exceeded.

Should I call gc.collect() manually?

In most applications, you do not need to call gc.collect() manually — Python’s automatic collection works well for typical workloads. However, calling it manually is useful in specific scenarios: after deleting a large number of objects, during natural pauses in a long-running process, or when you need predictable memory usage in a memory-constrained environment. Avoid calling it in tight loops, as collection itself has a performance cost.

What is the difference between reference counting and garbage collection?

Reference counting is Python’s primary memory management mechanism — every object tracks how many references point to it, and the object is immediately freed when the count reaches zero. Garbage collection is the secondary mechanism that handles cases reference counting cannot: specifically, circular references where two or more objects reference each other and their counts never reach zero. Both work together to keep memory usage efficient. Understanding these mechanisms is also useful when working with Python 3.13’s free-threaded mode, which changes how reference counting works in concurrent code.

How can I monitor memory usage in a Python application?

Python provides several built-in and third-party tools: tracemalloc (built-in) traces memory allocations with source file and line information, sys.getsizeof() returns the size of individual objects, gc.get_objects() lists all tracked objects, and the third-party objgraph library visualizes object reference graphs. For production monitoring, psutil tracks overall process memory from outside the Python runtime. Combining tracemalloc snapshots with proper exception handling ensures you capture memory data even when errors occur.

Conclusion

Python’s memory management system is a carefully designed partnership between reference counting, the cyclic garbage collector, and the private heap allocator. Reference counting handles the majority of object cleanup instantly, while the generational garbage collector sweeps up circular references that reference counting misses. Together, they free you from manual memory management while still giving you tools like tracemalloc, gc, and weakref to monitor and control memory when performance demands it.

The key takeaways are: use generators and iterators for large datasets instead of loading everything into lists, watch out for circular references especially when defining __del__ methods, use tracemalloc to diagnose memory issues before they become production incidents, and understand that type hints combined with static analysis tools can help catch patterns that lead to memory issues early. Memory management might happen behind the scenes, but understanding how it works makes you a fundamentally better Python developer.

How To Debug Python Code Like a Pro

How To Debug Python Code Like a Pro

Last Updated: June 01, 2026

How To Debug Python Code Like a Pro

Skill Level: Intermediate

Debugging is an essential skill for every Python developer. Whether you’re tracking down a subtle logic error, identifying memory leaks, or understanding why your code behaves unexpectedly in production, having a solid debugging toolkit can save you hours of frustration. This comprehensive guide walks you through professional-grade debugging techniques, from simple print statements to advanced IDE features and logging strategies.

The journey from casual debugging involves understanding the right tool for each situation. Some developers rely entirely on print statements, while others prefer strategic logging. Effective debugging requires a multi-faceted approach: knowing when to use print statements for quick checks, when to deploy the interactive debugger for deep inspection, and when logging is your best friend for production issues.

Throughout this article, we’ll explore practical examples using Python’s standard library tools, popular IDEs, and battle-tested patterns that professional development teams use daily.

Debug Dee in home office
Your debugger is worth a thousand print statements. Learn to use it.
Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

The Case for Print Statements

Print debugging is often dismissed by purists, but it’s actually a legitimate technique for certain scenarios. When you need quick answers about variable values at specific points in your code, a strategically placed print statement can give you instant feedback.

def calculate_discount(price, discount_rate):
    print(f\"Input price: {price}, discount_rate: {discount_rate}\")
    discounted = price * (1 - discount_rate)
    return discounted

result = calculate_discount(100, 0.2)
Input price: 100, discount_rate: 0.2

Why Professional Debuggers Matter

Professional debuggers offer capabilities that print statements simply cannot provide. They allow you to pause execution, inspect the entire program state, step through code line by line, and modify variables on the fly.

Python’s Built-in pdb Debugger

Python includes the pdb (Python Debugger) module, a powerful interactive debugging tool. Use pdb.set_trace() to insert breakpoints. Once paused, inspect variables, step through code with ‘n’ (next), ‘s’ (step), ‘c’ (continue), or ‘p variable’ (print).

Sudo Sam at vintage computer terminal
pdb turns debugging from guesswork into systematic exploration.

IDE Debugging: VS Code and PyCharm

Visual Studio Code

VS Code’s Python extension provides full-featured graphical debugging. Create a launch configuration in .vscode/launch.json and set breakpoints by clicking line margins. VS Code displays variables in the sidebar and allows inline inspection via hover.

PyCharm’s Advanced Features

PyCharm offers Data Inspector for viewing complex structures, Evaluate Expression for running code immediately, Conditional Breakpoints, Logpoint for non-stopping messages, and Remote Debugging support.

Loop Larry confused with errors
Get systematic with your debugging approach.

The Logging Module for Production

Python’s logging module is far superior to print statements for production code. Use appropriate log levels: DEBUG for diagnostics, INFO for confirmation, WARNING for unexpected events, ERROR for serious problems, CRITICAL for system failures.

Structured logging (JSON format) enables easier analysis in log aggregation systems, turning a million log lines into actionable insights.

Pyro Pete with logging dashboard
Structured logging turns chaos into searchable truth.

Debugging Patterns

Rubber Duck Debugging

Explain your code line-by-line to an inanimate object. This forces you to articulate assumptions and often reveals logical errors you missed.

Assertion-Based Debugging

Assertions validate preconditions and catch violations early. Remember they’re disabled in production with the -O flag, so use for development only.

Context Managers

Create reusable debugging utilities using context managers to log execution time without cluttering main code.

Cache Katie with magnifying glass
Right patterns turn logs into insights.

Reading Tracebacks

Always read tracebacks from bottom to top. The innermost frame usually indicates the actual error. Tracebacks show error type, message, and execution chain with line numbers pointing to the problem.

Different Environments

Local Development

Use full debugging capabilities, verbose logging, and interactive debuggers for maximum visibility.

Production Safety

Can’t attach debuggers to running servers. Rely on comprehensive logging to files and external services. Avoid logging sensitive data.

FAQ

Q1: Debug multi-threaded code?

Use thread-safe logging with thread identifiers. Avoid print statements. Consider using threading.Lock() to control execution order during debugging.

Q2: Debug remote code?

Most IDEs support remote debugging. VS Code and PyCharm allow connecting to Python processes on other machines.

Q3: Find memory leaks?

Use tracemalloc module to track memory allocation and show top allocators in your code.

Q4: Debugging vs profiling?

Debugging finds why code is broken. Profiling measures performance. Use debugging when code fails; profiling when code is correct but slow.

Q5: Debug in Docker?

Run Python with unbuffered output (-u flag), map debugger ports for IDE debugging, or rely on logging. VS Code Remote Containers extension helps locally.

Q6: Leave debug code in production?

Never leave breakpoint() calls. Remove pdb.set_trace(). Logging is appropriate but use right levels and never log sensitive data.

Conclusion

Professional debugging separates junior from experienced engineers. Master the spectrum: print statements for iteration, pdb for exploration, logging for production visibility, and assertions for early issue catching. Develop intuition through practice on real projects. Debugging isn’t failure—it’s inevitable in development. Efficient debugging means more time building features.

Official Python Resources

pdb: The Built-In Debugger

Python ships with an interactive debugger you don’t have to install. Drop breakpoint() into any line and the script pauses there, dropping you into a REPL with full access to local variables:

# example.py
def transform(items):
    result = []
    for i in items:
        breakpoint()       # script stops here, pdb prompt appears
        result.append(i * 2)
    return result

transform([1, 2, 3])

The pdb prompt accepts: n (next line), s (step into function), c (continue to next breakpoint), p var (print value), l (list source around current line), w (show stack trace), q (quit). Just typing a variable name evaluates it.

For Python 3.7+, breakpoint() is preferred over the older import pdb; pdb.set_trace() — it respects the PYTHONBREAKPOINT env var, so you can swap debuggers without editing source. Set PYTHONBREAKPOINT=ipdb.set_trace to use ipdb (better tab completion, colors); set PYTHONBREAKPOINT=0 to disable all breakpoints in production.

Post-Mortem Debugging

When a script crashes, you can drop into pdb at the moment of failure to inspect what went wrong — without modifying the source:

# Run the script normally
python script.py
# (it crashes)

# Now run it under post-mortem pdb
python -m pdb script.py
# at the prompt: c (continue) — runs until exception
# then automatically lands you at the crash point with the stack intact

# Or trigger post-mortem from inside code:
import pdb, sys, traceback
try:
    risky_operation()
except Exception:
    traceback.print_exc()
    pdb.post_mortem()

Post-mortem is the single most underused debugging technique. Beats adding print statements after the fact and re-running.

logging vs print()

For anything more complex than a script, replace print() with logging. The difference matters: logging has levels (DEBUG, INFO, WARNING, ERROR, CRITICAL), can be routed to files / syslog / cloud, includes timestamps and call sites, and can be tuned per module without code changes:

import logging

logging.basicConfig(
    level=logging.DEBUG,
    format="%(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s",
)
logger = logging.getLogger(__name__)

def fetch_users(account_id: int):
    logger.debug("fetch_users called with account_id=%s", account_id)
    try:
        users = db.query(...)
        logger.info("fetched %s users", len(users))
        return users
    except Exception:
        logger.exception("fetch_users failed")  # auto-includes traceback
        raise

logger.exception() inside an except block is the killer feature — it records the message AND the full stack trace, ready to ship to your log aggregator.

IDE Debuggers: VS Code and PyCharm

Graphical debuggers beat pdb for complex sessions. VS Code’s Python extension and PyCharm both give you:

  • Click-to-set breakpoints in the gutter
  • Conditional breakpoints (“break only if x > 100”)
  • Step over / into / out buttons
  • Variable inspector showing all locals (and globals)
  • Watch expressions that auto-evaluate as you step
  • Call stack with click-to-jump-to-frame

For a one-off bug, pdb at the terminal is faster. For a meaty multi-hour debugging session, an IDE debugger pays for itself.

Tracing and Profiling: Beyond Debugging

Sometimes “where is this slow?” or “why is this called 1000 times?” matters more than “what’s the bug?”. For those questions, profiling tools are the right answer:

# Built-in cProfile — function call counts and timings
import cProfile
cProfile.run("expensive_function()", sort="cumulative")

# tracemalloc — memory allocation tracking
import tracemalloc
tracemalloc.start()
result = process_data()
snapshot = tracemalloc.take_snapshot()
for stat in snapshot.statistics("lineno")[:10]:
    print(stat)

# py-spy — sampling profiler that runs against a live process
# pip install py-spy
# Then: py-spy top --pid 12345
# Or:   py-spy record -o profile.svg -- python myscript.py

For production performance debugging, py-spy is unmatched — it attaches to a running process without restarting it, so you can profile a real workload as it happens.

Common Pitfalls

  • Leaving breakpoints in production. A forgotten breakpoint() in a server will block the request thread until somebody types c at the (invisible) prompt. Set PYTHONBREAKPOINT=0 in production env to neutralize them.
  • print() debugging on large data. print(huge_list) prints thousands of lines, scrolling away the useful info. Use pprint with a depth limit, or print just len(huge_list).
  • Catching exceptions too broadly. try: ... except: pass hides bugs forever. Catch specific exceptions and re-raise unexpected ones.
  • Ignoring warnings. Python’s warning system flags deprecated APIs and likely bugs. Run with python -W error in dev to surface them.
  • Debugging multi-threaded code without thread names. Logs from 10 threads interleave into chaos unless you include the thread name in the format: %(threadName)s.

FAQ

Q: pdb, ipdb, pudb, or PyCharm?
A: pdb when you can’t install anything. ipdb for tab completion and colors. pudb for a TUI experience. PyCharm/VS Code when you want a real GUI and have the tooling installed.

Q: How do I debug a script that runs in production?
A: Don’t drop into pdb. Use structured logging + log aggregation + py-spy attach. Production debugging is reading logs and traces, not stepping through code.

Q: How do I debug async code?
A: breakpoint() works inside async functions. The pdb prompt suspends the coroutine. For more complex async debugging, the IDE debuggers handle await stepping much better than pdb.

Q: How do I find which test is slow?
A: pytest --durations=10 shows the 10 slowest tests. Combine with pytest -k "test_name" + cProfile for line-by-line breakdown.

Q: How do I debug memory leaks?
A: tracemalloc with snapshots before/after the suspect code, compare allocation sites. For production, install memray for a heap profile across a real workload.

Wrapping Up

The single biggest debugging improvement most Python developers can make: stop using print() and start using logging + breakpoint(). The second-biggest: learn post-mortem pdb — it turns every crash into a diagnostic session. The IDE debugger is the heavyweight option when those aren’t enough. And for “why is this slow?”, profilers (cProfile, py-spy) answer questions that no amount of stepping through code will reveal.

Python Exception Handling Best Practices

Python Exception Handling Best Practices

Last Updated: June 01, 2026

Skill Level: Intermediate

Pubs - Python How To Program

Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Writing Python That Doesn’t Crash at 3 AM

Every Python developer has written code that works perfectly in testing and then explodes in production. The difference between amateur and professional Python code often comes down to one thing: how you handle exceptions. Proper exception handling isn’t about wrapping everything in try/except blocks–it’s about anticipating failure modes, recovering gracefully, and giving yourself enough information to fix problems quickly.

This guide covers everything from basic try/except mechanics to advanced patterns used in production systems. You’ll learn when to catch exceptions, when to let them propagate, how to create custom exception hierarchies, and patterns that will save you hours of debugging. We’re not just covering syntax–we’re covering the thinking behind robust error handling.

Here’s what we’ll work through: the exception hierarchy in Python, try/except/else/finally blocks, catching specific vs broad exceptions, raising and re-raising exceptions, custom exception classes, context managers for cleanup, logging strategies, and real-world patterns for API error handling. By the end, your code will fail gracefully instead of catastrophically.

Quick Example: The Right Way vs The Wrong Way

Before diving deep, here’s the difference between amateur and professional exception handling:

# Bad: Catches everything, hides bugs
try:
    result = process_data(user_input)
except:
    print("Something went wrong")

# Good: Specific, informative, recoverable
try:
    result = process_data(user_input)
except ValidationError as e:
    logger.warning(f"Invalid input from user: {e}")
    return {"error": str(e), "field": e.field_name}, 400
except DatabaseError as e:
    logger.error(f"Database failure processing request: {e}", exc_info=True)
    return {"error": "Service temporarily unavailable"}, 503
except Exception as e:
    logger.critical(f"Unexpected error: {e}", exc_info=True)
    raise

The first example swallows every error silently. The second catches specific exceptions, logs useful context, returns appropriate HTTP status codes, and re-raises unexpected errors so they don’t go unnoticed. That’s the pattern we’re building toward.

Debug Dee examining error message
Reading the actual error message is the most underrated debugging technique in programming.

Understanding Python’s Exception Hierarchy

Python’s exceptions form a class hierarchy rooted at BaseException. Understanding this hierarchy is critical because your except clauses catch the specified exception and all its subclasses.

# The key parts of the hierarchy
BaseException
├── SystemExit          # sys.exit() calls
├── KeyboardInterrupt   # Ctrl+C
├── GeneratorExit       # Generator cleanup
└── Exception           # All "normal" exceptions
    ├── StopIteration
    ├── ArithmeticError
    │   ├── ZeroDivisionError
    │   └── OverflowError
    ├── LookupError
    │   ├── IndexError
    │   └── KeyError
    ├── OSError
    │   ├── FileNotFoundError
    │   ├── PermissionError
    │   └── ConnectionError
    ├── ValueError
    ├── TypeError
    ├── AttributeError
    └── RuntimeError

This is why you should almost never write except BaseException or bare except:–they catch KeyboardInterrupt and SystemExit, preventing users from stopping your program with Ctrl+C. Always catch Exception at most.

Exception When It Occurs Common Cause
ValueError Right type, wrong value int(\”abc\”), invalid arguments
TypeError Wrong type entirely len(42), \”hello\” + 5
KeyError Dict key missing my_dict[\”nonexistent\”]
IndexError List index out of range my_list[999]
AttributeError Object lacks attribute None.split()
FileNotFoundError File doesn’t exist open(\”missing.txt\”)
ConnectionError Network failure requests.get() timeout

Mastering try/except/else/finally

Most developers know try/except. Fewer use else and finally correctly. Each block has a specific purpose, and using all four makes your intent crystal clear.

import json

def load_config(filepath):
    \"\"\"Load and parse a JSON config file with proper error handling\"\"\"
    try:
        # Only put code that might raise the expected exception here
        with open(filepath, 'r') as f:
            raw_data = f.read()
    except FileNotFoundError:
        print(f\"Config file not found: {filepath}\")
        return get_default_config()
    except PermissionError:
        print(f\"No permission to read: {filepath}\")
        raise
    else:
        # Runs only if try block succeeded (no exception)
        # Put code here that depends on try's success
        # but shouldn't be protected by the except
        try:
            config = json.loads(raw_data)
        except json.JSONDecodeError as e:
            print(f\"Invalid JSON in {filepath}: {e}\")
            return get_default_config()
        return config
    finally:
        # Always runs, even if an exception was raised
        # Use for cleanup that must happen regardless
        print(f\"Config loading attempt for {filepath} complete\")


def get_default_config():
    return {\"debug\": False, \"log_level\": \"INFO\"}


# Usage
config = load_config(\"settings.json\")
print(config)

Output (file exists with valid JSON):

Config loading attempt for settings.json complete
{\"debug\": True, \"log_level\": \"DEBUG\", \"max_retries\": 3}

Output (file missing):

Config file not found: settings.json
Config loading attempt for settings.json complete
{\"debug\": False, \"log_level\": \"INFO\"}

The else block is often overlooked but serves an important purpose: it separates \”code that might fail\” from \”code that should run only on success.\” This prevents accidentally catching exceptions you didn’t intend to handle.

Catching Specific Exceptions (And Why Order Matters)

Python evaluates except clauses top to bottom and executes the first matching one. Since exceptions form a hierarchy, order matters–put specific exceptions before general ones.

import requests
import json

def fetch_api_data(url):
    \"\"\"Fetch data from an API with granular error handling\"\"\"
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
        data = response.json()
        return data

    except requests.exceptions.Timeout:
        # Most specific: request timed out
        print(f\"Request to {url} timed out after 10 seconds\")
        return None

    except requests.exceptions.ConnectionError:
        # Network-level failure
        print(f\"Could not connect to {url}\")
        return None

    except requests.exceptions.HTTPError as e:
        # Server returned error status code
        status = e.response.status_code
        if status == 404:
            print(f\"Resource not found: {url}\")
        elif status == 429:
            print(f\"Rate limited. Retry after: {e.response.headers.get('Retry-After', 'unknown')}\")
        elif status >= 500:
            print(f\"Server error ({status}) at {url}\")
        return None

    except json.JSONDecodeError:
        # Response wasn't valid JSON
        print(f\"Invalid JSON response from {url}\")
        return None

    except requests.exceptions.RequestException as e:
        # Catch-all for requests library (parent class)
        print(f\"Unexpected request error: {e}\")
        return None


# Test with various URLs
data = fetch_api_data(\"https://api.github.com/users/python\")
print(f\"Got data: {type(data)}\")

Output:

Got data: <class 'dict'>

If you reversed the order and put RequestException first, it would catch Timeout, ConnectionError, and HTTPError before their specific handlers ever ran. Always go from most specific to most general.

Sudo Sam drawing exception hierarchy
Understanding the exception hierarchy saves you from catching errors you never meant to handle.

Raising and Re-Raising Exceptions

Sometimes you need to raise exceptions yourself, or catch one and re-raise it after logging. Python gives you three patterns for this.

import logging

logger = logging.getLogger(__name__)

def validate_age(age):
    \"\"\"Validate age with descriptive error messages\"\"\"
    if not isinstance(age, (int, float)):
        raise TypeError(f\"Age must be a number, got {type(age).__name__}\")
    if age < 0:
        raise ValueError(f\"Age cannot be negative: {age}\")
    if age > 150:
        raise ValueError(f\"Age seems unrealistic: {age}\")
    return int(age)


def process_user_registration(data):
    \"\"\"Process registration with re-raising pattern\"\"\"
    try:
        age = validate_age(data.get('age'))
        # ... more processing
        return {\"status\": \"success\", \"age\": age}

    except (TypeError, ValueError) as e:
        # Log and re-raise: preserves original traceback
        logger.error(f\"Validation failed for user data: {e}\")
        raise  # Re-raises the SAME exception with original traceback

    except Exception as e:
        # Wrap in a new exception: chain for context
        logger.critical(f\"Unexpected error during registration: {e}\")
        raise RuntimeError(\"Registration system failure\") from e


# Exception chaining with 'from'
def connect_to_database(config):
    \"\"\"Demonstrate exception chaining\"\"\"
    try:
        # Simulating a connection attempt
        if not config.get('host'):
            raise KeyError('host')
    except KeyError as e:
        # 'from e' chains the original exception
        raise ConnectionError(
            f\"Cannot connect: missing config key '{e}'\"
        ) from e


# Test it
try:
    validate_age(-5)
except ValueError as e:
    print(f\"Caught: {e}\")

try:
    validate_age(\"twenty\")
except TypeError as e:
    print(f\"Caught: {e}\")

try:
    connect_to_database({})
except ConnectionError as e:
    print(f\"Caught: {e}\")
    print(f\"Caused by: {e.__cause__}\")

Output:

Caught: Age cannot be negative: -5
Caught: Age must be a number, got str
Caught: Cannot connect: missing config key 'host'
Caused by: 'host'

The raise without arguments re-raises the current exception with its original traceback intact. The raise X from Y syntax creates an exception chain, so the traceback shows both the new error and what caused it.

Creating Custom Exception Hierarchies

For any non-trivial application, create custom exceptions. They make your code self-documenting and let callers handle specific error conditions without parsing error messages.

class AppError(Exception):
    \"\"\"Base exception for our application\"\"\"
    def __init__(self, message, code=None, details=None):
        super().__init__(message)
        self.code = code
        self.details = details or {}


class ValidationError(AppError):
    \"\"\"Input validation failed\"\"\"
    def __init__(self, message, field=None, **kwargs):
        super().__init__(message, code=\"VALIDATION_ERROR\", **kwargs)
        self.field = field


class NotFoundError(AppError):
    \"\"\"Requested resource doesn't exist\"\"\"
    def __init__(self, resource_type, resource_id):
        message = f\"{resource_type} with id '{resource_id}' not found\"
        super().__init__(message, code=\"NOT_FOUND\")
        self.resource_type = resource_type
        self.resource_id = resource_id


class AuthenticationError(AppError):
    \"\"\"User authentication failed\"\"\"
    def __init__(self, message=\"Authentication required\"):
        super().__init__(message, code=\"AUTH_ERROR\")


class RateLimitError(AppError):
    \"\"\"Too many requests\"\"\"
    def __init__(self, retry_after=60):
        message = f\"Rate limit exceeded. Retry after {retry_after} seconds\"
        super().__init__(message, code=\"RATE_LIMIT\")
        self.retry_after = retry_after


# Using custom exceptions
def get_user(user_id):
    users = {\"1\": \"Alice\", \"2\": \"Bob\"}
    if not isinstance(user_id, str):
        raise ValidationError(\"User ID must be a string\", field=\"user_id\")
    if user_id not in users:
        raise NotFoundError(\"User\", user_id)
    return users[user_id]


def handle_request(user_id):
    \"\"\"Handler showing how custom exceptions simplify error responses\"\"\"
    try:
        user = get_user(user_id)
        return {\"status\": \"success\", \"user\": user}
    except ValidationError as e:
        return {\"error\": e.code, \"message\": str(e), \"field\": e.field}
    except NotFoundError as e:
        return {\"error\": e.code, \"message\": str(e)}
    except AppError as e:
        return {\"error\": e.code, \"message\": str(e)}


# Test
print(handle_request(\"1\"))
print(handle_request(\"99\"))
print(handle_request(42))

Output:

{'status': 'success', 'user': 'Alice'}
{'error': 'NOT_FOUND', 'message': \"User with id '99' not found\"}
{'error': 'VALIDATION_ERROR', 'message': 'User ID must be a string', 'field': 'user_id'}
Loop Larry tangled in exception blocks
Custom exception hierarchies turn something broke into this specific thing broke for this specific reason.

Context Managers for Guaranteed Cleanup

Context managers (the with statement) are Python’s best tool for ensuring cleanup happens even when exceptions occur. They’re cleaner than try/finally for resource management.

import sqlite3
from contextlib import contextmanager

@contextmanager
def database_connection(db_path):
    \"\"\"Context manager for database connections with automatic rollback\"\"\"
    conn = sqlite3.connect(db_path)
    try:
        yield conn
        conn.commit()  # Only commits if no exception occurred
    except Exception:
        conn.rollback()  # Rollback on any error
        raise  # Re-raise so caller knows something failed
    finally:
        conn.close()  # Always close the connection


@contextmanager
def temporary_file(filepath, mode='w'):
    \"\"\"Write to a temp file, then atomically rename on success\"\"\"
    import os
    temp_path = filepath + '.tmp'
    f = open(temp_path, mode)
    try:
        yield f
        f.close()
        os.replace(temp_path, filepath)  # Atomic rename
    except Exception:
        f.close()
        if os.path.exists(temp_path):
            os.remove(temp_path)  # Clean up temp file on failure
        raise


# Usage
with database_connection(\":memory:\") as conn:
    cursor = conn.cursor()
    cursor.execute(\"CREATE TABLE users (name TEXT, age INTEGER)\")
    cursor.execute(\"INSERT INTO users VALUES ('Alice', 30)\")
    cursor.execute(\"SELECT * FROM users\")
    print(cursor.fetchall())

# The connection is guaranteed to be closed, committed on success
# or rolled back on failure

Output:

[('Alice', 30)]

The @contextmanager decorator from contextlib lets you write context managers as generator functions. Everything before yield is your setup, and everything after is your cleanup. The try/except/finally inside ensures proper handling regardless of what happens.

Logging Exceptions Effectively

Print statements aren’t sufficient for production. Use Python’s logging module with structured information that helps you diagnose issues quickly.

import logging
import traceback
import sys

# Configure logging with useful format
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    handlers=[logging.StreamHandler(sys.stdout)]
)
logger = logging.getLogger(\"myapp\")


def process_payment(user_id, amount):
    \"\"\"Demonstrate different logging levels for exceptions\"\"\"
    try:
        if amount <= 0:
            raise ValueError(f\"Invalid amount: {amount}\")

        if amount > 10000:
            # Warning: not an error, but worth noting
            logger.warning(
                \"Large payment detected\",
                extra={\"user_id\": user_id, \"amount\": amount}
            )

        # Simulate processing
        if user_id == \"blocked\":
            raise PermissionError(\"User account is blocked\")

        logger.info(f\"Payment processed: user={user_id}, amount=${amount:.2f}\")
        return True

    except ValueError as e:
        # Client error: log as warning, don't need full traceback
        logger.warning(f\"Invalid payment request: {e}\")
        return False

    except PermissionError as e:
        # Expected business logic error
        logger.error(f\"Payment blocked: user={user_id}, reason={e}\")
        return False

    except Exception as e:
        # Unexpected error: log as critical WITH traceback
        logger.critical(
            f\"Payment system failure: user={user_id}, amount={amount}\",
            exc_info=True  # This includes the full traceback
        )
        raise


# Test scenarios
process_payment(\"user123\", 50.00)
process_payment(\"user456\", -10)
process_payment(\"blocked\", 100)

Output:

2026-04-08 10:30:00 - myapp - INFO - Payment processed: user=user123, amount=$50.00
2026-04-08 10:30:00 - myapp - WARNING - Invalid payment request: Invalid amount: -10
2026-04-08 10:30:00 - myapp - ERROR - Payment blocked: user=blocked, reason=User account is blocked

The key insight: use exc_info=True for unexpected exceptions where you need the full stack trace. For expected exceptions (validation errors, business logic), a simple message is sufficient. Don’t log full tracebacks for every caught exception–it creates noise that obscures real problems.

Pyro Pete setting up logging
Good logging turns a mystery crash into a five-minute fix.

Real-World Example: Resilient API Client with Retry Logic

Here’s a production-grade pattern combining everything we’ve covered: custom exceptions, context managers, logging, and retry logic for an API client.

import time
import logging
import json
from functools import wraps

logger = logging.getLogger(__name__)


class APIError(Exception):
    \"\"\"Base API exception\"\"\"
    def __init__(self, message, status_code=None, response_body=None):
        super().__init__(message)
        self.status_code = status_code
        self.response_body = response_body


class RetryableError(APIError):
    \"\"\"Error that should trigger a retry\"\"\"
    pass


class FatalError(APIError):
    \"\"\"Error that should NOT be retried\"\"\"
    pass


def retry_on_failure(max_retries=3, base_delay=1, backoff_factor=2):
    \"\"\"Decorator that retries on RetryableError with exponential backoff\"\"\"
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            last_exception = None

            for attempt in range(max_retries + 1):
                try:
                    return func(*args, **kwargs)
                except RetryableError as e:
                    last_exception = e
                    if attempt < max_retries:
                        delay = base_delay * (backoff_factor ** attempt)
                        logger.warning(
                            f\"Attempt {attempt + 1}/{max_retries + 1} failed: {e}. \"
                            f\"Retrying in {delay}s...\"
                        )
                        time.sleep(delay)
                    else:
                        logger.error(
                            f\"All {max_retries + 1} attempts failed for {func.__name__}\"
                        )
                except FatalError:
                    # Don't retry fatal errors
                    raise

            raise last_exception

        return wrapper
    return decorator


class APIClient:
    \"\"\"Resilient API client with proper exception handling\"\"\"

    def __init__(self, base_url, api_key):
        self.base_url = base_url
        self.api_key = api_key
        self._session = None

    def _classify_error(self, status_code, response_body):
        \"\"\"Classify HTTP errors as retryable or fatal\"\"\"
        if status_code in (429, 502, 503, 504):
            raise RetryableError(
                f\"Server returned {status_code}\",
                status_code=status_code,
                response_body=response_body
            )
        elif status_code in (400, 401, 403, 404, 422):
            raise FatalError(
                f\"Client error {status_code}: {response_body}\",
                status_code=status_code,
                response_body=response_body
            )
        else:
            raise APIError(
                f\"Unexpected status {status_code}\",
                status_code=status_code,
                response_body=response_body
            )

    @retry_on_failure(max_retries=3, base_delay=1)
    def get(self, endpoint):
        \"\"\"GET request with automatic retry on transient failures\"\"\"
        import urllib.request
        import urllib.error

        url = f\"{self.base_url}/{endpoint}\"
        req = urllib.request.Request(url)
        req.add_header('Authorization', f'Bearer {self.api_key}')

        try:
            with urllib.request.urlopen(req, timeout=10) as response:
                data = json.loads(response.read().decode())
                logger.info(f\"GET {endpoint}: success\")
                return data

        except urllib.error.HTTPError as e:
            body = e.read().decode() if e.fp else \"\"
            self._classify_error(e.code, body)

        except urllib.error.URLError as e:
            raise RetryableError(f\"Connection failed: {e.reason}\")

        except json.JSONDecodeError as e:
            raise FatalError(f\"Invalid JSON response: {e}\")

    @retry_on_failure(max_retries=2, base_delay=2)
    def post(self, endpoint, data):
        \"\"\"POST request with retry logic\"\"\"
        import urllib.request
        import urllib.error

        url = f\"{self.base_url}/{endpoint}\"
        payload = json.dumps(data).encode('utf-8')
        req = urllib.request.Request(url, data=payload, method='POST')
        req.add_header('Authorization', f'Bearer {self.api_key}')
        req.add_header('Content-Type', 'application/json')

        try:
            with urllib.request.urlopen(req, timeout=30) as response:
                result = json.loads(response.read().decode())
                logger.info(f\"POST {endpoint}: success\")
                return result

        except urllib.error.HTTPError as e:
            body = e.read().decode() if e.fp else \"\"
            self._classify_error(e.code, body)

        except urllib.error.URLError as e:
            raise RetryableError(f\"Connection failed: {e.reason}\")


# Usage example
client = APIClient(\"https://api.example.com\", \"my-api-key\")

try:
    users = client.get(\"users\")
    print(f\"Fetched {len(users)} users\")
except RetryableError as e:
    print(f\"Service unavailable after retries: {e}\")
except FatalError as e:
    print(f\"Request invalid: {e}\")
except APIError as e:
    print(f\"API error: {e}\")

This pattern handles transient failures (network issues, rate limits, server errors) with automatic retry and exponential backoff. Fatal errors like 401 or 404 fail immediately since retrying won't help. The decorator separates retry logic from business logic, keeping your code clean.

Cache Katie racing through retries
Exponential backoff is the polite way of saying I will keep trying, but I will wait longer each time.

Frequently Asked Questions

Should I use except Exception or except BaseException?

Almost always except Exception. The BaseException class includes SystemExit, KeyboardInterrupt, and GeneratorExit, which you almost never want to catch. Catching KeyboardInterrupt prevents Ctrl+C from working. Only use BaseException in top-level cleanup code where you truly need to catch everything.

When should I use EAFP vs LBYL style?

EAFP (Easier to Ask Forgiveness than Permission) means using try/except. LBYL (Look Before You Leap) means checking conditions first. Python idiomatically prefers EAFP. Use try/except when the check would be expensive or racy (like file existence checks). Use LBYL when the check is cheap and makes code clearer, like if key in dict.

Is it bad to use bare except clauses?

Yes, except: without specifying an exception type catches everything including SystemExit and KeyboardInterrupt. It also makes debugging harder because you don't know what went wrong. Always specify at least except Exception, and prefer more specific exception types.

How do I handle exceptions in async code?

Async exception handling uses the same try/except syntax inside async functions. For gathering multiple coroutines, use asyncio.gather(return_exceptions=True) to collect exceptions instead of failing on the first one. For task groups in Python 3.11+, use asyncio.TaskGroup which raises ExceptionGroup containing all child exceptions.

What's the performance cost of try/except?

In Python, entering a try block has virtually zero cost when no exception occurs. Exceptions are \"zero-cost\" on the happy path. However, actually raising and catching exceptions is expensive--roughly 10-100x slower than a simple if/else check. Don't use exceptions for normal control flow (like iterating with IndexError). Use them for truly exceptional conditions.

How do I test exception handling code?

Use pytest.raises as a context manager to verify exceptions are raised correctly. For testing retry logic, use unittest.mock.patch to simulate failures. Test both the happy path and each exception path. Verify that the right exception type, message, and attributes are set.

Wrapping Up

Exception handling separates production-ready code from scripts that work on your laptop. The patterns we covered--specific exception catching, custom hierarchies, context managers, logging strategies, and retry decorators--form the foundation of resilient Python applications.

The key principles to remember: catch specific exceptions rather than broad ones, use else and finally blocks intentionally, create custom exception classes for your domain, log with appropriate severity levels, and design your error recovery strategy before writing the happy path code.

Start applying these patterns incrementally. Pick one codebase and replace bare except clauses with specific ones. Add custom exceptions to your next project. Build a retry decorator for your API calls. Each improvement makes your code more debuggable and your production systems more reliable.

Official Resources

The Basic try/except Pattern

try:
    response = requests.get(url, timeout=5)
    response.raise_for_status()
    data = response.json()
except requests.Timeout:
    logger.warning("Request timed out")
    return None
except requests.HTTPError as e:
    logger.error("HTTP %s for %s", e.response.status_code, url)
    return None
except requests.RequestException:
    logger.exception("Network error fetching %s", url)
    return None

Catch the most specific exception first, then progressively broader ones. Bare except: catches EVERYTHING including KeyboardInterrupt — almost always wrong.

The Three Anti-Patterns

  • Bare except. except: with no class catches every exception, including system-level ones like KeyboardInterrupt. Use except Exception: as the fallback if you genuinely don't know.
  • Silently swallowing. except Exception: pass hides bugs forever. At minimum, log the exception. Better, re-raise after cleanup.
  • Catching too broadly. try: do_a(); do_b(); do_c() except Exception: can't tell you which call failed. Wrap each risky operation separately.

The else and finally Clauses

try:
    data = parse_user_input(raw)
except ValueError as e:
    logger.error("Invalid input: %s", e)
    return error_response(400)
else:
    # Runs ONLY if try block succeeded — no implicit catch of other errors
    save_to_db(data)
    notify_user(data)
finally:
    # Runs whether the try succeeded, failed, or raised
    close_connection()

The else block runs only on success — useful when you want code that's clearly NOT covered by the except. finally always runs — perfect for cleanup that must happen regardless.

Raising With Context

# Re-raise the same exception
try:
    risky()
except ValueError:
    log_diagnostics()
    raise  # re-raises the current exception

# Wrap with a higher-level exception
try:
    response = requests.get(url)
    return response.json()
except requests.RequestException as e:
    raise APIError("Could not fetch user data") from e

# Chain explicitly (preserves original traceback)
raise ValidationError("bad input") from original_exception

The from keyword preserves the chain — readers see "while handling X, Y happened" — invaluable for debugging.

Custom Exception Hierarchies

class APIError(Exception):
    pass

class APIConnectionError(APIError):
    pass

class APIRateLimit(APIError):
    def __init__(self, retry_after):
        self.retry_after = retry_after
        super().__init__("Rate limited, retry in %s seconds" % retry_after)

# Callers can catch broadly OR narrowly
try:
    call_api()
except APIRateLimit as e:
    time.sleep(e.retry_after)
except APIError:
    raise

# Or in one catch with different handling
try:
    call_api()
except APIError as e:
    if isinstance(e, APIRateLimit):
        retry(after=e.retry_after)
    else:
        raise

Exception Groups (Python 3.11+)

try:
    asyncio.run(parallel_tasks())
except* ValueError as eg:
    for e in eg.exceptions:
        log.warning("Validation failed: %s", e)
except* ConnectionError as eg:
    log.error("%s connections failed", len(eg.exceptions))

except* handles multiple errors raised in parallel — essential for async code where multiple tasks may fail simultaneously.

Common Pitfalls

  • except Exception as e: print(e). Loses the traceback. Use logger.exception(...) or traceback.print_exc().
  • Catching to silence linter warnings. A try/except that does nothing isn't a fix — it's hiding a bug.
  • Raising strings. raise "error" is a TypeError. Always raise an exception instance: raise ValueError("...").
  • Not closing resources on exception. Use with blocks (context managers) instead of try/finally for file handles, locks, DB sessions.
  • Catching system-exit exceptions. KeyboardInterrupt, SystemExit, GeneratorExit don't inherit from Exception. except Exception skips them — usually what you want.

FAQ

Q: Look before you leap, or ask forgiveness?
A: Python idiom is EAFP — try the operation, catch on failure. LBYL has race conditions (the file might be deleted between exists() and open()).

Q: When to raise vs return None?
A: Raise for programming errors (bad arguments, missing files, network failures the caller can't handle). Return None for "expected absence" (key not in dict, optional config value).

Q: How do I see the full traceback in production?
A: logger.exception("msg") inside the except block. Or set up a global handler with sys.excepthook.

Q: Should I catch and ignore in tests?
A: Use pytest.raises(ExpectedError) to assert errors. Don't catch errors in tests — they're meant to fail loudly.

Q: Performance cost of try/except?
A: Negligible if no exception fires. Raising is expensive — don't use exceptions for normal control flow.

Wrapping Up

Good exception handling is mostly about NOT catching things you shouldn't. Catch specific exceptions, never bare except, always log or re-raise. Use context managers for cleanup, custom exceptions for domain errors, and raise X from Y to preserve diagnostic chains. These patterns prevent the silent failures that cause production headaches.


How To Deploy a Python App to AWS Lambda

How To Deploy a Python App to AWS Lambda

Last Updated: June 01, 2026

Skill Level: Intermediate

Pubs - Python How To Program

Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Serverless Computing Meets Python

AWS Lambda has fundamentally changed how developers deploy backend applications. Instead of managing servers, worrying about scaling, or paying for idle compute time, you write functions and let AWS handle the rest. In this guide, you’ll learn how to take a Python application, package it with dependencies, and deploy it to Lambda where it’ll handle thousands of concurrent requests without you touching a single EC2 instance.

Don’t let the “serverless” terminology intimidate you. You’re still writing Python code–the infrastructure complexity is just abstracted away. We’ll walk through every step from local development to production deployment, and you’ll understand exactly what’s happening at each stage. By the end, you’ll have a working Lambda function integrated with API Gateway, ready to scale infinitely.

Here’s what we’re covering: setting up the AWS CLI, creating a Lambda handler function, managing dependencies with zip files and Lambda Layers, deploying via the command line, integrating with API Gateway for HTTP endpoints, configuring environment variables, and building a real-world URL shortener service. You’ll also see common pitfalls and how to avoid them.

Quick Example: Your First Lambda Function

Let’s skip the theory for a moment and get something working in five minutes. This is your first Lambda function:

# lambda_handler.py
import json
from datetime import datetime

def handler(event, context):
    """Simple Lambda handler that returns current time and a message"""
    return {
        'statusCode': 200,
        'body': json.dumps({
            'message': 'Hello from Lambda',
            'timestamp': datetime.now().isoformat()
        })
    }

Output:

{
  "statusCode": 200,
  "body": "{\"message\": \"Hello from Lambda\", \"timestamp\": \"2026-04-08T14:30:45.123456\"}"
}

That’s it. This function will run on Lambda, handles HTTP requests through API Gateway, and costs you nothing until someone actually invokes it. The event parameter contains request data, and context provides runtime information like request ID and remaining execution time.

Sudo Sam watching CloudWatch logs stream on monitors
Watching logs stream in real-time beats traditional server SSH sessions any day.

Understanding AWS Lambda Architecture

Lambda is Amazon’s serverless compute service. You upload code, set memory/timeout limits, and AWS auto-scales based on incoming requests. You pay only for execution time in 1ms increments. Unlike EC2 where you provision instances all day, Lambda spins up and down instantly.

The fundamental unit is the handler function. AWS calls this function whenever an event triggers it–could be an HTTP request through API Gateway, an S3 upload, a scheduled CloudWatch event, or an SQS message. Your function receives the event details and must return a response.

Aspect Lambda Traditional EC2 Containerized Services
Scaling Automatic, instant Manual or ASG Manual or orchestrator
Cold Starts 50-500ms first call None Can be optimized
Pricing Per invocation + duration Per instance-hour Per container hour
Management Code only Full OS control Container image
Ideal Use Bursty traffic, microservices Consistent workloads Complex deployments

Setting Up AWS CLI and Credentials

Before deploying anything, you need the AWS CLI configured with credentials. Install it via pip if you haven’t already, then create an IAM user with Lambda deployment permissions.

# terminal
pip install awscli --upgrade
aws --version

Output:

aws-cli/2.15.0 Python/3.11.7 Linux/6.1.0-20 botocore/2.32.1

Now configure credentials. Generate an access key in the AWS IAM console, then run:

# terminal
aws configure

You’ll be prompted for Access Key ID and Secret Access Key. Store them securely–never commit them to version control. The AWS CLI stores them in ~/.aws/credentials.

For deployment, your IAM user needs these permissions: lambda:CreateFunction, lambda:UpdateFunctionCode, iam:PassRole, apigateway:*. Create an inline policy or use the AWSLambdaFullAccess managed policy for development.

Stack Trace Steve reviewing IAM permissions disapprovingly
Granting overly broad permissions is how side projects become expensive side projects.

Creating Your Lambda Handler Function

A Lambda handler is any Python function that AWS Lambda invokes. The signature must accept two parameters: event (contains request data) and context (runtime metadata).

# app.py
import json
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    """
    Main Lambda handler function

    Args:
        event: dict containing request data from trigger source
        context: LambdaContext object with runtime info

    Returns:
        dict with statusCode and body for API Gateway
    """
    try:
        logger.info(f"Received event: {json.dumps(event)}")

        # Extract query parameters or body
        body = event.get('body', '{}')
        if isinstance(body, str):
            body = json.loads(body)

        name = body.get('name', 'World')

        response_data = {
            'message': f'Hello, {name}!',
            'request_id': context.request_id,
            'function_name': context.function_name
        }

        return {
            'statusCode': 200,
            'headers': {'Content-Type': 'application/json'},
            'body': json.dumps(response_data)
        }

    except Exception as e:
        logger.error(f"Error: {str(e)}")
        return {
            'statusCode': 500,
            'body': json.dumps({'error': 'Internal server error'})
        }

Output:

{
  "statusCode": 200,
  "headers": {"Content-Type": "application/json"},
  "body": "{\"message\": \"Hello, Alice!\", \"request_id\": \"12345-abcde\", \"function_name\": \"my-python-app\"}"
}

Notice the structure: we return a dict with statusCode, headers, and body. This format is specifically for API Gateway integration. The context object provides metadata like execution duration, memory limits, and the current request ID for logging.

Packaging Dependencies for Lambda

Lambda has a 50MB limit for unzipped deployment packages. Installing dependencies locally and zipping them is the standard approach. Many libraries like requests, numpy, and psycopg2 have compiled C extensions that must match Lambda’s Linux environment.

# terminal
mkdir lambda_package
cd lambda_package

pip install -r requirements.txt -t .

cat > requirements.txt << 'EOF'
requests==2.31.0
python-dateutil==2.8.2
boto3==1.28.0
EOF

zip -r function.zip .

Output:

  adding: app.py (deflated 45%)
  adding: requests/ (stored 0%)
  adding: requests/__init__.py (deflated 52%)
  ...
  adding: botocore/data/sts/2011-06-15/service-2.json (deflated 78%)
     31 files, 8.4 MB compressed into 2.1 MB

The key is installing dependencies with the -t flag, which places them in the current directory. Lambda's runtime will find them automatically when you import them. For larger packages (NumPy, TensorFlow), consider using Lambda Layers which support up to 5 layers of 50MB each.

Loop Larry shocked at growing zip file package size
Is your zip file under 50MB? You haven't installed pandas yet, have you.

Deploying via AWS CLI

With your code zipped and dependencies included, deploy it. First, create an IAM role that Lambda can assume.

# terminal
# Create the trust policy
cat > trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

# Create the role
aws iam create-role \\
  --role-name lambda-execution-role \\
  --assume-role-policy-document file://trust-policy.json

# Attach basic execution policy for CloudWatch logs
aws iam attach-role-policy \\
  --role-name lambda-execution-role \\
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

Output:

{
    "Role": {
        "RoleName": "lambda-execution-role",
        "Arn": "arn:aws:iam::123456789012:role/lambda-execution-role",
        "Path": "/",
        "CreateDate": "2026-04-08T10:30:00+00:00"
    }
}

Now deploy the function:

# terminal
aws lambda create-function \\
  --function-name my-python-app \\
  --runtime python3.11 \\
  --role arn:aws:iam::123456789012:role/lambda-execution-role \\
  --handler app.lambda_handler \\
  --zip-file fileb://function.zip \\
  --timeout 30 \\
  --memory-size 256

Output:

{
    "FunctionName": "my-python-app",
    "FunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:my-python-app",
    "Runtime": "python3.11",
    "Handler": "app.lambda_handler",
    "CodeSize": 2157812,
    "MemorySize": 256,
    "Timeout": 30,
    "LastModified": "2026-04-08T10:35:22.000+0000"
}

The --handler parameter points to your function: filename.function_name. Updates are just as easy:

# terminal
aws lambda update-function-code \\
  --function-name my-python-app \\
  --zip-file fileb://function.zip

Integrating with API Gateway

Raw Lambda invocations are fine for internal triggers, but to handle HTTP requests, you need API Gateway. This acts as the front door, converting HTTP requests into Lambda events.

# terminal
# Create REST API
API_ID=$(aws apigateway create-rest-api \\
  --name my-python-app-api \\
  --description "API for Python Lambda app" \\
  --query 'id' --output text)

echo "API ID: $API_ID"

# Get root resource
ROOT_ID=$(aws apigateway get-resources \\
  --rest-api-id $API_ID \\
  --query 'items[0].id' --output text)

# Create resource
RESOURCE_ID=$(aws apigateway create-resource \\
  --rest-api-id $API_ID \\
  --parent-id $ROOT_ID \\
  --path-part "greet" \\
  --query 'id' --output text)

# Create POST method
aws apigateway put-method \\
  --rest-api-id $API_ID \\
  --resource-id $RESOURCE_ID \\
  --http-method POST \\
  --authorization-type NONE

Then grant API Gateway permission to invoke Lambda:

# terminal
aws lambda add-permission \\
  --function-name my-python-app \\
  --statement-id AllowAPIGatewayInvoke \\
  --action lambda:InvokeFunction \\
  --principal apigateway.amazonaws.com

Wire the API to Lambda and deploy:

# terminal
# Set Lambda as integration
aws apigateway put-integration \\
  --rest-api-id $API_ID \\
  --resource-id $RESOURCE_ID \\
  --http-method POST \\
  --type AWS_PROXY \\
  --integration-http-method POST \\
  --uri arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:123456789012:function:my-python-app/invocations

# Deploy the API
aws apigateway create-deployment \\
  --rest-api-id $API_ID \\
  --stage-name prod

Output:

{
    "id": "abc123def456",
    "createdDate": "2026-04-08T11:00:00+00:00"
}

Your API endpoint is now available at https://{api-id}.execute-api.us-east-1.amazonaws.com/prod/greet. Send a POST request with JSON body:

# terminal
curl -X POST https://abc123.execute-api.us-east-1.amazonaws.com/prod/greet \\
  -H "Content-Type: application/json" \\
  -d '{"name": "Alice"}'

Output:

{"message":"Hello, Alice!","request_id":"req-12345","function_name":"my-python-app"}
Pyro Pete celebrating API Gateway response
That moment when your Lambda function actually responds over the internet.

Managing Environment Variables and Secrets

Never hardcode API keys or database passwords. Lambda supports environment variables for configuration.

# terminal
aws lambda update-function-configuration \\
  --function-name my-python-app \\
  --environment Variables="{ENVIRONMENT=production,LOG_LEVEL=INFO,DATABASE_HOST=mydb.us-east-1.rds.amazonaws.com}"

Access them in your code:

# app.py
import os
import json

def lambda_handler(event, context):
    env = os.getenv('ENVIRONMENT', 'development')
    log_level = os.getenv('LOG_LEVEL', 'INFO')
    db_host = os.getenv('DATABASE_HOST')

    return {
        'statusCode': 200,
        'body': json.dumps({
            'environment': env,
            'db_host': db_host
        })
    }

For sensitive data, use AWS Secrets Manager or Systems Manager Parameter Store instead of plaintext environment variables:

# app.py
import json
import boto3

secrets_client = boto3.client('secretsmanager')

def get_database_password():
    """Retrieve password from Secrets Manager"""
    try:
        response = secrets_client.get_secret_value(
            SecretId='prod/database/password'
        )
        return json.loads(response['SecretString'])['password']
    except Exception as e:
        print(f"Error retrieving secret: {e}")
        raise

def lambda_handler(event, context):
    db_password = get_database_password()
    # Use password safely
    return {'statusCode': 200, 'body': 'OK'}

Using Lambda Layers for Shared Dependencies

If you have multiple Lambda functions sharing libraries, Lambda Layers avoid duplication. A layer is a zip file containing code or libraries that all your functions can access.

# terminal
mkdir -p lambda_layer/python/lib/python3.11/site-packages

pip install requests numpy -t lambda_layer/python/lib/python3.11/site-packages/

cd lambda_layer
zip -r requests_numpy_layer.zip python
aws lambda publish-layer-version \\
  --layer-name shared-dependencies \\
  --zip-file fileb://requests_numpy_layer.zip \\
  --compatible-runtimes python3.11

Output:

{
    "LayerVersionArn": "arn:aws:lambda:us-east-1:123456789012:layer:shared-dependencies:1",
    "Version": 1,
    "CodeSize": 12457283
}

Now attach this layer to your function:

# terminal
aws lambda update-function-configuration \\
  --function-name my-python-app \\
  --layers arn:aws:lambda:us-east-1:123456789012:layer:shared-dependencies:1

Your function immediately gains access to requests and numpy without bundling them in your deployment package.

Cache Katie comparing package sizes with Lambda Layers
Layers are how you stop submitting 47MB zip files that contain the same pandas library across five functions.

Real-World Example: Serverless URL Shortener

Let's build a practical service that shortens URLs and redirects them. It uses DynamoDB for storage and API Gateway for HTTP endpoints.

# url_shortener.py
import json
import uuid
import boto3
import logging
from datetime import datetime, timedelta

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('url-mappings')
logger = logging.getLogger()

def generate_short_code(length=6):
    """Generate a random short code"""
    return str(uuid.uuid4())[:length]

def create_short_url(original_url, custom_alias=None):
    """Store mapping and return short code"""
    short_code = custom_alias or generate_short_code()

    try:
        table.put_item(
            Item={
                'short_code': short_code,
                'original_url': original_url,
                'created_at': datetime.now().isoformat(),
                'expires_at': (datetime.now() + timedelta(days=365)).isoformat(),
                'click_count': 0
            },
            ConditionExpression='attribute_not_exists(short_code)'
        )
        return short_code
    except Exception as e:
        logger.error(f"Error creating mapping: {e}")
        raise

def get_redirect_url(short_code):
    """Retrieve original URL and increment click count"""
    try:
        response = table.get_item(Key={'short_code': short_code})

        if 'Item' not in response:
            return None

        item = response['Item']

        # Increment click counter
        table.update_item(
            Key={'short_code': short_code},
            UpdateExpression='SET click_count = click_count + :inc',
            ExpressionAttributeValues={':inc': 1}
        )

        return item['original_url']
    except Exception as e:
        logger.error(f"Error retrieving URL: {e}")
        return None

def lambda_handler(event, context):
    """Handle shorten and redirect requests"""
    path = event.get('path', '/')
    method = event.get('httpMethod', 'GET')

    try:
        if path == '/shorten' and method == 'POST':
            body = json.loads(event.get('body', '{}'))
            original_url = body.get('url')
            custom_alias = body.get('alias')

            if not original_url:
                return {
                    'statusCode': 400,
                    'body': json.dumps({'error': 'URL is required'})
                }

            short_code = create_short_url(original_url, custom_alias)
            short_url = f"https://short.example.com/{short_code}"

            return {
                'statusCode': 201,
                'body': json.dumps({
                    'short_url': short_url,
                    'short_code': short_code
                })
            }

        elif path.startswith('/') and method == 'GET':
            short_code = path.lstrip('/')

            if not short_code:
                return {
                    'statusCode': 400,
                    'body': json.dumps({'error': 'Short code required'})
                }

            original_url = get_redirect_url(short_code)

            if not original_url:
                return {
                    'statusCode': 404,
                    'body': json.dumps({'error': 'Short URL not found'})
                }

            return {
                'statusCode': 301,
                'headers': {'Location': original_url}
            }

        else:
            return {
                'statusCode': 400,
                'body': json.dumps({'error': 'Invalid request'})
            }

    except Exception as e:
        logger.error(f"Handler error: {e}")
        return {
            'statusCode': 500,
            'body': json.dumps({'error': 'Internal server error'})
        }

Test Requests:

# Create short URL
curl -X POST https://api.example.com/shorten \\
  -H "Content-Type: application/json" \\
  -d '{"url": "https://www.example.com/very/long/article/path", "alias": "article123"}'

# Response
{"short_url":"https://short.example.com/article123","short_code":"article123"}

# Redirect
curl -L https://short.example.com/article123
# Follows 301 redirect to original URL

This example demonstrates several key Lambda patterns: DynamoDB integration, handling multiple HTTP methods, conditional writes to prevent duplicates, and atomic operations for click counting. Deploy it by creating a DynamoDB table first, then zipping the code with boto3 as a dependency.

API Alice pointing at analytics dashboard with URL metrics
Scaling a URL shortener to handle millions of clicks without touching a server feels like cheating.

Frequently Asked Questions

What causes cold starts and can I eliminate them?

Cold starts occur when Lambda initializes a new execution environment, adding 50-500ms latency. They happen when no warm containers are available. Provisioned Concurrency eliminates cold starts by keeping instances warm, but costs extra. Alternatively, keep your functions small and fast--warm starts (reusing existing containers) are virtually free.

Can I test Lambda functions locally?

Use the AWS SAM CLI (Serverless Application Model) to run functions locally with Lambda emulation. Install it, then run sam local start-api to test API Gateway integration. You can also invoke functions directly with sam local invoke. It's not 100% identical to production AWS Lambda, but it's close enough for development.

How do I handle long-running tasks in Lambda?

Lambda has a 15-minute timeout maximum. For longer tasks, decouple using SQS or SNS: receive a quick acknowledgment, queue the work, then process asynchronously. Or run your code on EC2 or ECS triggered by Lambda. For data processing, consider AWS Batch or Step Functions for orchestration.

What's the difference between Lambda and containers?

Lambda is fully managed--you upload code and don't worry about infrastructure. Containers (ECS/EKS) give you more control but require managing clusters. Lambda scales infinitely and automatically; containers require capacity planning. Use Lambda for bursty microservices; use containers for consistent workloads or when you need specific OS-level control.

How do I debug Lambda functions in production?

CloudWatch Logs capture all print statements and exceptions. Use aws logs tail /aws/lambda/my-function --follow to stream logs. Add structured logging with JSON output for easier parsing. Lambda Insights provides additional metrics and performance analysis. X-Ray integrates with Lambda to trace requests across services.

Can I use async/await with Lambda?

Yes, Python async/await works fine in Lambda. However, ensure your handler function itself is not async (Lambda doesn't await it). Call async functions using asyncio.run(). For truly asynchronous patterns, use SQS with Lambda's batch message processor or invoke Lambda asynchronously with InvocationType=Event.

Wrapping Up

Deploying Python to AWS Lambda eliminates infrastructure headaches. You've learned the complete pipeline: writing handlers, packaging dependencies, using Lambda Layers to share code, integrating with API Gateway for HTTP endpoints, managing secrets securely, and building real-world services like a URL shortener.

The serverless model isn't suitable for every workload--continuous background services are cheaper on EC2, and processes exceeding 15 minutes require different architectures. But for APIs, webhooks, microservices, and event-driven workflows, Lambda is unbeatable for simplicity and cost efficiency.

Next steps: explore Lambda@Edge for CDN functions, Step Functions for orchestrating multi-function workflows, and EventBridge for decoupling event sources. Check the official AWS Lambda documentation at docs.aws.amazon.com/lambda and the Boto3 documentation for Python SDK reference.

Official Resources

Continue Learning Python

Tutorials you might also find useful:


How To Dockerize a Python Application

How To Dockerize a Python Application

Last Updated: June 01, 2026

Intermediate

You have built a Python application that works perfectly on your machine. Then you deploy it to a server, and everything breaks — different Python version, missing system libraries, conflicting dependencies. This scenario plays out daily across development teams worldwide, and Docker solves it completely. By packaging your application with its exact runtime environment, Docker guarantees that what works on your laptop works identically in production.

Docker is free, runs on all major operating systems, and requires no special Python knowledge beyond what you already have. You will need Docker Desktop installed on your machine (available from docs.docker.com), and a basic Python application to containerize. If you do not have one, we will create a simple Flask app from scratch in this tutorial.

In this article, you will learn how to write a Dockerfile for Python applications, build and run Docker images, use multi-stage builds to keep images small, manage dependencies properly, set up Docker Compose for multi-container apps, and follow production-ready best practices. By the end, you will be able to containerize any Python project with confidence.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Dockerizing a Python App: Quick Example

Before we dive deep, here is the fastest way to containerize a Python script. Create these two files in an empty directory:

# app.py
from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello():
    return {"message": "Hello from Docker!", "status": "running"}

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)
# Dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 5000
CMD ["python", "app.py"]
# requirements.txt
flask==3.0.0

Now build and run it with two commands:

# terminal commands
docker build -t my-python-app .
docker run -p 5000:5000 my-python-app

Output:

 * Serving Flask app 'app'
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:5000
 * Running on http://172.17.0.2:5000

Visit http://localhost:5000 in your browser and you will see {"message": "Hello from Docker!", "status": "running"}. That is your Python app running inside a container — isolated, reproducible, and ready to deploy anywhere Docker runs. The rest of this article explains every piece in detail and shows you how to handle real-world scenarios.

What Is Docker and Why Use It for Python?

Docker is a platform that packages applications into lightweight, portable containers. A container includes your code, Python runtime, system libraries, and dependencies — everything needed to run your app. Unlike virtual machines, containers share the host OS kernel, making them start in seconds and use minimal resources.

For Python developers specifically, Docker solves several painful problems. It eliminates “works on my machine” issues by ensuring identical environments everywhere. It prevents dependency conflicts between projects without needing virtual environments on the host. It makes deployment as simple as shipping a single image file. And it lets you run different Python versions side by side without pyenv or system-level changes.

Docker vs virtual environments comparison for Python
Works on my machine. Ships on every machine.
ConceptDockerVirtual Environment (venv)
Isolates Python packagesYesYes
Isolates system librariesYesNo
Isolates Python versionYesNo
Isolates OSYesNo
Portable across machinesYesNo
Adds overheadMinimal (~50MB base)None
Learning curveModerateLow

Think of Docker as a virtual environment on steroids — it does not just isolate your pip packages, it isolates the entire operating system layer. This makes it the standard tool for deploying Python applications in production.

Understanding Dockerfiles: Line by Line

A Dockerfile is a text file with instructions that tell Docker how to build your image. Each instruction creates a layer, and Docker caches these layers to speed up subsequent builds. Let us break down every line from our quick example:

# Dockerfile
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 5000

CMD ["python", "app.py"]

Line-by-line explanation:

FROM python:3.12-slim — This sets the base image. The slim variant includes Python and minimal system packages, keeping your image small (~150MB vs ~900MB for the full image). Always pin your Python version to avoid surprises when a new release comes out.

WORKDIR /app — Sets the working directory inside the container. All subsequent commands run from this path. If the directory does not exist, Docker creates it.

COPY requirements.txt . — Copies only the requirements file first. This is a deliberate optimization — Docker caches each layer, so if your requirements have not changed, it skips the pip install step on rebuild.

RUN pip install --no-cache-dir -r requirements.txt — Installs dependencies. The --no-cache-dir flag prevents pip from storing downloaded packages in the cache, reducing image size.

COPY . . — Copies the rest of your application code. This comes after pip install so that code changes do not trigger a full dependency reinstall.

EXPOSE 5000 — Documents which port the container listens on. This does not actually publish the port — you still need -p 5000:5000 when running the container.

CMD ["python", "app.py"] — The default command that runs when the container starts. Use the exec form (JSON array) rather than shell form for proper signal handling.

Understanding Dockerfile layers and build caching
Layer by layer, cache hit by cache hit — that is how fast Docker builds are made.

Choosing the Right Python Base Image

The base image you choose significantly affects your container’s size, security, and compatibility. Python offers several official variants on Docker Hub:

Image TagSizeIncludesBest For
python:3.12~900MBFull Debian + build toolsApps needing C compilation
python:3.12-slim~150MBMinimal DebianMost web apps (recommended)
python:3.12-alpine~50MBAlpine Linux (musl libc)Tiny images (watch for compatibility)
python:3.12-bookworm~900MBDebian Bookworm + build toolsSpecific Debian version needed

For most Python web applications, python:3.12-slim is the best starting point. It includes enough system libraries to install common packages like psycopg2-binary and Pillow without being bloated. The alpine variant looks attractive at 50MB, but it uses musl libc instead of glibc, which can cause subtle compatibility issues with some Python packages — especially those with C extensions like numpy or pandas.

When Alpine Actually Makes Sense

Alpine works well for pure-Python applications with no C extensions. If your app only uses packages like Flask, requests, and click, Alpine gives you the smallest possible image. But the moment you need numpy, pandas, or any package that compiles C code, you will spend more time fighting build issues than you save on image size.

# alpine_example.py
# This Dockerfile works great for pure-Python apps
# FROM python:3.12-alpine
# WORKDIR /app
# COPY requirements.txt .
# RUN pip install --no-cache-dir -r requirements.txt
# COPY . .
# CMD ["python", "app.py"]

# But if requirements.txt contains numpy, you need:
# RUN apk add --no-cache gcc musl-dev linux-headers
# This adds complexity and build time

Managing Dependencies in Docker

Proper dependency management is the difference between a Docker image that builds reliably and one that breaks randomly. The key principle is deterministic builds — every build should install the exact same package versions.

Pin Every Version

Never use unpinned requirements in Docker. A requirements.txt that says flask without a version will install whatever the latest version is at build time. This means your image built today might behave differently from one built tomorrow.

# requirements_pinned.py
# BAD - unpinned versions
# flask
# requests
# sqlalchemy

# GOOD - pinned versions
# flask==3.0.0
# requests==2.31.0
# sqlalchemy==2.0.23

# BEST - use pip freeze to capture exact versions
# pip freeze > requirements.txt

Output:

# Output of pip freeze (example)
blinker==1.7.0
certifi==2023.11.17
charset-normalizer==3.3.2
click==8.1.7
flask==3.0.0
idna==3.6
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.3
requests==2.31.0
urllib3==2.1.0
Werkzeug==3.0.1

Running pip freeze captures every installed package including transitive dependencies. This guarantees reproducible builds. For even more control, consider using pip-compile from the pip-tools package, which generates a locked requirements file from a high-level requirements.in.

Pinning dependency versions in requirements.txt
Pin your versions, or your Friday deploy pins you to your desk.

Using .dockerignore

Just like .gitignore keeps files out of your repository, .dockerignore keeps files out of your Docker build context. Without it, Docker sends everything in your project directory to the Docker daemon, including large files that are not needed in the container.

# .dockerignore
__pycache__/
*.pyc
*.pyo
.git/
.gitignore
.env
.venv/
venv/
node_modules/
*.md
.pytest_cache/
.mypy_cache/
docker-compose*.yml
Dockerfile*
.dockerignore

This file reduces build context size and prevents sensitive files (like .env with secrets) from being copied into your image. Always create a .dockerignore file before building your first image.

Multi-Stage Builds for Smaller Images

Multi-stage builds are a Docker feature that lets you use multiple FROM statements in a single Dockerfile. This is powerful for Python because you can compile dependencies in a full build environment, then copy only the results into a slim runtime image.

# Dockerfile.multistage
# Stage 1: Build stage with full toolchain
FROM python:3.12 AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt

# Stage 2: Runtime stage with minimal image
FROM python:3.12-slim
WORKDIR /app

# Copy installed packages from builder
COPY --from=builder /install /usr/local

# Copy application code
COPY . .

EXPOSE 5000
CMD ["python", "app.py"]

Output (comparing image sizes):

# Single-stage build
my-app-single    latest    892MB

# Multi-stage build
my-app-multi     latest    167MB

The multi-stage build produces an image that is over 5 times smaller. The first stage (builder) has gcc, make, and other build tools needed to compile C extensions. The second stage only includes the compiled packages and your code. Build tools, source files, and pip cache are all left behind in the builder stage.

This technique is especially valuable when your dependencies include packages like psycopg2 (needs libpq-dev), Pillow (needs libjpeg), or cryptography (needs OpenSSL headers). You compile in the full image and run in the slim image.

Docker Compose for Multi-Container Apps

Real applications rarely run in isolation. Your Python app probably needs a database, a cache layer, or a message queue. Docker Compose lets you define and run multi-container applications with a single YAML file.

# docker-compose.yml
version: "3.9"

services:
  web:
    build: .
    ports:
      - "5000:5000"
    environment:
      - DATABASE_URL=postgresql://user:password@db:5432/myapp
      - REDIS_URL=redis://cache:6379/0
    depends_on:
      - db
      - cache
    volumes:
      - .:/app

  db:
    image: postgres:16-alpine
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=myapp
    volumes:
      - pgdata:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  cache:
    image: redis:7-alpine
    ports:
      - "6379:6379"

volumes:
  pgdata:

Output (docker compose up):

$ docker compose up
[+] Running 3/3
 - Container myapp-db-1     Started
 - Container myapp-cache-1  Started
 - Container myapp-web-1    Started
Attaching to myapp-cache-1, myapp-db-1, myapp-web-1
myapp-db-1     | PostgreSQL init process complete; ready for start up.
myapp-cache-1  | Ready to accept connections
myapp-web-1    |  * Running on http://0.0.0.0:5000

With one docker compose up command, you get a Python web app, PostgreSQL database, and Redis cache all running together. The depends_on directive ensures the database starts before your app. The volumes section persists database data between restarts and mounts your source code for live reloading during development.

Docker Compose orchestrating multiple services
docker compose up — one command, three services, zero excuses.

Production Best Practices

Development Dockerfiles and production Dockerfiles have different priorities. In development, you want fast rebuilds and live reloading. In production, you want small images, security, and reliability.

Run as Non-Root User

By default, containers run as root. This is a security risk — if an attacker exploits your app, they have root access inside the container. Always create and switch to a non-root user:

# Dockerfile.production
FROM python:3.12-slim

# Create non-root user
RUN groupadd -r appuser && useradd -r -g appuser appuser

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

# Change ownership and switch user
RUN chown -R appuser:appuser /app
USER appuser

EXPOSE 5000
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "--workers", "4", "app:app"]

Notice we also switched from the Flask development server to Gunicorn for production. The Flask dev server is single-threaded and not designed for production traffic. Gunicorn runs multiple worker processes to handle concurrent requests.

Add Health Checks

Health checks tell Docker (and orchestrators like Kubernetes) whether your application is actually working, not just running:

# Dockerfile.healthcheck
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
  CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:5000/health')" || exit 1

EXPOSE 5000
CMD ["python", "app.py"]

Output (docker inspect):

$ docker inspect --format='{{.State.Health.Status}}' my-container
healthy

The health check hits your /health endpoint every 30 seconds. If it fails 3 times in a row, Docker marks the container as unhealthy. Orchestrators can then automatically restart it or route traffic elsewhere.

Handle Secrets with Environment Variables

Never bake secrets into your Docker image. Anyone who pulls your image can extract them. Use environment variables instead:

# config.py
import os

DATABASE_URL = os.environ.get("DATABASE_URL", "sqlite:///local.db")
SECRET_KEY = os.environ.get("SECRET_KEY", "dev-only-secret")
DEBUG = os.environ.get("DEBUG", "false").lower() == "true"

print(f"Database: {DATABASE_URL.split('@')[-1] if '@' in DATABASE_URL else DATABASE_URL}")
print(f"Debug mode: {DEBUG}")

Output:

Database: db:5432/myapp
Debug mode: False

Pass environment variables at runtime with docker run -e SECRET_KEY=mysecret or through Docker Compose’s environment section. For sensitive values in production, use Docker secrets or your cloud provider’s secrets manager.

Real-Life Example: Dockerized Task Tracker API

Deploying Dockerized Python app to production
From localhost to the cloud in one docker push.

Let us build a complete, production-ready Dockerized application — a task tracker API with Flask, SQLite, and proper project structure:

# task_tracker.py
from flask import Flask, request, jsonify
import sqlite3
import os
from datetime import datetime

app = Flask(__name__)
DB_PATH = os.environ.get("DB_PATH", "/app/data/tasks.db")


def get_db():
    os.makedirs(os.path.dirname(DB_PATH), exist_ok=True)
    conn = sqlite3.connect(DB_PATH)
    conn.row_factory = sqlite3.Row
    return conn


def init_db():
    db = get_db()
    db.execute("""
        CREATE TABLE IF NOT EXISTS tasks (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            title TEXT NOT NULL,
            description TEXT DEFAULT '',
            completed BOOLEAN DEFAULT 0,
            created_at TEXT DEFAULT CURRENT_TIMESTAMP
        )
    """)
    db.commit()
    db.close()


@app.route("/health")
def health():
    return {"status": "healthy", "timestamp": datetime.now().isoformat()}


@app.route("/tasks", methods=["GET"])
def list_tasks():
    db = get_db()
    tasks = db.execute("SELECT * FROM tasks ORDER BY created_at DESC").fetchall()
    db.close()
    return jsonify([dict(t) for t in tasks])


@app.route("/tasks", methods=["POST"])
def create_task():
    data = request.get_json()
    if not data or not data.get("title"):
        return {"error": "Title is required"}, 400

    db = get_db()
    cursor = db.execute(
        "INSERT INTO tasks (title, description) VALUES (?, ?)",
        (data["title"], data.get("description", ""))
    )
    db.commit()
    task_id = cursor.lastrowid
    task = db.execute("SELECT * FROM tasks WHERE id = ?", (task_id,)).fetchone()
    db.close()
    return jsonify(dict(task)), 201


@app.route("/tasks/<int:task_id>/complete", methods=["PATCH"])
def complete_task(task_id):
    db = get_db()
    db.execute("UPDATE tasks SET completed = 1 WHERE id = ?", (task_id,))
    db.commit()
    task = db.execute("SELECT * FROM tasks WHERE id = ?", (task_id,)).fetchone()
    db.close()
    if task is None:
        return {"error": "Task not found"}, 404
    return jsonify(dict(task))


init_db()

if __name__ == "__main__":
    port = int(os.environ.get("PORT", 5000))
    app.run(host="0.0.0.0", port=port, debug=os.environ.get("DEBUG") == "true")

Output (testing the API):

$ curl -X POST http://localhost:5000/tasks \
    -H "Content-Type: application/json" \
    -d '{"title": "Learn Docker", "description": "Complete the tutorial"}'
{
  "id": 1,
  "title": "Learn Docker",
  "description": "Complete the tutorial",
  "completed": 0,
  "created_at": "2026-04-08 10:30:00"
}

$ curl http://localhost:5000/tasks
[
  {
    "id": 1,
    "title": "Learn Docker",
    "description": "Complete the tutorial",
    "completed": 0,
    "created_at": "2026-04-08 10:30:00"
  }
]

$ curl http://localhost:5000/health
{"status": "healthy", "timestamp": "2026-04-08T10:30:15.123456"}

This application demonstrates several production patterns: health check endpoint for container orchestration, environment variable configuration, proper database initialization, input validation, and error handling. The Docker setup uses volumes to persist the SQLite database and a non-root user for security. You can extend this by adding authentication, switching to PostgreSQL via Docker Compose, or deploying to a cloud container service.

Essential Docker Commands Reference

CommandWhat It Does
docker build -t name .Build image from Dockerfile in current directory
docker run -p 5000:5000 nameRun container, map port 5000
docker run -d nameRun container in background (detached)
docker psList running containers
docker logs container_idView container logs
docker exec -it container_id bashOpen shell inside running container
docker stop container_idStop a running container
docker imagesList all local images
docker system pruneRemove unused images, containers, networks
docker compose upStart all services in docker-compose.yml
docker compose downStop and remove all services

Frequently Asked Questions

Do I still need a virtual environment inside Docker?

No. The Docker container itself provides isolation, so there is no risk of conflicting with system Python or other projects. Some teams still use venv inside Docker for consistency with their local development workflow, but it adds no practical benefit. Skip it and install directly with pip install in your Dockerfile.

Why is my Docker build so slow?

The most common cause is poor layer caching. If you COPY . . before RUN pip install, any code change invalidates the pip install cache. Always copy requirements.txt first and install dependencies before copying the rest of your code. Also check that your .dockerignore excludes large directories like .git/, node_modules/, and __pycache__/.

Should I use Alpine Linux for my Python Docker image?

Only if your app uses pure-Python packages with no C extensions. Alpine uses musl libc instead of glibc, which causes build failures and subtle runtime issues with packages like numpy, pandas, and psycopg2. The slim variant is only ~100MB larger than Alpine and avoids these compatibility headaches entirely.

How do I reduce my Docker image size?

Use multi-stage builds to separate build tools from your runtime image. Use python:3.12-slim as your runtime base. Add --no-cache-dir to pip install commands. Create a thorough .dockerignore file. Remove unnecessary system packages with apt-get clean and rm -rf /var/lib/apt/lists/* after installing system dependencies.

What is the difference between Dockerfile and docker-compose.yml?

A Dockerfile defines how to build a single image — what base to start from, what to install, what to copy, and what command to run. Docker Compose defines how to run multiple containers together — which images to use, how they connect, what ports to expose, and what volumes to mount. You need a Dockerfile for each custom service and a docker-compose.yml to orchestrate them all.

How do I get hot-reload working in Docker during development?

Mount your source code as a volume so changes on your host are reflected inside the container immediately. In your docker-compose.yml, add volumes: [".:/app"] under your web service. Then run Flask with debug=True or use a tool like watchdog to restart on file changes. This gives you the fast feedback loop of local development with the consistency of Docker.

Conclusion

You have learned how to containerize Python applications with Docker from scratch. We covered writing Dockerfiles with proper layer caching, choosing the right base image, pinning dependencies for reproducible builds, using multi-stage builds to shrink image size, orchestrating multi-container apps with Docker Compose, and following production best practices like non-root users and health checks.

Try extending the task tracker example by adding PostgreSQL via Docker Compose, or deploy it to a cloud service like AWS ECS, Google Cloud Run, or Railway. The skills you have learned here apply to any Python application — from simple scripts to complex microservice architectures.

For the official Docker documentation, visit docs.docker.com. For Python-specific Docker guidance, see the official Python Docker images documentation.

How To Use Pydantic V2 for Data Validation in Python

How To Use Pydantic V2 for Data Validation in Python

Last Updated: June 01, 2026

Intermediate

Every Python application that receives data from the outside world — API requests, configuration files, CSV imports, form submissions — faces the same problem: how do you guarantee that the data matches what your code expects? A missing field crashes your function. A string where you expected an integer causes silent bugs. An email address without an “@” slips into your database. Manual validation with if-else chains works for one or two fields, but it does not scale.

Pydantic V2 solves this by letting you define data shapes as Python classes with type hints, then validating and converting incoming data automatically. Released in mid-2023, V2 is a complete rewrite of Pydantic with a Rust-powered core that runs 5-50x faster than V1. It is the validation engine behind FastAPI, and it works just as well standalone in any Python project. Install it with pip install pydantic.

This tutorial covers everything you need to use Pydantic V2 effectively: defining models with type annotations, using built-in validators and constraints, writing custom validation logic, working with nested models, serializing data to dictionaries and JSON, and handling validation errors gracefully. By the end, you will be able to validate any data structure your application encounters.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

Pydantic Validation in 30 Seconds

Here is the smallest useful Pydantic model. It validates a user’s data and converts types automatically.

# quick_example.py
from pydantic import BaseModel

class User(BaseModel):
    name: str
    age: int
    email: str

# Valid data -- works perfectly
user = User(name="Alice", age=30, email="alice@example.com")
print(user)
print(user.model_dump())

# Type coercion -- "25" becomes int 25
user2 = User(name="Bob", age="25", email="bob@example.com")
print(f"Bob's age: {user2.age} (type: {type(user2.age).__name__})")

Output:

name='Alice' age=30 email='alice@example.com'
{'name': 'Alice', 'age': 30, 'email': 'alice@example.com'}
Bob's age: 25 (type: int)

Notice that Pydantic automatically converted the string "25" to the integer 25 because the age field is typed as int. This type coercion is one of Pydantic’s most practical features — it handles the messy reality of data that comes in as strings from JSON, forms, or environment variables.

What Is Pydantic and Why Use It?

Pydantic is a data validation library that uses Python type hints to define data structures and validate them at runtime. When you create a Pydantic model instance, it checks every field against its declared type, applies any constraints you have defined, and raises a detailed error if anything is wrong.

ApproachLines of CodeType CoercionError MessagesNested Validation
Manual if-elseManyManualYou write themYou build it
dataclassesFewNoneBasic TypeErrorNone
Pydantic V2FewAutomaticDetailed, structuredBuilt-in
marshmallowModerateConfigurableDetailedBuilt-in

The key advantage of Pydantic over alternatives like dataclasses or attrs is that it validates at runtime. A dataclass with age: int happily accepts age="hello" — it only declares the type hint without enforcing it. Pydantic actually checks and converts the value, raising a ValidationError if conversion fails.

Creating Pydantic models
Twelve lines of __init__ boilerplate, or one BaseModel. Pick wisely.

Built-in Field Types and Constraints

Pydantic supports all standard Python types plus specialized types for common validation patterns. The Field function adds constraints like minimum length, numeric ranges, and regex patterns.

# field_types.py
from pydantic import BaseModel, Field, EmailStr
from typing import Optional
from datetime import datetime

class Product(BaseModel):
    name: str = Field(..., min_length=1, max_length=100)
    price: float = Field(..., gt=0, description="Price in dollars")
    quantity: int = Field(default=0, ge=0)
    sku: str = Field(..., pattern=r"^[A-Z]{2}-\d{4}$")
    description: Optional[str] = None
    created_at: datetime = Field(default_factory=datetime.now)

# Valid product
product = Product(name="Widget", price=19.99, sku="AB-1234")
print(product.model_dump())

# Invalid -- price is negative
try:
    bad = Product(name="Widget", price=-5, sku="AB-1234")
except Exception as e:
    print(f"Error: {e}")

Output:

{'name': 'Widget', 'price': 19.99, 'quantity': 0, 'sku': 'AB-1234', 'description': None, 'created_at': '2026-04-07T...'}
Error: 1 validation error for Product
price
  Input should be greater than 0 [type=greater_than, input_value=-5, input_type=int]

The Field function is where you add constraints beyond basic type checking. The ... (Ellipsis) means the field is required. The gt=0 constraint rejects zero and negative numbers. The pattern constraint validates the SKU format with a regex. All of these constraints are checked automatically when you create the model instance.

Common Pydantic Types

Pydantic provides specialized types that go beyond basic Python types. These handle common validation patterns that would otherwise require custom code.

TypeWhat It ValidatesExample
EmailStrValid email format"user@example.com"
HttpUrlValid HTTP/HTTPS URL"https://example.com"
IPvAnyAddressValid IPv4 or IPv6 address"192.168.1.1"
SecretStrString hidden in repr/logs"s3cr3t" (shows "**********")
PositiveIntInteger greater than 042
FutureDatetimeDatetime in the future"2027-01-01T00:00:00"
constrConstrained string (length, pattern)constr(min_length=3)

To use EmailStr, install the optional dependency: pip install pydantic[email]. The other types are available in the core package.

Custom Validators

When built-in constraints are not enough, Pydantic V2 provides the @field_validator and @model_validator decorators for custom validation logic.

Field Validators

# custom_validators.py
from pydantic import BaseModel, field_validator

class Registration(BaseModel):
    username: str
    password: str
    confirm_password: str

    @field_validator("username")
    @classmethod
    def username_must_be_alphanumeric(cls, v: str) -> str:
        if not v.isalnum():
            raise ValueError("Username must contain only letters and numbers")
        if len(v) < 3:
            raise ValueError("Username must be at least 3 characters")
        return v.lower()  # Normalize to lowercase

    @field_validator("password")
    @classmethod
    def password_strength(cls, v: str) -> str:
        if len(v) < 8:
            raise ValueError("Password must be at least 8 characters")
        if not any(c.isupper() for c in v):
            raise ValueError("Password must contain an uppercase letter")
        if not any(c.isdigit() for c in v):
            raise ValueError("Password must contain a digit")
        return v

# Valid registration
reg = Registration(username="Alice42", password="Secure1Pass", confirm_password="Secure1Pass")
print(f"Username: {reg.username}")

# Invalid username
try:
    Registration(username="a!", password="Secure1Pass", confirm_password="Secure1Pass")
except Exception as e:
    print(f"Error: {e}")

Output:

Username: alice42
Error: 1 validation error for Registration
username
  Value error, Username must contain only letters and numbers [type=value_error, ...]

Field validators receive the raw value and can either return a transformed value (like v.lower()) or raise a ValueError with a descriptive message. The @classmethod decorator is required in V2.

Pydantic field validators
@field_validator stamps your data with approval — or stamps it into the ground.

Model Validators

Model validators check relationships between multiple fields. Use them when validation depends on more than one field at a time.

# model_validator.py
from pydantic import BaseModel, model_validator

class DateRange(BaseModel):
    start_date: str
    end_date: str

    @model_validator(mode="after")
    def check_dates(self):
        if self.start_date >= self.end_date:
            raise ValueError("end_date must be after start_date")
        return self

class Registration(BaseModel):
    password: str
    confirm_password: str

    @model_validator(mode="after")
    def passwords_match(self):
        if self.password != self.confirm_password:
            raise ValueError("Passwords do not match")
        return self

# Valid
dates = DateRange(start_date="2026-01-01", end_date="2026-12-31")
print(f"Range: {dates.start_date} to {dates.end_date}")

# Invalid -- passwords don't match
try:
    Registration(password="Secret1Pass", confirm_password="Different1Pass")
except Exception as e:
    print(f"Error: {e}")

Output:

Range: 2026-01-01 to 2026-12-31
Error: 1 validation error for Registration
  Value error, Passwords do not match [type=value_error, ...]

The mode="after" parameter means the validator runs after individual field validation is complete, so you can safely access all fields. Use mode="before" when you need to transform the raw input data before field-level validation runs.

Nested Models

Real-world data is rarely flat. Pydantic handles nested structures by composing models inside other models. Validation cascades through every level automatically.

# nested_models.py
from pydantic import BaseModel, EmailStr
from typing import Optional

class Address(BaseModel):
    street: str
    city: str
    state: str
    zip_code: str

class ContactInfo(BaseModel):
    email: str
    phone: Optional[str] = None
    address: Address

class Employee(BaseModel):
    name: str
    title: str
    department: str
    contact: ContactInfo

# Create from nested dictionaries
data = {
    "name": "Alice Johnson",
    "title": "Senior Developer",
    "department": "Engineering",
    "contact": {
        "email": "alice@company.com",
        "phone": "555-0123",
        "address": {
            "street": "123 Main St",
            "city": "Melbourne",
            "state": "VIC",
            "zip_code": "3000"
        }
    }
}

employee = Employee(**data)
print(f"Name: {employee.name}")
print(f"City: {employee.contact.address.city}")
print(f"Email: {employee.contact.email}")

Output:

Name: Alice Johnson
City: Melbourne
Email: alice@company.com

Pydantic validates every level of the nested structure. If the zip code is missing from the address, you get an error pointing to the exact path: contact -> address -> zip_code. This nested validation is especially valuable when parsing complex JSON from APIs or configuration files.

Serialization: Models to Dictionaries and JSON

Pydantic models are not just for validation -- they also handle serialization. The model_dump() and model_dump_json() methods convert models back to dictionaries and JSON strings with fine-grained control.

# serialization.py
from pydantic import BaseModel, Field
from typing import Optional
from datetime import datetime

class Article(BaseModel):
    title: str
    content: str
    author: str
    published: bool = False
    created_at: datetime = Field(default_factory=datetime.now)
    internal_notes: Optional[str] = None

article = Article(
    title="Pydantic V2 Guide",
    content="Learn data validation...",
    author="Alice",
    internal_notes="Draft needs review"
)

# Full dictionary
print("Full:", article.model_dump())

# Exclude internal fields
public = article.model_dump(exclude={"internal_notes", "created_at"})
print("Public:", public)

# Only include specific fields
summary = article.model_dump(include={"title", "author", "published"})
print("Summary:", summary)

# Skip fields with None values
clean = article.model_dump(exclude_none=True)
print("Clean:", clean)

# JSON string
json_str = article.model_dump_json(indent=2)
print("JSON:", json_str[:80], "...")

Output:

Full: {'title': 'Pydantic V2 Guide', 'content': 'Learn data validation...', 'author': 'Alice', 'published': False, 'created_at': datetime(...), 'internal_notes': 'Draft needs review'}
Public: {'title': 'Pydantic V2 Guide', 'content': 'Learn data validation...', 'author': 'Alice', 'published': False}
Summary: {'title': 'Pydantic V2 Guide', 'author': 'Alice', 'published': False}
Clean: {'title': 'Pydantic V2 Guide', 'content': 'Learn data validation...', 'author': 'Alice', 'published': False, 'created_at': datetime(...), 'internal_notes': 'Draft needs review'}
JSON: {
  "title": "Pydantic V2 Guide",
  "content": "Learn data validation..." ...

The exclude and include parameters give you control over which fields appear in the output. This is essential for APIs where internal fields like notes or timestamps should not be sent to clients.

Handling validation errors
Square peg, round hole. Pydantic will tell you exactly which corner doesn't fit.

Handling Validation Errors

When validation fails, Pydantic raises a ValidationError with detailed information about every problem. You can catch this exception and extract structured error data for API responses or logging.

# error_handling.py
from pydantic import BaseModel, Field, ValidationError

class Order(BaseModel):
    product: str = Field(..., min_length=1)
    quantity: int = Field(..., gt=0)
    price: float = Field(..., gt=0)
    email: str

# Multiple validation errors at once
try:
    Order(product="", quantity=-5, price="free", email="not-an-email")
except ValidationError as e:
    print(f"Error count: {e.error_count()}")
    print()
    for error in e.errors():
        print(f"Field: {error['loc']}")
        print(f"Message: {error['msg']}")
        print(f"Type: {error['type']}")
        print()

Output:

Error count: 3

Field: ('product',)
Message: String should have at least 1 character
Type: string_too_short

Field: ('quantity',)
Message: Input should be greater than 0
Type: greater_than

Field: ('price',)
Message: Input should be a valid number, unable to parse string as a number
Type: float_parsing

Pydantic collects all validation errors rather than stopping at the first one. Each error includes the field path (loc), a human-readable message (msg), and a machine-readable error type (type). In a FastAPI application, these errors are automatically converted to 422 responses with this same structured format.

Model Configuration

Pydantic V2 uses model_config to customize model behavior. This replaces the inner class Config from V1.

# model_config.py
from pydantic import BaseModel, ConfigDict

class StrictUser(BaseModel):
    model_config = ConfigDict(
        str_strip_whitespace=True,    # Strip leading/trailing whitespace
        str_min_length=1,             # No empty strings allowed
        frozen=True,                  # Immutable after creation
        extra="forbid",               # No extra fields allowed
    )

    name: str
    email: str

# Whitespace gets stripped automatically
user = StrictUser(name="  Alice  ", email="alice@example.com")
print(f"Name: '{user.name}'")

# Extra fields are rejected
try:
    StrictUser(name="Bob", email="bob@example.com", role="admin")
except Exception as e:
    print(f"Extra field error: {e}")

# Immutable -- cannot change after creation
try:
    user.name = "Charlie"
except Exception as e:
    print(f"Frozen error: {e}")

Output:

Name: 'Alice'
Extra field error: 1 validation error for StrictUser
role
  Extra inputs are not permitted [type=extra_forbidden, ...]
Frozen error: 1 validation error for StrictUser
name
  Instance is frozen [type=frozen_instance, ...]

The extra="forbid" setting is especially important for security -- it prevents attackers from injecting unexpected fields into your data models. The frozen=True setting makes models behave like named tuples, which is useful for configuration objects that should not be modified after creation.

Real-Life Example: Application Configuration Manager

Pydantic V2 performance
V2 rewrote the core in Rust. Your validation just got a turbocharger.

Here is a practical example: a type-safe application configuration system that reads from environment variables, validates every setting at startup, and provides clean access throughout your application.

# config_manager.py
import os
from pydantic import BaseModel, Field, field_validator, model_validator
from pydantic import ConfigDict
from typing import Optional

class DatabaseConfig(BaseModel):
    host: str = "localhost"
    port: int = Field(default=5432, ge=1, le=65535)
    name: str
    user: str
    password: str
    pool_size: int = Field(default=10, ge=1, le=100)

    @property
    def connection_url(self) -> str:
        return f"postgresql://{self.user}:{self.password}@{self.host}:{self.port}/{self.name}"

class CacheConfig(BaseModel):
    enabled: bool = True
    ttl_seconds: int = Field(default=300, ge=0)
    max_size: int = Field(default=1000, ge=1)

class LoggingConfig(BaseModel):
    level: str = "INFO"
    format: str = "json"

    @field_validator("level")
    @classmethod
    def validate_level(cls, v: str) -> str:
        allowed = {"DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"}
        v_upper = v.upper()
        if v_upper not in allowed:
            raise ValueError(f"Log level must be one of {allowed}")
        return v_upper

    @field_validator("format")
    @classmethod
    def validate_format(cls, v: str) -> str:
        if v not in ("json", "text"):
            raise ValueError("Format must be 'json' or 'text'")
        return v

class AppConfig(BaseModel):
    model_config = ConfigDict(frozen=True)

    app_name: str = "MyApp"
    debug: bool = False
    api_version: str = "v1"
    database: DatabaseConfig
    cache: CacheConfig = CacheConfig()
    logging: LoggingConfig = LoggingConfig()

    @model_validator(mode="after")
    def warn_debug_in_production(self):
        if self.debug and self.logging.level not in ("DEBUG", "INFO"):
            raise ValueError("Debug mode requires log level DEBUG or INFO")
        return self

# Create config from a settings dictionary
settings = {
    "app_name": "BookStore API",
    "debug": True,
    "database": {
        "host": "db.example.com",
        "port": 5432,
        "name": "bookstore",
        "user": "app_user",
        "password": "secure_password_here",
    },
    "cache": {"ttl_seconds": 600, "max_size": 5000},
    "logging": {"level": "debug", "format": "json"},
}

config = AppConfig(**settings)
print(f"App: {config.app_name}")
print(f"DB URL: {config.database.connection_url}")
print(f"Cache TTL: {config.cache.ttl_seconds}s")
print(f"Log Level: {config.logging.level}")
print(f"Debug: {config.debug}")

Output:

App: BookStore API
DB URL: postgresql://app_user:secure_password_here@db.example.com:5432/bookstore
Cache TTL: 600s
Log Level: DEBUG
Debug: True

This configuration model validates every setting at application startup. If the database port is out of range, the log level is invalid, or debug mode conflicts with the log level, you get a clear error immediately instead of a mysterious failure at runtime. The frozen=True config prevents accidental modification after initialization.

Frequently Asked Questions

What changed between Pydantic V1 and V2?

The biggest changes are: .dict() became .model_dump(), .json() became .model_dump_json(), inner class Config became model_config = ConfigDict(...), validators use @field_validator instead of @validator, and the core validation engine was rewritten in Rust for major performance improvements. The migration guide at docs.pydantic.dev/latest/migration covers every change.

How much faster is V2 than V1?

Pydantic V2 is 5-50x faster than V1 depending on the operation. Simple model creation is about 5x faster, while complex nested validation can see 50x improvements. The speed comes from the Rust core (pydantic-core) that handles parsing and validation natively.

Should I use Pydantic or dataclasses?

Use Pydantic when you need runtime validation (API inputs, config files, external data). Use dataclasses when you need simple data containers for internal application state where the data is already trusted. Pydantic also has a @pydantic.dataclasses.dataclass decorator that adds validation to standard dataclass syntax.

How does Pydantic integrate with FastAPI?

FastAPI uses Pydantic models for request body validation, query parameter validation, and response serialization. When you define a FastAPI endpoint parameter as a Pydantic model, FastAPI automatically validates incoming JSON against it and returns 422 errors with Pydantic's structured error format.

Can I use Pydantic with SQLAlchemy or Django ORM?

Yes. Use model_config = ConfigDict(from_attributes=True) (formerly orm_mode) to create Pydantic models from ORM objects. This lets you validate and serialize database records through Pydantic models: UserSchema.model_validate(db_user) converts an ORM object into a validated Pydantic model.

Conclusion

Pydantic V2 is the most practical data validation library in the Python ecosystem. Its combination of type hint-driven validation, automatic type coercion, structured error reporting, and Rust-powered performance makes it the right choice for any project that handles external data. The configuration manager example shows how a well-designed model catches errors at startup instead of letting them surface as runtime crashes.

Start by replacing your manual validation code with Pydantic models, then explore advanced features like computed fields, generic models, and custom types. The official documentation at docs.pydantic.dev is comprehensive and well-organized.

How To Add Authentication to FastAPI with OAuth2 and JWT

How To Add Authentication to FastAPI with OAuth2 and JWT

Last Updated: June 01, 2026

Intermediate

You have built a FastAPI application with clean endpoints and Pydantic validation. Everything works perfectly — until you realize anyone on the internet can call your API. No login required, no identity checks, no permission controls. This is the exact moment every API developer reaches: your endpoints need authentication, and you need it done properly without building a security framework from scratch.

FastAPI has built-in support for OAuth2 with Password flow and integrates cleanly with JSON Web Tokens (JWT). You will need three packages beyond FastAPI itself: python-jose[cryptography] for creating and verifying JWT tokens, passlib[bcrypt] for secure password hashing, and python-multipart for handling form data in the login endpoint. Install them with pip install python-jose[cryptography] passlib[bcrypt] python-multipart.

This tutorial walks you through the complete authentication flow: hashing and verifying passwords, creating JWT access tokens, protecting endpoints with dependency injection, extracting the current user from tokens, and implementing role-based access control. By the end, you will have a reusable authentication system that you can drop into any FastAPI project.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

JWT Authentication in 30 Seconds

Part of the Python Web Frameworks Hub. See the full hub for related Python tutorials.

Part of the Modern Python AI Stack series. See the full tutorial hub for all 23 tutorials on LangGraph, MCP, Pydantic AI, Polars, FastAPI, Litestar, Typer, and more.

Here is the simplest possible protected endpoint in FastAPI. This gives you the core pattern before we build out the full system.

# quick_auth.py
from fastapi import FastAPI, Depends, HTTPException
from fastapi.security import OAuth2PasswordBearer

app = FastAPI()
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="login")

@app.get("/protected")
def protected_route(token: str = Depends(oauth2_scheme)):
    # In a real app, you would decode and verify the JWT here
    if token != "secret-token":
        raise HTTPException(status_code=401, detail="Invalid token")
    return {"message": "You have access!", "token": token}

Output (without token):

{"detail": "Not authenticated"}

Output (with valid token in Authorization header):

{"message": "You have access!", "token": "secret-token"}

The OAuth2PasswordBearer dependency automatically extracts the token from the Authorization: Bearer <token> header. If the header is missing, FastAPI returns a 401 response before your function even runs. The tokenUrl parameter tells Swagger UI where to send login requests.

How JWT Authentication Works

Before writing the full implementation, it helps to understand the flow. JWT (JSON Web Token) authentication works in four steps: the client sends credentials (username and password), the server verifies them and returns a signed token, the client includes that token in every subsequent request, and the server verifies the token signature on each protected endpoint.

StepWhoWhat Happens
1. LoginClientSends username + password to /login
2. Token creationServerVerifies password, creates signed JWT
3. Authenticated requestClientSends JWT in Authorization: Bearer header
4. Token verificationServerDecodes JWT, checks signature and expiry

The JWT itself contains encoded JSON with the user’s identity (called “claims”), an expiration time, and a cryptographic signature. The server signs the token with a secret key, so any tampering is detectable without a database lookup. This makes JWT ideal for stateless APIs where you do not want to store session data on the server.

OAuth2 authentication flow
OAuth2 is just a series of handshakes. Miss one and the bouncer won’t let you in.

Password Hashing with Passlib

Never store passwords in plain text. Use passlib with the bcrypt algorithm to hash passwords before storing them and verify them during login.

# password_utils.py
from passlib.context import CryptContext

pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")

def hash_password(password: str) -> str:
    """Hash a plain text password for storage."""
    return pwd_context.hash(password)

def verify_password(plain_password: str, hashed_password: str) -> bool:
    """Check a plain text password against a stored hash."""
    return pwd_context.verify(plain_password, hashed_password)

# Example usage
hashed = hash_password("my_secure_password")
print(f"Hashed: {hashed}")
print(f"Verify correct: {verify_password('my_secure_password', hashed)}")
print(f"Verify wrong: {verify_password('wrong_password', hashed)}")

Output:

Hashed: $2b$12$LJ3m4ys3Lg...Kz8dHJKe (unique each time)
Verify correct: True
Verify wrong: False

The CryptContext handles algorithm selection, salt generation, and hash verification. The deprecated="auto" setting means passlib will automatically upgrade old hashes to the current scheme when passwords are verified. Each hash is unique even for the same password because bcrypt generates a random salt internally.

Creating and Decoding JWT Tokens

The python-jose library creates and verifies JWT tokens. Each token contains a payload (claims) signed with your secret key.

# jwt_utils.py
from datetime import datetime, timedelta, timezone
from jose import jwt, JWTError

SECRET_KEY = "your-secret-key-keep-this-safe-and-long"
ALGORITHM = "HS256"
ACCESS_TOKEN_EXPIRE_MINUTES = 30

def create_access_token(data: dict, expires_delta: timedelta = None) -> str:
    """Create a JWT access token with an expiration time."""
    to_encode = data.copy()
    expire = datetime.now(timezone.utc) + (expires_delta or timedelta(minutes=ACCESS_TOKEN_EXPIRE_MINUTES))
    to_encode.update({"exp": expire})
    return jwt.encode(to_encode, SECRET_KEY, algorithm=ALGORITHM)

def decode_access_token(token: str) -> dict:
    """Decode and verify a JWT token. Raises JWTError if invalid."""
    return jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])

# Example
token = create_access_token({"sub": "alice", "role": "admin"})
print(f"Token: {token[:50]}...")

payload = decode_access_token(token)
print(f"Payload: {payload}")

Output:

Token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzd...
Payload: {'sub': 'alice', 'role': 'admin', 'exp': 1712534400}

The sub (subject) claim is a JWT standard for identifying the user. The exp claim sets when the token expires — after this time, decode_access_token raises a JWTError automatically. In production, store SECRET_KEY in an environment variable, not in your source code.

JWT token creation
Three Base64 segments walk into a bar. The signature picks up the tab.

Building the Complete Authentication System

Now let us combine password hashing and JWT tokens into a complete FastAPI authentication system with a user database, login endpoint, and protected routes.

# auth_app.py
from datetime import datetime, timedelta, timezone
from fastapi import FastAPI, Depends, HTTPException, status
from fastapi.security import OAuth2PasswordBearer, OAuth2PasswordRequestForm
from jose import jwt, JWTError
from passlib.context import CryptContext
from pydantic import BaseModel

# Configuration
SECRET_KEY = "your-secret-key-change-in-production"
ALGORITHM = "HS256"
ACCESS_TOKEN_EXPIRE_MINUTES = 30

# Password hashing
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")

# OAuth2 scheme
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="login")

app = FastAPI(title="Auth Demo")

# Simulated user database
fake_users_db = {
    "alice": {
        "username": "alice",
        "hashed_password": pwd_context.hash("alice123"),
        "role": "admin",
    },
    "bob": {
        "username": "bob",
        "hashed_password": pwd_context.hash("bob456"),
        "role": "user",
    },
}

# Pydantic models
class Token(BaseModel):
    access_token: str
    token_type: str

class User(BaseModel):
    username: str
    role: str

# Helper functions
def authenticate_user(username: str, password: str):
    user = fake_users_db.get(username)
    if not user or not pwd_context.verify(password, user["hashed_password"]):
        return None
    return user

def create_access_token(data: dict, expires_delta: timedelta = None) -> str:
    to_encode = data.copy()
    expire = datetime.now(timezone.utc) + (expires_delta or timedelta(minutes=ACCESS_TOKEN_EXPIRE_MINUTES))
    to_encode.update({"exp": expire})
    return jwt.encode(to_encode, SECRET_KEY, algorithm=ALGORITHM)

async def get_current_user(token: str = Depends(oauth2_scheme)) -> User:
    credentials_exception = HTTPException(
        status_code=status.HTTP_401_UNAUTHORIZED,
        detail="Could not validate credentials",
        headers={"WWW-Authenticate": "Bearer"},
    )
    try:
        payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])
        username = payload.get("sub")
        if username is None:
            raise credentials_exception
    except JWTError:
        raise credentials_exception
    user = fake_users_db.get(username)
    if user is None:
        raise credentials_exception
    return User(username=user["username"], role=user["role"])

# Endpoints
@app.post("/login", response_model=Token)
async def login(form_data: OAuth2PasswordRequestForm = Depends()):
    user = authenticate_user(form_data.username, form_data.password)
    if not user:
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Incorrect username or password",
            headers={"WWW-Authenticate": "Bearer"},
        )
    token = create_access_token(data={"sub": user["username"], "role": user["role"]})
    return {"access_token": token, "token_type": "bearer"}

@app.get("/me", response_model=User)
async def read_current_user(current_user: User = Depends(get_current_user)):
    return current_user

@app.get("/admin")
async def admin_only(current_user: User = Depends(get_current_user)):
    if current_user.role != "admin":
        raise HTTPException(status_code=403, detail="Admin access required")
    return {"message": f"Welcome admin {current_user.username}!"}

@app.get("/public")
async def public_endpoint():
    return {"message": "This endpoint is open to everyone"}

Testing the login flow:

# Step 1: Login to get a token
curl -X POST http://127.0.0.1:8000/login \
  -d "username=alice&password=alice123"

# Response:
{"access_token": "eyJhbGciOiJIUzI1NiI...", "token_type": "bearer"}

# Step 2: Access protected endpoint with the token
curl http://127.0.0.1:8000/me \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiI..."

# Response:
{"username": "alice", "role": "admin"}

# Step 3: Access admin-only endpoint
curl http://127.0.0.1:8000/admin \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiI..."

# Response:
{"message": "Welcome admin alice!"}

The get_current_user dependency is the heart of this system. It extracts the token from the header, decodes it, looks up the user, and returns a User object — all before your endpoint function runs. Any endpoint that includes current_user: User = Depends(get_current_user) is automatically protected.

Password hashing with bcrypt
bcrypt turns your password into unrecognizable mush. That’s the point.

Role-Based Access Control

The admin endpoint above uses a simple role check, but you can make this reusable with a dependency factory that creates role-checking dependencies on the fly.

# role_checker.py
from fastapi import Depends, HTTPException, status

def require_role(required_role: str):
    """Create a dependency that checks if the user has the required role."""
    async def role_checker(current_user: User = Depends(get_current_user)):
        if current_user.role != required_role:
            raise HTTPException(
                status_code=status.HTTP_403_FORBIDDEN,
                detail=f"Role '{required_role}' required. You have '{current_user.role}'.",
            )
        return current_user
    return role_checker

# Usage in endpoints
@app.get("/admin/dashboard")
async def admin_dashboard(admin: User = Depends(require_role("admin"))):
    return {"message": "Admin dashboard", "user": admin.username}

@app.get("/editor/publish")
async def editor_publish(editor: User = Depends(require_role("editor"))):
    return {"message": "Editor publish page", "user": editor.username}

Output (bob tries /admin/dashboard):

{"detail": "Role 'admin' required. You have 'user'."}

The require_role function returns a new dependency for each role. This pattern scales cleanly — add as many roles as your application needs without duplicating validation logic in every endpoint.

Token Refresh Strategy

Access tokens should be short-lived (15-30 minutes) for security. But you do not want users logging in every 30 minutes. The standard solution is a refresh token with a longer lifespan that can generate new access tokens.

# token_refresh.py
REFRESH_TOKEN_EXPIRE_DAYS = 7

def create_refresh_token(data: dict) -> str:
    to_encode = data.copy()
    expire = datetime.now(timezone.utc) + timedelta(days=REFRESH_TOKEN_EXPIRE_DAYS)
    to_encode.update({"exp": expire, "type": "refresh"})
    return jwt.encode(to_encode, SECRET_KEY, algorithm=ALGORITHM)

@app.post("/login", response_model=dict)
async def login_with_refresh(form_data: OAuth2PasswordRequestForm = Depends()):
    user = authenticate_user(form_data.username, form_data.password)
    if not user:
        raise HTTPException(status_code=401, detail="Invalid credentials")
    access_token = create_access_token({"sub": user["username"], "role": user["role"]})
    refresh_token = create_refresh_token({"sub": user["username"]})
    return {
        "access_token": access_token,
        "refresh_token": refresh_token,
        "token_type": "bearer",
    }

@app.post("/refresh", response_model=Token)
async def refresh_access_token(refresh_token: str):
    try:
        payload = jwt.decode(refresh_token, SECRET_KEY, algorithms=[ALGORITHM])
        if payload.get("type") != "refresh":
            raise HTTPException(status_code=401, detail="Invalid token type")
        username = payload.get("sub")
        new_token = create_access_token({"sub": username, "role": fake_users_db[username]["role"]})
        return {"access_token": new_token, "token_type": "bearer"}
    except JWTError:
        raise HTTPException(status_code=401, detail="Invalid refresh token")

The refresh token has a type claim set to "refresh" so it cannot be used as an access token. When the access token expires, the client sends the refresh token to /refresh to get a new access token without re-entering credentials.

Security Best Practices

Authentication code is one place where shortcuts cause real damage. Here are the practices that matter most for a production FastAPI application.

PracticeWhy It Matters
Store SECRET_KEY in env varsKeys in source code end up in Git history
Use bcrypt (not MD5/SHA)Bcrypt is intentionally slow, resistant to brute force
Set short token expiry (15-30 min)Limits damage if a token is stolen
Use HTTPS in productionTokens sent over HTTP can be intercepted
Validate token claims (sub, exp)Missing validation lets forged tokens through
Return generic error messages“Invalid credentials” not “User not found” prevents user enumeration
Token refresh handling
Access tokens expire. Refresh tokens expire slower. Your patience expires fastest.

Real-Life Example: Protected Notes API

Let us build a practical application: a notes API where each user can only see and modify their own notes. This demonstrates how authentication integrates with real CRUD operations.

# notes_api.py
from datetime import datetime, timedelta, timezone
from fastapi import FastAPI, Depends, HTTPException, status
from fastapi.security import OAuth2PasswordBearer, OAuth2PasswordRequestForm
from jose import jwt, JWTError
from passlib.context import CryptContext
from pydantic import BaseModel
from typing import Optional

SECRET_KEY = "notes-app-secret-change-me"
ALGORITHM = "HS256"
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="login")

app = FastAPI(title="Protected Notes API")

# Databases
users_db = {
    "alice": {"username": "alice", "hashed_password": pwd_context.hash("alice123")},
    "bob": {"username": "bob", "hashed_password": pwd_context.hash("bob456")},
}
notes_db = {}
next_note_id = 1

# Models
class Token(BaseModel):
    access_token: str
    token_type: str

class NoteCreate(BaseModel):
    title: str
    content: str

class NoteResponse(BaseModel):
    id: int
    title: str
    content: str
    owner: str
    created_at: str

# Auth helpers
def create_token(username: str) -> str:
    expire = datetime.now(timezone.utc) + timedelta(minutes=30)
    return jwt.encode({"sub": username, "exp": expire}, SECRET_KEY, algorithm=ALGORITHM)

async def get_current_user(token: str = Depends(oauth2_scheme)) -> str:
    try:
        payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])
        username = payload.get("sub")
        if username is None or username not in users_db:
            raise HTTPException(status_code=401, detail="Invalid token")
        return username
    except JWTError:
        raise HTTPException(status_code=401, detail="Invalid token")

# Endpoints
@app.post("/login", response_model=Token)
async def login(form_data: OAuth2PasswordRequestForm = Depends()):
    user = users_db.get(form_data.username)
    if not user or not pwd_context.verify(form_data.password, user["hashed_password"]):
        raise HTTPException(status_code=401, detail="Invalid credentials")
    return {"access_token": create_token(form_data.username), "token_type": "bearer"}

@app.post("/notes", response_model=NoteResponse, status_code=201)
async def create_note(note: NoteCreate, username: str = Depends(get_current_user)):
    global next_note_id
    note_data = {
        "id": next_note_id,
        "title": note.title,
        "content": note.content,
        "owner": username,
        "created_at": datetime.now(timezone.utc).isoformat(),
    }
    notes_db[next_note_id] = note_data
    next_note_id += 1
    return note_data

@app.get("/notes", response_model=list[NoteResponse])
async def list_my_notes(username: str = Depends(get_current_user)):
    return [n for n in notes_db.values() if n["owner"] == username]

@app.delete("/notes/{note_id}", status_code=204)
async def delete_note(note_id: int, username: str = Depends(get_current_user)):
    note = notes_db.get(note_id)
    if not note:
        raise HTTPException(status_code=404, detail="Note not found")
    if note["owner"] != username:
        raise HTTPException(status_code=403, detail="Not your note")
    del notes_db[note_id]

Testing the flow:

# Login as Alice
curl -X POST http://127.0.0.1:8000/login -d "username=alice&password=alice123"
# {"access_token": "eyJ...", "token_type": "bearer"}

# Create a note (use Alice's token)
curl -X POST http://127.0.0.1:8000/notes \
  -H "Authorization: Bearer eyJ..." \
  -H "Content-Type: application/json" \
  -d '{"title": "Shopping List", "content": "Milk, eggs, bread"}'
# {"id": 1, "title": "Shopping List", "content": "Milk, eggs, bread", "owner": "alice", ...}

# Bob cannot see Alice's notes
curl http://127.0.0.1:8000/notes -H "Authorization: Bearer BOB_TOKEN"
# [] (empty list -- Bob has no notes)

Each note is tagged with its owner, and the list_my_notes endpoint filters by the authenticated user. The delete endpoint checks ownership before allowing deletion. This pattern of tying data to the authenticated user is fundamental to building multi-tenant APIs.

Frequently Asked Questions

Should I use sessions or JWT for my API?

Use JWT for stateless APIs (especially mobile or SPA clients) where you do not want to maintain server-side session storage. Use sessions for traditional server-rendered web applications where the backend controls the full page lifecycle. JWT works well for microservices because any service can verify the token independently without a shared session store.

How should I generate and store the SECRET_KEY?

Generate a random key with openssl rand -hex 32 in your terminal. Store it in an environment variable and read it with os.environ["SECRET_KEY"]. Never commit it to version control. In production, use a secrets manager like AWS Secrets Manager, HashiCorp Vault, or your platform’s built-in secrets.

How do I implement password reset?

Create a /forgot-password endpoint that generates a short-lived token (10-15 minutes) and sends it to the user’s email. Create a /reset-password endpoint that accepts the token and the new password. Use the same JWT mechanism but with a different token type claim so reset tokens cannot be used for API access.

Can I add Google or GitHub login?

Yes. Use the authlib or httpx-oauth library to implement OAuth2 authorization code flow. FastAPI’s dependency injection makes it straightforward to add multiple authentication methods. The user logs in through the provider’s OAuth flow, and your server exchanges the authorization code for user profile data.

How do I test authenticated endpoints?

Use FastAPI’s TestClient with the headers parameter. Create a test user, generate a token in your test setup, and include it in requests: client.get("/me", headers={"Authorization": f"Bearer {token}"}). For dependency overrides, use app.dependency_overrides[get_current_user] = lambda: test_user.

Conclusion

FastAPI’s dependency injection system makes OAuth2 and JWT authentication clean and reusable. The get_current_user dependency is the single point of authentication that you inject into any endpoint that needs protection. Combined with passlib for secure password hashing and python-jose for token management, you have a production-ready auth system in under 100 lines of code.

Start with the Protected Notes API example and extend it with a real database, email verification, and OAuth2 social login. The official FastAPI security documentation at fastapi.tiangolo.com/tutorial/security covers advanced patterns including scopes and multiple authentication schemes.

FastAPI vs Flask: Which Python Framework Should You Choose?

FastAPI vs Flask: Which Python Framework Should You Choose?

Last Updated: June 01, 2026

Intermediate

You have a new Python project that needs a web API. You open Google, type “best Python web framework,” and immediately drown in opinions. Flask has been the go-to choice for over a decade. FastAPI showed up in 2018 and climbed to 75,000+ GitHub stars faster than almost any Python project in history. Both can build APIs. Both are lightweight. So which one should you actually pick for your next project?

The answer depends on what you are building. Flask gives you maximum flexibility and a massive ecosystem of extensions. FastAPI gives you automatic data validation, async support out of the box, and self-documenting endpoints. Neither is universally better — they solve different problems in different ways, and understanding those differences saves you from rewriting code six months later.

In this article, we compare Flask and FastAPI across every dimension that matters: setup and routing, data validation, async support, performance, automatic documentation, ecosystem maturity, and real-world project structure. Every comparison includes runnable code so you can see the differences yourself. By the end, you will have a clear decision framework for choosing the right tool.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program — 270+ in-depth tutorials covering the modern Python stack.

View all tutorials by Pubs →

FastAPI vs Flask: Quick Comparison

Part of the Python Web Frameworks Hub. See the full hub for related Python tutorials.

Part of the Modern Python AI Stack series. See the full tutorial hub for all 23 tutorials on LangGraph, MCP, Pydantic AI, Polars, FastAPI, Litestar, Typer, and more.

Before we dig into code, here is a high-level comparison table covering the key differences between Flask and FastAPI.

FeatureFlaskFastAPI
First release20102018
Async supportLimited (Flask 2.0+)Native, built on Starlette
Data validationManual or Flask-MarshmallowBuilt-in via Pydantic
Auto documentationRequires Flask-RESTX or FlasggerBuilt-in Swagger and ReDoc
Type hintsOptional, no runtime effectRequired, drive validation and docs
Learning curveVery gentleGentle (steeper if new to type hints)
Ecosystem sizeHuge (thousands of extensions)Growing fast, fewer extensions
WSGI/ASGIWSGI (Werkzeug)ASGI (Starlette + Uvicorn)
Best forTraditional web apps, prototypes, server-rendered pagesModern APIs, microservices, async workloads

Now let us see how these differences play out in actual code.

FastAPI async speed advantage
async/await turns your API into a rocket. Flask takes the scenic route.

Hello World: Setup and First Route

The fastest way to feel the difference between Flask and FastAPI is to build the simplest possible endpoint in each. Both frameworks let you go from zero to running server in under 10 lines.

Flask Hello World

Flask uses the @app.route decorator and returns plain strings or dictionaries. Install it with pip install flask and run with the built-in development server.

# flask_hello.py
from flask import Flask, jsonify

app = Flask(__name__)

@app.route("/hello")
def hello():
    return jsonify({"message": "Hello from Flask!"})

if __name__ == "__main__":
    app.run(debug=True, port=5000)

Output (when you visit http://localhost:5000/hello):

{"message": "Hello from Flask!"}

FastAPI Hello World

FastAPI uses the same decorator pattern but returns dictionaries directly — no need for jsonify. Install it with pip install fastapi uvicorn and run with Uvicorn.

# fastapi_hello.py
from fastapi import FastAPI

app = FastAPI()

@app.get("/hello")
async def hello():
    return {"message": "Hello from FastAPI!"}

# Run with: uvicorn fastapi_hello:app --reload --port 8000

Output (when you visit http://localhost:8000/hello):

{"message": "Hello from FastAPI!"}

The syntax is nearly identical. The key differences: FastAPI uses HTTP method decorators (@app.get instead of @app.route), supports async def natively, and returns dicts without a wrapper function. Flask requires jsonify() to return proper JSON responses.

Data Validation: Where FastAPI Pulls Ahead

Data validation is where the two frameworks diverge most sharply. Flask leaves validation entirely up to you. FastAPI makes it automatic through Python type hints and Pydantic models. This single difference changes how much boilerplate you write for every endpoint.

Flask: Manual Validation

In Flask, you parse the request body yourself, check each field manually, and return error responses when something is wrong. Here is a typical pattern for creating a user.

# flask_validation.py
from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/users", methods=["POST"])
def create_user():
    data = request.get_json()

    # Manual validation -- you write every check
    if not data:
        return jsonify({"error": "Request body required"}), 400
    if "name" not in data or not isinstance(data["name"], str):
        return jsonify({"error": "name must be a string"}), 400
    if "email" not in data or "@" not in data["email"]:
        return jsonify({"error": "Valid email required"}), 400
    if "age" in data and not isinstance(data["age"], int):
        return jsonify({"error": "age must be an integer"}), 400

    return jsonify({"status": "created", "user": data}), 201

if __name__ == "__main__":
    app.run(debug=True, port=5000)

Output (POST with valid data):

{"status": "created", "user": {"name": "Alice", "email": "alice@example.com", "age": 30}}

FastAPI: Pydantic Does the Work

FastAPI validates request data automatically using Pydantic models. You define a model with type hints, and FastAPI rejects invalid requests before your function even runs.

# fastapi_validation.py
from fastapi import FastAPI
from pydantic import BaseModel, EmailStr

app = FastAPI()

class User(BaseModel):
    name: str
    email: EmailStr
    age: int | None = None

@app.post("/users", status_code=201)
async def create_user(user: User):
    return {"status": "created", "user": user.model_dump()}

# Run with: uvicorn fastapi_validation:app --reload --port 8000

Output (POST with invalid email):

{
  "detail": [
    {
      "type": "value_error",
      "loc": ["body", "email"],
      "msg": "value is not a valid email address"
    }
  ]
}

The FastAPI version is half the code and catches more errors. Pydantic validates types, checks email format, handles optional fields with defaults, and returns structured error responses automatically. In Flask, you would need to add a library like Marshmallow or Cerberus to get similar functionality, and even then it requires more setup.

Flask traditional approach
Flask has been battle-tested since 2010. That’s either reassuring or terrifying.

Async Support: Native vs Retrofitted

Modern APIs often need to handle many concurrent connections — waiting on database queries, calling external services, or streaming responses. Async programming lets your server handle thousands of these waiting operations without blocking. This is where the architectural difference between Flask and FastAPI matters most.

FastAPI was built on ASGI (Asynchronous Server Gateway Interface) from the start. Every route handler can be an async def function, and the framework coordinates concurrency through Python’s asyncio event loop. Flask was built on WSGI (Web Server Gateway Interface), which is synchronous by design. Flask 2.0 added async def support, but it runs each async view in a separate thread rather than using a true event loop.

FastAPI Async Example

# fastapi_async.py
import asyncio
from fastapi import FastAPI

app = FastAPI()

async def fetch_from_database():
    """Simulate a slow database query."""
    await asyncio.sleep(1)
    return {"users": ["Alice", "Bob", "Charlie"]}

async def fetch_from_cache():
    """Simulate a cache lookup."""
    await asyncio.sleep(0.5)
    return {"cached": True}

@app.get("/dashboard")
async def dashboard():
    # Both calls run concurrently -- total time ~1 second, not 1.5
    db_task = asyncio.create_task(fetch_from_database())
    cache_task = asyncio.create_task(fetch_from_cache())
    db_result = await db_task
    cache_result = await cache_task
    return {**db_result, **cache_result}

# Run with: uvicorn fastapi_async:app --reload --port 8000

Output:

{"users": ["Alice", "Bob", "Charlie"], "cached": true}

The two async calls run concurrently using asyncio.create_task(), so the total response time is about 1 second instead of 1.5 seconds. This pattern scales beautifully when your API calls multiple microservices or databases per request.

Flask Async (Limited)

# flask_async.py
import asyncio
from flask import Flask, jsonify

app = Flask(__name__)

@app.route("/dashboard")
async def dashboard():
    # Flask 2.0+ supports async views, but runs them in threads
    await asyncio.sleep(1)
    return jsonify({"users": ["Alice", "Bob", "Charlie"], "note": "async works but limited"})

if __name__ == "__main__":
    app.run(debug=True, port=5000)

Flask’s async support works for simple cases, but it does not give you the same concurrency benefits as FastAPI’s native ASGI approach. For I/O-heavy workloads with many concurrent connections, FastAPI’s architecture has a clear advantage.

Automatic API Documentation

One of FastAPI’s most impressive features is automatic interactive documentation. The moment you define an endpoint with type hints, FastAPI generates Swagger UI and ReDoc pages at /docs and /redoc respectively. No configuration needed.

Flask has no built-in documentation generator. You can add it with extensions like Flask-RESTX (which includes Swagger) or Flasgger, but they require additional decorators and configuration. Here is what you need in Flask to get what FastAPI gives you for free.

Docs FeatureFlaskFastAPI
Swagger UIFlask-RESTX or Flasgger (install + configure)Built-in at /docs
ReDocManual setupBuilt-in at /redoc
OpenAPI schemaGenerated by extensionBuilt-in at /openapi.json
Request body docsManual schema definitionsAuto-generated from Pydantic models
Response examplesManualAuto-generated from return type hints

This is a significant productivity win for teams building APIs. Frontend developers, QA testers, and external consumers can explore your API interactively without reading source code or maintaining a separate Postman collection.

Comparing FastAPI and Flask features
Feature comparison: where one framework’s ceiling is the other’s floor.

Project Structure and Scalability

Both frameworks support clean project organization through modular patterns, but they use different terminology. Flask uses Blueprints to split a large application into reusable modules. FastAPI uses APIRouter for the same purpose. The concepts are nearly identical.

Flask Blueprints

# flask_blueprint.py
from flask import Flask, Blueprint, jsonify

# Define a blueprint for user routes
users_bp = Blueprint("users", __name__, url_prefix="/users")

@users_bp.route("/")
def list_users():
    return jsonify({"users": ["Alice", "Bob"]})

@users_bp.route("/<int:user_id>")
def get_user(user_id):
    return jsonify({"user_id": user_id, "name": "Alice"})

# Main app registers the blueprint
app = Flask(__name__)
app.register_blueprint(users_bp)

if __name__ == "__main__":
    app.run(debug=True, port=5000)

Output (GET /users/):

{"users": ["Alice", "Bob"]}

FastAPI APIRouter

# fastapi_router.py
from fastapi import FastAPI, APIRouter

# Define a router for user routes
users_router = APIRouter(prefix="/users", tags=["users"])

@users_router.get("/")
async def list_users():
    return {"users": ["Alice", "Bob"]}

@users_router.get("/{user_id}")
async def get_user(user_id: int):
    return {"user_id": user_id, "name": "Alice"}

# Main app includes the router
app = FastAPI()
app.include_router(users_router)

# Run with: uvicorn fastapi_router:app --reload --port 8000

Output (GET /users/):

{"users": ["Alice", "Bob"]}

The patterns are almost identical. The main difference is that FastAPI’s router automatically includes the routes in the generated documentation under the specified tag, while Flask Blueprints need additional configuration for documentation.

Ecosystem and Community

Flask has been around since 2010, which gives it a massive head start in ecosystem size. If you need a feature, chances are someone built a Flask extension for it: Flask-Login for authentication, Flask-SQLAlchemy for ORM integration, Flask-Mail for email, Flask-CORS for cross-origin requests, Flask-Migrate for database migrations, and hundreds more.

FastAPI’s ecosystem is smaller but growing rapidly. It leans on the broader Python ecosystem rather than framework-specific extensions. For databases, you use SQLAlchemy or Tortoise-ORM directly. For authentication, you use python-jose and passlib. For CORS, FastAPI has built-in middleware. This approach means fewer “FastAPI-specific” packages but more flexibility in choosing your tools.

NeedFlask ExtensionFastAPI Approach
AuthenticationFlask-Login, Flask-JWT-ExtendedBuilt-in OAuth2 + python-jose
Database ORMFlask-SQLAlchemySQLAlchemy (async) or SQLModel
CORSFlask-CORSBuilt-in CORSMiddleware
Form handlingFlask-WTFPydantic models
Admin panelFlask-AdminSQLAdmin or Starlette-Admin
Rate limitingFlask-Limiterslowapi
Migration strategy
Migrating frameworks mid-project is like changing engines mid-flight. Plan accordingly.

When to Use Each Framework

After comparing code, features, and ecosystems, here is a practical decision framework. Neither framework is objectively better — the right choice depends on what you are building and who is building it.

Choose Flask When

Flask is the better choice when you are building server-rendered web applications with HTML templates (Jinja2 is deeply integrated), when your team is new to web development and benefits from Flask’s minimal learning curve, when you need a specific Flask extension that has no equivalent in FastAPI’s ecosystem, when you are prototyping quickly and do not need type-enforced validation, or when you are maintaining an existing Flask codebase and migration is not justified.

Choose FastAPI When

FastAPI is the better choice when you are building a pure REST API or microservice (no server-rendered HTML), when you need automatic request/response validation and do not want to write it yourself, when your API handles many concurrent I/O operations (database calls, external API requests), when you want auto-generated interactive documentation for your team or API consumers, or when you are starting a new project and your team is comfortable with Python type hints.

Real-Life Example: Todo API in Both Frameworks

To make the comparison concrete, here is the same Todo API built in both frameworks. This gives you a side-by-side view of how the same requirements translate into code.

Choosing the right framework
The best framework is the one that ships your project. The second best is the one you actually know.

Flask Version

# flask_todo.py
from flask import Flask, request, jsonify

app = Flask(__name__)
todos = []
next_id = 1

@app.route("/todos", methods=["GET"])
def list_todos():
    return jsonify(todos)

@app.route("/todos", methods=["POST"])
def create_todo():
    global next_id
    data = request.get_json()
    if not data or "title" not in data:
        return jsonify({"error": "title is required"}), 400
    todo = {
        "id": next_id,
        "title": data["title"],
        "done": data.get("done", False)
    }
    next_id += 1
    todos.append(todo)
    return jsonify(todo), 201

@app.route("/todos/<int:todo_id>", methods=["PUT"])
def update_todo(todo_id):
    data = request.get_json()
    for todo in todos:
        if todo["id"] == todo_id:
            todo["title"] = data.get("title", todo["title"])
            todo["done"] = data.get("done", todo["done"])
            return jsonify(todo)
    return jsonify({"error": "Not found"}), 404

@app.route("/todos/<int:todo_id>", methods=["DELETE"])
def delete_todo(todo_id):
    global todos
    todos = [t for t in todos if t["id"] != todo_id]
    return jsonify({"status": "deleted"})

if __name__ == "__main__":
    app.run(debug=True, port=5000)

FastAPI Version

# fastapi_todo.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()
todos = []
next_id = 1

class TodoCreate(BaseModel):
    title: str
    done: bool = False

class TodoResponse(BaseModel):
    id: int
    title: str
    done: bool

@app.get("/todos", response_model=list[TodoResponse])
async def list_todos():
    return todos

@app.post("/todos", response_model=TodoResponse, status_code=201)
async def create_todo(todo: TodoCreate):
    global next_id
    new_todo = {"id": next_id, "title": todo.title, "done": todo.done}
    next_id += 1
    todos.append(new_todo)
    return new_todo

@app.put("/todos/{todo_id}", response_model=TodoResponse)
async def update_todo(todo_id: int, todo: TodoCreate):
    for t in todos:
        if t["id"] == todo_id:
            t["title"] = todo.title
            t["done"] = todo.done
            return t
    raise HTTPException(status_code=404, detail="Not found")

@app.delete("/todos/{todo_id}")
async def delete_todo(todo_id: int):
    global todos
    todos = [t for t in todos if t["id"] != todo_id]
    return {"status": "deleted"}

# Run with: uvicorn fastapi_todo:app --reload --port 8000

Both versions do the same thing, but the FastAPI version gives you automatic validation on every POST and PUT request, typed response models that document exactly what each endpoint returns, and interactive Swagger docs at /docs — all without a single extra line of configuration. The Flask version requires you to validate manually and document separately.

Frequently Asked Questions

Is Flask dead now that FastAPI exists?

Not at all. Flask remains one of the most popular Python web frameworks with active development, regular releases, and a massive ecosystem. Flask 3.0 introduced further improvements and the framework continues to evolve. Many production applications run Flask successfully, and it remains an excellent choice for server-rendered web applications, prototypes, and projects that benefit from its extensive extension library.

Can I migrate from Flask to FastAPI incrementally?

Yes, but it is not a simple find-and-replace. The routing syntax is similar, but data validation patterns, middleware, and extension usage differ significantly. The most practical approach is to build new endpoints in FastAPI while keeping existing Flask endpoints running, then migrate route by route. Libraries like a2wsgi can help run both frameworks during the transition period.

Is FastAPI really faster than Flask?

In benchmarks, FastAPI running on Uvicorn typically handles 2-3x more requests per second than Flask running on Gunicorn for async workloads. For synchronous, CPU-bound tasks, the difference is smaller. The real performance gain comes from FastAPI’s native async support, which lets a single worker handle many concurrent I/O-bound requests without blocking.

Which should I learn first as a beginner?

Flask is often recommended as a first framework because it has fewer concepts to learn upfront. You can build a working web app without understanding type hints, Pydantic models, or async/await. Once you are comfortable with Flask and HTTP concepts, learning FastAPI becomes straightforward because you already understand routing, request handling, and middleware.

What about Django? How does it compare?

Django is a “batteries-included” full-stack framework with an ORM, admin panel, authentication system, and template engine built in. Flask and FastAPI are both “micro” frameworks that let you choose your own components. If you need a full web application with user management, an admin dashboard, and server-rendered pages, Django is worth considering. For pure REST APIs and microservices, Flask or FastAPI are typically more appropriate.

Conclusion

Flask and FastAPI are both excellent Python web frameworks that serve different needs. Flask gives you simplicity, flexibility, and the largest extension ecosystem in the Python web world. FastAPI gives you automatic validation, native async support, and self-documenting APIs with zero extra configuration. The code examples in this article show that the syntax is similar enough to switch between them comfortably.

For your next project, start by asking: “Am I building a REST API or a full web application?” If you are building a pure API with typed data flowing in and out, FastAPI will save you hours of boilerplate validation and documentation. If you are building a traditional web app with HTML templates, or you need a specific Flask extension, Flask remains the proven choice.

You can explore the official documentation for both frameworks to go deeper: Flask documentation and FastAPI documentation.