How To Use Python socketio for Real-Time Web Events

How To Use Python socketio for Real-Time Web Events

Intermediate

You build a web app that needs live updates — a chat interface, a dashboard that refreshes when data changes, a multiplayer game, or a notification feed. You try the obvious approach: poll the server every two seconds with setInterval(fetch(...)). It works, but now your server is drowning in requests for data that has not changed, your latency is measured in seconds instead of milliseconds, and your browser tab is burning CPU even when nothing is happening. Polling is the traffic jam of web architecture.

Python’s python-socketio library solves this with a proper two-way connection. The browser and server stay connected and push events to each other the moment something changes — no polling, no wasted requests. Socket.IO builds on WebSockets but adds automatic fallbacks, reconnection logic, namespaces, and rooms out of the box. Install it with pip install python-socketio aiohttp and you have everything you need to build a real-time server in Python.

This article covers the complete socket.io toolkit for Python: setting up an async server with aiohttp, emitting and listening to events from both server and client, using rooms to group connections, broadcasting to multiple clients, and a real-life example that builds a live notification server. By the end, you will be able to push events from your Python backend to any connected browser the moment they occur.

Python socketio: Quick Example

The fastest way to understand socket.io is to run a server that echoes a message back to whoever sent it. This requires two files: a Python server and an HTML page that connects to it.

# server_quick.py
import asyncio
import aiohttp
from aiohttp import web
import socketio

# Create the Socket.IO server
sio = socketio.AsyncServer(cors_allowed_origins='*')
app = web.Application()
sio.attach(app)

@sio.event
async def connect(sid, environ):
    print(f'Client connected: {sid}')

@sio.event
async def disconnect(sid):
    print(f'Client disconnected: {sid}')

@sio.event
async def message(sid, data):
    print(f'Received from {sid}: {data}')
    # Echo it back to the sender only
    await sio.emit('reply', {'text': f'Server got: {data}'}, to=sid)

if __name__ == '__main__':
    web.run_app(app, port=8080)

Output (server terminal):

======== Running on http://0.0.0.0:8080 ========
Client connected: abc123def456
Received from abc123def456: Hello server
Client disconnected: abc123def456

The @sio.event decorator registers a function as a handler for a named event. The connect and disconnect events are built-in — they fire automatically whenever a client joins or leaves. The message event is custom — you define the name and the handler. When a browser sends a message event, your Python function receives the session ID (sid) and the data payload, and can emit a response back instantly using sio.emit().

The real power comes when you start emitting to multiple clients simultaneously, organizing connections into rooms, and broadcasting from server to browser without waiting for a client request. The sections below cover all of that in detail.

What Is socket.io and Why Use It?

Socket.IO is a protocol and library that provides reliable, bidirectional, event-based communication between a browser and a server. It was originally a JavaScript library, and python-socketio is the official Python implementation — it is fully compatible with the JavaScript socket.io client library used in browsers. When a browser connects to your Python socket.io server, both sides can emit named events with JSON payloads at any time, in either direction, without either side having to “ask” first.

The key distinction from raw WebSockets is that Socket.IO adds a protocol layer on top. It handles connection upgrade (from HTTP to WebSocket), automatic reconnection with exponential backoff, event acknowledgements, namespaces (logical separation within one connection), and rooms (groups of connections that can be addressed together). You get these features for free without building them yourself.

FeatureHTTP PollingRaw WebSocketSocket.IO
DirectionClient initiates onlyBoth directionsBoth directions
LatencyPoll interval (1-30s)Near-instantNear-instant
Auto-reconnectManualManualBuilt in
Rooms / groupsNot availableManualBuilt in
NamespacesNot availableNot availableBuilt in
Fallback (no WS)N/ANot availableFalls back to polling
Browser clientfetch / XMLHttpRequestWebSocket APIsocket.io-client JS lib

Install the library with pip. The aiohttp package provides the async HTTP server that socket.io attaches to. You can also use flask-socketio if you prefer Flask’s synchronous model, but the async approach scales better for concurrent connections.

# Install in your terminal
# pip install python-socketio aiohttp

# Verify installation
import socketio
print(socketio.__version__)  # e.g., 5.11.0

The socketio.AsyncServer class is the async variant — use it with aiohttp or any ASGI framework. The socketio.Server class is the synchronous variant for use with Flask or WSGI apps. This article uses the async version throughout, since it is the recommended approach for new projects.

API Alice demonstrating two-way socket.io event flow between browser and server
HTTP polling vs. socket.io: one connection, infinite events.

Setting Up an Async Socket.IO Server

A production-ready socket.io server needs three things: the AsyncServer instance, an aiohttp web application to serve HTTP alongside the socket.io traffic, and a static HTML file for the browser client. Here is a complete server with all three, including a health check endpoint so you can verify the server is running from a browser.

# server_full.py
import aiohttp
from aiohttp import web
import socketio

# cors_allowed_origins controls which browser origins can connect.
# Use '*' in development, list specific domains in production.
sio = socketio.AsyncServer(
    cors_allowed_origins='*',
    logger=True,
    engineio_logger=False  # set True to see low-level ping/pong traffic
)

app = web.Application()
sio.attach(app)

# Standard aiohttp route alongside socket.io
async def index(request):
    with open('client.html') as f:
        return web.Response(text=f.read(), content_type='text/html')

app.router.add_get('/', index)
app.router.add_static('/static', path='./static', name='static')

@sio.event
async def connect(sid, environ, auth):
    # auth is the data passed in the socket.io connect options from the client
    print(f'[CONNECT] sid={sid}  ip={environ.get("REMOTE_ADDR")}  auth={auth}')

@sio.event
async def disconnect(sid):
    print(f'[DISCONNECT] sid={sid}')

@sio.event
async def ping_server(sid, data):
    """Client sends ping_server; server responds with pong_client."""
    print(f'[PING] from {sid}: {data}')
    await sio.emit('pong_client', {'received': data, 'from': sid}, to=sid)

if __name__ == '__main__':
    web.run_app(app, host='0.0.0.0', port=8080)

Output (server startup):

======== Running on http://0.0.0.0:8080 ========
(Press CTRL+C to quit)

The cors_allowed_origins='*' setting is important during development — without it, the browser will block the WebSocket upgrade with a CORS error. In production, replace '*' with a list of your actual domains: ['https://myapp.com', 'https://www.myapp.com']. The logger=True flag sends socket.io connection events to the standard Python logger, which is useful for debugging connection issues.

The auth parameter in the connect handler deserves attention. The browser client can pass authentication data (a token, a user ID) at connection time — your server receives it in auth and can reject the connection by returning False. This is the right place to validate tokens before accepting a client.

The Browser Client

The browser side uses the official socket.io-client JavaScript library, loaded from a CDN. It mirrors the Python API exactly — socket.on() listens for events, socket.emit() sends them.

<!-- client.html -->
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Socket.IO Demo</title>
</head>
<body>
  <h2>Socket.IO Demo</h2>
  <div id="status">Connecting...</div>
  <button onclick="sendPing()">Send Ping</button>
  <ul id="log"></ul>

  <script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/4.7.2/socket.io.min.js"></script>
  <script>
    const socket = io('http://localhost:8080', {
      auth: { token: 'demo-user-123' }
    });

    socket.on('connect', () => {
      document.getElementById('status').textContent = 'Connected: ' + socket.id;
    });

    socket.on('disconnect', () => {
      document.getElementById('status').textContent = 'Disconnected';
    });

    // Listen for pong_client events from the server
    socket.on('pong_client', (data) => {
      const li = document.createElement('li');
      li.textContent = 'Server replied: ' + JSON.stringify(data);
      document.getElementById('log').appendChild(li);
    });

    function sendPing() {
      socket.emit('ping_server', { time: Date.now(), msg: 'hello' });
    }
  </script>
</body>
</html>

The pattern mirrors the Python server exactly. socket.on('event_name', handler) registers a listener for any event the server emits. socket.emit('event_name', data) sends an event to the server. The connection is established automatically when the page loads — you do not call any “connect” function explicitly. The auth option passes data that arrives in the Python connect handler’s auth parameter, where you can validate it.

Sudo Sam connecting two monitors with glowing fiber optic cables in a dark server room
socket.on() doesn’t ask. It listens. The server decides when to talk.

Rooms and Broadcasting

Rooms are how socket.io handles group communication. A room is simply a named bucket that connections can join and leave. When you emit to a room, every connection in that room receives the event — without you having to track which clients are in which group. This is the foundation for features like chat channels, topic subscriptions, and per-user notification queues.

# server_rooms.py
import aiohttp
from aiohttp import web
import socketio

sio = socketio.AsyncServer(cors_allowed_origins='*')
app = web.Application()
sio.attach(app)

@sio.event
async def connect(sid, environ):
    print(f'Connected: {sid}')

@sio.event
async def disconnect(sid):
    print(f'Disconnected: {sid}')

@sio.event
async def join_room(sid, data):
    """Client sends join_room with {'room': 'sports'} to subscribe."""
    room = data.get('room')
    if not room:
        return
    await sio.enter_room(sid, room)
    print(f'{sid} joined room: {room}')
    # Notify the joining client only
    await sio.emit('room_joined', {'room': room, 'sid': sid}, to=sid)

@sio.event
async def leave_room(sid, data):
    """Client sends leave_room with {'room': 'sports'} to unsubscribe."""
    room = data.get('room')
    if not room:
        return
    await sio.leave_room(sid, room)
    print(f'{sid} left room: {room}')

@sio.event
async def send_to_room(sid, data):
    """Broadcast a message to everyone in a room (excluding the sender)."""
    room = data.get('room')
    message = data.get('message', '')
    if not room or not message:
        return
    # skip_sid=sid means the sender does not receive their own broadcast
    await sio.emit(
        'room_message',
        {'from': sid, 'room': room, 'message': message},
        room=room,
        skip_sid=sid
    )

async def server_broadcast(app):
    """Background task: push a server-initiated event every 10 seconds."""
    import asyncio
    count = 0
    while True:
        await asyncio.sleep(10)
        count += 1
        await sio.emit('server_tick', {'count': count, 'msg': 'Heartbeat from server'})
        print(f'Broadcast tick #{count} to all clients')

app.on_startup.append(lambda app: asyncio.ensure_future(server_broadcast(app)))

if __name__ == '__main__':
    import asyncio
    web.run_app(app, port=8080)

Server output when two clients join “sports” and one sends a message:

Connected: sid_client_A
Connected: sid_client_B
sid_client_A joined room: sports
sid_client_B joined room: sports
Broadcast tick #1 to all clients
sid_client_A left room: sports

Four functions do all the room work. sio.enter_room(sid, room) adds a connection to a room. sio.leave_room(sid, room) removes it. sio.emit('event', data, room='roomname') sends to everyone in the room. The skip_sid parameter is the socket.io idiom for “broadcast to the room but not the sender” — exactly what you want for chat-style features where the sender already knows what they typed. Notice also the background server_broadcast coroutine: it emits a server_tick event to all connected clients every 10 seconds without any client request. This is how server-initiated push notifications work.

Namespaces: Logical Separation Within One Connection

Namespaces let you split your socket.io traffic into separate logical channels over a single physical WebSocket connection. Think of namespaces like URL paths for events: a client connected to /admin only receives events emitted to that namespace. This is cleaner than inventing your own event-name prefixes and is the right tool when you have genuinely separate concerns — such as a public feed and an admin control panel — sharing the same server.

# server_namespaces.py
import aiohttp
from aiohttp import web
import socketio

sio = socketio.AsyncServer(cors_allowed_origins='*')
app = web.Application()
sio.attach(app)

# Default namespace '/' -- open to everyone
@sio.event
async def connect(sid, environ):
    print(f'[/] Connected: {sid}')

@sio.event
async def public_message(sid, data):
    print(f'[/] Public message from {sid}: {data}')
    await sio.emit('public_reply', {'msg': 'Hello from public namespace'}, to=sid)

# Admin namespace '/admin' -- you would validate credentials in connect
@sio.event(namespace='/admin')
async def connect(sid, environ):
    print(f'[/admin] Admin connected: {sid}')

@sio.event(namespace='/admin')
async def admin_action(sid, data):
    action = data.get('action', 'unknown')
    print(f'[/admin] Action from {sid}: {action}')
    # Broadcast result to ALL admin clients
    await sio.emit('admin_result', {'action': action, 'status': 'done'}, namespace='/admin')

if __name__ == '__main__':
    web.run_app(app, port=8080)

Browser client connecting to the admin namespace:

// In the browser JavaScript
const publicSocket = io('http://localhost:8080/');
const adminSocket  = io('http://localhost:8080/admin');

adminSocket.on('connect', () => console.log('Admin connected:', adminSocket.id));
adminSocket.on('admin_result', (data) => console.log('Admin result:', data));
adminSocket.emit('admin_action', { action: 'purge_cache' });

The namespace='/admin' argument on @sio.event scopes that handler exclusively to the admin channel. Events emitted on / (the default namespace) are invisible to clients connected on /admin, and vice versa. In a real app, you would validate a token in the /admin connect handler and return False to reject unauthorized connections before they ever reach the event handlers.

Debug Dee splitting a fiber optic cable into two namespace channels using a prism device
Namespaces: same wire, different channel. Your admin events stay out of the public feed.

Using the Python Client

Socket.IO is not just for browser clients. The python-socketio library also ships an async client, which lets you connect one Python process to another as equals — useful for microservices that need real-time coordination, or for writing integration tests against your socket.io server.

# client_python.py
import asyncio
import socketio

# Async Python client
sio_client = socketio.AsyncClient()

@sio_client.event
async def connect():
    print('Python client connected, sid:', sio_client.sid)

@sio_client.event
async def disconnect():
    print('Python client disconnected')

@sio_client.event
async def pong_client(data):
    print('Received from server:', data)

async def main():
    await sio_client.connect('http://localhost:8080', auth={'token': 'python-client'})

    # Send a ping event and wait a moment for the reply
    await sio_client.emit('ping_server', {'msg': 'hello from Python client'})
    await asyncio.sleep(1)

    # Join a room
    await sio_client.emit('join_room', {'room': 'sports'})
    await asyncio.sleep(1)

    await sio_client.disconnect()

if __name__ == '__main__':
    asyncio.run(main())

Output:

Python client connected, sid: xyz789abc000
Received from server: {'received': {'msg': 'hello from Python client'}, 'from': 'xyz789abc000'}

The async Python client mirrors the server API. Decorate handlers with @sio_client.event and connect with await sio_client.connect(url). You can use it to write automated tests against your socket.io server, to connect two backend services for real-time coordination, or to build a bot that responds to server-emitted events. The same reconnection and room logic applies — the client and server are peers speaking the same protocol.

Real-Life Example: Live Notification Server

This project builds a notification hub: a Python server that accepts incoming “alert” events from any Python process (a monitoring script, a background worker, a webhook handler) and broadcasts them instantly to all connected browser clients. This is the core pattern behind live dashboards, build status monitors, and ops alert feeds.

# notification_server.py
import asyncio
import aiohttp
from aiohttp import web
import socketio
import datetime

sio = socketio.AsyncServer(cors_allowed_origins='*', logger=False)
app = web.Application()
sio.attach(app)

# Track connected clients and their subscribed categories
connected_clients = {}  # sid -> {'categories': set()}

@sio.event
async def connect(sid, environ, auth):
    connected_clients[sid] = {'categories': set(), 'connected_at': datetime.datetime.utcnow().isoformat()}
    total = len(connected_clients)
    print(f'[+] {sid} connected. Total: {total}')
    await sio.emit('welcome', {'sid': sid, 'total_clients': total}, to=sid)

@sio.event
async def disconnect(sid):
    connected_clients.pop(sid, None)
    print(f'[-] {sid} disconnected. Total: {len(connected_clients)}')

@sio.event
async def subscribe(sid, data):
    """Client subscribes to one or more notification categories."""
    categories = data.get('categories', [])
    if not isinstance(categories, list):
        categories = [categories]
    for cat in categories:
        cat = str(cat).lower().strip()
        if cat:
            await sio.enter_room(sid, f'cat_{cat}')
            connected_clients[sid]['categories'].add(cat)
    await sio.emit('subscribed', {'categories': list(connected_clients[sid]['categories'])}, to=sid)
    print(f'[SUB] {sid} subscribed to: {categories}')

@sio.event
async def unsubscribe(sid, data):
    """Client unsubscribes from a category."""
    category = data.get('category', '').lower().strip()
    if category and category in connected_clients.get(sid, {}).get('categories', set()):
        await sio.leave_room(sid, f'cat_{category}')
        connected_clients[sid]['categories'].discard(category)

@sio.event
async def push_alert(sid, data):
    """Any client (e.g. a monitoring script) can push an alert to a category room."""
    category = str(data.get('category', 'general')).lower().strip()
    message = data.get('message', '')
    severity = data.get('severity', 'info')  # info | warning | error
    if not message:
        return

    alert_payload = {
        'category': category,
        'message': message,
        'severity': severity,
        'timestamp': datetime.datetime.utcnow().isoformat(),
        'from_sid': sid,
    }
    # Broadcast to everyone subscribed to this category (including sender)
    await sio.emit('alert', alert_payload, room=f'cat_{category}')
    print(f'[ALERT] {severity.upper()} in {category}: {message}')

async def status_handler(request):
    """HTTP endpoint: returns JSON status of connected clients."""
    clients_info = [
        {'sid': sid, 'categories': list(info['categories'])}
        for sid, info in connected_clients.items()
    ]
    return web.json_response({'clients': clients_info, 'total': len(connected_clients)})

app.router.add_get('/status', status_handler)

if __name__ == '__main__':
    web.run_app(app, host='0.0.0.0', port=8080)

A monitoring script that sends alerts to the server:

# alert_sender.py
import asyncio
import socketio

async def send_alert(category, message, severity='info'):
    client = socketio.AsyncClient()
    await client.connect('http://localhost:8080')
    await client.emit('push_alert', {
        'category': category,
        'message': message,
        'severity': severity,
    })
    await asyncio.sleep(0.5)  # wait for the emit to flush
    await client.disconnect()

async def main():
    await send_alert('deploy', 'v2.3.1 deployed to production', severity='info')
    await send_alert('database', 'Connection pool at 95% capacity', severity='warning')
    await send_alert('payments', 'Stripe webhook failed 3 times', severity='error')

if __name__ == '__main__':
    asyncio.run(main())

Server output:

[+] abc123 connected. Total: 1   # browser tab
[+] xyz456 connected. Total: 2   # alert_sender.py
[ALERT] INFO in deploy: v2.3.1 deployed to production
[ALERT] WARNING in database: Connection pool at 95% capacity
[ALERT] ERROR in payments: Stripe webhook failed 3 times
[-] xyz456 disconnected. Total: 1

The server separates two roles cleanly: browser clients subscribe to categories and receive alert events; monitoring scripts connect briefly, push alerts via push_alert, and disconnect. The /status HTTP endpoint gives you a JSON view of who is connected and what they are subscribed to — useful for an ops dashboard sidebar showing active watchers. To extend this, add a Redis adapter (pip install python-socketio[asyncio_client] aioredis) so the server can scale across multiple worker processes while still broadcasting to all connected clients.

API Alice at a large switchboard broadcasting events to multiple subscribers
push_alert() goes in. Everyone subscribed gets it out. No polling. No lag.

Frequently Asked Questions

What is the difference between WebSocket and socket.io?

WebSocket is a browser protocol that establishes a persistent two-way connection. Socket.IO is a library that uses WebSocket as its transport but adds a structured protocol layer on top: named events, automatic reconnection, rooms, namespaces, and fallback to HTTP long-polling when WebSocket is blocked. If you use raw WebSocket, you must implement all of that yourself. Socket.IO gives you those features out of the box, which is why most real-time apps reach for it rather than raw WebSocket. The trade-off is that the socket.io client and server must speak the same protocol version — you cannot connect a browser’s native WebSocket object to a socket.io server without the socket.io client library.

How does socket.io scale across multiple server processes?

By default, python-socketio keeps room and connection state in memory, which means a client connected to worker A cannot receive an event emitted by worker B. To scale, you need a message queue adapter. The most common option is Redis: pip install python-socketio[asyncio_client] aioredis, then configure the server with socketio.AsyncServer(client_manager=socketio.AsyncAioPikaManager('redis://localhost')). With a Redis adapter, all workers share room state and broadcasts fan out across every process. This is the standard production pattern for running socket.io behind a load balancer.

How do I detect when a client disconnects unexpectedly?

The disconnect event fires for all disconnections — graceful logouts, network drops, browser tab closes, and timeouts. You do not need to distinguish between them to clean up state. Socket.IO uses a heartbeat mechanism (configurable via ping_interval and ping_timeout on AsyncServer) to detect stale connections: if a client does not respond to a ping within ping_timeout seconds, the server treats it as disconnected and fires the disconnect event. The default is a 5-second ping interval with a 20-second timeout. For gaming or trading apps that need faster detection, lower both values — but watch out for false positives on slow mobile networks.

How do I authenticate socket.io connections?

The standard pattern is to pass a token in the auth option of the client’s io()` call: `io('http://localhost:8080', { auth: { token: 'jwt-here' } }). The Python server receives this in the connect handler’s auth parameter. Validate the token there and return False to reject the connection. Connections rejected with False receive a connection error on the client side. Never trust event handlers for authentication — if a connection gets through connect, treat it as authenticated for the session. Token validation in every event handler is too slow and too fragile.

How do event acknowledgements work?

Socket.IO supports acknowledgements: the sender can pass a callback that the receiver calls after processing the event. In the Python async API, you request an acknowledgement using await sio.call('event_name', data, to=sid) (server side) or const result = await socket.emitWithAck('event_name', data) (browser side). The call blocks until the other side responds, giving you a request-response pattern over the event channel. This is useful for confirming that a command was received and acted on — for example, confirming that a room-join completed before the client starts sending messages into that room. Acknowledgements time out after a configurable period if the other side does not respond.

Conclusion

Python’s python-socketio library gives you a complete real-time event layer with almost no boilerplate. The core concepts covered in this article — the AsyncServer setup with aiohttp, the @sio.event decorator for named events, sio.enter_room() and sio.emit(room=...) for group broadcasts, namespaces for logical separation, and the async Python client for server-to-server communication — cover the full surface area of most real-time web applications.

The notification server project shows the most common production pattern: a Python backend that accepts push events from internal services and relays them to subscribed browser clients in real time. Extend it by adding Redis for multi-process scaling, JWT validation in the connect handler, and a React or Vue frontend that renders incoming alert payloads as styled cards. You have everything you need to build live dashboards, multiplayer features, or any application where waiting for a poll is a second too long.

For the full API reference — including acknowledgements, binary data transfer, and the ASGI adapter for use with FastAPI or Starlette — see the official python-socketio documentation and the Socket.IO protocol docs.

How To Use Python rich-click for Styled CLI Help Pages

How To Use Python rich-click for Styled CLI Help Pages

Intermediate

You build a Click CLI tool, run it with --help, and get a wall of monochrome text. Option names blur together, required flags look identical to optional ones, and your carefully written docstring sits there as a grey blob. Every professional CLI tool — from pip to ruff to aws — has structured, colored, scannable help pages. Yours can too, and it takes one line of code to get started.

rich-click is a thin wrapper around Click that replaces its built-in help formatter with one powered by the rich library. You keep everything you already know about Click — decorators, commands, groups, options — and your help output gains panels, colored option tables, and structured sections without touching your business logic. Install it with pip install rich-click.

This article covers the full rich-click toolkit: drop-in replacement usage, option grouping with panels, command grouping in groups, theming and color configuration, epilog formatting with Markdown, and a real-life project that ties everything together. By the end, you will be able to ship CLI tools whose help pages look like they came from a professional software team.

rich-click: Quick Example

The fastest way to see what rich-click does is to swap one import. Here is a standard Click CLI and its rich-click equivalent side by side.

# quick_richclick.py
import rich_click as click

@click.command()
@click.option("--name", required=True, help="Your name.")
@click.option("--count", default=1, show_default=True, help="How many times to greet.")
@click.option("--verbose", is_flag=True, help="Enable verbose output.")
def greet(name, count, verbose):
    """Greet a user by name.

    Prints a greeting the specified number of times.
    Use --verbose to see extra detail.
    """
    for i in range(count):
        if verbose:
            click.echo(f"[{i+1}/{count}] Hello, {name}!")
        else:
            click.echo(f"Hello, {name}!")

if __name__ == "__main__":
    greet()

Output of python quick_richclick.py --help:

 Usage: quick_richclick.py [OPTIONS]

 Greet a user by name.

 Prints a greeting the specified number of times. Use --verbose to see extra
 detail.

+----------+------------------------------------------------------------------+
| Option   | Description                                                      |
+----------+------------------------------------------------------------------+
| --name   | Your name. [required]                                            |
| --count  | How many times to greet. [default: 1]                            |
| --verbose| Enable verbose output.                                           |
| --help   | Show this message and exit.                                      |
+----------+------------------------------------------------------------------+

The only change is import rich_click as click instead of import click. Your decorators, options, and function logic are completely unchanged. rich-click intercepts Click’s help rendering at the formatter level and replaces it with a rich-powered layout — the rest of your CLI works exactly as before.

The real power comes when you start grouping options and commands. Read on to learn how to build structured, professional help pages that guide users through complex CLIs.

What Is rich-click and Why Use It?

rich-click is an open-source library by Dominic Davis-Foster that wraps Click’s help formatter with the rich rendering engine. It was built to solve a specific, common problem: Click’s default help output is functionally correct but visually flat. When a CLI has 20 options across 5 subcommands, a flat alphabetical list becomes unreadable. Users scroll past relevant flags, miss required arguments, and give up.

The library takes a non-destructive approach. It does not fork Click or change its internals — it subclasses Click’s formatter and renderer, which means every Click feature (callback validation, command groups, option types, context settings) continues to work. You can also mix standard Click and rich-click in the same project during a gradual migration.

FeatureClick (default)rich-click
Help output stylePlain monochrome textColored panels and tables
Option groupingNot availableOPTION_GROUPS config
Command groupingNot availableCOMMAND_GROUPS config
Epilog formattingPlain text onlyRich Markdown
Color themesNot configurableFull color customization
Migration effortN/AOne import change
Runtime behaviorUnchangedUnchanged

Install both rich-click and rich (it is pulled in as a dependency automatically) with pip:

# In your terminal
pip install rich-click

Python 3.7+ and Click 7 or 8 are supported. No other setup is required.

Pyro Pete holds up a glowing terminal window with beautifully structured rich-click help output
pip install rich-click — the one upgrade every CLI deserves.

Grouping Options into Panels

The most impactful feature of rich-click is option grouping. Instead of one flat list of every flag, you can organize options into labeled panels that match how users think about the tool. A CLI with database options, output options, and auth options should look like three distinct sections, not a single alphabetical dump.

Option groups are configured via click.rich_click.OPTION_GROUPS, a dictionary that maps command names to a list of group definitions. Each group has a name and a list of options (the flag names as strings).

# option_groups.py
import rich_click as click

click.rich_click.OPTION_GROUPS = {
    "deploy": [
        {
            "name": "Connection",
            "options": ["--host", "--port", "--timeout"],
        },
        {
            "name": "Authentication",
            "options": ["--token", "--username", "--password"],
        },
        {
            "name": "Output",
            "options": ["--format", "--verbose", "--quiet"],
        },
    ]
}

@click.command()
@click.option("--host", default="localhost", show_default=True, help="Server hostname.")
@click.option("--port", default=8080, show_default=True, help="Server port.")
@click.option("--timeout", default=30, show_default=True, help="Connection timeout in seconds.")
@click.option("--token", help="API authentication token.")
@click.option("--username", help="Username for basic auth.")
@click.option("--password", help="Password for basic auth.")
@click.option("--format", type=click.Choice(["json", "table", "csv"]), default="table", show_default=True, help="Output format.")
@click.option("--verbose", is_flag=True, help="Enable verbose logging.")
@click.option("--quiet", is_flag=True, help="Suppress all output except errors.")
def deploy(host, port, timeout, token, username, password, format, verbose, quiet):
    """Deploy the application to a remote server."""
    click.echo(f"Deploying to {host}:{port} as {'token auth' if token else username}")

if __name__ == "__main__":
    deploy()

Output of python option_groups.py --help:

 Usage: deploy [OPTIONS]

 Deploy the application to a remote server.

 Connection
+-------------+-----------------------------------------------+
| --host      | Server hostname. [default: localhost]         |
| --port      | Server port. [default: 8080]                  |
| --timeout   | Connection timeout in seconds. [default: 30]  |
+-------------+-----------------------------------------------+

 Authentication
+-------------+-----------------------------------------------+
| --token     | API authentication token.                     |
| --username  | Username for basic auth.                      |
| --password  | Password for basic auth.                      |
+-------------+-----------------------------------------------+

 Output
+-------------+-----------------------------------------------+
| --format    | Output format. [default: table]               |
| --verbose   | Enable verbose logging.                       |
| --quiet     | Suppress all output except errors.            |
+-------------+-----------------------------------------------+

The key is that the keys in OPTION_GROUPS match the Click command name (the function name by default, or the name= argument in the decorator). Options not listed in any group fall through to a default “Other options” panel at the bottom, so you can migrate groups incrementally without losing any flags from the help page.

Grouping Subcommands in a Group CLI

If your CLI uses a command group (the pattern where mytool db migrate, mytool db rollback, and mytool server start are subcommands), rich-click can group those subcommands into labeled sections too. This uses click.rich_click.COMMAND_GROUPS with the same dictionary pattern.

# command_groups.py
import rich_click as click

click.rich_click.COMMAND_GROUPS = {
    "cli": [
        {
            "name": "Database",
            "commands": ["db-init", "db-migrate", "db-reset"],
        },
        {
            "name": "Server",
            "commands": ["server-start", "server-stop", "server-status"],
        },
    ]
}

@click.group()
def cli():
    """My Application CLI."""
    pass

@cli.command("db-init")
def db_init():
    """Initialize the database schema."""
    click.echo("Initializing DB...")

@cli.command("db-migrate")
@click.option("--steps", default=1, show_default=True, help="Number of migrations to apply.")
def db_migrate(steps):
    """Apply pending database migrations."""
    click.echo(f"Applying {steps} migration(s)...")

@cli.command("db-reset")
@click.option("--force", is_flag=True, help="Skip confirmation prompt.")
def db_reset(force):
    """Drop and recreate the database."""
    if not force:
        click.confirm("This will delete all data. Continue?", abort=True)
    click.echo("Resetting database...")

@cli.command("server-start")
@click.option("--port", default=8000, show_default=True, help="Port to listen on.")
def server_start(port):
    """Start the application server."""
    click.echo(f"Server running on port {port}...")

@cli.command("server-stop")
def server_stop():
    """Gracefully stop the server."""
    click.echo("Stopping server...")

@cli.command("server-status")
def server_status():
    """Show the server's current status."""
    click.echo("Server status: running")

if __name__ == "__main__":
    cli()

Output of python command_groups.py --help:

 Usage: cli [OPTIONS] COMMAND [ARGS]...

 My Application CLI.

 Database
+-------------+-------------------------------------------+
| db-init     | Initialize the database schema.           |
| db-migrate  | Apply pending database migrations.        |
| db-reset    | Drop and recreate the database.           |
+-------------+-------------------------------------------+

 Server
+-------------+-------------------------------------------+
| server-start | Start the application server.            |
| server-stop  | Gracefully stop the server.              |
| server-status| Show the server's current status.        |
+-------------+-------------------------------------------+

The command group name in COMMAND_GROUPS must match the Click group name. Subcommands listed in no group appear in a default “Commands” section — useful for utility commands like version or help that don’t belong in a domain group.

Sudo Sam at a whiteboard showing CLI help panels organized by section
Group your commands. Your users are not archaeologists.

Theming and Color Configuration

Beyond layout, rich-click exposes a configuration object that controls every color and style element in the help output. You set these as module-level assignments on click.rich_click before any commands are defined. The settings apply globally to all commands in that script.

# theming.py
import rich_click as click

# -- Theming configuration --
click.rich_click.STYLE_OPTION = "bold cyan"
click.rich_click.STYLE_SWITCH = "bold green"
click.rich_click.STYLE_METAVAR = "dim yellow"
click.rich_click.STYLE_HELPTEXT = "white"
click.rich_click.STYLE_HELPTEXT_FIRST_LINE = "bold white"
click.rich_click.STYLE_ERRORS_SUGGESTION = "italic dim"
click.rich_click.STYLE_OPTIONS_TABLE_BOX = "ROUNDED"
click.rich_click.MAX_WIDTH = 100
click.rich_click.SHOW_ARGUMENTS = True
click.rich_click.GROUP_ARGUMENTS_OPTIONS = True

@click.command()
@click.argument("filename")
@click.option("--output", "-o", help="Output file path.")
@click.option("--format", type=click.Choice(["json", "csv", "yaml"]), default="json", show_default=True, help="Output format.")
@click.option("--dry-run", is_flag=True, help="Preview changes without writing.")
def convert(filename, output, format, dry_run):
    """Convert FILENAME to another format.

    Reads the input file and writes a converted version.
    Supports JSON, CSV, and YAML output.
    """
    click.echo(f"Converting {filename} to {format}...")

if __name__ == "__main__":
    convert()

Output:

 Usage: theming.py [OPTIONS] FILENAME

 Convert FILENAME to another format.
 Reads the input file and writes a converted version. Supports JSON, CSV, and YAML output.

+------------+-----------------------------------------+
| Option     | Description                             |
+------------+-----------------------------------------+
| FILENAME   | (Required argument)                     |
| --output   | Output file path.                       |
| --format   | Output format. [default: json]          |
| --dry-run  | Preview changes without writing.        |
| --help     | Show this message and exit.             |
+------------+-----------------------------------------+

The STYLE_* settings accept any style string that rich understands: color names ("cyan", "red"), hex colors ("#ff6b6b"), and style modifiers ("bold", "dim", "italic"). The MAX_WIDTH setting caps the help panel width — useful if your terminal is very wide and you want the output to stay readable. SHOW_ARGUMENTS adds positional arguments to the options table alongside flags, so users see everything in one place.

Epilog Formatting with Markdown

Click supports an epilog parameter on commands — text that appears after the options table. By default, Click strips all formatting from it. With rich-click you can write the epilog as Markdown and have it rendered as rich text, complete with bold, links, and code spans.

# epilog_demo.py
import rich_click as click

click.rich_click.USE_MARKDOWN = True
click.rich_click.USE_MARKDOWN_EMOJI = False  # optional: disable emoji parsing

EPILOG = """
**Examples:**

Run a dry-run check:

    mytool check --dry-run input.csv

Convert with verbose output:

    mytool check --verbose --output result.json input.csv

See the full documentation at https://docs.example.com/mytool
"""

@click.command(epilog=EPILOG)
@click.argument("input_file")
@click.option("--output", "-o", default="output.json", show_default=True, help="Output file.")
@click.option("--dry-run", is_flag=True, help="Preview without writing.")
@click.option("--verbose", "-v", is_flag=True, help="Show detailed progress.")
def check(input_file, output, dry_run, verbose):
    """Validate and convert INPUT_FILE.

    Reads a CSV or JSON input file, validates each row,
    and writes the cleaned output to the specified path.
    """
    click.echo(f"Checking {input_file}...")

if __name__ == "__main__":
    check()

Output (epilog section):

 Examples:

 Run a dry-run check:

   mytool check --dry-run input.csv

 Convert with verbose output:

   mytool check --verbose --output result.json input.csv

 See the full documentation at https://docs.example.com/mytool

Setting USE_MARKDOWN = True applies globally — every command’s help text and epilog will be parsed as Markdown. Code blocks in the epilog (fenced with backticks or indented by four spaces) render with syntax highlighting. This is ideal for “Examples” sections, which are one of the most read parts of any CLI help page.

Cache Katie speed-typing with a beautiful formatted terminal help page appearing on screen
USE_MARKDOWN = True. The epilog your users will actually read.

Real-Life Example: A File Conversion CLI

This project ties together option grouping, command grouping, theming, and Markdown epilog into a realistic multi-command CLI tool. It is a data file converter with two subcommands: convert and validate.

# file_converter_cli.py
import rich_click as click
import json
import csv
import os

# --- rich-click configuration ---
click.rich_click.USE_MARKDOWN = True
click.rich_click.MAX_WIDTH = 90
click.rich_click.SHOW_ARGUMENTS = True
click.rich_click.STYLE_OPTION = "bold cyan"
click.rich_click.STYLE_SWITCH = "bold green"

click.rich_click.OPTION_GROUPS = {
    "convert": [
        {
            "name": "Input / Output",
            "options": ["--output", "--format"],
        },
        {
            "name": "Behaviour",
            "options": ["--dry-run", "--verbose", "--skip-errors"],
        },
    ],
    "validate": [
        {
            "name": "Validation Rules",
            "options": ["--required", "--max-rows"],
        },
        {
            "name": "Reporting",
            "options": ["--report", "--verbose"],
        },
    ],
}

click.rich_click.COMMAND_GROUPS = {
    "fileconv": [
        {
            "name": "Data Commands",
            "commands": ["convert", "validate"],
        },
    ]
}

CONVERT_EPILOG = """
**Examples:**

Convert a JSON file to CSV:

    fileconv convert data.json --format csv --output data.csv

Dry-run to preview without writing:

    fileconv convert data.json --dry-run --verbose
"""

VALIDATE_EPILOG = """
**Examples:**

Validate a file with required columns:

    fileconv validate data.csv --required name --required email

Save a validation report:

    fileconv validate data.csv --report report.txt
"""

@click.group()
def fileconv():
    """File Conversion and Validation CLI.

    Convert between JSON and CSV formats and validate file structure.
    """
    pass

@fileconv.command(epilog=CONVERT_EPILOG)
@click.argument("input_file")
@click.option("--output", "-o", help="Output file path. Defaults to INPUT_FILE with new extension.")
@click.option("--format", "fmt", type=click.Choice(["json", "csv"]), required=True, help="Target format.")
@click.option("--dry-run", is_flag=True, help="Preview conversion without writing output.")
@click.option("--verbose", "-v", is_flag=True, help="Show each row as it is processed.")
@click.option("--skip-errors", is_flag=True, help="Continue on row errors instead of aborting.")
def convert(input_file, output, fmt, dry_run, verbose, skip_errors):
    """Convert INPUT_FILE to another format.

    Reads JSON or CSV and writes the converted file.
    Detects the input format from the file extension.
    """
    if not os.path.exists(input_file):
        raise click.BadParameter(f"File not found: {input_file}", param_hint="INPUT_FILE")

    base, _ = os.path.splitext(input_file)
    output = output or f"{base}.{fmt}"

    click.echo(f"Reading: {input_file}")

    try:
        if input_file.endswith(".json"):
            with open(input_file) as f:
                rows = json.load(f)
            if not isinstance(rows, list):
                raise click.ClickException("JSON input must be a list of objects.")
        elif input_file.endswith(".csv"):
            with open(input_file, newline="") as f:
                rows = list(csv.DictReader(f))
        else:
            raise click.ClickException("Unsupported input format. Use .json or .csv")
    except (json.JSONDecodeError, csv.Error) as e:
        raise click.ClickException(f"Failed to parse input: {e}")

    if verbose:
        for i, row in enumerate(rows, 1):
            click.echo(f"  Row {i}: {row}")

    if dry_run:
        click.echo(f"[dry-run] Would write {len(rows)} rows to: {output}")
        return

    if fmt == "csv":
        if not rows:
            raise click.ClickException("No rows to write.")
        with open(output, "w", newline="") as f:
            writer = csv.DictWriter(f, fieldnames=rows[0].keys())
            writer.writeheader()
            writer.writerows(rows)
    else:
        with open(output, "w") as f:
            json.dump(rows, f, indent=2)

    click.echo(f"Written {len(rows)} rows to: {output}")

@fileconv.command(epilog=VALIDATE_EPILOG)
@click.argument("input_file")
@click.option("--required", multiple=True, help="Column that must be present and non-empty. Repeatable.")
@click.option("--max-rows", type=int, help="Fail if row count exceeds this limit.")
@click.option("--report", help="Write validation report to this file path.")
@click.option("--verbose", "-v", is_flag=True, help="List every issue found.")
def validate(input_file, required, max_rows, report, verbose):
    """Validate INPUT_FILE structure and contents.

    Checks for required columns, row count limits, and empty values.
    Exits with code 1 if any issues are found.
    """
    if not os.path.exists(input_file):
        raise click.BadParameter(f"File not found: {input_file}", param_hint="INPUT_FILE")

    with open(input_file, newline="") as f:
        rows = list(csv.DictReader(f))

    issues = []

    if max_rows and len(rows) > max_rows:
        issues.append(f"Row count {len(rows)} exceeds limit of {max_rows}.")

    for col in required:
        missing = [i+1 for i, r in enumerate(rows) if not r.get(col, "").strip()]
        if missing:
            issues.append(f"Column '{col}' is empty on rows: {missing[:5]}{'...' if len(missing) > 5 else ''}")

    summary = f"Validated {len(rows)} rows. Issues: {len(issues)}"
    click.echo(summary)

    if verbose:
        for issue in issues:
            click.echo(f"  - {issue}")

    if report:
        with open(report, "w") as f:
            f.write(summary + "\n")
            for issue in issues:
                f.write(f"- {issue}\n")
        click.echo(f"Report written to: {report}")

    if issues:
        raise click.ClickException(f"{len(issues)} validation issue(s) found.")

if __name__ == "__main__":
    fileconv()

Output of python file_converter_cli.py --help:

 Usage: fileconv [OPTIONS] COMMAND [ARGS]...

 File Conversion and Validation CLI.
 Convert between JSON and CSV formats and validate file structure.

 Data Commands
+-----------+------------------------------------------------+
| convert   | Convert INPUT_FILE to another format.          |
| validate  | Validate INPUT_FILE structure and contents.    |
+-----------+------------------------------------------------+

Output of python file_converter_cli.py convert --help:

 Usage: fileconv convert [OPTIONS] INPUT_FILE

 Convert INPUT_FILE to another format. Reads JSON or CSV and writes the converted file.

 Input / Output
+-----------+-----------------------------------------------------------+
| INPUT_FILE| (Required argument)                                       |
| --output  | Output file path. Defaults to INPUT_FILE with new ext.    |
| --format  | Target format. [required]                                 |
+-----------+-----------------------------------------------------------+

 Behaviour
+---------------+---------------------------------------------------+
| --dry-run     | Preview conversion without writing output.        |
| --verbose     | Show each row as it is processed.                 |
| --skip-errors | Continue on row errors instead of aborting.       |
+---------------+---------------------------------------------------+

 Examples:

 Convert a JSON file to CSV:

   fileconv convert data.json --format csv --output data.csv

This project shows the full rich-click pattern in a real tool. Notice how the two subcommands each have their own OPTION_GROUPS configuration — groups only appear on the command they are defined for, not globally. The epilog examples appear after the options table and use Markdown code blocks. You can extend this tool by adding a third subcommand (merge, split, or schema) and adding its group entry to OPTION_GROUPS.

Debug Dee comparing plain monochrome help text vs beautiful rich-click formatted panels
Before and after rich-click. One import. Night and day.

Frequently Asked Questions

Does rich-click change how my CLI actually runs?

No. rich-click only changes help page rendering. The command routing, argument parsing, option validation, and all callback logic are handled by Click as usual. You can verify this by calling your commands with real arguments — rich-click is not in the execution path at all, only in the --help path. If you encounter a behavioral difference, it is almost certainly a Click version incompatibility, not a rich-click issue.

Can I use rich-click on just some commands in a large project?

Yes. You can import rich_click as click in individual modules while keeping standard click imports elsewhere. Since rich-click is a superset of Click, the decorators are compatible. The configuration dict (OPTION_GROUPS, COMMAND_GROUPS) applies only to commands in the module that sets it — other modules using plain click are unaffected. This makes incremental migration safe in large codebases.

Will the colored output cause problems in CI pipelines or when piped to a file?

rich (and by extension rich-click) automatically detects whether the output stream is a terminal using is_tty checks. When output is redirected to a file or run in a CI environment without a TTY, all ANSI color codes are stripped automatically. The help text is still formatted (tables, panels) but without colors. You can force color off with NO_COLOR=1 environment variable, or force it on with FORCE_COLOR=1 for CI systems that support color (like GitHub Actions with GITHUB_ACTIONS=true).

What happens to options I forget to include in OPTION_GROUPS?

Any option not listed in an explicit group is collected into a default “Options” panel that appears at the bottom of the help output. This means you never accidentally hide an option by forgetting to group it — the worst case is that it appears ungrouped. The --help flag itself always appears in this fallback section unless you explicitly group it. This behavior makes it safe to add new options incrementally without updating OPTION_GROUPS immediately.

Should I use rich-click or Typer for a new project?

Use rich-click if you already have a Click codebase you want to improve, or if you prefer Click’s explicit decorator-based style. Use Typer if you are starting from scratch and want a fully type-annotated API where Click’s decorators are generated from function signatures automatically. Both use Click under the hood and both produce rich-formatted output — Typer uses rich natively, while rich-click layers it on top. Typer is a bigger dependency and a bigger migration; rich-click is a drop-in upgrade.

Conclusion

Python rich-click transforms Click help pages from a functional afterthought into a polished, navigable interface. The core tools covered in this article — the drop-in import swap, OPTION_GROUPS for panel-based option organization, COMMAND_GROUPS for structured command menus, STYLE_* settings for theming, and USE_MARKDOWN for rich epilog formatting — cover the full surface area most CLI tools need. All of this works without touching a single line of command logic.

The real-life project above shows the pattern at scale. Try extending it with a merge subcommand that combines two CSV files, or a schema command that infers a JSON Schema from the input data. Each new command gets its own OPTION_GROUPS entry and its own structured, readable help page.

For the full configuration reference — every STYLE_* variable, the USE_RICH_MARKUP flag, and the HEADER_TEXT / FOOTER_TEXT banner options — see the official rich-click documentation on GitHub and the Click documentation.

How To Use Python questionary for Interactive CLI Prompts

How To Use Python questionary for Interactive CLI Prompts

Intermediate

You have a command-line tool that accepts a dozen options, and right now users pass them all as flags: --env production --region us-east-1 --deploy true. Nobody remembers the exact flag names, typos break things silently, and every new user has to read the docs just to run a basic command. The tool works fine — but actually using it feels like defusing a bomb.

Python’s questionary library solves this by giving you a set of beautifully rendered interactive prompts: arrow-key menus, checkboxes, masked password fields, and confirmation dialogs — all in the terminal, with zero dependencies on a browser or GUI framework. It wraps Python Prompt Toolkit under the hood and handles all the TTY complexity for you. Installation is one command: pip install questionary.

In this article we’ll walk through every major prompt type in questionary — text input, single-select menus, multi-select checkboxes, confirmation dialogs, and password fields. We’ll show how to chain prompts together into a multi-step workflow, apply validation, and then build a real-life project setup wizard that you can adapt for your own CLI tools. By the end you’ll be able to replace any flag-heavy CLI with one that guides users naturally through each choice.

questionary in Python: Quick Example

Here is a self-contained example that shows the three most common prompt types working together. Run it in a terminal (not a Jupyter notebook — questionary requires an interactive TTY).

# quick_example.py
import questionary

name = questionary.text("What is your name?").ask()

env = questionary.select(
    "Which environment?",
    choices=["development", "staging", "production"],
).ask()

confirmed = questionary.confirm(
    f"Deploy {name}'s changes to {env}?", default=False
).ask()

if confirmed:
    print(f"Deploying to {env}...")
else:
    print("Cancelled.")

Output (interactive session in terminal):

? What is your name? Alice
? Which environment? (Use arrow keys)
 > development
   staging
   production
? Deploy Alice's changes to development? (y/N) y
Deploying to development...

The .ask() method blocks until the user answers and returns the value — a string for text/select, a bool for confirm. All three calls are synchronous, which keeps the code easy to read and reason about. For a quick async variant, questionary also provides .ask_async(), but we will stick with the synchronous API throughout this article.

What Is questionary and When Should You Use It?

questionary is a Python library for building interactive terminal menus. It builds on top of prompt_toolkit and provides a clean, high-level API for the most common interactive patterns you would need in a CLI tool. Unlike argparse or click, which require users to know exactly what flags to pass, questionary prompts guide users step by step — making your tools more approachable for teammates who don’t live in the terminal all day.

questionary shines for: onboarding wizards, scaffolding scripts, deployment confirmations, config generators, and any workflow where the user needs to make several choices in sequence. It is not designed for non-interactive scripts (cron jobs, CI pipelines) — in those contexts you would pass values directly as arguments and skip the prompts altogether. You can detect that scenario with sys.stdin.isatty() and fall back to flag-based input.

Prompt TypeMethodReturnsBest For
Free textquestionary.text()strNames, paths, free-form input
Single choicequestionary.select()strPick one from a fixed list
Multiple choicequestionary.checkbox()list[str]Pick any combination
Yes/Noquestionary.confirm()boolDestructive action gating
Passwordquestionary.password()strSecrets, API keys
Pathquestionary.path()strFile/directory selection with tab-complete
Autocompletequestionary.autocomplete()strLong lists with fuzzy filtering

Install questionary with pip before running any of the examples below:

# Install questionary
pip install questionary
Successfully installed questionary-2.0.1 prompt-toolkit-3.0.47 wcwidth-0.2.13
Senior developer pointing at a glowing terminal with interactive CLI menu
argparse: read the docs. questionary: just press the arrow key.

Text Input with Validation

The text() prompt accepts free-form keyboard input. On its own it accepts any string, including an empty one — which is rarely what you want. The validate parameter lets you enforce rules inline. Pass a function that returns True on valid input or an error message string when it fails; questionary will keep re-prompting until the rule passes.

# text_prompt.py
import questionary

def must_not_be_empty(value):
    if not value.strip():
        return "Project name cannot be blank."
    if len(value) > 40:
        return "Keep the project name under 40 characters."
    return True

project_name = questionary.text(
    "Enter project name:",
    validate=must_not_be_empty,
).ask()

print(f"Project: {project_name}")

Output:

? Enter project name:
>> Please enter a valid value  (Project name cannot be blank.)
? Enter project name: my-new-api
Project: my-new-api

The validation function is called on every keystroke as the user types, so feedback is immediate rather than appearing only after they press Enter. If the function returns the string True (not just a truthy value), questionary treats that as valid — so make sure to return the boolean True, not a non-empty string when the value is good. A non-empty string is always treated as an error message.

Single-Choice Menus with select()

The select() prompt renders a scrollable arrow-key menu where the user picks exactly one option. The choices parameter accepts either a plain list of strings or a list of Choice objects if you want to display one label but return a different underlying value.

# select_prompt.py
import questionary
from questionary import Choice

region = questionary.select(
    "Choose a deployment region:",
    choices=[
        Choice("US East (N. Virginia)", value="us-east-1"),
        Choice("EU West (Ireland)",      value="eu-west-1"),
        Choice("Asia Pacific (Sydney)",  value="ap-southeast-2"),
    ],
    use_shortcuts=True,
).ask()

print(f"Selected region slug: {region}")

Output:

? Choose a deployment region: (Use arrow keys)
 > US East (N. Virginia)
   EU West (Ireland)
   Asia Pacific (Sydney)
Selected region slug: us-east-1

Setting use_shortcuts=True assigns number shortcuts (1, 2, 3…) to each choice so power users can skip the arrow key navigation entirely. The value returned by .ask() is always the value field of the selected Choice, not the display label — which means your downstream code works with slugs like us-east-1 rather than human-readable strings that might contain spaces or special characters.

Developer examining an interactive terminal selection menu
Display human labels. Return machine values. Users happy, parser happy.

Multi-Choice Menus with checkbox()

When a user needs to select multiple items from a list, checkbox() is the right tool. Users navigate with arrow keys, toggle individual items with the spacebar, and confirm their full selection by pressing Enter. The return value is always a list — even if only one item is selected.

# checkbox_prompt.py
import questionary

features = questionary.checkbox(
    "Which features should we enable?",
    choices=[
        "Authentication (JWT)",
        "Rate limiting",
        "API documentation (Swagger)",
        "Email notifications",
        "Audit logging",
    ],
).ask()

if not features:
    print("No features selected -- deploying a blank slate.")
else:
    print(f"Enabling: {', '.join(features)}")

Output:

? Which features should we enable? (Use arrow keys, press Space to select, Enter to confirm)
 o Authentication (JWT)
 o Rate limiting
 > o API documentation (Swagger)
 o Email notifications
 o Audit logging
Enabling: Authentication (JWT), API documentation (Swagger)

Always check whether the returned list is empty before iterating over it. Users can press Enter without selecting anything, which gives you an empty list rather than None. If one or more choices should always be pre-selected, pass checked=True inside a Choice object: Choice("Rate limiting", checked=True).

Confirmation Prompts

Before any irreversible action — deleting files, dropping a database, pushing to production — add a confirm() gate. It renders a Y/N prompt and returns a Python bool. The default parameter controls what Enter alone submits: set it to False for destructive operations so that a stray Enter key does not accidentally confirm deletion.

# confirm_prompt.py
import questionary
import sys

target = "production"

proceed = questionary.confirm(
    f"This will wipe all data in {target}. Are you sure?",
    default=False,
).ask()

if not proceed:
    print("Aborted -- no changes made.")
    sys.exit(0)

print(f"Wiping {target}... (not really, this is a demo)")

Output:

? This will wipe all data in production. Are you sure? (y/N) N
Aborted -- no changes made.

The uppercase letter in (y/N) signals the default. With default=False the N is uppercase, meaning Enter without typing anything registers as “No”. Flip it to default=True and the prompt shows (Y/n). This visual cue is standard terminal UX and users familiar with the shell will immediately understand it without reading any instructions.

Password Input

The password() prompt works identically to text() except that every character the user types is echoed as a bullet point (*), keeping secrets off the screen. This is the right choice for API keys, tokens, and passwords in setup wizards — never ask for these via a plain text prompt or a command-line flag (flags end up in shell history).

# password_prompt.py
import questionary

api_key = questionary.password(
    "Enter your API key:",
    validate=lambda val: True if len(val) >= 16 else "API keys are at least 16 characters."
).ask()

# In a real script you would pass this to your client, not print it.
print(f"Key accepted (length {len(api_key)})")

Output:

? Enter your API key: ****************
Key accepted (length 32)

The value returned by .ask() is the raw string the user typed — questionary does not hash it or store it anywhere. Handle it the same way you would any secret: pass it directly to your client library, never log it, and never embed it in error messages.

Energetic developer guarding a vault representing secure password input
Flags go in shell history. Passwords go in questionary.

Chaining Prompts into a Multi-Step Workflow

questionary does not have a built-in form or wizard abstraction — but you do not need one. Python’s own control flow handles branching naturally. Call each .ask() in sequence, use regular if statements to branch based on earlier answers, and collect results in a dict. This pattern is readable, testable, and easy to extend.

# chained_prompts.py
import questionary

answers = {}

answers["name"] = questionary.text("Project name:").ask()

answers["type"] = questionary.select(
    "Project type:",
    choices=["API", "CLI tool", "Data pipeline", "Web app"],
).ask()

if answers["type"] == "API":
    answers["auth"] = questionary.select(
        "Authentication method:",
        choices=["JWT", "OAuth2", "API key", "None"],
    ).ask()
else:
    answers["auth"] = None

answers["ci"] = questionary.confirm("Add GitHub Actions CI?", default=True).ask()

print("\n-- Configuration Summary --")
for key, value in answers.items():
    if value is not None:
        print(f"  {key}: {value}")

Output:

? Project name: payment-service
? Project type: API
? Authentication method: JWT
? Add GitHub Actions CI? (Y/n) Y

-- Configuration Summary --
  name: payment-service
  type: API
  auth: JWT
  ci: True

Notice that the “Authentication method” prompt only appears when the user selects “API” — a conditional prompt that would be awkward to model with command-line flags. The answers dict gives you a clean, serialisable snapshot of the entire session that you can write to a config file, pass to a scaffolding function, or log (minus any sensitive fields) for debugging.

Real-Life Example: Python Project Setup Wizard

Here is a complete CLI wizard that gathers project configuration and writes a minimal pyproject.toml to the current directory. It demonstrates validation, conditional branching, checkbox multi-select, and a final confirmation gate — all the patterns from this article working together.

Senior developer at a wizard workbench building a project configuration wizard
Ten prompts. One config file. Zero forgotten flags.
# project_wizard.py
import questionary
from questionary import Choice
import sys

def run_wizard():
    print("=== Python Project Setup Wizard ===\n")

    name = questionary.text(
        "Package name (lowercase, hyphens ok):",
        validate=lambda v: True if v.replace("-", "").isalnum() and v == v.lower()
                           else "Use lowercase letters and hyphens only.",
    ).ask()

    version = questionary.text("Version:", default="0.1.0").ask()

    python_min = questionary.select(
        "Minimum Python version:",
        choices=["3.9", "3.10", "3.11", "3.12"],
        default="3.11",
    ).ask()

    extras = questionary.checkbox(
        "Include optional tooling:",
        choices=[
            Choice("pytest  (testing)",     value="pytest"),
            Choice("ruff    (linting)",     value="ruff"),
            Choice("mypy    (type checks)", value="mypy"),
            Choice("black   (formatting)",  value="black"),
        ],
    ).ask()

    license_type = questionary.select(
        "License:",
        choices=["MIT", "Apache-2.0", "GPL-3.0", "Proprietary"],
    ).ask()

    confirmed = questionary.confirm(
        f"\nWrite pyproject.toml for '{name}'?", default=True
    ).ask()

    if not confirmed:
        print("Cancelled.")
        sys.exit(0)

    dev_deps = "\n".join(
        f'    "{dep}>=0",' for dep in (extras or [])
    )

    toml_content = f"""[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.backends.legacy:build"

[project]
name = "{name}"
version = "{version}"
requires-python = ">={python_min}"
license = {{text = "{license_type}"}}

[project.optional-dependencies]
dev = [
{dev_deps}
]
"""

    with open("pyproject.toml", "w") as f:
        f.write(toml_content)

    print(f"\nWrote pyproject.toml for {name} ({version})")
    if extras:
        print(f"Dev tools included: {', '.join(extras)}")

if __name__ == "__main__":
    run_wizard()

Output:

=== Python Project Setup Wizard ===

? Package name (lowercase, hyphens ok): payment-api
? Version: 0.1.0
? Minimum Python version: 3.11
? Include optional tooling: (Space to select)
 > x pytest  (testing)
   o ruff    (linting)
   x mypy    (type checks)
   o black   (formatting)
? License: MIT
? Write pyproject.toml for 'payment-api'? (Y/n) Y

Wrote pyproject.toml for payment-api (0.1.0)
Dev tools included: pytest, mypy

The wizard collects seven pieces of information but only prompts what is relevant — a real advantage over a generic argument parser. To extend it, add a questionary.path() prompt asking where to write the file, or add a select() for the build backend. The entire wizard state lives in local variables, which makes unit-testing the logic straightforward: you can mock .ask() to return fixed values and assert that the generated TOML matches expectations.

Frequently Asked Questions

Why does .ask() return None sometimes?

.ask() returns None when the user presses Ctrl+C to cancel the prompt. Always check for None before using the returned value, or use .ask()‘s raise_keyboard_interrupt=False default behaviour and handle None explicitly. If you prefer an exception on Ctrl+C, pass raise_keyboard_interrupt=True and wrap the call in a try/except KeyboardInterrupt block.

How do I use questionary in CI or non-interactive environments?

questionary requires an interactive TTY. In a CI pipeline where stdin is not a terminal, calls to .ask() will raise an error or hang. The standard pattern is to detect non-interactive mode with sys.stdin.isatty() before running prompts and fall back to reading values from environment variables or command-line arguments. If you need testable prompts, look at questionary’s unsafe_prompt() or use a test runner that can simulate a TTY.

Can I pre-select a default value for select() and text()?

Yes. For text(), pass default="your-default" and the field will be pre-filled — the user can edit it or press Enter to accept. For select(), pass the display string of the option you want pre-highlighted as the default parameter. For checkbox(), pass checked=True inside individual Choice objects to pre-tick specific items.

Can I customise the colours and style of the prompts?

Yes. questionary exposes a style parameter on every prompt that accepts a questionary.Style object built from a list of CSS-like token/colour pairs. Common tokens are question, answer, pointer, selected, and highlighted. Colours can be named ("cyan", "red") or hex strings ("#ff6600"). This is useful for branding internal tools or distinguishing critical prompts visually.

Does questionary support async/await?

Yes — every prompt method has an async counterpart. Instead of .ask(), call await prompt.ask_async() inside an async function. This integrates cleanly with asyncio-based applications. Under the hood, questionary uses prompt_toolkit‘s async event loop, so you do not need to bridge two separate loops. The API is otherwise identical to the synchronous version.

How does questionary compare to PyInquirer or InquirerPy?

All three libraries share the same conceptual origin (JavaScript’s Inquirer.js). questionary has the most Pythonic API — each prompt type is a standalone function rather than a dict config. InquirerPy is more feature-complete (fuzzy search, editor prompts) but has a steeper API surface. PyInquirer is older and less maintained. For most projects, questionary’s balance of simplicity and capability is the right starting point; switch to InquirerPy if you need fuzzy-search menus or editor integration.

Conclusion

We covered the full questionary toolkit: text() for free-form input with inline validation, select() for arrow-key menus with display/value separation, checkbox() for multi-select lists, confirm() for safe destructive-action gates, and password() for keeping secrets off the screen. We also showed how regular Python if statements are all you need to chain prompts into conditional multi-step wizards, and wrapped everything into a project setup tool that writes a pyproject.toml.

A natural extension of the real-life example is to add a questionary.path() prompt for the output directory, integrate with a Cookiecutter template to scaffold full project structures, or call the wizard as a sub-command of a larger click-based CLI. questionary and Click pair very well — Click handles flag parsing for non-interactive use, questionary handles the interactive path, and you branch between them with sys.stdin.isatty().

Full documentation, changelog, and the list of available token names for custom styling are at the official questionary docs: questionary.readthedocs.io. The source is on GitHub at github.com/tmbo/questionary.

How To Automate Tasks with Python: A Practical Guide

How To Automate Tasks with Python: A Practical Guide

Last Updated: June 14, 2026

Beginner to Intermediate

You have a folder full of downloaded files named document(1).pdf, screenshot_2024.png, and report_final_v3_FINAL.xlsx. Every week you spend 20 minutes sorting them by hand. Or maybe you copy data from a website into a spreadsheet every morning, or you run the same three terminal commands every time you start work. These are the tasks Python was built to eliminate. With a few dozen lines of code you can turn a painful weekly chore into something that runs itself while you drink coffee.

Python ships with a rich standard library for automation — pathlib for file operations, subprocess for running system commands, smtplib for sending email — and the broader ecosystem adds libraries like schedule for periodic jobs and requests for web data collection. No special setup is needed beyond a standard Python 3.8+ install. For scheduling, you will need to install schedule with pip, but everything else in this article is built in.

In this guide we will cover four practical automation categories: organizing files and folders with pathlib and shutil, collecting web data with requests and BeautifulSoup, scheduling jobs to run automatically with the schedule library, and running system commands with subprocess. Each section ends with working code you can adapt to your own situation. By the end you will have a toolkit of reusable automation patterns and a complete script that organizes a messy downloads folder automatically.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program.

Automate Tasks with Python: Quick Example

Here is a self-contained script that renames and moves files in a folder based on their extension — one of the most common automation tasks you will ever write:

# sort_downloads.py
from pathlib import Path
import shutil

FOLDER = Path.home() / "Downloads"
DESTINATIONS = {
    ".pdf":  "Documents",
    ".png":  "Images",
    ".jpg":  "Images",
    ".xlsx": "Spreadsheets",
    ".csv":  "Spreadsheets",
    ".zip":  "Archives",
}

for file in FOLDER.iterdir():
    if file.is_file() and file.suffix in DESTINATIONS:
        dest_folder = FOLDER / DESTINATIONS[file.suffix]
        dest_folder.mkdir(exist_ok=True)
        shutil.move(str(file), dest_folder / file.name)
        print(f"Moved: {file.name} -> {DESTINATIONS[file.suffix]}/")

Output (example):

Moved: invoice_march.pdf -> Documents/
Moved: screenshot_2024.png -> Images/
Moved: sales_report.xlsx -> Spreadsheets/

The script uses Path.home() to get the user’s home directory regardless of operating system, iterdir() to loop over every item in the folder, and shutil.move() to relocate the file. The mkdir(exist_ok=True) call creates the destination folder if it does not already exist — no crash if it is there, no error if it is not. We will build a more complete version in the real-life example section, including duplicate detection and logging.

What Is Python Automation and When Should You Use It?

Automation means writing code that performs a repetitive task so you do not have to. The rule of thumb is: if you have done something manually more than three times, it is worth automating. Python is the go-to language for automation because it has concise syntax, a massive standard library, and third-party packages that cover almost every automation use case out of the box.

The table below maps common repetitive tasks to the Python tools that handle them:

Task TypePython ToolWhen to Use
File/folder operationspathlib, shutilRenaming, moving, copying, deleting files
Reading/writing filesbuilt-in open(), csvLog parsing, report generation, data transformation
Web data collectionrequests, BeautifulSoupPulling prices, headlines, tables from websites
Scheduled jobsschedule, APSchedulerRunning tasks daily, hourly, or on a cron-like schedule
System commandssubprocessRunning CLI tools, shell scripts, git, ffmpeg
Email sendingsmtplib, yagmailAutomated reports, alerts, notifications

A good automation script is idempotent — running it twice produces the same result as running it once. It handles edge cases (missing files, network errors, duplicate names) without crashing. And it logs what it did so you can review the results later. Keep these principles in mind as we work through each section.

Python developer sorting files automatically with pathlib and shutil
pathlib.iterdir() — because sorting files by hand is how you waste a Tuesday.

Automating File and Folder Operations

The pathlib module (Python 3.4+) provides an object-oriented interface for working with file paths that is far more readable than the older os.path approach. Combined with shutil for copy/move operations, these two modules cover 90% of file automation tasks.

Finding and Filtering Files with pathlib

The glob() and rglob() methods on a Path object let you find files matching a pattern across an entire directory tree. glob() searches one level deep; rglob() (recursive glob) searches all subdirectories:

# find_files.py
from pathlib import Path

base = Path("/tmp/project")  # change to your actual folder

# Find all Python files in this folder only
py_files = list(base.glob("*.py"))
print("Python files (top-level):", [f.name for f in py_files])

# Find all log files anywhere in the tree
log_files = list(base.rglob("*.log"))
print("Log files (all depths):", [f.relative_to(base) for f in log_files])

# Find files larger than 1 MB
large_files = [f for f in base.rglob("*") if f.is_file() and f.stat().st_size > 1_000_000]
print("Files over 1MB:", [f.name for f in large_files])

Output (example):

Python files (top-level): ['app.py', 'utils.py', 'config.py']
Log files (all depths): [PosixPath('logs/app.log'), PosixPath('logs/error.log')]
Files over 1MB: ['dataset.csv', 'backup.tar.gz']

f.stat().st_size returns the file size in bytes. The expression f.relative_to(base) strips the base directory from the path so you see logs/app.log instead of the full absolute path. Both are useful for building reports of what your script found before it starts moving anything.

Renaming and Copying Files Safely

Before moving or renaming files in an automation script, always check whether the destination already exists. Blindly overwriting a file can cause data loss that is impossible to reverse:

# safe_copy.py
from pathlib import Path
import shutil

def safe_copy(src: Path, dest_dir: Path) -> Path:
    """Copy src into dest_dir, appending a counter if the name already exists."""
    dest_dir.mkdir(parents=True, exist_ok=True)
    dest = dest_dir / src.name

    if dest.exists():
        counter = 1
        while dest.exists():
            stem = src.stem
            dest = dest_dir / f"{stem}_{counter}{src.suffix}"
            counter += 1

    shutil.copy2(str(src), dest)   # copy2 preserves metadata (timestamps)
    return dest

# Demo
src_file = Path("/tmp/report.pdf")
src_file.write_text("dummy content")   # create test file
result = safe_copy(src_file, Path("/tmp/archive"))
print(f"Copied to: {result}")

result2 = safe_copy(src_file, Path("/tmp/archive"))  # simulate duplicate
print(f"Duplicate handled: {result2}")

Output:

Copied to: /tmp/archive/report.pdf
Duplicate handled: /tmp/archive/report_1.pdf

The shutil.copy2() function copies the file content AND preserves the original modification time, which is important when you want archive copies to retain their original dates. The counter loop ensures you never silently overwrite an existing file — a critical safety net for any file automation script.

Developer inspecting duplicate files during automation
Duplicate detected. Counter incremented. Crisis averted.

Automating Web Data Collection

Web scraping lets your scripts pull data from websites automatically. The standard approach uses requests to download HTML and BeautifulSoup to parse it. Install both with pip install requests beautifulsoup4.

Fetching and Parsing a Web Page

We will use quotes.toscrape.com, a site built specifically for scraping practice. It serves reliable HTML with a stable structure, so this code will continue to work without modification.

Here is the HTML structure of each quote on that page, so you can see exactly what the selectors are targeting:

<!-- HTML structure of each quote on quotes.toscrape.com -->
<div class="quote">
    <span class="text">"The world as we have created it..."</span>
    <span>
        by <small class="author">Albert Einstein</small>
    </span>
    <div class="tags">
        <a class="tag" href="/tag/change/page/1/">change</a>
    </div>
</div>

Now the scraping code:

# scrape_quotes.py
import requests
from bs4 import BeautifulSoup

def scrape_quotes(url: str) -> list[dict]:
    response = requests.get(url, timeout=10)
    response.raise_for_status()   # raises HTTPError for 4xx/5xx responses

    soup = BeautifulSoup(response.text, "html.parser")
    results = []

    for quote_div in soup.select("div.quote"):
        text_elem = quote_div.select_one("span.text")
        author_elem = quote_div.select_one("small.author")
        tag_elems = quote_div.select("a.tag")

        # Defensive: check before accessing .text
        text   = text_elem.text.strip()   if text_elem   else "Unknown"
        author = author_elem.text.strip() if author_elem else "Unknown"
        tags   = [t.text for t in tag_elems]

        results.append({"quote": text, "author": author, "tags": tags})

    return results

quotes = scrape_quotes("https://quotes.toscrape.com")
for q in quotes[:3]:
    print(f'{q["author"]}: {q["quote"][:60]}...')
    print(f'  Tags: {", ".join(q["tags"])}')
    print()

Output:

Albert Einstein: "The world as we have created it is a process of...
  Tags: change, deep-thoughts, thinking, world

J.K. Rowling: "It is our choices, Harry, that show what we truly a...
  Tags: abilities, choices

Albert Einstein: "There are only two ways to live your life. One is...
  Tags: inspirational, life, live, miracle, miracles

response.raise_for_status() is a one-line safety net — it raises a requests.HTTPError if the server returns a 4xx or 5xx status code instead of silently continuing with bad data. The defensive checks (text_elem.text if text_elem else "Unknown") protect against pages where an element is missing, which happens constantly on real-world sites.

Saving Scraped Data to CSV

Collecting data is only half the job — you need to store it somewhere useful. Writing to CSV with Python’s built-in csv module keeps the output format-agnostic and readable in any spreadsheet application:

# save_to_csv.py
import csv
import requests
from bs4 import BeautifulSoup

def scrape_quotes(url):
    resp = requests.get(url, timeout=10)
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "html.parser")
    results = []
    for div in soup.select("div.quote"):
        text   = div.select_one("span.text")
        author = div.select_one("small.author")
        results.append({
            "quote":  text.text.strip()   if text   else "",
            "author": author.text.strip() if author else "",
        })
    return results

quotes = scrape_quotes("https://quotes.toscrape.com")

output_file = "quotes.csv"
with open(output_file, "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["author", "quote"])
    writer.writeheader()
    writer.writerows(quotes)

print(f"Saved {len(quotes)} quotes to {output_file}")

Output:

Saved 10 quotes to quotes.csv

Always pass encoding="utf-8" when writing CSV files — without it, non-ASCII characters (curly quotes, accented letters, em-dashes) will cause encoding errors or garbled output on Windows. The newline="" argument is also required by Python’s csv module to prevent extra blank lines on Windows.

Python web scraping with requests and BeautifulSoup
BeautifulSoup.select() — structured extraction from unstructured chaos.

Running Automated Jobs on a Schedule

Writing the automation code is only half the job. You also need it to run at the right time, without you having to remember to start it. The schedule library provides a clean Python API for defining when jobs should run — every minute, every day at 9am, every Monday, and so on. Install it with pip install schedule.

Basic Scheduling with the schedule Library

The schedule library works with a simple event loop: you register jobs with schedule.every(), then call schedule.run_pending() in a loop to check whether any jobs are due:

# basic_schedule.py
import schedule
import time
from datetime import datetime

def morning_report():
    print(f"[{datetime.now():%H:%M:%S}] Good morning! Running daily report...")
    # your actual report logic goes here

def hourly_check():
    print(f"[{datetime.now():%H:%M:%S}] Hourly check complete.")

# Schedule the jobs
schedule.every().day.at("09:00").do(morning_report)
schedule.every().hour.do(hourly_check)
schedule.every(30).minutes.do(lambda: print("30-min heartbeat"))

print("Scheduler running. Press Ctrl+C to stop.")
while True:
    schedule.run_pending()
    time.sleep(30)   # check every 30 seconds to save CPU

Output (example at 09:00:00):

Scheduler running. Press Ctrl+C to stop.
[09:00:00] Good morning! Running daily report...
[09:00:00] Hourly check complete.
[09:00:00] 30-min heartbeat
[09:30:00] 30-min heartbeat
[10:00:00] Hourly check complete.

The time.sleep(30) inside the loop is important — without a sleep, the loop burns 100% of one CPU core doing nothing. Sleeping 30 seconds means jobs can fire up to 30 seconds late (acceptable for most automation), while using negligible CPU. If you need second-level precision, use time.sleep(1) instead.

Handling Errors in Scheduled Jobs

When a job in a scheduled loop raises an unhandled exception, the whole process crashes and no more jobs run. Wrap your job functions with a try/except to log errors and keep the loop running:

# robust_schedule.py
import schedule
import time
import logging
from datetime import datetime

logging.basicConfig(
    filename="automation.log",
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)

def run_safely(job_func):
    """Decorator that catches exceptions and logs them without crashing the loop."""
    def wrapper():
        try:
            job_func()
            logging.info(f"{job_func.__name__} completed successfully")
        except Exception as exc:
            logging.error(f"{job_func.__name__} failed: {exc}", exc_info=True)
    return wrapper

def daily_scrape():
    # Simulating a job that sometimes fails
    import random
    if random.random() < 0.3:
        raise ConnectionError("Network unavailable")
    print("Scraped data successfully.")

schedule.every().day.at("08:00").do(run_safely(daily_scrape))

while True:
    schedule.run_pending()
    time.sleep(60)

The run_safely decorator wraps any function so that exceptions are caught, logged to a file, and the scheduler continues. The exc_info=True argument tells Python's logging module to include the full traceback in the log file -- essential for debugging failures that happen at 3am while you are asleep.

Python schedule library running automated jobs on a timer
schedule.run_pending() -- the world's most patient while loop.

Running System Commands with subprocess

Sometimes the right tool for a job is a command-line program, not a Python library. subprocess.run() lets your Python script launch any system command and capture its output, making it easy to orchestrate CLI tools like git, ffmpeg, or database utilities.

Basic subprocess Usage

# run_commands.py
import subprocess

# Run a command and capture its output
result = subprocess.run(
    ["python3", "--version"],
    capture_output=True,
    text=True,       # decode bytes to str automatically
    check=False,     # don't raise on non-zero exit code
)

print("Return code:", result.returncode)
print("Stdout:", result.stdout.strip())
print("Stderr:", result.stderr.strip())

# Run a command that lists files (works on macOS/Linux)
ls_result = subprocess.run(
    ["ls", "-la", "/tmp"],
    capture_output=True,
    text=True,
)
print("\nFirst 3 lines of /tmp listing:")
for line in ls_result.stdout.strip().split("\n")[:3]:
    print(" ", line)

Output:

Return code: 0
Stdout: Python 3.11.4
Stderr:

First 3 lines of /tmp listing:
  total 0
  drwxrwxrwt  20 root  wheel  640 Jun 12 09:31 .
  drwxr-xr-x  20 root  wheel  640 May 18 11:02 ..

Always use a list for the command argument (["ls", "-la", "/tmp"]) rather than a string ("ls -la /tmp"). The list form avoids shell injection vulnerabilities -- if any part of the command comes from user input, a string passed to shell=True can execute arbitrary shell commands. The list form is always safer.

Practical Example: Automating Git Operations

Here is a real-world use case -- a script that automatically stages, commits, and pushes changes in a git repository. This is useful for automating backup commits or syncing generated files:

# git_autocommit.py
import subprocess
from datetime import datetime
from pathlib import Path

def git_run(args: list[str], cwd: Path) -> subprocess.CompletedProcess:
    """Run a git command in the specified directory."""
    return subprocess.run(
        ["git"] + args,
        capture_output=True,
        text=True,
        cwd=cwd,
    )

def auto_commit(repo_path: Path, message: str = None) -> bool:
    """Stage all changes and commit if there is anything to commit."""
    # Check for changes
    status = git_run(["status", "--porcelain"], repo_path)
    if not status.stdout.strip():
        print("No changes to commit.")
        return False

    if message is None:
        message = f"Auto-commit {datetime.now():%Y-%m-%d %H:%M}"

    # Stage all changes
    git_run(["add", "-A"], repo_path)

    # Commit
    commit = git_run(["commit", "-m", message], repo_path)
    if commit.returncode == 0:
        print(f"Committed: {message}")
        return True
    else:
        print(f"Commit failed: {commit.stderr.strip()}")
        return False

# Usage (change to your actual repo path)
repo = Path("/tmp/my-project")
repo.mkdir(exist_ok=True)
auto_commit(repo, "Automated daily backup")

Output (when changes exist):

Committed: Automated daily backup

The helper function git_run() takes the git subcommand as a list and prepends "git", keeping the calling code clean. The cwd=cwd argument tells subprocess where to run the command -- without it, git would operate on whatever directory the script itself lives in, which is almost never what you want.

Real-Life Example: Automated Downloads Folder Organizer

We will now build a complete, production-ready script that watches your Downloads folder and organizes files into subfolders by type. It handles duplicates, logs every action, and can be scheduled to run automatically.

Python automation script organizing downloads folder with logging
Automation log: 47 files sorted. 3 duplicates handled. 0 manual clicks.
# downloads_organizer.py
import shutil
import logging
from pathlib import Path
from datetime import datetime

# --- Configuration ---
DOWNLOADS_DIR = Path.home() / "Downloads"
LOG_FILE = Path.home() / "downloads_organizer.log"

RULES = {
    "Documents": [".pdf", ".doc", ".docx", ".txt", ".rtf"],
    "Images":    [".jpg", ".jpeg", ".png", ".gif", ".svg", ".webp", ".heic"],
    "Videos":    [".mp4", ".mov", ".mkv", ".avi", ".m4v"],
    "Audio":     [".mp3", ".m4a", ".flac", ".wav", ".aac"],
    "Archives":  [".zip", ".tar", ".gz", ".rar", ".7z"],
    "Code":      [".py", ".js", ".html", ".css", ".json", ".sh", ".ipynb"],
    "Data":      [".csv", ".xlsx", ".xls", ".tsv", ".parquet"],
}

# Build reverse lookup: extension -> folder name
EXT_MAP = {ext: folder for folder, exts in RULES.items() for ext in exts}

# --- Logging setup ---
logging.basicConfig(
    filename=LOG_FILE,
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)

def unique_dest(dest: Path) -> Path:
    """Append a counter to avoid overwriting existing files."""
    if not dest.exists():
        return dest
    counter = 1
    while True:
        candidate = dest.parent / f"{dest.stem}_{counter}{dest.suffix}"
        if not candidate.exists():
            return candidate
        counter += 1

def organize(dry_run: bool = False) -> dict:
    """Move files from Downloads into categorized subfolders."""
    stats = {"moved": 0, "skipped": 0, "unknown": 0}

    for item in DOWNLOADS_DIR.iterdir():
        if not item.is_file():
            continue

        folder_name = EXT_MAP.get(item.suffix.lower())
        if folder_name is None:
            logging.info(f"SKIP (unknown type): {item.name}")
            stats["unknown"] += 1
            continue

        dest_dir = DOWNLOADS_DIR / folder_name
        dest = unique_dest(dest_dir / item.name)

        if dry_run:
            print(f"[DRY RUN] Would move: {item.name} -> {folder_name}/")
        else:
            dest_dir.mkdir(exist_ok=True)
            shutil.move(str(item), dest)
            logging.info(f"MOVED: {item.name} -> {folder_name}/{dest.name}")
            stats["moved"] += 1

    return stats

if __name__ == "__main__":
    print(f"Organizing {DOWNLOADS_DIR} ...")
    result = organize(dry_run=False)
    summary = f"Done. Moved: {result['moved']}, Unknown: {result['unknown']}"
    print(summary)
    logging.info(summary)

Output (example):

Organizing /Users/alice/Downloads ...
Done. Moved: 12, Unknown: 3

The dry_run=True mode lets you preview what the script would do without actually moving anything -- run it first to confirm the output looks right. The unique_dest() function guarantees you never silently overwrite a file with the same name in the destination folder. You can schedule this script to run daily using the schedule library covered earlier, or on macOS/Linux you can add it to your crontab with crontab -e for OS-level scheduling without a Python process running continuously.

Frequently Asked Questions

What is the difference between shutil and pathlib for file operations?

pathlib is for path manipulation and simple operations: checking if a file exists, reading its metadata, renaming it within the same filesystem. shutil is for heavier operations: copying files (with or without metadata), moving files across filesystems, and deleting entire directory trees. In practice you often use both -- pathlib to build and check paths, shutil to actually move or copy the files. Use shutil.move() for moves (it handles cross-filesystem moves gracefully) and shutil.copy2() when you want to preserve file modification times.

Should I use the schedule library or cron for scheduled tasks?

It depends on your setup. The schedule library is pure Python and works identically on Windows, macOS, and Linux -- great for scripts you want to be portable or that run within an existing Python process. Cron (on Linux/macOS) and Task Scheduler (on Windows) are OS-level schedulers that are more reliable for long-running production tasks because they survive reboots automatically and do not require a Python process to stay running. For personal automation scripts on a development machine, schedule is simpler. For server deployments, lean on cron or a process manager like systemd.

When is it safe to use subprocess with shell=True?

Use shell=True only when the entire command is a string literal you control completely -- for example, a hardcoded one-liner like subprocess.run("ls -la /tmp | wc -l", shell=True). Never pass user input, command-line arguments, or any external data into a shell=True command string; doing so opens a shell injection vulnerability where an attacker can execute arbitrary commands. The list form (["ls", "-la", "/tmp"]) is safe with external data because each list element is passed directly to the OS without going through a shell interpreter.

How do I avoid getting blocked when scraping websites?

The main causes of blocks are too-fast request rates, missing request headers, and large-volume scraping. Add a delay between requests using time.sleep(random.uniform(1, 3)) -- randomized delays look more human than a fixed interval. Always set a User-Agent header in your requests session to identify your scraper politely: session.headers.update({"User-Agent": "MyBot/1.0 (research project)"}). Always check the site's robots.txt file before scraping -- if the path you want to scrape is listed as disallowed, respect it. For sites that require JavaScript to load content, switch to a tool like playwright or selenium instead of requests.

Why should automation scripts log to a file instead of just printing?

When a script runs unattended -- on a schedule overnight or as a background process -- there is no terminal to see the output. File logging means you can review what happened after the fact, including any errors. Python's built-in logging module automatically records timestamps, log levels (INFO, WARNING, ERROR), and full tracebacks on exceptions. Set up a RotatingFileHandler for long-running scripts to cap the log file size so it does not grow indefinitely: from logging.handlers import RotatingFileHandler, then RotatingFileHandler("script.log", maxBytes=1_000_000, backupCount=3) keeps the last 3MB of logs and discards older entries automatically.

How do I make an automation script safe to run multiple times?

Design for idempotency -- the script should produce the same result whether it runs once or ten times. For file organization scripts, this means checking if a file already exists in the destination before moving it (and using a counter suffix for duplicates, as shown in the real-life example). For database or API writes, check for existing records before inserting. For web scraping pipelines, track which pages or records have already been collected in a CSV or SQLite database, and skip them on subsequent runs. The general pattern is: check state first, act only if the desired state is not already present.

Conclusion

We have covered four practical automation categories in this guide: file and folder operations with pathlib and shutil, web data collection with requests and BeautifulSoup, scheduled jobs with the schedule library, and system command automation with subprocess. The real-life Downloads Organizer script ties these concepts together into a complete, production-ready tool with duplicate handling, configurable rules, dry-run mode, and file logging.

The best next step is to adapt the Downloads Organizer to your own situation -- add more file types, change the destination folders, or hook it into schedule to run automatically every morning. Once you have the pattern down, you will start seeing automation opportunities everywhere: renaming podcast downloads, archiving old project folders, pulling daily exchange rates from an API, or auto-committing generated reports to git.

For deeper reading, the official Python documentation covers pathlib, shutil, and subprocess in full detail. The schedule library docs have examples for every scheduling pattern you might need. And BeautifulSoup4's documentation is an excellent reference for parsing more complex HTML structures.

How To Use PyArrow for Parquet File Processing in Python

How To Use PyArrow for Parquet File Processing in Python

Intermediate

You have a CSV file with ten million rows of sales data. You read it into pandas every morning, filter down to the last 30 days, and compute some aggregates. The read step alone takes 45 seconds. Now imagine it took less than two. That is the promise of Parquet — a columnar binary file format designed for exactly this kind of workload. Instead of reading every row to find the values in a single column, Parquet lets the engine skip straight to the data it actually needs.

PyArrow is the Python library that makes Parquet accessible without any special infrastructure. It ships with pandas, integrates with polars, and lets you read and write Parquet files in a handful of lines of code. PyArrow also handles schema enforcement, Snappy and Zstandard compression, and partitioned datasets spread across directories — the same building blocks that power data lakes at companies like Uber and Netflix, available on your laptop.

In this article we will cover how to install PyArrow and write your first Parquet file, how the columnar format differs from CSV and JSON, how to control schema and compression, how to filter data at read time without loading the whole file, how to work with partitioned Parquet datasets, and how to build a real-world pipeline that stores transaction data efficiently. By the end you will have a working toolkit for replacing slow CSV pipelines with fast, schema-safe Parquet files.

PyArrow Parquet in Python: Quick Example

Here is a self-contained example that creates a table, writes it to Parquet, reads it back, and prints the result — all in about 15 lines:

# quick_parquet.py
import pyarrow as pa
import pyarrow.parquet as pq

# Create an Arrow table from Python lists
table = pa.table({
    "product": ["Widget A", "Widget B", "Widget C"],
    "quantity": [100, 250, 75],
    "price": [9.99, 4.49, 19.99],
})

# Write to Parquet
pq.write_table(table, "products.parquet")

# Read it back
result = pq.read_table("products.parquet")
print(result.to_pandas())

Output:

    product  quantity  price
0  Widget A       100   9.99
1  Widget B       250   4.49
2  Widget C        75  19.99

The two key functions are pq.write_table() and pq.read_table(). Both take a file path as their second argument. pa.table() creates an Arrow table from a dictionary of column names to Python lists — PyArrow infers the data types automatically. You get a typed, compressed binary file on disk and a clean DataFrame-compatible object back on read.

That is the core loop. The rest of this article shows you how to control compression, enforce schemas, filter at read time, and partition data across directories for datasets that do not fit in memory.

Python debugger character examining Parquet row group filtering
CSV reads every row. Parquet reads only the columns you asked for.

What Is Parquet and Why Is It Faster Than CSV?

Parquet is a columnar storage format originally developed at Twitter and Cloudera for the Hadoop ecosystem. Instead of storing data row by row (like CSV or JSON), Parquet stores all the values in each column together. This has three major consequences: better compression, faster column scans, and cheaper predicate pushdown.

When values in the same column are stored together, they tend to be similar in type and often in value — a “country” column full of repeated strings like “AU” or “US” compresses dramatically compared to mixed-content rows. A Parquet file is also self-describing: it embeds a schema with column names and types, so you never have to guess whether a field is a string or a number.

Here is how the main file formats compare for typical data engineering workloads:

FeatureCSVJSONParquet
Storage formatRow-oriented textRow-oriented textColumn-oriented binary
Schema embeddedNoNoYes
CompressionNone (plain text)NoneSnappy, Gzip, Zstd
Read a single columnMust scan entire fileMust scan entire fileJumps directly to column
Predicate pushdownNot supportedNot supportedRow group filtering
Typical file sizeLargeLarger3x-10x smaller
Human readableYesYesNo (binary)

The tradeoff is binary format — you cannot open a Parquet file in a text editor. But for data pipelines that read the same file repeatedly, the speed and size advantages far outweigh the loss of human readability. PyArrow bridges this gap by making the format as easy to work with as CSV.

Writing Parquet Files with PyArrow

You can create Parquet files from PyArrow tables, pandas DataFrames, or plain Python dictionaries. The most direct path is pa.table() followed by pq.write_table(). PyArrow automatically infers the schema from your data types unless you supply one explicitly.

# write_parquet.py
import pyarrow as pa
import pyarrow.parquet as pq
import pandas as pd

# Method 1: from a pandas DataFrame
df = pd.DataFrame({
    "user_id": [1001, 1002, 1003, 1004],
    "country": ["AU", "US", "AU", "GB"],
    "spend": [120.50, 88.00, 200.75, 44.20],
    "active": [True, False, True, True],
})
table = pa.Table.from_pandas(df)
pq.write_table(table, "users.parquet")

# Method 2: directly from an Arrow table
orders = pa.table({
    "order_id": pa.array([5001, 5002, 5003], type=pa.int32()),
    "amount": pa.array([29.99, 14.50, 99.00], type=pa.float64()),
    "status": pa.array(["shipped", "pending", "delivered"], type=pa.string()),
})
pq.write_table(orders, "orders.parquet", compression="snappy")
print("Files written.")

Output:

Files written.

Notice compression="snappy" in the second call. Snappy is the default when compression is not specified — it is fast to decode and gives reasonable size reduction. You can also use "gzip" for better compression at the cost of slower reads, or "zstd" for the best balance of speed and compression ratio. For data you read frequently, Snappy is usually the right choice. For archival data you read rarely, Zstd at a higher compression level saves more disk space.

Reading Parquet Files and Filtering Columns

Reading a full Parquet file is one line. The real power comes from reading only the columns you need — PyArrow loads only those column chunks from disk, skipping the rest entirely. For a 100-column dataset where your query uses only 3 columns, this can be a 30x reduction in I/O.

# read_parquet.py
import pyarrow.parquet as pq

# Read all columns
full = pq.read_table("users.parquet")
print("Full table:")
print(full.to_pandas())
print()

# Read only specific columns -- PyArrow skips the others on disk
subset = pq.read_table("users.parquet", columns=["user_id", "country"])
print("Subset (2 of 4 columns):")
print(subset.to_pandas())

Output:

Full table:
   user_id country   spend  active
0     1001      AU  120.50    True
1     1002      US   88.00   False
2     1003      AU  200.75    True
3     1004      GB   44.20    True

Subset (2 of 4 columns):
   user_id country
0     1001      AU
1     1002      US
2     1003      AU
3     1004      GB

PyArrow also supports predicate pushdown via the filters parameter. You can push filter conditions into the read operation so that row groups that cannot satisfy the filter are skipped without being decoded. For large files split into many row groups, this can cut read time dramatically.

# read_filtered.py
import pyarrow.parquet as pq

# Only return rows where country == "AU"
# Parquet row group statistics let PyArrow skip non-matching groups
au_users = pq.read_table(
    "users.parquet",
    filters=[("country", "==", "AU")],
)
print(au_users.to_pandas())

Output:

   user_id country   spend  active
0     1001      AU  120.50    True
1     1003      AU  200.75    True

The filters parameter accepts a list of tuples in the form (column, operator, value). Valid operators include ==, !=, <, >, <=, >=, in, and not in. For maximum effectiveness, sort your data by the filter column before writing — that way each row group contains a contiguous range and PyArrow can skip most groups on a range filter.

Python developer character comparing Snappy, Gzip, and Zstd Parquet compression
filters=[(‘country’, ‘==’, ‘AU’)] — 98% of the file just stays on disk.

Defining and Enforcing Schemas

One of Parquet’s greatest strengths over CSV is explicit schema. When you read a CSV with mixed data, you often get silent type coercions — a column that should be integers becomes floats because one row had a decimal. With Parquet, the schema is embedded in the file and enforced on write. You can also define the schema explicitly and have PyArrow reject data that does not conform.

# schema_parquet.py
import pyarrow as pa
import pyarrow.parquet as pq

# Define an explicit schema with precise types
schema = pa.schema([
    pa.field("transaction_id", pa.int64()),
    pa.field("amount", pa.float64()),
    pa.field("currency", pa.string()),
    pa.field("timestamp", pa.timestamp("ms")),
])

# Create table with matching data
import pandas as pd
from datetime import datetime

df = pd.DataFrame({
    "transaction_id": [9001, 9002, 9003],
    "amount": [150.00, 32.50, 899.99],
    "currency": ["AUD", "USD", "EUR"],
    "timestamp": [
        datetime(2026, 6, 1, 9, 0),
        datetime(2026, 6, 1, 10, 15),
        datetime(2026, 6, 1, 11, 30),
    ],
})

# Cast to explicit schema -- raises if types are incompatible
table = pa.Table.from_pandas(df, schema=schema)
pq.write_table(table, "transactions.parquet")

# Read back and inspect schema
t = pq.read_table("transactions.parquet")
print(t.schema)

Output:

transaction_id: int64
amount: double
currency: string
timestamp: timestamp[ms]

The pa.schema() call creates a strict type contract for the file. If you try to write a DataFrame where transaction_id is a string, PyArrow raises a ArrowInvalid error at write time — not a silent runtime bug hours later. The pa.timestamp("ms") type stores timestamps as milliseconds since Unix epoch, which Parquet encodes compactly and PyArrow converts back to Python datetime objects automatically on read.

Choosing the Right Compression

Parquet supports several compression codecs. Each has different tradeoffs between file size and read/write speed. The right choice depends on your access pattern: if you write once and read many times (analytical workloads), optimize for read speed. If you archive data rarely accessed, optimize for size.

# compression_comparison.py
import pyarrow as pa
import pyarrow.parquet as pq
import os

table = pa.table({
    "category": ["electronics"] * 50000 + ["clothing"] * 50000,
    "value": list(range(100000)),
    "note": ["sample data for compression test"] * 100000,
})

codecs = ["none", "snappy", "gzip", "zstd"]

for codec in codecs:
    fname = f"data_{codec}.parquet"
    pq.write_table(table, fname, compression=codec)
    size_kb = os.path.getsize(fname) / 1024
    print(f"{codec:8s} -> {size_kb:8.1f} KB")

Output:

none     ->   2841.2 KB
snappy   ->    312.4 KB
gzip     ->    204.8 KB
zstd     ->    189.3 KB

Snappy achieves a 9x size reduction over uncompressed and is fast enough that decompression overhead is barely noticeable. Gzip and Zstd go further — typically 12x to 15x smaller — but take slightly longer to read. For most production pipelines, snappy is the default for a good reason. Switch to zstd (with compression_level=3) for cold archival storage where read latency matters less.

Python developer character running through Hive-partitioned directory tree
Snappy: 9x smaller, instant reads. Zstd: 15x smaller, slightly slower. Pick one and commit.

Working with Partitioned Datasets

When a dataset grows beyond what fits in a single file, you can partition it — split it across multiple Parquet files organized in a directory hierarchy. Hive-style partitioning uses subdirectory names like country=AU/ or year=2026/month=06/ to encode partition values. PyArrow reads partitioned datasets as if they were a single table, pushing partition filters at the directory level so entire directories are skipped without opening any files.

# partitioned_write.py
import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.parquet as pq
import shutil, os

table = pa.table({
    "user_id": [1, 2, 3, 4, 5, 6],
    "country": ["AU", "US", "AU", "GB", "US", "AU"],
    "spend": [120.5, 88.0, 200.75, 44.2, 310.0, 55.0],
})

output_dir = "partitioned_users"
if os.path.exists(output_dir):
    shutil.rmtree(output_dir)

# Write partitioned by country -- creates subdirs like country=AU/
pq.write_to_dataset(table, root_path=output_dir, partition_cols=["country"])

# Show what was created
for root, dirs, files in os.walk(output_dir):
    for f in files:
        path = os.path.join(root, f)
        print(path)

Output:

partitioned_users/country=AU/part-0.parquet
partitioned_users/country=GB/part-0.parquet
partitioned_users/country=US/part-0.parquet

Reading the partitioned dataset back is equally simple. PyArrow discovers all the partition directories and reconstructs the full table. You can also filter at the partition level — when you filter on country == "AU", PyArrow only opens the country=AU/ directory and never touches the others.

# partitioned_read.py
import pyarrow.dataset as ds

# Read only AU records -- other country dirs are skipped entirely
dataset = ds.dataset("partitioned_users", format="parquet", partitioning="hive")
au_data = dataset.to_table(filter=ds.field("country") == "AU")
print(au_data.to_pandas())

Output:

   user_id country   spend
0        1      AU  120.50
1        3      AU  200.75
2        6      AU   55.00

The partitioning="hive" argument tells PyArrow to expect Hive-style directory names. The ds.field("country") == "AU" expression is evaluated at the directory level — PyArrow reads the directory listing, sees that only country=AU/ matches, and skips country=US/ and country=GB/ completely.

Python developer character orchestrating a CSV to Parquet data pipeline
read_csv() every morning, or read_parquet() once in 400ms. This is not a hard choice.

PyArrow and pandas: Interoperability

If you already use pandas, PyArrow integrates cleanly in both directions. pandas.read_parquet() and pandas.DataFrame.to_parquet() use PyArrow under the hood when PyArrow is installed. You can pass PyArrow-specific options through pandas using the engine="pyarrow" argument.

# pandas_parquet.py
import pandas as pd

# Write from pandas
df = pd.DataFrame({
    "product_id": [101, 102, 103],
    "name": ["Bolt M6", "Nut M6", "Washer M6"],
    "stock": [500, 1200, 800],
    "unit_price": [0.10, 0.05, 0.03],
})
df.to_parquet("products_pd.parquet", engine="pyarrow", compression="snappy", index=False)

# Read from pandas -- column pruning still works
df_back = pd.read_parquet(
    "products_pd.parquet",
    engine="pyarrow",
    columns=["product_id", "name", "stock"],
)
print(df_back)
print(df_back.dtypes)

Output:

   product_id       name  stock
0         101    Bolt M6    500
1         102     Nut M6   1200
2         103  Washer M6    800
product_id      int64
name           object
stock           int64
dtype: object

Note index=False in the to_parquet() call. By default, pandas writes the DataFrame index as an extra column. For most datasets the index is just a sequential integer you do not need, so index=False keeps the file clean. If you are using a meaningful index (like timestamps), omit that argument and PyArrow will preserve it.

Real-Life Example: Daily Sales Archive Pipeline

Here is a realistic pipeline that reads daily sales data from CSV, validates the schema, writes partitioned Parquet files by date, and reads back aggregate statistics — the kind of pattern you would use to replace a slow nightly ETL job.

# sales_pipeline.py
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds
import pandas as pd
import io, os, shutil

# --- Simulate raw CSV data arriving daily ---
raw_csv = """order_id,date,region,product,quantity,revenue
1001,2026-06-01,AU,Widget A,10,99.90
1002,2026-06-01,US,Widget B,5,22.45
1003,2026-06-02,AU,Widget A,3,29.97
1004,2026-06-02,GB,Widget C,8,159.92
1005,2026-06-03,US,Widget A,12,119.88
1006,2026-06-03,AU,Widget C,2,39.98
"""

# --- Step 1: Load and validate schema ---
schema = pa.schema([
    pa.field("order_id", pa.int64()),
    pa.field("date", pa.string()),
    pa.field("region", pa.string()),
    pa.field("product", pa.string()),
    pa.field("quantity", pa.int32()),
    pa.field("revenue", pa.float64()),
])

df = pd.read_csv(io.StringIO(raw_csv))
table = pa.Table.from_pandas(df, schema=schema)
print(f"Loaded {len(table)} rows with schema:")
print(table.schema)

# --- Step 2: Write partitioned by date ---
archive_dir = "sales_archive"
if os.path.exists(archive_dir):
    shutil.rmtree(archive_dir)

pq.write_to_dataset(
    table,
    root_path=archive_dir,
    partition_cols=["date"],
    compression="snappy",
)
print(f"\nPartitioned archive written to: {archive_dir}/")
for root, dirs, files in os.walk(archive_dir):
    for f in files:
        print(f"  {os.path.join(root, f)}")

# --- Step 3: Query AU region totals across all dates ---
dataset = ds.dataset(archive_dir, format="parquet", partitioning="hive")
au_table = dataset.to_table(
    filter=ds.field("region") == "AU",
    columns=["date", "product", "revenue"],
)

au_df = au_table.to_pandas()
summary = (
    au_df.groupby("product")["revenue"]
    .sum()
    .sort_values(ascending=False)
    .reset_index()
)
summary.columns = ["product", "total_revenue_AU"]
print("\nAU Revenue by Product:")
print(summary.to_string(index=False))

Output:

Loaded 6 rows with schema:
order_id: int64
date: string
region: string
product: string
quantity: int32
revenue: double

Partitioned archive written to: sales_archive/
  sales_archive/date=2026-06-01/part-0.parquet
  sales_archive/date=2026-06-02/part-0.parquet
  sales_archive/date=2026-06-03/part-0.parquet

AU Revenue by Product:
  product  total_revenue_AU
 Widget A            129.87
 Widget C             39.98

This pipeline handles four real-world requirements in one script: schema validation at ingest time, partitioned storage for efficient querying, column pruning on read (only loading date, product, and revenue), and partition pruning (skipping the US and GB partitions entirely). You can extend it by adding a date range filter like ds.field("date") >= "2026-06-02" or by switching the partition key to region if your most common query is by geography rather than date.

Python developer orchestrating CSV to Parquet data pipeline with PyArrow
read_csv() every morning, or read_parquet() once in 400ms. This is not a hard choice.

Frequently Asked Questions

How do I install PyArrow?

Run pip install pyarrow. That is all — no system-level Hadoop or JVM dependencies. If you are already using pandas, you may have it already since pandas uses PyArrow as an optional engine. Verify with import pyarrow; print(pyarrow.__version__). For partitioned dataset support (pyarrow.dataset), any version 4.0 or later works reliably; version 14+ is recommended for current features.

When should I use Parquet instead of CSV?

Use Parquet when any of these apply: your files are over 50 MB (compression alone justifies it), you read the same file more than once (you only pay the write cost once), you always query a subset of columns (column pruning is a free speed boost), or you need schema guarantees (no more “is this field a string or a number?”). Stick with CSV for small files, one-off outputs that humans need to read, or tools that do not support Parquet.

What is the difference between engine=”pyarrow” and engine=”fastparquet” in pandas?

Both engines read and write Parquet from pandas. PyArrow is maintained by the Apache Arrow project, has broader compression support (including Zstd), handles timestamps more reliably, and supports the latest Parquet spec. Fastparquet is a pure-Python alternative that can be lighter to install in some environments. For new projects, use engine="pyarrow" — it is the more actively developed option and handles edge cases better, especially with nested types and partitioned datasets.

What is a row group and why does it matter?

A Parquet file is internally split into row groups — chunks of consecutive rows, typically 128 MB each. Each row group stores min/max statistics for every column. When you use filters=[("amount", ">", 1000)], PyArrow reads the row group statistics first and skips any group where the max amount is below 1000, without decoding those rows at all. Smaller row groups mean finer-grained filtering but more metadata overhead. The default row_group_size parameter in pq.write_table() works well in most cases — tune it only if you have very specific query patterns.

Can Parquet store nested data like lists or structs?

Yes — this is one of Parquet’s advantages over CSV. PyArrow supports pa.list_(), pa.struct(), and pa.map_() types. For example, you can store a column of lists of strings with pa.field("tags", pa.list_(pa.string())). When you round-trip this through pandas, it becomes a column of Python lists. Nested types are stored efficiently in columnar format using Dremel encoding, the same approach used internally by Google BigQuery.

How should I handle Parquet files that are larger than RAM?

Use pyarrow.dataset.Dataset with lazy evaluation instead of pq.read_table(). With ds.dataset(path).to_batches(batch_size=100_000), you get an iterator of record batches that processes data in chunks. Each batch is decoded and processed independently, so only one chunk lives in memory at a time. Combine this with column pruning (columns=[...]) and partition filters (filter=...) and you can query multi-terabyte partitioned datasets on a laptop with 16 GB of RAM without loading the full dataset.

Conclusion

PyArrow turns Parquet from a format you hear about at data conferences into something you can use in a single afternoon. We covered the core read/write loop with pq.read_table() and pq.write_table(), how to define explicit schemas to catch type errors at write time, how column pruning and predicate pushdown reduce I/O without changing your query logic, how compression codecs like Snappy and Zstd shrink files 9x to 15x, and how Hive-style partitioned datasets let you skip entire directories on filter queries. The real-life example tied these together into a pipeline pattern you can adapt directly for nightly ETL jobs, analytics archives, or any workload where you read the same data more than once.

The best next step is to pick one of your existing CSV files and convert it: measure the size before and after, then benchmark a common query on both. The difference is usually dramatic enough that you will want to migrate everything. From there, explore pyarrow.dataset.Dataset.to_batches() for out-of-core processing and pa.RecordBatchReader for streaming write patterns.

Official documentation: PyArrow Parquet Documentation and PyArrow Datasets.

How To Use Python pexpect for Automating Terminal Interactions

How To Use Python pexpect for Automating Terminal Interactions

Last Updated: June 12, 2026

Intermediate

Some programs simply refuse to cooperate with normal Python automation. They prompt for passwords interactively, ask for confirmation before running a destructive command, or display a menu you have to navigate before they give you the data you need. You cannot pipe stdin to them or parse their output with subprocess alone because they are designed to talk to a human through a terminal. This is exactly where pexpect comes in — a Python library that lets your script pretend to be a human sitting at a keyboard, interacting with any terminal-based program as if it were typing commands and reading the screen.

The pexpect library works by spawning a child process attached to a pseudo-terminal (PTY) and then letting you define patterns to wait for and responses to send. It handles the low-level terminal I/O so you can focus on the conversation logic. It runs on any Unix-like system (Linux, macOS) and Python 3.4 or later. You install it with a single pip install pexpect command. There is no system daemon to configure and no special OS permissions needed for basic use.

In this article we will cover what pexpect is and how it differs from subprocess, how to spawn a child process and wait for output, how to send input and handle timeouts, how to capture output between interactions, how to handle multiple expected patterns with a list, and how to use the built-in pxssh module to automate SSH sessions. By the end you will be able to script any interactive terminal program and build reliable automation for tasks that previously required a human at the keyboard.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program.

Python pexpect: Quick Example

Here is a complete working example that spawns the system python3 interpreter, sends a few commands, reads the output, and exits cleanly — all under pexpect’s control:

# quick_pexpect.py
import pexpect

# Spawn a Python 3 interpreter as a child process
child = pexpect.spawn('python3', encoding='utf-8', timeout=10)

# Wait for the Python REPL prompt
child.expect('>>>')

# Send a command and wait for the next prompt
child.sendline('print("Hello from pexpect!")')
child.expect('>>>')

# Everything the process printed since the last expect() is in child.before
output = child.before.strip()
print("Child said:", output)

# Exit the interpreter cleanly
child.sendline('exit()')
child.expect(pexpect.EOF)

print("Session complete.")

Output:

Child said: Hello from pexpect!
Session complete.

The key methods here are pexpect.spawn(), which starts the child process, child.expect(), which waits until a pattern appears in the output, and child.sendline(), which types text followed by a newline. Everything the child printed between the last two expect() calls is stored in child.before. The encoding='utf-8' argument makes all I/O return Python strings instead of bytes, which is almost always what you want.

The sections below cover each piece in depth — pattern matching, timeouts, capturing multi-line output, and real-world SSH automation.

What Is pexpect and When Should You Use It?

pexpect is a pure Python module that automates interactive terminal applications by controlling them through a pseudo-terminal (PTY). A PTY is a kernel-level device that makes a process believe it is talking to a real terminal. This matters because many command-line programs behave differently when they detect a terminal versus a pipe — they disable color output, stop prompting for input, or buffer output differently. By using a PTY, pexpect gets the same output a human would see at a real terminal prompt.

The library’s name combines “expect” (the classic Unix scripting tool that inspired it) with Python. The core idea is simple: wait for an expected string or pattern to appear in the output, then send a response. This “expect and respond” loop is what lets you drive interactive programs programmatically.

ScenarioUse subprocess?Use pexpect?
Run a command and capture outputYes — simplerOverkill
Program asks for a password interactivelyNo — it won’t promptYes
Navigate a menu-driven CLINoYes
Automate an SSH login and run commandsFragileYes — use pxssh
Control a program that uses curses/ncursesNoYes
Run a script and pipe its stdoutYesNot needed

Use pexpect when the program you need to control was designed for a human at a terminal. Use subprocess when the program is script-friendly and just produces output that you can capture with stdout/stderr. Getting this distinction right will save you hours of frustration.

Sudo Sam at a terminal control panel like an air traffic controller
subprocess is a hammer. pexpect is a conversation.

Installing pexpect

pexpect is not part of the Python standard library, so you need to install it before you can use it. The install is a single command and brings in no heavy dependencies:

# install_pexpect.sh
pip install pexpect

Output:

Successfully installed pexpect-4.9.0 ptyprocess-0.7.0

The only dependency is ptyprocess, which handles the low-level PTY creation. You can verify the install worked correctly by importing pexpect in a Python shell and checking the version:

# verify_install.py
import pexpect
print(pexpect.__version__)

Output:

4.9.0

Note that pexpect only runs on Unix-like systems — Linux, macOS, and WSL on Windows. On native Windows without WSL, pexpect cannot create PTYs. If you are on Windows, the wexpect or winexpect packages are alternatives, but most teams use WSL or Docker for pexpect-based automation instead.

Spawning a Child Process with pexpect.spawn()

The entry point to every pexpect session is pexpect.spawn(). You pass it the command to run, along with options like encoding and timeout, and it returns a child process object that you use for all subsequent interaction:

# spawn_basics.py
import pexpect

# Spawn the 'bc' command-line calculator
child = pexpect.spawn('bc', encoding='utf-8', timeout=5)

# bc prints a copyright header first -- wait for the prompt
child.expect(r'\$|>')  # bc shows a bare prompt on some systems

# Send a calculation
child.sendline('2 + 2')
child.expect(r'\$|>')

print("Result:", child.before.strip())

child.sendline('quit')
child.expect(pexpect.EOF)

Output:

Result: 4

The timeout parameter controls how many seconds expect() will wait before raising a pexpect.TIMEOUT exception. Setting it at spawn time applies it globally, but you can also pass a different timeout to individual expect() calls. The encoding='utf-8' argument tells pexpect to decode bytes to strings automatically — without it, all I/O is in bytes and you need to write patterns as byte literals like b'>>>'. Always use the encoding argument for new code; it makes pattern writing much cleaner.

Waiting for Output with child.expect()

The expect() method is the heart of pexpect. It reads from the child process’s output until it finds a match for the pattern you give it, then returns. The pattern can be a plain string (exact match), a compiled regex, or a list of patterns to match any of them at once:

# expect_patterns.py
import pexpect
import re

child = pexpect.spawn('python3', encoding='utf-8', timeout=10)

# Wait for the >>> prompt using an exact string
child.expect('>>>')

# Send a command that might print a result OR raise an error
child.sendline('1 / 0')

# Match whichever appears first -- a result or an error
index = child.expect(['>>>', 'ZeroDivisionError', pexpect.TIMEOUT])

if index == 0:
    print("Got a result:", child.before.strip())
elif index == 1:
    print("Got an error:", child.before.strip())
else:
    print("Timed out waiting for output")

child.sendline('exit()')
child.expect(pexpect.EOF)

Output:

Got an error: Traceback (most recent call last):
  File "<stdin>", line 1, in <module>

When you pass a list to expect(), the return value is the index of the pattern that matched. This is the standard way to branch based on what the child process says next — you handle each possible response differently. The two special sentinel values pexpect.EOF (the process closed its output) and pexpect.TIMEOUT (time ran out) can always be included in the list to avoid uncaught exceptions. After a successful match, child.before contains everything printed before the match, and child.after contains the matched text itself.

Debug Dee using a magnifying glass to inspect pattern matches in terminal output
child.before is gold. child.after is the match. The rest is noise.

Sending Input with sendline() and send()

pexpect gives you two ways to send text to a child process. sendline() appends a newline character after the text (simulating pressing Enter), while send() sends the raw text without a newline. You use send() when you need to type individual characters or when the program reads character-by-character rather than line-by-line:

# sending_input.py
import pexpect

# Automate the 'ftp' command to connect and list files
# Using a public anonymous FTP server for testing
child = pexpect.spawn('ftp ftp.dlptest.com', encoding='utf-8', timeout=15)

child.expect('Name.*:')
child.sendline('dlpuser')   # username

child.expect('Password:')
child.sendline('rNrKYTX9zDd27W7')   # public test password

child.expect('ftp>')
child.sendline('ls')

child.expect('ftp>')
print("Directory listing:")
print(child.before.strip())

child.sendline('quit')
child.expect(pexpect.EOF)

Output:

Directory listing:
-rw-r--r--    1 0        0               0 Sep 01 00:00 .gitkeep
-rw-r--r--    1 0        0            1024 Sep 01 00:00 test1.txt
-rw-r--r--    1 0        0            2048 Sep 01 00:00 test2.txt

Notice how the password is sent with sendline() even though it is a sensitive value — pexpect does not mask passwords in transit because it is sending them directly to the child process’s PTY. For production automation of anything security-sensitive, ensure that your script itself is protected (not world-readable, stored with proper permissions, or using a secrets manager to supply the value at runtime rather than hardcoding it).

Capturing Output Between Interactions

The output a child process produces between two expect() calls accumulates in child.before. This is the primary way you extract data from interactive programs. However, child.before can contain terminal control codes (color sequences, cursor movements) that you usually want to strip out:

# capture_output.py
import pexpect
import re

def strip_ansi(text):
    """Remove ANSI terminal escape sequences from a string."""
    ansi_escape = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
    return ansi_escape.sub('', text)

child = pexpect.spawn('bash', encoding='utf-8', timeout=10)
child.expect(r'\$')   # Wait for the shell prompt

# Run a command and capture its output
child.sendline('df -h /')
child.expect(r'\$')   # Wait for the next prompt

raw_output = child.before
clean_output = strip_ansi(raw_output).strip()

print("Disk usage for /:")
# Skip the first line (the command echo) and print the rest
lines = [l for l in clean_output.splitlines() if l.strip()]
for line in lines[1:]:   # lines[0] is the echoed command
    print(line)

child.sendline('exit')
child.expect(pexpect.EOF)

Output:

Disk usage for /:
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   18G   30G  38% /

The strip_ansi() helper is worth keeping in your toolkit. Many terminal programs emit ANSI escape codes even when you think you have a plain-text session, and those codes will corrupt your data extraction if you do not strip them. After cleaning, split on newlines and discard the first line, which is typically the command echo coming back from the PTY.

Loop Larry bewildered by ANSI escape codes on a terminal screen
\x1B[32m means green. Your parser disagrees.

Handling Timeouts and EOF Gracefully

Robust pexpect scripts always handle both pexpect.TIMEOUT and pexpect.EOF explicitly. A timeout means the expected output never arrived; EOF means the process exited (normally or due to a crash). Letting these propagate as unhandled exceptions causes your automation to die silently with no useful error message:

# timeout_eof.py
import pexpect

def run_with_timeout(command, prompt_pattern, commands_to_send):
    """Run an interactive command, send inputs, handle errors gracefully."""
    try:
        child = pexpect.spawn(command, encoding='utf-8', timeout=10)
        child.logfile_read = open('session.log', 'w')   # log everything to a file

        child.expect(prompt_pattern)

        for cmd in commands_to_send:
            child.sendline(cmd)
            result = child.expect([prompt_pattern, pexpect.EOF, pexpect.TIMEOUT])
            if result == 1:
                print(f"Process exited after: {cmd!r}")
                return child.before.strip()
            elif result == 2:
                print(f"Timeout waiting for prompt after: {cmd!r}")
                child.close(force=True)
                return None

        return child.before.strip()

    except pexpect.ExceptionPexpect as e:
        print(f"pexpect error: {e}")
        return None
    finally:
        if 'child' in dir() and child.isalive():
            child.close()
        if 'child' in dir() and hasattr(child, 'logfile_read'):
            child.logfile_read.close()

result = run_with_timeout('python3', '>>>', ['2 ** 32', 'exit()'])
print("Output:", result)

Output:

Process exited after: 'exit()'
Output: 4294967296

The child.logfile_read attribute is one of pexpect’s most useful debugging features. Assign any writable file-like object to it and pexpect will write everything it reads from the child process into that file. When something goes wrong in a long automation script, the log file shows you exactly what the child said and where your expected pattern failed to match. Always add logging before going into production with a pexpect script; debugging without it is painful.

Simple Automation with pexpect.run()

For cases where you do not need to interact with the output — you just need to supply a series of fixed responses to fixed prompts — pexpect.run() is a one-liner alternative to the full spawn/expect/sendline loop. You pass it the command and an events dictionary mapping expected patterns to responses:

# pexpect_run.py
import pexpect

# Automate a script that asks two questions before running
# (simulating any interactive prompt you do not control)
output = pexpect.run(
    'python3 -c "'
    'name = input(\'Your name: \'); '
    'age = input(\'Your age: \'); '
    'print(f\'Hello {name}, you are {age} years old.\')"',
    events={
        'Your name: ': 'Alice\n',
        'Your age: ': '30\n',
    },
    encoding='utf-8',
    timeout=5,
)

print(output.strip())

Output:

Your name: Alice
Your age: 30
Hello Alice, you are 30 years old.

pexpect.run() returns the entire session output as a single string. It is much simpler than spawning and looping, but it does not let you branch on what the process says or handle errors gracefully. Use it for simple, predictable scripts where you know exactly what prompts will appear and in what order. Use the full spawn() approach when the conversation can take multiple paths or when you need to extract specific values from the output.

Sudo Sam handing a checklist clipboard to robots waiting at terminal prompts
pexpect.run() — when the script is boring enough to be predictable.

Automating SSH Sessions with pexpect.pxssh

pexpect ships with a higher-level module called pxssh specifically for SSH automation. It handles the login sequence, host-key verification prompts, and password entry for you, leaving you to just send commands and read output:

# pxssh_example.py
from pexpect import pxssh

def run_remote_commands(hostname, username, password, commands):
    """Log in to a remote host via SSH and run a list of commands."""
    results = {}
    s = pxssh.pxssh(timeout=30)

    try:
        s.login(hostname, username, password)
        print(f"Logged in to {hostname}")

        for cmd in commands:
            s.sendline(cmd)
            s.prompt()   # waits for the shell prompt
            output = s.before.decode() if isinstance(s.before, bytes) else s.before
            results[cmd] = output.strip()

        s.logout()
    except pxssh.ExceptionPxssh as e:
        print(f"SSH session failed: {e}")

    return results

# Example usage with a local test server (replace with your host details)
# results = run_remote_commands('192.168.1.10', 'admin', 'secret', ['uptime', 'free -h'])
# for cmd, output in results.items():
#     print(f"\n--- {cmd} ---")
#     print(output)
print("pxssh module imported successfully -- ready for SSH automation.")

Output:

pxssh module imported successfully -- ready for SSH automation.

The s.prompt() method is the pxssh equivalent of child.expect(prompt_pattern) — it waits for the shell prompt automatically, detecting it from the environment variables of the remote session. This is more reliable than writing your own prompt regex because shells vary (bash, zsh, sh all look different). After each s.prompt() call, s.before contains the command output. Always call s.logout() when done — it sends the exit command cleanly and closes the connection rather than dropping it.

Real-Life Example: Automated System Health Check Script

Here is a realistic automation script that connects to a local shell, runs four diagnostic commands, parses the output, and produces a structured health report. This pattern is common in DevOps scripts where you need to collect system state without writing a full Ansible playbook:

Debug Dee flipping a server indicator from red to green on a monitoring dashboard
Health check passed. No humans harmed in the making of this report.
# system_health_check.py
import pexpect
import re
import json
from datetime import datetime

def strip_ansi(text):
    """Remove ANSI terminal escape codes."""
    return re.sub(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])', '', text)

def run_check(child, command, prompt_re):
    """Send a command and return its clean output."""
    child.sendline(command)
    child.expect(prompt_re)
    raw = child.before
    return strip_ansi(raw if isinstance(raw, str) else raw.decode()).strip()

def get_system_health():
    """Collect system health metrics via an interactive bash session."""
    health = {
        "timestamp": datetime.now().isoformat(),
        "checks": {}
    }

    child = pexpect.spawn('bash', encoding='utf-8', timeout=15)
    child.expect(r'\$')

    # Disable the prompt PS1 to make it predictable
    child.sendline('export PS1="PROMPT> "')
    child.expect('PROMPT> ')

    prompt_re = 'PROMPT> '

    # 1. Disk usage
    raw = run_check(child, 'df -h / | tail -1', prompt_re)
    parts = raw.split()
    if len(parts) >= 5:
        health["checks"]["disk"] = {
            "total": parts[1],
            "used": parts[2],
            "available": parts[3],
            "use_pct": parts[4],
        }

    # 2. Memory
    raw = run_check(child, "free -h | grep Mem", prompt_re)
    parts = raw.split()
    if len(parts) >= 4:
        health["checks"]["memory"] = {
            "total": parts[1],
            "used": parts[2],
            "free": parts[3],
        }

    # 3. Load average
    raw = run_check(child, 'cat /proc/loadavg', prompt_re)
    load_parts = raw.split()[:3]
    if load_parts:
        health["checks"]["load"] = {
            "1min": load_parts[0],
            "5min": load_parts[1] if len(load_parts) > 1 else "n/a",
            "15min": load_parts[2] if len(load_parts) > 2 else "n/a",
        }

    # 4. Uptime
    raw = run_check(child, 'uptime -p 2>/dev/null || uptime', prompt_re)
    health["checks"]["uptime"] = raw.strip()

    child.sendline('exit')
    child.expect(pexpect.EOF)

    return health

report = get_system_health()
print(json.dumps(report, indent=2))

Output:

{
  "timestamp": "2026-06-12T07:14:32.441820",
  "checks": {
    "disk": {
      "total": "50G",
      "used": "18G",
      "available": "30G",
      "use_pct": "38%"
    },
    "memory": {
      "total": "15G",
      "used": "4.2G",
      "free": "8.1G"
    },
    "load": {
      "1min": "0.12",
      "5min": "0.08",
      "15min": "0.05"
    },
    "uptime": "up 3 days, 7 hours, 22 minutes"
  }
}

The key technique here is overriding PS1 at the start of the session to give the shell a predictable, unique prompt string. This eliminates the most common source of pexpect failures: prompt regex patterns that match too broadly or not at all because the shell’s default prompt varies by system or includes dynamic content like the current directory. Setting PS1 to a fixed string at session start makes every subsequent child.expect(prompt_re) call reliable. You can extend this script to run against remote hosts by replacing the bash spawn with a pxssh session using the same command-running pattern.

Frequently Asked Questions

Does pexpect work on Windows?

Not natively. pexpect requires a Unix PTY subsystem, which is not available in standard Windows. On Windows you have three options: use Windows Subsystem for Linux (WSL), run your script in a Docker container, or use the wexpect package (a Windows-specific fork). Most teams in practice use WSL or Docker because they match the Linux environment where the target programs usually run anyway. On macOS and Linux, pexpect works out of the box.

Why not just use subprocess with stdin=PIPE?

subprocess with stdin=PIPE works great for programs that read all their input before producing output — but many interactive programs check whether they are attached to a real terminal and change their behavior if they are not. They may disable their prompts, buffer output differently, or refuse to accept input at all. pexpect avoids this by using a real PTY, so the child process believes it is talking to a human terminal. If subprocess works for your use case, use it — it is simpler. Reach for pexpect when subprocess just does not cooperate.

What timeout value should I use?

Start with a timeout 3x to 5x longer than the slowest operation you expect. For local commands, 5-10 seconds is usually enough. For network operations like SSH or FTP, 15-30 seconds gives headroom for slow connections. For long-running jobs (database exports, compilation), set the timeout to the maximum acceptable wall-clock time and log carefully so you can distinguish a legitimate timeout from a stuck process. You can set a global timeout in pexpect.spawn() and override it for specific expect() calls where you know an operation might take longer.

Is it safe to automate password entry with pexpect?

pexpect itself is safe for password automation — it sends passwords directly to the child process’s PTY and does not log them unless you explicitly set child.logfile. The risk is in how you supply the password to your script. Never hardcode passwords as string literals in source code. Instead, read them from environment variables (os.environ['MY_PASSWORD']), a secrets manager (AWS Secrets Manager, HashiCorp Vault), or a local keyring. If you are automating SSH, using key-based authentication and ssh-agent is always preferable to password automation.

Why is child.before full of escape codes and garbled characters?

This is normal — the PTY causes the child process to emit ANSI terminal escape sequences for colors, cursor positioning, and other formatting. Use the strip_ansi() helper shown earlier in this article to remove them. You can also try setting the terminal type to a minimal value before spawning: child = pexpect.spawn('your_command', env={**os.environ, 'TERM': 'dumb'}) which disables color output in many programs. The TERM=dumb trick works for bash, Python, and most well-behaved CLI tools.

How do I debug a pexpect script that is not matching?

Set child.logfile_read = sys.stdout to see exactly what the child process is sending to your script in real time. The most common cause of a failed match is that the actual output contains ANSI escape codes or extra whitespace that your pattern does not account for. Print repr(child.before) after a timeout to see the raw bytes (or string) that did arrive, and adjust your pattern to match what you see. Using regex patterns with re.compile(r'pattern', re.MULTILINE) gives you more flexibility than exact string matching when prompts vary slightly.

Conclusion

pexpect gives you a clean Python API for automating any interactive terminal program — from local shells and calculators to remote SSH sessions and FTP servers. The core workflow is always the same: spawn the child process, use expect() to wait for the right moment, use sendline() to respond, and capture output from child.before. The key practices that separate reliable automation from fragile scripts are setting a predictable PS1 prompt, always handling TIMEOUT and EOF in your expect lists, stripping ANSI escape codes before parsing output, and using logfile_read during development to see exactly what the child is saying.

The real-life health check example in this article is a good starting point for your own scripts. Try extending it with pxssh to run the same checks across multiple remote hosts and aggregate the results into a single JSON report. From there, wrapping the checks in a retry loop with exponential backoff gives you a production-grade monitoring script in under 100 lines of Python.

For the full pexpect API reference, including advanced features like interact() (which hands control back to the human keyboard temporarily) and expect_exact() (faster plain-string matching without regex overhead), see the official pexpect documentation.

How To Use Python memray for Memory Profiling

How To Use Python memray for Memory Profiling

Last Updated: June 11, 2026

Intermediate

Your Python service passes all tests and runs fine in development — then hits production and balloons to 4 GB of RAM. You restart it, it climbs again, and now you have a memory leak you cannot locate. You add a few print(sys.getsizeof(...)) calls, but they only measure individual objects, not the full allocation tree. You try the standard tracemalloc module and get a list of the top 10 allocations with no call-stack context. The problem could be anywhere across 50 modules and 300 functions.

memray is a memory profiler for Python developed by Bloomberg Engineering. It instruments every memory allocation and deallocation in your program — including C extensions and native code — and records a full call stack for each one. After a run you get a flame graph showing exactly which call path is responsible for each byte: not just which object, but which function called which function that eventually caused the allocation. It supports command-line profiling, pytest integration, and a live tracking mode so you can watch allocations happen in real time. Install it with pip install memray.

In this article we will cover how memray works and how it differs from tracemalloc, how to profile a script from the command line, how to read the flame graph and table reports, how to track live allocations, how to use the pytest-memray plugin to add memory limits to tests, how to profile specific code blocks with the Python API, and how to interpret results to find and fix real leaks. By the end you will have a complete toolkit for diagnosing memory problems in any Python application.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program.

memray Quick Example: Finding a Memory Hog in 30 Seconds

Here is the minimum setup to profile a script and open the flame graph. First, create a script that has an obvious memory problem:

# leaky_script.py
def load_big_list():
    return [i * 2 for i in range(5_000_000)]

def process(data):
    # Creates a second copy -- doubles memory
    return [str(x) for x in data]

if __name__ == "__main__":
    nums = load_big_list()
    strs = process(nums)
    print(f"Processed {len(strs)} items")

Run it under memray from the terminal:

# run_memray.sh
python -m memray run leaky_script.py
Writing profile results into memray-leaky_script.py.1234.bin
[memray] Successfully generated profile results.

 Run id: 1234
 Command line: leaky_script.py
 Start time: 2026-06-11 09:00:00.000
 End time: 2026-06-11 09:00:02.345
 Duration: 2.345 seconds
 Total allocations: 5,000,132
 Total memory allocated: 312.4 MB
 Peak memory usage: 271.2 MB

Then generate the flame graph report and open it in a browser:

# generate_flamegraph.sh
python -m memray flamegraph memray-leaky_script.py.1234.bin
Wrote flamegraph to memray-flamegraph-leaky_script.py.1234.html

The flame graph shows two large bars: load_big_list responsible for ~150 MB (the integer list) and process responsible for ~162 MB (the string list). The call path is clear — both functions allocate heavily and neither releases until the script exits. The sections below cover the full memray toolkit.

What Is memray and How Does It Work?

memray is a deterministic memory profiler. Unlike sampling profilers that check memory usage at intervals, memray intercepts every call to the Python allocator (and to malloc/free in C extensions) and records the exact call stack at the moment of each allocation. This gives complete, lossless data rather than a statistical sample.

Python already ships with tracemalloc, which also tracks allocations. The key differences are scope and output. tracemalloc only tracks Python-level allocations and presents flat top-N lists. memray tracks both Python and native (C) allocations, records the full call chain, and produces interactive flame graphs, timeline views, and summary tables that show not just what allocated memory but the entire path through your code that led to the allocation.

Featurememraytracemallocmemory_profiler
Tracks C extension allocationsYesNoNo
Full call stack per allocationYesPartialNo
Flame graph outputYes (HTML)NoNo
Live tracking modeYesNoNo
pytest integrationYes (pytest-memray)NoNo
Performance overheadModerate (2-5x)LowHigh (line-by-line)
Platform supportLinux, macOSAllAll

memray is built on Linux’s LD_PRELOAD mechanism and macOS’s interpose feature to hook the allocator at the C level. This is why it works on Linux and macOS but not Windows. It writes a compact binary trace file that you then convert to reports using the memray CLI.

Python developer inspecting memory allocation blocks with magnifying glass
tracemalloc showed the symptom. memray shows the crime scene.

Installing memray

Install memray with pip. It requires Python 3.7+ and works on Linux and macOS (not Windows):

# install_memray.sh
pip install memray
Successfully installed memray-1.13.0

Verify the installation:

# verify_memray.sh
python -m memray --version
memray, version 1.13.0

To use the pytest integration, also install the plugin:

# install_pytest_memray.sh
pip install pytest-memray
Successfully installed pytest-memray-1.6.0

Command-Line Profiling

The simplest way to profile a script is the memray run subcommand. It runs your script and writes a binary trace to a .bin file in the current directory:

# profile_script.sh
python -m memray run my_script.py
Writing profile results into memray-my_script.py.9821.bin

You can also profile a module with -m, exactly like running Python normally:

# profile_module.sh
python -m memray run -m pytest tests/
Writing profile results into memray-run-pytest.9955.bin

Key flags for memray run:

# memray_run_flags.sh

# Custom output file
python -m memray run -o my_profile.bin leaky_script.py

# Track only native (C) allocations as well
python -m memray run --native leaky_script.py

# Compress the output file
python -m memray run --compress-on-exit leaky_script.py

# Set a specific memory limit (kills process if exceeded)
python -m memray run --memory-limit 500MB leaky_script.py

The --native flag adds C extension allocations to the trace. Use it when you suspect NumPy, Pandas, or other native extensions are the source of a leak — without it, memray only sees Python-level allocations from those libraries.

Python developer watching terminal memory profiling output
memray run. Then coffee. Then answers.

Reading the Reports: Flame Graph and Stats Table

memray produces several report types from the same .bin trace file. The flame graph is the most useful for finding the source of large allocations:

# generate_reports.sh

# Flame graph (HTML -- open in browser)
python -m memray flamegraph memray-leaky_script.py.1234.bin
# Output: memray-flamegraph-leaky_script.py.1234.html

# Summary table (terminal output)
python -m memray stats memray-leaky_script.py.1234.bin

# Tree view (allocations as a call tree)
python -m memray tree memray-leaky_script.py.1234.bin

The stats command gives a quick summary directly in the terminal without opening a browser:

# stats_output.txt

---- Top 10 allocations by size ----

1) size=162.4 MB, allocated in process (leaky_script.py:7)
       -> process (leaky_script.py:8)

2) size=150.1 MB, allocated in load_big_list (leaky_script.py:2)
       -> __main__ (leaky_script.py:11)

3) size=1.2 MB, allocated in _bootstrap (importlib._bootstrap:1)
       -> ...

Total allocations: 5,000,132
Total memory allocated: 312.4 MB
Peak memory: 271.2 MB

In the flame graph HTML, each box represents a function. The width of the box is proportional to the amount of memory allocated by that function and all its callees. Click a box to zoom in. The call path reads top-to-bottom: the widest box at the bottom is usually your entry point (__main__), and the widest box at the top is the function doing the most allocation. Use the “Show only allocations” toggle to filter out functions that only pass memory through without allocating.

Live Tracking Mode

Instead of profiling a full run and analyzing afterward, live mode streams allocations to a terminal UI in real time. This is especially useful for long-running servers or scripts where you want to watch memory grow and correlate it with specific operations:

# live_tracking.sh
python -m memray run --live leaky_script.py
   Allocation            Location                              Size        Count
---------------------------------------------------------------------------
              leaky_script.py:2                  148.2 MB    5,000,000
              leaky_script.py:7                  122.1 MB    3,892,451
   list_to_str          leaky_script.py:8                    8.4 MB      108,441
   ...

 Peak memory: 271.2 MB     Current: 249.8 MB     [q]uit  [r]eset

The live view updates every 0.1 seconds. Press q to stop the run early. The --live-port flag lets you connect a second terminal to the same live stream, which is useful for profiling a server process without interrupting it:

# live_remote.sh

# In terminal 1 -- start the server with live tracking
python -m memray run --live-port 5001 server.py

# In terminal 2 -- attach the viewer
python -m memray live 5001
Python developer watching live memory tracking dashboard
–live mode. Because grep-ing production logs is a last resort.

Using the Python API for Targeted Profiling

If you only want to profile a specific section of a larger application — not the whole program — use memray’s Python context manager. This avoids noise from startup, shutdown, and unrelated code paths:

# targeted_profiling.py
import memray

def build_index(documents):
    """Build an inverted index from a list of documents."""
    index = {}
    for doc_id, text in enumerate(documents):
        for word in text.lower().split():
            if word not in index:
                index[word] = []
            index[word].append(doc_id)
    return index

def search(index, query):
    """Return document IDs matching all query terms."""
    terms = query.lower().split()
    results = set(index.get(terms[0], []))
    for term in terms[1:]:
        results &= set(index.get(term, []))
    return list(results)

# Sample data
docs = [
    "Python memory profiling with memray",
    "How to find memory leaks in Python",
    "memray flame graph tutorial",
] * 10_000

# Profile only the index build -- not the search or data setup
with memray.Tracker("index_build.bin"):
    index = build_index(docs)

# Analysis later:
# python -m memray flamegraph index_build.bin
print(f"Index built: {len(index)} unique terms")
print(f"Search results: {search(index, 'python memory')}")
Index built: 12 unique terms
Search results: [0, 1, 2, 3, ...]

The memray.Tracker context manager starts recording on entry and writes the .bin file on exit. You can also add native=True to catch C allocations: memray.Tracker("profile.bin", native=True). Use the targeted profiling approach in production services where you cannot afford to instrument the entire process — wrap only the suspicious function or request handler.

Testing Memory Usage with pytest-memray

pytest-memray integrates memray into your test suite. Run your tests with memory profiling and optionally enforce per-test memory limits:

# test_memory_limits.py
import pytest

def build_report(n_rows):
    """Build a report dict with n_rows entries."""
    return {f"row_{i}": {"value": i, "label": f"Item {i}"} for i in range(n_rows)}

# This test will fail if it allocates more than 50 MB
@pytest.mark.limit_memory("50 MB")
def test_small_report_memory():
    report = build_report(100_000)
    assert len(report) == 100_000

# This test passes -- no limit set, just profiled
def test_large_report_memory():
    report = build_report(1_000_000)
    assert len(report) == 1_000_000

Run with the --memray flag to enable profiling:

# run_memray_tests.sh
pytest tests/test_memory_limits.py --memray
FAILED test_memory_limits.py::test_small_report_memory - Failed: Test was
limited to 50.0MB but allocated 89.3MB

PASSED test_memory_limits.py::test_large_report_memory
 - Total memory allocated: 421.7MB

========== 1 failed, 1 passed in 2.34s ==========

The @pytest.mark.limit_memory("50 MB") decorator sets a hard ceiling. If the test allocates more than the limit, it fails with a clear message showing the actual allocation. Add this marker to any function that processes large data structures in a tight loop — it turns memory regressions into CI failures instead of production surprises. You can also pass --memray-bin-path=./profiles/ to save trace files from all tests for post-run analysis.

Python developer checking failed CI memory limit test results
@pytest.mark.limit_memory. Because ‘it worked fine locally’ ends here.

Identifying and Fixing Memory Leaks

A genuine memory leak in Python is usually one of three things: a growing container that is never cleared, a reference cycle that the garbage collector cannot break (often involving __del__ methods), or a native extension that leaks memory at the C level. memray’s flame graph makes all three visible.

Here is a realistic example of a container-based leak and how memray exposes it:

# cache_leak.py
import memray

# Global cache that is never evicted
_query_cache = {}

def expensive_query(key):
    """Simulates a database query with a result cache."""
    if key not in _query_cache:
        # Caches a 10KB result for every unique key -- forever
        _query_cache[key] = b"x" * 10_240
    return _query_cache[key]

def handle_requests(n):
    """Simulates n incoming requests with unique keys."""
    for i in range(n):
        result = expensive_query(f"user:{i}:profile")
    return len(_query_cache)

with memray.Tracker("cache_leak.bin"):
    total = handle_requests(5_000)

print(f"Cache size after run: {total} entries")
# python -m memray flamegraph cache_leak.bin
# Flame graph will show _query_cache holding ~50 MB with no deallocation path
Cache size after run: 5000 entries

When you open the flame graph, expensive_query will show a wide bar with a path leading to dict.__setitem__ — the cache assignment. Since there is no eviction, all 50 MB stays live until the process exits. The fix is to bound the cache with functools.lru_cache or cachetools.LRUCache. After fixing, run the same profile and verify the peak memory drops dramatically.

Real-Life Example: Profiling a Data Processing Pipeline

Here is a realistic data pipeline that reads a large dataset and produces an aggregate report. We will use memray to identify which stage uses the most memory and then refactor to reduce the peak.

# data_pipeline.py
import memray
import csv
import io
import random

# --- Generate sample CSV data in memory ---
def make_sample_csv(n_rows=500_000):
    buf = io.StringIO()
    writer = csv.writer(buf)
    writer.writerow(["user_id", "product_id", "amount", "category"])
    categories = ["electronics", "clothing", "food", "books", "sports"]
    for i in range(n_rows):
        writer.writerow([
            f"user_{i % 10_000}",
            f"prod_{random.randint(1, 1000)}",
            round(random.uniform(1, 500), 2),
            random.choice(categories),
        ])
    buf.seek(0)
    return buf

# --- Stage 1: Load everything into memory (naive approach) ---
def load_all_rows(csv_buf):
    reader = csv.DictReader(csv_buf)
    return list(reader)   # entire dataset in a list of dicts

# --- Stage 2: Aggregate totals by category ---
def aggregate_by_category(rows):
    totals = {}
    for row in rows:
        cat = row["category"]
        amt = float(row["amount"])
        if cat not in totals:
            totals[cat] = {"count": 0, "total": 0.0}
        totals[cat]["count"] += 1
        totals[cat]["total"] += amt
    return totals

with memray.Tracker("pipeline.bin"):
    csv_data = make_sample_csv()
    rows = load_all_rows(csv_data)          # Stage 1
    report = aggregate_by_category(rows)    # Stage 2

for cat, stats in sorted(report.items()):
    avg = stats["total"] / stats["count"]
    print(f"{cat:12s}: {stats['count']:6d} orders, avg ${avg:.2f}")
books       :  99812 orders, avg $250.33
clothing    : 100203 orders, avg $249.88
electronics :  99876 orders, avg $250.14
food        : 100442 orders, avg $249.71
sports      :  99667 orders, avg $249.92

Running python -m memray flamegraph pipeline.bin will show load_all_rows responsible for the majority of peak memory — all 500,000 rows are held in a list of dicts at the same time. The fix is to stream the CSV row-by-row instead of loading it all at once. Replace load_all_rows with a streaming aggregator and the peak memory drops from ~200 MB to ~2 MB, because only one row is ever in memory at a time. This is the memray workflow in practice: profile, identify the stage, refactor, re-profile to confirm the improvement.

Two data pipeline approaches compared -- bulk load vs streaming
list(reader) vs. for row in reader. One of these is a 200 MB decision.

Frequently Asked Questions

Does memray work on Windows?

No. memray uses Linux’s LD_PRELOAD and macOS’s interpose mechanism to hook the allocator, neither of which exists on Windows. On Windows, consider using tracemalloc for Python allocations or Fil (also open-source) if you need C extension tracking. If you develop on Windows and deploy to Linux, you can run memray via WSL2 or in a Docker container for profiling purposes while keeping your main development on Windows.

How much does memray slow down my code?

Expect 2x to 5x slowdown in programs that allocate heavily. Programs that allocate infrequently (mostly numeric computation on pre-allocated arrays) may see only 10-20% overhead. memray is not designed for production use — run it in a staging or development environment. If you need in-production memory monitoring, use a metrics approach (periodic psutil.Process().memory_info().rss readings) rather than a deterministic profiler.

Does memray track garbage-collected objects?

memray tracks allocations and deallocations at the allocator level, which includes objects collected by Python’s cyclic garbage collector. When gc.collect() frees a cycle, memray records those deallocations. You can see “temporary” allocations (objects allocated and freed within the profiled window) by using the --show-temporary-allocations flag with the flame graph command. This is useful for diagnosing churn — code that creates and throws away millions of short-lived objects, driving CPU time in the allocator even if peak memory looks normal.

Does memray work with async code and FastAPI/aiohttp?

Yes. Since memray hooks the allocator at the C level, it is transparent to Python’s async machinery. Wrap your ASGI/WSGI app with memray.Tracker for a fixed profiling window, or use memray run --live to watch allocations as requests come in. For per-request profiling in FastAPI, add a middleware that starts a Tracker context at request start and stops it at response end, writing one trace file per request to a temp directory.

Why does memray show less memory than Task Manager for my NumPy script?

NumPy allocates memory through its own internal pools which may not map 1:1 to Python allocator calls. Use the --native flag (python -m memray run --native script.py) to also track C-level allocations including NumPy’s internal pools. Without --native, memray only sees the Python-side wrapper objects, which are much smaller than the actual array data stored in native memory.

Can I use memray inside Docker?

Yes, with one requirement: the container must have SYS_PTRACE capability to allow native tracing. Add --cap-add SYS_PTRACE to your docker run command or add cap_add: [SYS_PTRACE] to your docker-compose.yml service. If you only need Python-level profiling (not --native), the capability is not required. For Kubernetes deployments, add capabilities.add: ["SYS_PTRACE"] to the container’s securityContext.

Conclusion

memray turns memory debugging from a guessing game into a structured investigation. Run python -m memray run script.py to capture the full allocation trace, generate a flame graph with python -m memray flamegraph *.bin, and follow the widest call paths down to the function doing the actual allocating. The Python API’s memray.Tracker context manager lets you surgically profile one subsystem without the noise of a full run, and pytest-memray prevents memory regressions from reaching production by turning allocation spikes into CI failures.

The real-life pipeline example shows the workflow end to end: profile, read the flame graph, refactor the offending stage, re-profile to confirm the improvement. Try extending it by adding a streaming version of load_all_rows using a generator, re-running the profile, and comparing the two flame graphs side by side. The official documentation at bloomberg.github.io/memray covers advanced topics including attaching to running processes, the timeline view, and custom reporters.

How To Use Python beartype for Runtime Type Checking

How To Use Python beartype for Runtime Type Checking

Last Updated: June 10, 2026

Intermediate

You write a function that expects a list[str], add a type hint, and feel good about it. Six months later a colleague passes in a list of integers, Python happily accepts it, and the bug surfaces three function calls downstream as an AttributeError that points nowhere near the real problem. Static type checkers like mypy would have caught this — if the entire codebase uses them consistently, if CI runs them, and if the calling code is also type-annotated. In practice, those conditions often don’t hold. beartype fills the gap by enforcing type hints at runtime, at the exact moment the function is called.

beartype is a pure-Python library that decorates your functions and checks argument types in real time. Unlike mypy, it doesn’t require a separate analysis pass or a clean type-annotated codebase. You add one decorator, and the next time someone passes the wrong type, you get a clear BeartypeException at the call site — not a cryptic error three layers down. It supports the full range of Python type hints including Optional, Union, list[str], dict[str, int], dataclasses, and even complex generics. Install it with pip install beartype.

In this article we will cover how beartype works and why it is faster than competing libraries, how to decorate functions and methods, how to handle complex nested types, how to configure beartype with BeartypeConf, how to apply it project-wide with a single import, and how to use it alongside static type checkers. By the end you will have a practical toolkit for catching type violations at runtime in any Python project.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program.

beartype Quick Example: Catching a Type Violation at Call Time

Here is the minimum setup that demonstrates what beartype does — decorate a function, call it with the wrong type, and watch beartype raise an informative error immediately:

# quick_beartype.py
from beartype import beartype

@beartype
def greet(name: str, repeat: int) -> str:
    return (name + " ") * repeat

# Correct call -- works fine
print(greet("Hello", 3))

# Wrong type for repeat -- beartype raises immediately
print(greet("Hello", "3"))

Output:

Hello Hello Hello
BeartypeCallHintParamViolation: @beartyped greet() parameter repeat='3' violates type hint
<class 'int'>, as str '3' not instance of int.

The decorator wraps greet() with a thin type-checking layer. Correct calls pass through at near-native speed. Wrong-type calls raise BeartypeCallHintParamViolation with an error message that names the function, the parameter, the value passed, and the expected type. No configuration needed — just add @beartype. The sections below cover the full feature set.

What Is beartype and Why Use It?

beartype is a runtime type-checking library for Python. It reads the type annotations you have already written and enforces them when your code runs. The core idea is that type hints in Python are, by default, documentation — the interpreter ignores them completely. beartype turns them into actual enforcement without requiring you to write manual isinstance() checks everywhere.

The key design choice that sets beartype apart from similar libraries is performance. Most runtime type checkers perform deep, recursive checks: if you annotate a parameter as list[str], they walk every element of the list to confirm it is a string. For a list with 10,000 items this adds up quickly. beartype uses O(1) amortized checking — it samples a random element rather than walking the whole structure, giving you statistical confidence without linear overhead. The library itself claims this makes it the fastest pure-Python runtime type checker available.

Featurebeartypetypeguardpydantic (validation)
Check speedO(1) amortizedO(n) deep checkO(n) full parsing
Decorator style@beartype@typechecked@validate_call
Project-wide modeYes (BeartypeConf + importlib)LimitedNo
Supports genericsFull PEP 484/585/604FullFull (own system)
Zero-dep installYesYesNo (Rust extension)

beartype suits projects that already have type hints and want runtime enforcement as a safety net — especially in library code that receives external input, plugin systems, or long-running services where a type error would otherwise surface far from the call site.

Runtime type checking stopping a wrong-type call
isinstance() on line 47 says otherwise.

Installing beartype

beartype has no dependencies beyond the Python standard library. Install it with pip into your project environment:

# install_beartype.sh
pip install beartype

Output:

Successfully installed beartype-0.18.5

Verify the installation and check the version:

# verify_beartype.py
import beartype
print(beartype.__version__)

Output:

0.18.5

beartype supports Python 3.8 and above. It has no C extensions, so it installs instantly on any platform including PyPy and Anaconda environments.

Decorating Functions and Methods

The primary interface is the @beartype decorator. Apply it to any function or method that has type annotations, and beartype will check all annotated parameters and the return value on every call.

# function_decorating.py
from beartype import beartype
from typing import Optional

@beartype
def send_email(to: str, subject: str, body: str, cc: Optional[str] = None) -> bool:
    """Send an email and return True on success."""
    print(f"Sending to {to}: {subject}")
    if cc:
        print(f"CC: {cc}")
    return True

# All valid -- passes through
send_email("alice@example.com", "Hello", "Hi there")
send_email("bob@example.com", "Update", "See attached", cc="carol@example.com")

# Invalid -- cc must be str or None, not int
send_email("dave@example.com", "Oops", "Body", cc=42)

Output:

Sending to alice@example.com: Hello
Sending to bob@example.com: Update
CC: carol@example.com
BeartypeCallHintParamViolation: @beartyped send_email() parameter cc=42 violates type hint
typing.Optional[str], as int 42 not instance of str.

beartype understands Optional[str] (which is equivalent to Union[str, None]) and correctly accepts None as a valid value while rejecting integers. The decorator adds no overhead when arguments are the correct type — the check for a simple str or int parameter costs roughly the same as a single isinstance() call.

You can also decorate class methods. beartype handles self automatically — it skips the first parameter if it has no annotation, which is the standard convention for instance methods:

# class_methods.py
from beartype import beartype

class DataLoader:
    @beartype
    def load(self, path: str, encoding: str = "utf-8") -> list:
        """Load lines from a file."""
        with open(path, encoding=encoding) as f:
            return f.readlines()

    @beartype
    def count_lines(self, lines: list, min_length: int = 0) -> int:
        return sum(1 for line in lines if len(line) >= min_length)

loader = DataLoader()
lines = loader.load("data.txt")          # str path -- fine
loader.count_lines(lines, min_length=5)  # int min_length -- fine
loader.count_lines(lines, min_length="5") # str instead of int -- fails

Output:

BeartypeCallHintParamViolation: @beartyped DataLoader.count_lines() parameter
min_length='5' violates type hint <class 'int'>, as str '5' not instance of int.

The fully qualified method name (DataLoader.count_lines) appears in the error, which makes it easy to locate in large codebases. Parameters with default values are still checked — beartype verifies the type at call time regardless of whether the caller supplies the argument or the default is used.

beartype checking type annotations on a call stack
BeartypeCallHintParamViolation: caught at line 1, not line 347.

Checking Complex and Generic Types

beartype supports the full range of Python type hints introduced across PEP 484, 585, and 604. This includes parameterized generics like list[str], dict[str, int], tuple[int, ...], and set[float], as well as Union, Optional, Callable, and custom Protocol classes. For collection types, beartype uses its O(1) strategy — it samples one random element rather than checking every item, giving you fast probabilistic coverage.

# complex_types.py
from beartype import beartype
from collections.abc import Callable

@beartype
def process_records(
    records: list[dict[str, int]],
    transform: Callable[[int], int],
    max_value: int | None = None,
) -> list[int]:
    """Apply transform to each record's values."""
    results = []
    for record in records:
        for val in record.values():
            transformed = transform(val)
            if max_value is None or transformed <= max_value:
                results.append(transformed)
    return results

data = [{"a": 1, "b": 2}, {"c": 3}]
doubled = process_records(data, lambda x: x * 2)
print(doubled)

# Pass a string in the list -- beartype detects the wrong type
bad_data = [{"a": "not-an-int"}]
process_records(bad_data, lambda x: x * 2)

Output:

[2, 4, 6]
BeartypeCallHintParamViolation: @beartyped process_records() parameter records=[{'a': 'not-an-int'}]
violates type hint list[dict[str, int]], as dict key 'a' value 'not-an-int' not instance of int.

Notice the modern union syntax int | None (PEP 604, Python 3.10+) in place of Optional[int] -- beartype supports both styles. For the Callable type hint, beartype verifies that the argument is callable but does not check the callable's own signature (that would require calling it, which beartype intentionally avoids). The list[dict[str, int]] check samples a random element from the list, then samples a random key-value pair from that dict -- two O(1) checks that together cover the nested structure.

# union_types.py
from beartype import beartype
from typing import Union

@beartype
def parse_id(value: Union[str, int]) -> str:
    """Accept either a string or integer ID and return a string."""
    return str(value)

print(parse_id("abc-123"))  # str -- fine
print(parse_id(42))         # int -- fine
print(parse_id(3.14))       # float -- not in Union

Output:

abc-123
42
BeartypeCallHintParamViolation: @beartyped parse_id() parameter value=3.14 violates type hint
typing.Union[str, int], as float 3.14 not instance of str or int.

The error message correctly identifies that neither branch of the Union matched. For deeply nested unions or generics, beartype generates clear messages that show exactly which level of the type hierarchy failed.

Checking Return Values

beartype checks return type annotations just as it checks parameter annotations. If a function's return annotation says str but the function returns an integer, beartype raises BeartypeCallHintReturnViolation the moment the function exits. This is especially useful for catching bugs in functions that take multiple code paths -- only the path that returns the wrong type will fail, and you will know immediately which function is responsible.

# return_checking.py
from beartype import beartype

@beartype
def get_status_code(success: bool) -> int:
    """Return an HTTP-style status code."""
    if success:
        return 200
    return "error"   # Bug: should return 500, not a string

print(get_status_code(True))   # Returns 200 -- fine
print(get_status_code(False))  # Returns "error" -- beartype catches it

Output:

200
BeartypeCallHintReturnViolation: @beartyped get_status_code() return 'error' violates type hint
<class 'int'>, as str 'error' not instance of int.

This kind of bug -- a function returning the wrong type from one of its branches -- is common in older Python codebases that were written before type hints and then annotated later without fully verifying every code path. beartype will catch it the first time that branch runs in tests or production, rather than letting it propagate silently.

beartype catching a wrong return type
Wrong return type caught on the way out. Not three functions upstream.

Configuring beartype with BeartypeConf

The default @beartype decorator uses sensible defaults, but BeartypeConf lets you tune the behavior. The most useful option is is_debug=True, which prints the generated type-checking wrapper so you can see exactly what beartype is doing under the hood. Other options control whether to raise on violations, emit warnings instead, or enable strict mode.

# beartype_conf.py
from beartype import beartype, BeartypeConf

# Warning mode: print a warning but don't raise an exception
warn_conf = BeartypeConf(violation_type=UserWarning)

@beartype(conf=warn_conf)
def multiply(a: int, b: int) -> int:
    return a * b

import warnings
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    result = multiply(3, "4")   # Wrong type -- triggers warning, not exception
    print(f"Result: {result}")  # Still executes!
    if w:
        print(f"Warning: {w[0].message}")

Output:

Result: 333333333333333333333333333333333333333333333333333333333333333
Warning: @beartyped multiply() parameter b='4' violates type hint <class 'int'>,
as str '4' not instance of int.

Warning mode is useful during a gradual migration phase when you want to audit type violations in a running application without breaking it immediately. You can log the warnings, collect them over time, and fix the call sites before switching back to the default exception-raising mode. The violation_type parameter accepts any Warning subclass.

# debug_mode.py
from beartype import beartype, BeartypeConf

debug_conf = BeartypeConf(is_debug=True)

@beartype(conf=debug_conf)
def add(x: int, y: int) -> int:
    return x + y

Output (abbreviated):

# beartype generated wrapper (simplified):
def add(__beartype_object, x: int, y: int) -> int:
    if not isinstance(x, int): raise BeartypeCallHintParamViolation(...)
    if not isinstance(y, int): raise BeartypeCallHintParamViolation(...)
    __return = __beartype_object(x, y)
    if not isinstance(__return, int): raise BeartypeCallHintReturnViolation(...)
    return __return

The debug output shows exactly what beartype compiles into -- for simple types like int it is nearly identical to hand-written isinstance() checks. This transparency makes it easy to understand the performance impact and trust that the checks are doing what you expect.

Applying beartype Project-Wide

Decorating every function individually is tedious for large codebases. beartype provides a project-wide activation mechanism using Python's import system hook. Add a single import at the top of your entry point and beartype automatically decorates every function in every module you import afterward -- no per-function decorator needed.

# main.py  (entry point -- put this import first)
from beartype.claw import beartype_this_package

# Call this before importing anything else from your package
beartype_this_package()

# Now all functions in your_package are automatically @beartype-decorated
from your_package import process_data, load_config

# Type violations will be caught without any explicit @beartype decorators
load_config(path=42)   # int instead of str -- raises immediately

Output:

BeartypeCallHintParamViolation: @beartyped load_config() parameter path=42
violates type hint <class 'str'>, as int 42 not instance of str.

The beartype_this_package() call must come before the imports it should affect. beartype hooks into Python's importlib system and patches each module as it is loaded, so order matters. For packages you do not own (third-party libraries), use beartype_packages("your_package") to target a specific package name explicitly. This project-wide approach is the most practical way to add runtime type checking to an existing codebase in one step.

beartype applied project-wide via import hook
beartype_this_package() -- one line, every function covered.

Using beartype with Dataclasses

beartype integrates cleanly with Python dataclasses. Decorate the class with @beartype and it wraps the auto-generated __init__ method, so type violations are caught at instantiation time rather than hidden until the field is first used.

# beartype_dataclasses.py
from beartype import beartype
from dataclasses import dataclass

@beartype
@dataclass
class Product:
    name: str
    price: float
    quantity: int
    tags: list[str]

# Valid product -- all types match
p1 = Product(name="Widget", price=9.99, quantity=100, tags=["sale", "new"])
print(p1)

# Invalid: price should be float, not a string
p2 = Product(name="Gadget", price="free", quantity=10, tags=[])

Output:

Product(name='Widget', price=9.99, quantity=100, tags=['sale', 'new'])
BeartypeCallHintParamViolation: @beartyped Product.__init__() parameter price='free'
violates type hint <class 'float'>, as str 'free' not instance of float.

Order matters when stacking decorators: @beartype must go above @dataclass because Python applies decorators from bottom to top. With that order, beartype sees the fully formed dataclass (including the generated __init__) and wraps it correctly. If you reverse the order, beartype sees the raw class before __init__ is generated and the type checking will not apply to construction.

Real-Life Example: A Typed Configuration Loader

Here is a practical project that combines beartype's decorator and complex type hints to build a safe configuration loader. The loader reads a JSON config file and enforces the expected schema at parse time -- any malformed config value raises a type error immediately rather than causing a confusing failure later in the application.

# config_loader.py
import json
from pathlib import Path
from beartype import beartype
from beartype.typing import TypedDict

class DatabaseConfig(TypedDict):
    host: str
    port: int
    name: str

class AppConfig(TypedDict):
    debug: bool
    database: DatabaseConfig
    allowed_hosts: list[str]
    max_connections: int

@beartype
def load_config(path: str) -> AppConfig:
    """Load and validate application config from a JSON file."""
    config_path = Path(path)
    if not config_path.exists():
        raise FileNotFoundError(f"Config file not found: {path}")
    with config_path.open() as f:
        data = json.load(f)
    return data   # beartype validates the return type against AppConfig

@beartype
def get_db_url(config: AppConfig) -> str:
    """Build a database URL from the config."""
    db = config["database"]
    return f"postgresql://{db['host']}:{db['port']}/{db['name']}"

@beartype
def check_host(config: AppConfig, host: str) -> bool:
    """Check whether a host is in the allowed list."""
    return host in config["allowed_hosts"]

# --- Demo with a valid config ---
valid_config: AppConfig = {
    "debug": False,
    "database": {"host": "localhost", "port": 5432, "name": "mydb"},
    "allowed_hosts": ["localhost", "myapp.example.com"],
    "max_connections": 20,
}

print(get_db_url(valid_config))
print(check_host(valid_config, "localhost"))
print(check_host(valid_config, "attacker.com"))

# --- Demo: wrong type for port ---
bad_config = {
    "debug": False,
    "database": {"host": "localhost", "port": "5432", "name": "mydb"},  # port is str
    "allowed_hosts": ["localhost"],
    "max_connections": 10,
}
get_db_url(bad_config)  # beartype catches port: str instead of int

Output:

postgresql://localhost:5432/mydb
True
False
BeartypeCallHintParamViolation: @beartyped get_db_url() parameter config[...] violates
type hint -- database.port expected int, got str '5432'.

This pattern is valuable for applications that load configuration at startup. Instead of crashing with an obscure TypeError or KeyError deep in the application logic, beartype raises a precise error at the boundary where the config enters your typed code. You can extend this example by adding more config fields to the TypedDict classes or by adding a validate_config() function that checks business logic (port range, non-empty hostnames) on top of the structural type checks beartype provides.

beartype validating a typed config dict
Port 5432 is an int. Port '5432' is a footgun.

Frequently Asked Questions

Does beartype slow down my application significantly?

For simple types like str, int, float, and bool, beartype adds roughly the overhead of one isinstance() call per parameter -- typically under a microsecond. For complex generic types like list[str], beartype samples one random element rather than traversing the entire collection, keeping the check O(1) regardless of size. In practice, the overhead is negligible compared to network I/O, disk access, or any real computation your function performs. If you profile and find beartype is a bottleneck, you can disable it in production by setting BEARTYPE_IS_COLOR=0 or by using BeartypeConf(is_pep484_tower=False) to narrow the checks.

Should I use beartype instead of mypy?

They are complementary, not competing. mypy is a static type checker -- it analyzes your code without running it and catches type errors at development time. beartype is a runtime type checker -- it runs your code and catches type errors when functions are actually called. Use both: mypy catches the majority of errors before you run anything, and beartype catches the remaining cases where runtime data violates your annotations (external API responses, user input, config files). Think of mypy as a spell checker and beartype as a grammar checker -- they operate on different passes of the same text.

Can beartype check types in third-party library code?

beartype can only check code it wraps -- either via @beartype decorators or the importlib hook. It cannot check inside third-party libraries you call because it has no way to wrap their functions at import time unless you explicitly include them in beartype_packages(). However, you can protect your own code against bad values coming from third-party libraries by annotating your functions' return types carefully and decorating the boundaries between your code and library code with @beartype.

Can I customize the exception type beartype raises?

Yes. Pass BeartypeConf(violation_type=MyCustomError) to use your own exception class. The custom exception must inherit from either Exception or Warning. Raising a Warning subclass puts beartype into warning mode, where violations are logged but not raised. Raising an Exception subclass lets you integrate beartype violations into your existing error-handling infrastructure -- for example, returning a 400 Bad Request in a web framework by catching BeartypeCallHintViolation at the top-level request handler.

Does beartype support Protocol and ABC types?

Yes. beartype checks structural subtype compatibility for typing.Protocol classes using Python's runtime isinstance() mechanism, provided the protocol is decorated with @runtime_checkable. Abstract base classes from collections.abc (like Sequence, Mapping, Iterable) work out of the box because they support isinstance() natively. For protocols without @runtime_checkable, beartype falls back to a duck-typing check that verifies the object has the expected attributes and methods.

How do I disable beartype in production for zero overhead?

Set the environment variable BEARTYPE_IS_COLOR=0 before starting your application -- this only disables ANSI color codes in error messages, not checking. To fully disable checks, use BeartypeConf(strategy=BeartypeStrategy.O0) which makes every check a no-op while keeping the decorators in place. Alternatively, wrap your decorators in a helper that checks an environment variable: ENABLE_TYPE_CHECKS=false returns a no-op decorator, true returns @beartype. This gives you a single toggle for all checks in the codebase.

Conclusion

beartype bridges the gap between Python's optional type annotations and actual enforcement. It turns the str, int | None, and list[dict[str, int]] hints you have already written into executable contracts -- raising precise, actionable errors at the exact call site where the type violation occurs, rather than letting wrong values propagate silently until they cause a cryptic failure elsewhere. The decorator pattern is one line per function; the project-wide hook is one line per package; the performance overhead is O(1) regardless of data size.

The real-life config loader example shows the most practical use case: enforcing structure at the boundary where external data (JSON files, API responses, user input) enters your typed Python code. Extend that example by adding more TypedDict levels, swapping in BeartypeConf(violation_type=UserWarning) during a gradual migration, or combining it with beartype_this_package() to cover your entire application in one import. The official documentation at beartype.readthedocs.io covers advanced topics including custom validators, PEP 593 Annotated metadata, and the full BeartypeConf reference.

Continue Learning Python

Tutorials you might also find useful:

How To Use Python nox for Automated Test Sessions

How To Use Python nox for Automated Test Sessions

Last Updated: June 09, 2026

Intermediate

You push your code, CI turns red, and the error reads: “ModuleNotFoundError: No module named ‘pytest‘.” But you ran the tests locally and everything passed. The difference? Your laptop has months of accumulated packages that silently filled in the gaps. The CI machine doesn’t. This is the “works on my machine” problem, and it bites Python projects of every size. Running tests in a dirty local environment is like spell-checking a document while autocorrect is on — you’ll miss things that would fail for everyone else.

nox solves this by creating a fresh, isolated virtual environment every time you run your tests. It reads a plain Python file called noxfile.py to know what to install and what to run. There’s no TOML config syntax to memorize, no INI format quirks — just Python functions decorated with @nox.session. If you already know how to write Python, you already know how to write nox sessions. Installation requires nothing more than pip install nox.

In this article we’ll cover everything you need to get productive with nox: writing your first noxfile, running tests with pytest in an isolated environment, testing across multiple Python versions, adding linting sessions, and using parameterized sessions to reduce duplication. By the end you’ll have a complete noxfile that mirrors what real-world Python projects use in CI/CD pipelines.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program.

nox Quick Example: Running Tests in Isolation

Before diving into every feature, here is the minimum nox setup that gets your pytest tests running in a clean virtual environment every time:

# noxfile.py
import nox

@nox.session
def tests(session):
    session.install("pytest")
    session.install(".")          # install your own package
    session.run("pytest", "tests/")

With this file saved at the root of your project, run the session from your terminal:

nox -s tests
nox > Running session tests
nox > Creating virtual environment (virtualenv) using python3.12 in .nox/tests
nox > python -m pip install pytest
nox > python -m pip install .
nox > pytest tests/
========================= test session starts ==========================
collected 4 items

tests/test_math.py ....                                          [100%]

========================= 4 passed in 0.32s ============================
nox > Session tests was successful.

nox created a fresh virtual environment in .nox/tests/, installed your dependencies from scratch, and ran pytest. The session.install() calls map directly to pip install, and session.run() maps to running a command inside that environment. Every subsequent nox -s tests call rebuilds the environment by default, so you’re always testing against a clean slate.

The sections below cover the full nox feature set — parameterized sessions, multi-version testing, linting, and more — so you can replace a fragile shell script or a long tox config with a single readable Python file.

Python packages floating in an isolated environment bubble
pip install everything and pray. Or use nox.

What is nox and Why Use It?

nox is a command-line tool for automating Python testing. You define sessions — each session is a Python function that installs dependencies and runs commands. When you execute a session, nox creates a dedicated virtual environment for it so nothing from your system Python leaks in.

The closest comparison is tox, which has served the Python ecosystem well for years. The key difference is that tox uses a configuration file (tox.ini or pyproject.toml sections) while nox uses a Python file. This means you can use loops, conditionals, environment variables, and any Python logic you want to control your sessions. tox’s config syntax eventually becomes a DSL of its own; nox stays Python.

Featurenoxtox
Configuration formatPython (noxfile.py)INI / TOML config file
Conditional logicPlain Python if/for statementsLimited config-level conditionals
Multi-version testing@nox.session(python=["3.11","3.12"])[tox] envlist = py311,py312
Reuse environmentsnox -r flag-e / skip-missing-interpreters
Learning curveLow — just PythonMedium — needs config knowledge

nox is the default automation tool for Google’s open-source Python projects and is used in CPython’s own test infrastructure. If your project is small, a simple noxfile with two sessions (tests + lint) is enough. If it grows, you can extend the same file rather than learning a new syntax.

Installing nox

Install nox globally with pip so it’s available across all your projects:

# install_nox.sh
pip install nox
Successfully installed nox-2024.4.15 virtualenv-20.26.0

Verify the installation:

# verify_nox.sh
nox --version
2024.4.15

nox is intentionally installed outside your project’s virtualenv. It’s a task runner, not a dependency. Your noxfile.py tells it what to install inside each session’s isolated environment.

Writing Your First noxfile

Create a file named noxfile.py at the root of your project — the same directory that contains your pyproject.toml or setup.py. A complete noxfile for a simple Python project looks like this:

# noxfile.py
import nox

# Reuse the virtual environment between runs with `nox -r` to speed things up
nox.options.reuse_existing_virtualenvs = True

@nox.session(python="3.12")
def tests(session):
    """Run the test suite with pytest."""
    # Install test dependencies
    session.install("pytest", "pytest-cov")
    # Install the project itself in editable mode
    session.install("-e", ".")
    # Run pytest with coverage
    session.run(
        "pytest",
        "--cov=mypackage",
        "--cov-report=term-missing",
        "tests/",
    )

@nox.session(python="3.12")
def lint(session):
    """Check code style with flake8."""
    session.install("flake8")
    session.run("flake8", "mypackage/", "tests/")

Run both sessions sequentially with nox (no arguments), or run a specific one with nox -s tests. The nox.options.reuse_existing_virtualenvs = True line tells nox to skip rebuilding the environment if it already exists — this makes repeated local runs much faster. On CI you’d omit this setting so every run starts clean.

nox
nox > Running session tests
nox > Re-using existing virtual environment at .nox/tests.
nox > pytest --cov=mypackage --cov-report=term-missing tests/
========================= test session starts ==========================
collected 8 items

tests/test_utils.py ........                                     [100%]

---------- coverage: platform darwin, python 3.12 ----------
Name                    Stmts   Miss  Cover
-------------------------------------------
mypackage/utils.py         24      2    92%

nox > Session tests was successful.
nox > Running session lint
nox > Re-using existing virtual environment at .nox/lint.
nox > flake8 mypackage/ tests/
nox > Session lint was successful.
2 sessions ran successfully.

The session.install() method accepts the same arguments as pip install: package names, version pins ("pytest>=7.0"), and flags like "-e", "." for editable installs. The session.run() method takes the command as separate string arguments — this is intentional since it avoids shell injection issues and makes the call easy to read.

nox session structure as isolated hexagonal environments
@nox.session is just a Python function. The config IS the code.

Testing Across Multiple Python Versions

One of nox’s most powerful features is running the same test suite against multiple Python versions without any extra tooling. Pass a list of version strings to the python parameter of @nox.session, and nox creates a separate environment for each:

# noxfile.py  (multi-version excerpt)
import nox

@nox.session(python=["3.10", "3.11", "3.12", "3.13"])
def tests(session):
    """Run tests on all supported Python versions."""
    session.install("pytest", "pytest-cov")
    session.install("-e", ".")
    session.run("pytest", "tests/")
nox -s tests
nox > Running session tests-3.10
nox > Creating virtual environment (virtualenv) using python3.10 in .nox/tests-3.10
...
nox > Session tests-3.10 was successful.
nox > Running session tests-3.11
nox > Creating virtual environment (virtualenv) using python3.11 in .nox/tests-3.11
...
nox > Session tests-3.11 was successful.
nox > Running session tests-3.12
...
nox > Session tests-3.12 was successful.
nox > Running session tests-3.13
...
nox > Session tests-3.13 was successful.
4 sessions ran successfully.

If a Python version isn’t installed on the machine, nox will skip it by default (or fail — use nox --no-error-on-missing-interpreters to keep going regardless). On CI you can set up your matrix to install exactly the versions you specify, then run nox -s tests and let nox handle the rest.

Run only a specific version by appending it to the session name: nox -s "tests-3.12". This is useful when you’re debugging a failure that only shows up on one Python version.

Adding Linting and Formatting Sessions

Testing sessions are the most common use of nox, but code quality checks are a natural second session. Here is a realistic setup using flake8 for linting and black for formatting:

# noxfile.py  (linting session)
import nox

@nox.session(python="3.12")
def lint(session):
    """Run flake8 for style issues."""
    session.install("flake8", "flake8-bugbear")
    session.run("flake8", "mypackage/", "tests/", "--max-line-length=88")

@nox.session(python="3.12")
def format_check(session):
    """Check if black would reformat any file."""
    session.install("black")
    # --check exits non-zero if any file would be reformatted
    session.run("black", "--check", "mypackage/", "tests/")

@nox.session(python="3.12")
def type_check(session):
    """Run mypy for static type checking."""
    session.install("mypy")
    session.install("-e", ".")
    session.run("mypy", "mypackage/")
nox -s lint format_check
nox > Running session lint
nox > Creating virtual environment (virtualenv) using python3.12 in .nox/lint
nox > python -m pip install flake8 flake8-bugbear
nox > flake8 mypackage/ tests/ --max-line-length=88
nox > Session lint was successful.
nox > Running session format_check
nox > Creating virtual environment (virtualenv) using python3.12 in .nox/format_check
nox > python -m pip install black
nox > black --check mypackage/ tests/
All done! -- 6 files would be left unchanged.
nox > Session format_check was successful.
2 sessions ran successfully.

Each session installs only the tools it needs. The lint session doesn’t install black, and the format_check session doesn’t install flake8. This keeps each environment minimal and avoids version conflicts between tools.

Code linting checking Python code for style issues
flake8 found 47 issues. Your pre-commit hook found zero. Useful.

Parameterized Sessions to Avoid Repetition

When you need to run the same session with different arguments — for example, testing against multiple database backends or different dependency combinations — use @nox.parametrize:

# noxfile.py  (parameterized session)
import nox

DJANGO_VERSIONS = ["4.2", "5.0", "5.1"]

@nox.session(python="3.12")
@nox.parametrize("django", DJANGO_VERSIONS)
def tests_django(session, django):
    """Test against multiple Django versions."""
    session.install(f"django=={django}", "pytest", "pytest-django")
    session.install("-e", ".")
    session.run("pytest", "tests/")
nox -s "tests_django(django='5.1')"
nox > Running session tests_django(django='5.1')
nox > Creating virtual environment (virtualenv) using python3.12 in .nox/tests_django-django-5-1
nox > python -m pip install django==5.1 pytest pytest-django
nox > python -m pip install -e .
nox > pytest tests/
========================= test session starts ==========================
collected 12 items

tests/test_views.py ............                                 [100%]

========================= 12 passed in 0.91s ============================
nox > Session tests_django(django='5.1') was successful.

@nox.parametrize generates one session per value in the list. Run all three with nox -s tests_django, or target one with the quoted name shown above. This pattern is far cleaner than duplicating the same session function three times with different package pins.

Passing Arguments Through to pytest

When debugging a specific test, you want to pass flags like -k test_login or -x directly to pytest without modifying the noxfile. nox supports this via the session.posargs mechanism:

# noxfile.py  (posargs example)
import nox

@nox.session(python="3.12")
def tests(session):
    """Run tests. Pass extra pytest args after '--': nox -s tests -- -k test_login -v"""
    session.install("pytest")
    session.install("-e", ".")
    # session.posargs contains anything after '--' on the command line
    session.run("pytest", "tests/", *session.posargs)
nox -s tests -- -k "test_login" -v
nox > Running session tests
nox > pytest tests/ -k test_login -v
========================= test session starts ==========================
collected 8 items / 7 deselected / 1 selected

tests/test_auth.py::test_login PASSED                            [100%]

========================= 1 passed, 7 deselected in 0.14s ==============
nox > Session tests was successful.

The double dash -- separates nox arguments from pytest arguments. session.posargs is just a list — when it’s empty (no -- on the command line), the unpacking adds nothing. When it has values, they’re appended to the pytest command. This pattern works for any underlying tool, not just pytest.

Parallel test runs across multiple Python versions
nox -p 4. Because waiting for CI is a personality trait you can fix.

Real-Life Example: A Complete Project noxfile

Here is a production-ready noxfile for a Python library that needs tests, coverage, linting, type checking, and documentation. This is modeled on the pattern used by real open-source packages:

# noxfile.py  (complete project noxfile)
import nox

# Reuse envs locally, but not in CI
nox.options.reuse_existing_virtualenvs = True

# Default sessions run when you type 'nox' with no arguments
nox.options.sessions = ["tests", "lint"]

SOURCE_DIR = "src/mylib"
TEST_DIR = "tests"

@nox.session(python=["3.11", "3.12", "3.13"])
def tests(session):
    """Run the full test suite."""
    session.install("pytest", "pytest-cov", "pytest-xdist")
    session.install("-e", ".[dev]")  # install with dev extras
    session.run(
        "pytest",
        "--cov=" + SOURCE_DIR,
        "--cov-report=term-missing",
        "--cov-fail-under=85",
        "-n", "auto",            # parallel tests via pytest-xdist
        TEST_DIR,
        *session.posargs,
    )

@nox.session(python="3.12")
def lint(session):
    """Run flake8, black check, and isort check."""
    session.install("flake8", "flake8-bugbear", "black", "isort")
    session.run("flake8", SOURCE_DIR, TEST_DIR, "--max-line-length=88")
    session.run("black", "--check", SOURCE_DIR, TEST_DIR)
    session.run("isort", "--check-only", "--profile=black", SOURCE_DIR)

@nox.session(python="3.12")
def typing(session):
    """Run mypy for type checking."""
    session.install("mypy", "types-requests")
    session.install("-e", ".")
    session.run("mypy", SOURCE_DIR)

@nox.session(python="3.12")
def docs(session):
    """Build the Sphinx documentation."""
    session.install("sphinx", "furo")
    session.install("-e", ".")
    session.run("sphinx-build", "-W", "docs/", "docs/_build/html/")

@nox.session(python="3.12")
def release_check(session):
    """Verify the package builds cleanly before releasing."""
    session.install("build", "twine")
    session.run("python", "-m", "build")
    session.run("twine", "check", "dist/*")
nox
nox > Running session tests-3.11
...
nox > Session tests-3.11 was successful.
nox > Running session tests-3.12
...
nox > Session tests-3.12 was successful.
nox > Running session tests-3.13
...
nox > Session tests-3.13 was successful.
nox > Running session lint
...
nox > Session lint was successful.
4 sessions ran successfully.

This noxfile covers the typical CI pipeline in one readable file: parallel tests on three Python versions with coverage enforcement, code style checks, and type checking. The release_check session is only run manually when you’re preparing a release — it doesn’t appear in nox.options.sessions so it’s skipped in the normal flow. You can extend this by adding sessions for integration tests, database migrations, or any project-specific tasks.

All Python versions passing in CI with green checkmarks
Green across 3.11, 3.12, and 3.13. Merge button unlocked.

Frequently Asked Questions

When should I use nox instead of tox?

Use nox when you want to express your test automation as Python code rather than configuration. nox is a better fit if you have conditional logic in your sessions (for example, running a session only on Linux), if you need to loop over a dynamic list of parameters, or if your team already writes Python well and finds INI-format config files annoying to maintain. tox is a mature choice if you prefer convention over configuration and your project’s needs are simple enough to express in a static config file.

How do I speed up nox when environment creation is slow?

Add nox.options.reuse_existing_virtualenvs = True to your noxfile to skip recreating environments that already exist. When dependencies change, run nox --reuse-existing-virtualenvs False -s tests or simply delete the .nox/ directory to force a clean rebuild. On CI, never reuse environments — always start clean to catch dependency drift early.

Can I install from a requirements.txt file inside a nox session?

Yes. Use session.install("-r", "requirements.txt"). You can combine this with pinned package installs in the same session. For projects that use pip-tools or pip-compile, the noxfile can also be the place where you regenerate your lock files — run pip-compile as a separate session and document it in the same file as your tests.

How do I set environment variables inside a nox session?

Use session.env to pass a dictionary of environment variables to a specific session.run() call: session.run("pytest", "tests/", env={"DATABASE_URL": "sqlite:///:memory:"}). To set variables for the entire session, assign to session.env at the start of the function: session.env["DEBUG"] = "1". This is cleaner than setting environment variables globally because each session’s environment is isolated from the others.

How do I run nox in GitHub Actions?

Use the setup-python action to install your target Python versions, then install nox with pip install nox, and finally run nox -s tests. For multi-version testing, use a matrix strategy to set up each version on a separate runner and pass nox -s "tests-${{ matrix.python-version }}" to target only the relevant session. This mirrors what you’d test locally but runs in parallel across the matrix.

How do I conditionally skip a session?

Call session.skip("reason") anywhere inside the session function to stop and mark the session as skipped rather than failed. A common pattern is to skip platform-specific sessions: if sys.platform != "linux": session.skip("Linux only"). This lets you define all sessions in one noxfile and let each environment (Windows, macOS, Linux) naturally skip what doesn’t apply to it.

Conclusion

nox removes the gap between “tests pass on my machine” and “tests pass everywhere.” By creating a fresh virtual environment for each session, it guarantees that every test run starts from the same known state — no stale packages, no version mismatches, no implicit dependencies inherited from your system Python. The noxfile.py is just Python, so all the conditional logic, loops, and parameterization you already know applies directly without learning a config DSL.

The real-life noxfile in this article is a solid foundation for most Python projects. Try extending it: add a session that runs your integration tests against a real database, add a @nox.parametrize decorator to test against multiple versions of a key dependency, or wire it into your GitHub Actions matrix. The official nox documentation at https://nox.thea.codes/en/stable/ covers advanced topics like session groups, backend selection (virtualenv vs. venv vs. conda), and the full configuration reference.

How To Use Python Typer and Click for Building Complex CLIs

How To Use Python Typer and Click for Building Complex CLIs

Last Updated: June 08, 2026

Intermediate

Your Python script takes a filename, a mode, and an optional verbose flag. You wire it up with argparse and two hours later you have 80 lines of parser setup just to read three arguments. Then the requirements grow: now you need subcommands — process, validate, export — each with its own flags and help text. At that point, argparse stops being a tool and starts being a project in itself. There is a better way.

Click and Typer solve this cleanly. Click is a mature, decorator-based library that turns Python functions into CLI commands with a single line. Typer builds on top of Click, using Python type annotations to generate argument parsers automatically — you write normal typed function signatures and Typer does the rest. Both ship via pip, both integrate with the same testing toolchain, and both produce CLIs that behave the way users expect: proper help text, error messages with context, tab completion support.

This article covers everything you need to build complex, production-ready CLIs in Python. We start with a quick working example, then walk through Typer’s command system, Click’s group and subcommand architecture, type validation, shared context, and callback hooks. The article closes with a multi-command project manager CLI that demonstrates all the patterns together.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program.

Building a CLI with Typer: Quick Example

The fastest path to a working CLI is Typer. Install it with pip install typer, then write a typed function decorated with @app.command(). Typer reads the type annotations and builds the argument parser for you.

# greet_cli.py
import typer

app = typer.Typer()

@app.command()
def greet(
    name: str,
    count: int = 1,
    loud: bool = typer.Option(False, "--loud", "-l", help="Print in uppercase"),
):
    """Greet a user by name, optionally multiple times."""
    message = f"Hello, {name}!"
    if loud:
        message = message.upper()
    for _ in range(count):
        typer.echo(message)

if __name__ == "__main__":
    app()
$ python greet_cli.py Alice
Hello, Alice!

$ python greet_cli.py Alice --count 3
Hello, Alice!
Hello, Alice!
Hello, Alice!

$ python greet_cli.py Alice --loud
HELLO, ALICE!

$ python greet_cli.py --help
Usage: greet_cli.py [OPTIONS] NAME

  Greet a user by name, optionally multiple times.

Arguments:
  NAME  [required]

Options:
  --count      INTEGER  [default: 1]
  --loud   -l           Print in uppercase
  --help                Show this message and exit.

Three things to notice: name: str becomes a required positional argument, count: int = 1 becomes an optional argument with a default, and typer.Option(False, "--loud", "-l") creates a boolean flag with a short alias. The --help text is generated from the function’s docstring. No parser setup, no add_argument calls, no parse_args(). The deeper sections below show how to extend this into a multi-command CLI with subcommands, shared state, and validation.

What Is Typer and How Does It Relate to Click?

Typer is a thin wrapper around Click that replaces Click’s decorator-based argument definitions with Python type annotations. Where Click requires you to explicitly declare each option’s type and metadata, Typer infers them from your function signature. The underlying parser, error handling, and shell integration are all Click — Typer just removes the boilerplate.

Here is the same two-argument command written in both libraries so you can see the difference:

# comparison.py
# --- Click version ---
import click

@click.command()
@click.argument("filename")
@click.option("--lines", default=10, help="Number of lines to show")
def tail_click(filename, lines):
    """Show the last N lines of a file."""
    with open(filename) as f:
        content = f.readlines()
    for line in content[-lines:]:
        click.echo(line, nl=False)

# --- Typer version ---
import typer

app = typer.Typer()

@app.command()
def tail_typer(
    filename: str,
    lines: int = typer.Option(10, help="Number of lines to show"),
):
    """Show the last N lines of a file."""
    with open(filename) as f:
        content = f.readlines()
    for line in content[-lines:]:
        typer.echo(line, nl=False)

Both produce identical CLIs. The Typer version is slightly more concise and the type annotations double as IDE hints — hover over filename anywhere in your code and your editor knows it is a str. Click is the better choice when you need low-level control over parser behavior or are extending an existing Click-based CLI. Typer is the better choice for new projects where speed of development matters.

FeatureClickTyper
Argument definitionDecorators (@click.argument)Type annotations + defaults
Type validationExplicit type=click.INTInferred from annotation
Subcommands@cli.group() + @cli.command()typer.Typer() + app.add_typer()
Underlying engineClick itselfClick
IDE autocompletionLimited (decorator metadata)Full (native type annotations)
Best forExisting Click projects, fine controlNew projects, typed codebases

Building Multi-Command Apps with Typer

A single-command CLI works for simple scripts, but real tools have subcommands. The pattern in Typer is to create a main Typer() app and add sub-apps to it using app.add_typer(). Each sub-app is its own Typer() instance with its own commands. This mirrors how tools like git, docker, and aws are organized.

# multi_app.py
import typer

app = typer.Typer(help="A simple multi-command file tool.")
convert_app = typer.Typer(help="File conversion commands.")
info_app = typer.Typer(help="File information commands.")

app.add_typer(convert_app, name="convert")
app.add_typer(info_app, name="info")

@convert_app.command("to-upper")
def convert_upper(filename: str, output: str = typer.Option(None, help="Output file")):
    """Convert a text file to uppercase."""
    with open(filename) as f:
        content = f.read().upper()
    dest = output or filename + ".upper"
    with open(dest, "w") as f:
        f.write(content)
    typer.echo(f"Written to {dest}")

@convert_app.command("count-words")
def convert_count(filename: str):
    """Count words in a file."""
    with open(filename) as f:
        words = len(f.read().split())
    typer.echo(f"{filename}: {words} words")

@info_app.command()
def size(filename: str):
    """Show the size of a file in bytes."""
    import os
    bytes_ = os.path.getsize(filename)
    typer.echo(f"{filename}: {bytes_} bytes")

@info_app.command()
def lines(filename: str):
    """Count lines in a file."""
    with open(filename) as f:
        n = sum(1 for _ in f)
    typer.echo(f"{filename}: {n} lines")

if __name__ == "__main__":
    app()
$ python multi_app.py --help
Usage: multi_app.py [OPTIONS] COMMAND [ARGS]...

  A simple multi-command file tool.

Commands:
  convert  File conversion commands.
  info     File information commands.

$ python multi_app.py convert --help
Commands:
  to-upper     Convert a text file to uppercase.
  count-words  Count words in a file.

$ echo "hello world foo bar" > test.txt
$ python multi_app.py info lines test.txt
test.txt: 1 lines

$ python multi_app.py convert count-words test.txt
test.txt: 4 words

The app.add_typer(sub_app, name="convert") call registers the sub-app as a command group. Each @convert_app.command() becomes a sub-subcommand under convert. The help text at each level is auto-generated from the Typer(help="...")) argument and each function’s docstring. Adding a new subcommand is just adding a new decorated function — no manual registration needed.

Typer CLI subcommands architecture diagram
add_typer() — because –help with twenty flags stopped being help.

Click Command Groups and Subcommands

Click uses a @click.group() decorator to create command groups. A group is itself a command that dispatches to subcommands. You attach subcommands with @group_name.command(). This is the Click-native way to build the same structure that Typer’s add_typer creates under the hood.

# click_groups.py
import click
import os

@click.group()
def cli():
    """Developer toolkit: db and cache management."""
    pass

@cli.group()
def db():
    """Database operations."""
    pass

@cli.group()
def cache():
    """Cache operations."""
    pass

@db.command()
@click.argument("table")
@click.option("--limit", default=100, show_default=True, help="Row limit")
def query(table, limit):
    """Run a SELECT query on TABLE."""
    click.echo(f"SELECT * FROM {table} LIMIT {limit}")
    # In a real tool, execute the query here

@db.command()
@click.argument("table")
@click.confirmation_option(prompt="This will delete all rows. Are you sure?")
def truncate(table):
    """Truncate TABLE (requires confirmation)."""
    click.echo(f"TRUNCATE TABLE {table}")

@cache.command()
def flush():
    """Flush the entire cache."""
    click.echo("Cache flushed.")

@cache.command()
@click.argument("key")
def delete(key):
    """Delete a single cache key."""
    click.echo(f"Deleted key: {key}")

if __name__ == "__main__":
    cli()
$ python click_groups.py --help
Usage: click_groups.py [OPTIONS] COMMAND [ARGS]...

  Developer toolkit: db and cache management.

Commands:
  cache  Cache operations.
  db     Database operations.

$ python click_groups.py db query users --limit 50
SELECT * FROM users LIMIT 50

$ python click_groups.py db truncate orders
This will delete all rows. Are you sure? [y/N]: y
TRUNCATE TABLE orders

$ python click_groups.py cache delete session:abc123
Deleted key: session:abc123

The @click.confirmation_option decorator on truncate shows one of Click’s built-in validation helpers — it prompts the user to confirm before running the command, and aborts cleanly if they decline. Click provides similar helpers for passwords (@click.password_option), version information (@click.version_option), and verbose mode. These cover the most common patterns so you do not have to implement them yourself.

Passing Context Between Commands

Real CLIs often need to share state between a group command and its subcommands — a database connection, a config file path, or a verbosity flag set at the top level. Both Click and Typer support this through a context object that flows down the command chain.

# context_demo.py
import click

@click.group()
@click.option("--config", default="config.json", help="Config file path")
@click.option("--verbose", is_flag=True, help="Enable verbose output")
@click.pass_context
def cli(ctx, config, verbose):
    """App with shared config context."""
    # Store shared state in ctx.obj
    ctx.ensure_object(dict)
    ctx.obj["config"] = config
    ctx.obj["verbose"] = verbose
    if verbose:
        click.echo(f"[verbose] Using config: {config}")

@cli.command()
@click.argument("username")
@click.pass_context
def create_user(ctx, username):
    """Create a new user."""
    config = ctx.obj["config"]
    verbose = ctx.obj["verbose"]
    if verbose:
        click.echo(f"[verbose] Reading from {config}")
    click.echo(f"Created user: {username}")

@cli.command()
@click.pass_context
def list_users(ctx):
    """List all users."""
    config = ctx.obj["config"]
    click.echo(f"Listing users from {config}")

if __name__ == "__main__":
    cli()
$ python context_demo.py --verbose create-user alice
[verbose] Using config: config.json
[verbose] Reading from config.json
Created user: alice

$ python context_demo.py --config /etc/myapp.json list-users
Listing users from /etc/myapp.json

The @click.pass_context decorator injects the Click Context object as the first argument to the function. The group command populates ctx.obj (a plain dict by convention) with whatever shared state the subcommands need. Each subcommand retrieves that state via its own ctx.obj reference. This pattern cleanly separates top-level configuration (parsed once) from per-command logic (parsed independently).

Click context passing between commands
ctx.obj — global state that travels with the command, not bolted onto a global variable.

Type Validation, Choices, and Custom Types

Both Click and Typer validate argument types before your function runs. If a user passes a string where an integer is expected, they get a clear error message and the command exits — no exception traceback, no partially executed logic. You can extend this with choices, path validation, and custom types.

# validation_demo.py
import typer
from enum import Enum
from pathlib import Path

class OutputFormat(str, Enum):
    json = "json"
    csv = "csv"
    table = "table"

app = typer.Typer()

@app.command()
def export(
    source: Path = typer.Argument(..., exists=True, help="Input file (must exist)"),
    output: Path = typer.Option(Path("output.txt"), help="Output file"),
    fmt: OutputFormat = typer.Option(OutputFormat.table, help="Output format"),
    limit: int = typer.Option(100, min=1, max=10000, help="Row limit (1-10000)"),
):
    """Export data from SOURCE in the given format."""
    typer.echo(f"Reading: {source}")
    typer.echo(f"Format:  {fmt.value}")
    typer.echo(f"Limit:   {limit}")
    typer.echo(f"Writing: {output}")

if __name__ == "__main__":
    app()
$ echo "id,name" > data.csv

$ python validation_demo.py data.csv --fmt json --limit 50
Reading: data.csv
Format:  json
Limit:   50
Writing: output.txt

$ python validation_demo.py missing.csv
Error: Invalid value for 'SOURCE': Path 'missing.csv' does not exist.

$ python validation_demo.py data.csv --limit 99999
Error: Invalid value for '--limit': 99999 is not in the range 1<=x<=10000.

$ python validation_demo.py data.csv --fmt xml
Error: Invalid value for '--fmt': 'xml' is not one of 'json', 'csv', 'table'.

The typer.Argument(..., exists=True) parameter tells Typer to validate that the path exists before running the function. The min=1, max=10000 parameters on the integer option enforce a numeric range. The Enum subclass restricts fmt to a fixed set of values. All of this validation happens before your function body runs -- and crucially, the error messages tell the user exactly what went wrong and what the valid values are. Compare this to catching a ValueError inside your function and printing a custom error: the built-in validators produce more consistent, user-friendly output.

Callbacks, Version Flags, and Eager Options

Sometimes you need an option that exits immediately after running -- a --version flag or a --list-formats option that prints information and stops. Both Click and Typer support this through "eager" options and callbacks.

# callbacks_demo.py
import typer
from typing import Optional

__version__ = "1.4.2"

def version_callback(value: bool):
    if value:
        typer.echo(f"mytool version {__version__}")
        raise typer.Exit()

app = typer.Typer()

@app.callback()
def main(
    version: Optional[bool] = typer.Option(
        None, "--version", "-v",
        callback=version_callback,
        is_eager=True,
        help="Show version and exit.",
    )
):
    """mytool -- a demonstration CLI."""

@app.command()
def run(task: str, workers: int = 4):
    """Run TASK with N workers."""
    typer.echo(f"Running '{task}' with {workers} workers")

@app.command()
def status():
    """Check current status."""
    typer.echo("Status: OK")

if __name__ == "__main__":
    app()
$ python callbacks_demo.py --version
mytool version 1.4.2

$ python callbacks_demo.py run "build images" --workers 8
Running 'build images' with 8 workers

$ python callbacks_demo.py status
Status: OK

$ python callbacks_demo.py --help
Usage: callbacks_demo.py [OPTIONS] COMMAND [ARGS]...

  mytool -- a demonstration CLI.

Options:
  -v, --version  Show version and exit.
  --help         Show this message and exit.

Commands:
  run     Run TASK with N workers.
  status  Check current status.

The is_eager=True parameter tells Typer (and Click underneath) to process this option before any command runs -- including before validating other arguments. The callback function receives the option value and raises typer.Exit() to stop execution cleanly. The @app.callback() decorator attaches options to the root app rather than to a specific command, which is the right place for global flags like --version, --config, and --verbose.

Typer Exit callback control panel
raise typer.Exit() -- the only acceptable use of control flow as error handling.

Real-Life Example: A Task Manager CLI

This project implements a multi-command task manager CLI with Typer: add tasks, list them, mark as done, and delete them. Tasks are stored in a JSON file. The project uses sub-apps, type validation, colored output, and the callback context pattern.

# tasks_cli.py
import typer
import json
import os
from enum import Enum
from pathlib import Path
from typing import Optional

DB_PATH = Path("tasks.json")

def load_tasks():
    if not DB_PATH.exists():
        return []
    with open(DB_PATH) as f:
        return json.load(f)

def save_tasks(tasks):
    with open(DB_PATH, "w") as f:
        json.dump(tasks, f, indent=2)

class Priority(str, Enum):
    low = "low"
    medium = "medium"
    high = "high"

app = typer.Typer(help="A simple terminal task manager.")

@app.command()
def add(
    title: str = typer.Argument(..., help="Task title"),
    priority: Priority = typer.Option(Priority.medium, help="Task priority"),
    tag: Optional[str] = typer.Option(None, help="Optional tag"),
):
    """Add a new task."""
    tasks = load_tasks()
    task_id = max((t["id"] for t in tasks), default=0) + 1
    task = {"id": task_id, "title": title, "priority": priority.value,
            "tag": tag, "done": False}
    tasks.append(task)
    save_tasks(tasks)
    typer.echo(f"[+] Added task #{task_id}: {title} [{priority.value}]")

@app.command(name="list")
def list_tasks(
    show_done: bool = typer.Option(False, "--done", help="Include completed tasks"),
    priority: Optional[Priority] = typer.Option(None, help="Filter by priority"),
):
    """List tasks."""
    tasks = load_tasks()
    if not show_done:
        tasks = [t for t in tasks if not t["done"]]
    if priority:
        tasks = [t for t in tasks if t["priority"] == priority.value]
    if not tasks:
        typer.echo("No tasks found.")
        return
    for t in tasks:
        status = "[x]" if t["done"] else "[ ]"
        tag = f" #{t['tag']}" if t.get("tag") else ""
        typer.echo(f"  {status} #{t['id']:3d}  {t['priority']:6s}  {t['title']}{tag}")

@app.command()
def done(task_id: int = typer.Argument(..., help="Task ID to mark complete")):
    """Mark a task as done."""
    tasks = load_tasks()
    for t in tasks:
        if t["id"] == task_id:
            t["done"] = True
            save_tasks(tasks)
            typer.echo(f"[ok] Task #{task_id} marked done.")
            return
    typer.echo(f"Task #{task_id} not found.", err=True)
    raise typer.Exit(1)

@app.command()
def delete(
    task_id: int = typer.Argument(..., help="Task ID to delete"),
    force: bool = typer.Option(False, "--force", "-f", help="Skip confirmation"),
):
    """Delete a task."""
    tasks = load_tasks()
    task = next((t for t in tasks if t["id"] == task_id), None)
    if not task:
        typer.echo(f"Task #{task_id} not found.", err=True)
        raise typer.Exit(1)
    if not force:
        typer.confirm(f"Delete task #{task_id}: '{task['title']}'?", abort=True)
    tasks = [t for t in tasks if t["id"] != task_id]
    save_tasks(tasks)
    typer.echo(f"[-] Deleted task #{task_id}.")

if __name__ == "__main__":
    app()
$ python tasks_cli.py add "Write unit tests" --priority high --tag testing
[+] Added task #1: Write unit tests [high]

$ python tasks_cli.py add "Update README"
[+] Added task #2: Update README [medium]

$ python tasks_cli.py add "Fix login bug" --priority high
[+] Added task #3: Fix login bug [high]

$ python tasks_cli.py list
  [ ]   #1  high    Write unit tests #testing
  [ ]   #2  medium  Update README
  [ ]   #3  high    Fix login bug

$ python tasks_cli.py list --priority high
  [ ]   #1  high    Write unit tests #testing
  [ ]   #3  high    Fix login bug

$ python tasks_cli.py done 1
[ok] Task #1 marked done.

$ python tasks_cli.py list --done
  [x]   #1  high    Write unit tests #testing
  [ ]   #2  medium  Update README
  [ ]   #3  high    Fix login bug

$ python tasks_cli.py delete 2
Delete task #2: 'Update README'? [y/N]: y
[-] Deleted task #2.

This project demonstrates the key Typer patterns: enum-based choices for Priority, optional filtering arguments with Optional[Priority], a command named list using name="list" (since list is a Python builtin), graceful error exits with raise typer.Exit(1) for error conditions, and typer.confirm() for destructive operations. Extend it by replacing the JSON store with SQLite via sqlite3, adding a typer.progressbar() for bulk operations, or adding shell completion with typer install-completion.

CLI task manager checklist
JSON flat-file storage: perfectly adequate until the day it isn't.

Frequently Asked Questions

When should I use Typer versus Click directly?

Use Typer for new projects, especially if you are already using type annotations elsewhere in your codebase (Pydantic models, FastAPI endpoints, mypy). Typer's annotation-based API is faster to write, works with IDE autocompletion out of the box, and produces the same underlying CLI as Click. Use Click directly when you are extending or maintaining an existing Click-based project, need low-level access to Click internals, or are building a library that other Click users will import and extend.

What does Click/Typer offer that argparse does not?

Click and Typer use a decorator pattern that is far less verbose than argparse's imperative API. More importantly, they support nested command groups natively -- building a git-style CLI with five subcommand groups in argparse requires significant manual plumbing. Click and Typer also provide built-in support for password prompts, progress bars, colored terminal output (typer.style()), shell completion generation, and confirmation prompts. These are all DIY in argparse.

How do I test Click and Typer CLIs?

Click ships a CliRunner that invokes commands without spawning a subprocess, capturing stdout and stderr for assertions. Typer exposes the same runner via its test module. Create a runner, call runner.invoke(app, ["add", "my-task"]), and check result.output and result.exit_code. This approach is fast, does not require a real terminal, and works in pytest without any special plugins. Test edge cases like missing required arguments and invalid types -- the error messages from Click/Typer validation are worth asserting on.

How do I add shell tab completion?

Typer can generate and install completion scripts automatically. Run python your_cli.py --install-completion and Typer writes a completion script for the current shell (bash, zsh, fish) to the appropriate location. Click has a similar mechanism -- set the COMP_WORDS and COMP_CWORD environment variables and add the completion script via eval "$(_CLI_COMPLETE=bash_source your_cli)". For both libraries, completion works on command names, subcommand names, and option names automatically -- you only need custom completion logic if an argument's valid values come from a dynamic source like a database query.

How do I structure a large CLI with many subcommands?

Split command groups into separate modules. Create a file per sub-app (e.g., commands/db.py, commands/cache.py), define each Typer app or Click group in its module, and import and register them in a central cli.py entry point. This keeps each module focused and testable independently. Use app.add_typer(db_app, name="db") in your entry point to stitch them together. For very large projects (50+ commands), consider using Click's lazy loading pattern -- registered commands are loaded only when invoked, so startup time stays fast regardless of how many commands are defined.

Conclusion

Click and Typer cover the full range of Python CLI complexity -- from a single-function script to a multi-level command hierarchy with shared context, type validation, and shell completion. Typer's annotation-based approach eliminates most of the setup code, while Click's decorator model gives finer-grained control when needed. The two are fully interoperable: a Typer app is a Click app, so you can mix patterns in the same project. We covered Typer multi-command apps with add_typer, Click groups with @cli.group(), context passing via ctx.obj, type validation with Enums and path constraints, eager callbacks for version flags, and a real task manager project pulling the patterns together.

The task manager in the real-life example is a good starting point -- add persistent storage with sqlite3, plug in rich for colored tabular output, or wire the done command to a webhook when tasks are completed. The full Typer documentation at typer.tiangolo.com covers advanced topics including parameter callback validation, testing with CliRunner, and generating man pages. The Click documentation remains the definitive reference for anything Typer does not expose directly.

How To Use Python Dask for Parallel Data Processing

How To Use Python Dask for Parallel Data Processing

Last Updated: June 07, 2026

Intermediate

Your pandas script runs fine on 500 MB of data. Then the dataset grows to 50 GB and suddenly you’re waiting 40 minutes, swapping to disk, or hitting a MemoryError before the job even starts. The standard advice — “use chunking” or “filter first” — only gets you so far. What you actually need is a way to spread the work across all your CPU cores, or even multiple machines, without rewriting your entire pipeline. That’s exactly what Dask does.

Dask is a parallel computing library that feels like pandas and NumPy on the outside but distributes work under the hood. It breaks your data and computations into a task graph, schedules those tasks across available workers, and assembles the results. It ships with the standard pip/conda toolchain — no cluster required to get started. You can run it on a laptop using all your cores and scale the same code to a cloud cluster later with minimal changes.

This article covers everything you need to use Dask productively. We’ll start with a quick working example, then walk through Dask’s four main interfaces — DataFrames, Arrays, Bags, and Delayed — with runnable code for each. We’ll cover the Dask scheduler, how to monitor progress with the dashboard, and when NOT to use Dask. The article closes with a real-life log-processing pipeline that ties all the pieces together.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program.

Dask DataFrames: Quick Example

The fastest way to see Dask in action is to swap pandas for dask.dataframe and read a large CSV. Dask reads the file lazily — building a task graph instead of loading everything into memory — and only computes when you call .compute().

# dask_quick.py
import dask.dataframe as dd

# Read a large CSV (or multiple CSVs with glob patterns)
df = dd.read_csv("sales_data_*.csv")

# Operations look identical to pandas
high_value = df[df["amount"] > 1000]
summary = high_value.groupby("region")["amount"].sum()

# Nothing runs yet -- Dask builds a task graph

# Call .compute() to trigger execution across all CPU cores
result = summary.compute()
print(result)
region
East     4823910.50
North    3197642.75
South    5012388.00
West     2945100.25
Name: amount, dtype: float64

The key difference from pandas: operations are lazy. Calling df["amount"] > 1000 doesn’t filter anything immediately — it records the intent and adds it to the task graph. .compute() executes the entire graph using all available CPU cores. For small datasets this adds overhead; for large ones it’s the difference between finishing in seconds versus crashing with OOM.

What Is Dask and When Should You Use It?

Dask is a Python library for parallel and out-of-core computation. “Out-of-core” means it can process data larger than your RAM by loading only the pieces it needs at any moment. It achieves this by representing your computation as a directed acyclic graph (DAG) of tasks, then scheduling those tasks across workers — whether those workers are threads, processes, or machines on a cluster.

Dask’s four main interfaces each mirror a familiar library:

Dask InterfaceMirrorsBest For
dask.dataframepandasLarge tabular data, CSV/Parquet files
dask.arrayNumPyLarge numerical arrays, image stacks, ML preprocessing
dask.bagPySpark RDDs / itertoolsUnstructured data, JSON, log files, text corpora
dask.delayedconcurrent.futuresCustom pipelines, arbitrary Python functions

Dask is the right tool when your data exceeds available RAM, your computation is embarrassingly parallel (many independent chunks), or you want to scale from a laptop to a cluster without a rewrite. Dask is NOT the right tool when your dataset fits comfortably in memory (pandas is faster with less overhead), when you need ACID transactions, or when your bottleneck is a single sequential algorithm that can’t be parallelized.

Dask DataFrames for Large Tabular Data

A Dask DataFrame is a collection of pandas DataFrames partitioned across rows. Each partition is a regular pandas DataFrame that fits in memory. Dask schedules operations across partitions in parallel and assembles the results.

# dask_dataframe.py
import dask.dataframe as dd
import pandas as pd
import numpy as np

# Create a sample dataset to work with
# In practice, this would be a large CSV or Parquet file on disk
records = []
for i in range(200000):
    records.append({
        "order_id": i,
        "product": np.random.choice(["A", "B", "C", "D"]),
        "region": np.random.choice(["East", "West", "North", "South"]),
        "amount": round(np.random.uniform(10, 5000), 2),
        "month": np.random.randint(1, 13),
    })

pdf = pd.DataFrame(records)
pdf.to_csv("orders.csv", index=False)

# Load with Dask -- npartitions controls how many chunks
df = dd.read_csv("orders.csv")

print(f"Number of partitions: {df.npartitions}")
print(f"Columns: {list(df.columns)}")

# Aggregation -- exactly like pandas, but parallel
monthly_totals = df.groupby("month")["amount"].sum().compute()
print("\nMonthly totals:")
print(monthly_totals.sort_index())

# Filter then aggregate
top_regions = (
    df[df["amount"] > 2000]
    .groupby("region")["amount"]
    .agg(["sum", "count"])
    .compute()
)
print("\nHigh-value orders by region:")
print(top_regions)
Number of partitions: 1
Columns: ['order_id', 'product', 'region', 'amount', 'month']

Monthly totals:
month
1     434823.51
2     399217.88
3     412088.92
...
12    408761.23
Name: amount, dtype: float64

High-value orders by region:
              sum  count
region
East    1298843.62  998
North   1201007.44  922
South   1299217.83  1001
West    1248391.10  956

For real-world large files, use dd.read_csv("data_*.csv") to glob multiple files into one logical DataFrame, or dd.read_parquet("data/")> for columnar files. Parquet is strongly preferred for Dask work because it stores column metadata and allows Dask to skip irrelevant columns at read time, dramatically reducing I/O.

One Dask DataFrame operation that differs from pandas is len(df) — it triggers a full scan. If you need row counts frequently, store them externally. Also, operations that require a full sort (like sort_values) are expensive because they require shuffling data across partitions. Filter and aggregate first; sort last, and only if necessary.

Dask Arrays for Numerical Computing

Dask Arrays wrap NumPy arrays, splitting them into chunks and applying NumPy operations in parallel. This is particularly useful for large scientific datasets, image stacks, or machine learning preprocessing where the full array doesn’t fit in RAM.

# dask_array.py
import dask.array as da
import numpy as np

# Create a large Dask array from a NumPy array (in practice, from disk or a lazy loader)
# chunks=(1000, 1000) means each chunk is 1000x1000 elements
x = da.from_array(np.random.random((10000, 10000)), chunks=(2000, 2000))

print(f"Array shape:  {x.shape}")
print(f"Chunk shape:  {x.chunksize}")
print(f"Total chunks: {x.npartitions}")

# NumPy operations work identically
mean_val = x.mean().compute()
std_val  = x.std().compute()
row_sums = x.sum(axis=1).compute()

print(f"\nMean: {mean_val:.6f}")
print(f"Std:  {std_val:.6f}")
print(f"Row sums shape: {row_sums.shape}")

# Linear algebra -- parallelized across chunks
y = x.T @ x             # Lazy matrix multiplication
result = y.compute()
print(f"\nMatrix product shape: {result.shape}")
Array shape:  (10000, 10000)
Chunk shape:  (2000, 2000)
Total chunks: 25

Mean: 0.499983
Std:  0.288681
Row sums shape: (10000,)

Matrix product shape: (10000, 10000)

The chunks parameter is the most important tuning knob for Dask Arrays. Aim for chunks between 100 MB and 1 GB each — too small and you spend more time on scheduler overhead than actual computation; too large and you lose parallelism. For time-series data, you typically chunk along the time axis. For image stacks, chunk along the image dimension so each chunk is one or a few complete images.

Dask Delayed for Custom Pipelines

Dask Delayed is the lowest-level interface and the most flexible. It wraps any Python function so that calls to it return a “delayed object” — a placeholder that records what should happen without running it. Assemble a collection of delayed objects, then call dask.compute() to execute them all in parallel.

# dask_delayed.py
import dask
import time
import random

def fetch_data(source):
    """Simulates fetching data from a slow source."""
    time.sleep(random.uniform(0.5, 1.0))
    return {"source": source, "records": random.randint(100, 1000)}

def process(data):
    """Simulates processing a chunk of data."""
    time.sleep(random.uniform(0.2, 0.5))
    return data["records"] * 1.5

def aggregate(results):
    """Combines all processed results."""
    return sum(results)

sources = ["db_shard_1", "db_shard_2", "db_shard_3", "db_shard_4"]

# --- Sequential version ---
start = time.perf_counter()
results_seq = []
for src in sources:
    raw = fetch_data(src)
    processed = process(raw)
    results_seq.append(processed)
total_seq = aggregate(results_seq)
elapsed_seq = time.perf_counter() - start
print(f"Sequential: {total_seq:.0f} in {elapsed_seq:.2f}s")

# --- Dask Delayed version ---
# Wrap functions with @dask.delayed (or call dask.delayed(fn)(...))
delayed_fetch   = dask.delayed(fetch_data)
delayed_process = dask.delayed(process)
delayed_agg     = dask.delayed(aggregate)

start = time.perf_counter()
delayed_results = [delayed_process(delayed_fetch(src)) for src in sources]
total_dask = delayed_agg(delayed_results)

result = total_dask.compute()   # All tasks run in parallel
elapsed_dask = time.perf_counter() - start
print(f"Dask:       {result:.0f} in {elapsed_dask:.2f}s")
print(f"Speedup: {elapsed_seq / elapsed_dask:.1f}x")
Sequential: 4234 in 6.84s
Dask:       4234 in 1.87s
Speedup: 3.7x

The same result in a third of the time, with zero changes to the actual business logic. dask.delayed is the right tool when you have a pipeline of functions that process independent chunks, when your data isn’t tabular (so Dask DataFrame doesn’t apply), or when you want to parallelize existing code without a full rewrite. You can visualize the task graph with total_dask.visualize() — this renders a PNG showing exactly which tasks depend on which, which is invaluable for debugging complex pipelines.

Dask Bag for Unstructured Data

Dask Bag handles unstructured or semi-structured data — JSON logs, plain text files, collections of records with varying shapes. It’s the Dask equivalent of Python’s itertools or PySpark RDDs: a distributed collection of arbitrary Python objects.

# dask_bag.py
import dask.bag as db
import json
import tempfile
import os

# Create sample log files (JSON Lines format)
log_dir = tempfile.mkdtemp()
for i in range(5):
    path = os.path.join(log_dir, f"logs_{i}.jsonl")
    with open(path, "w") as f:
        for j in range(1000):
            record = {
                "event_id": i * 1000 + j,
                "level": ["INFO", "WARNING", "ERROR"][j % 3],
                "service": ["auth", "api", "worker", "db"][j % 4],
                "duration_ms": 10 + (j % 500),
            }
            f.write(json.dumps(record) + "\n")

# Load all log files as a Dask Bag
bag = db.read_text(os.path.join(log_dir, "*.jsonl")).map(json.loads)

print(f"Type: {type(bag)}")

# Filter to errors only
errors = bag.filter(lambda r: r["level"] == "ERROR")
error_count = errors.count().compute()
print(f"Total records: {bag.count().compute()}")
print(f"Error records: {error_count}")

# Compute average duration by service for errors
def extract_service_duration(record):
    return (record["service"], record["duration_ms"])

durations = errors.map(extract_service_duration)

# foldby for group aggregation on a Bag
from dask.bag import foldby
grouped = durations.foldby(
    key=lambda x: x[0],
    binop=lambda acc, x: (acc[0] + x[1], acc[1] + 1),
    initial=(0, 0),
    combine=lambda a, b: (a[0] + b[0], a[1] + b[1]),
    combine_initial=(0, 0),
).compute()

print("\nAverage duration (ms) by service for ERROR events:")
for service, (total, count) in sorted(grouped):
    print(f"  {service:8s}: {total/count:.1f} ms ({count} errors)")
Type: <class 'dask.bag.core.Bag'>
Total records: 5000
Error records: 1667

Average duration (ms) by service for ERROR events:
  api     : 256.0 ms (416 errors)
  auth    : 256.3 ms (417 errors)
  db      : 255.6 ms (417 errors)
  worker  : 256.1 ms (417 errors)

Dask Bag is the right choice when your data doesn’t have a uniform schema — think application logs, web crawl output, or datasets where records have optional fields. For data that is uniform and tabular, prefer dask.dataframe for better performance. For JSON-Lines data that IS uniform, dd.read_json is often faster than Bag because it can use vectorized operations internally.

Choosing the Right Scheduler

Dask’s behavior is controlled by the scheduler. The default scheduler works well for most cases, but understanding your options lets you optimize for your specific workload.

SchedulerWhen to UseHow to Activate
SynchronousDebugging — runs tasks sequentially, full Python tracebacksdask.config.set(scheduler='synchronous')
Threaded (default)NumPy/pandas workloads that release the GILdask.config.set(scheduler='threads')
MultiprocessingPure Python code that holds the GILdask.config.set(scheduler='processes')
DistributedMulti-machine clusters, advanced monitoring, futures APIfrom dask.distributed import Client; client = Client()
# dask_scheduler.py
import dask
import dask.array as da
import numpy as np

x = da.from_array(np.random.random((5000, 5000)), chunks=(1000, 1000))

# Threaded scheduler (default for arrays -- NumPy releases GIL)
with dask.config.set(scheduler='threads'):
    result_t = x.mean().compute()

# Synchronous scheduler -- useful for debugging
with dask.config.set(scheduler='synchronous'):
    result_s = x.mean().compute()

print(f"Threaded result:    {result_t:.6f}")
print(f"Synchronous result: {result_s:.6f}")
print("Results match:", abs(result_t - result_s) < 1e-10)
Threaded result:    0.499987
Synchronous result: 0.499987
Results match: True

For debugging, always switch to the synchronous scheduler first. It runs tasks one at a time in the current process, so you get full Python tracebacks and can use pdb normally. The threaded scheduler is the best default for pandas and NumPy workloads because both libraries release Python’s Global Interpreter Lock (GIL) for most operations, so true parallelism is achieved with threads. For pure Python code that holds the GIL, the multiprocessing scheduler launches separate processes and achieves genuine CPU parallelism at the cost of pickling overhead.

Dask Distributed: Monitoring and Scaling

The dask.distributed package provides a richer scheduler with a web dashboard, better fault tolerance, and support for multi-machine clusters. You can start it locally with zero configuration and it immediately gives you a real-time view of task progress.

# dask_distributed.py
from dask.distributed import Client
import dask.dataframe as dd
import pandas as pd
import numpy as np

# Start a local cluster (uses all available CPU cores by default)
client = Client()  # Opens dashboard at http://localhost:8787/status

print(f"Dashboard: {client.dashboard_link}")
print(f"Workers:   {len(client.scheduler_info()['workers'])}")

# Generate sample data
df_pandas = pd.DataFrame({
    "x": np.random.random(500000),
    "y": np.random.random(500000),
    "group": np.random.choice(list("ABCDE"), 500000),
})
df_pandas.to_parquet("sample.parquet")

# Load and process -- Dask Distributed automatically uses all workers
df = dd.read_parquet("sample.parquet")

result = (
    df.assign(z=df["x"] * df["y"])
    .groupby("group")["z"]
    .agg(["mean", "std", "count"])
    .compute()
)

print("\nResult:")
print(result.round(4))

client.close()
Dashboard: http://127.0.0.1:8787/status
Workers:   8

Result:
         mean     std  count
group
A      0.2498  0.2040  99832
B      0.2500  0.2041  100064
C      0.2500  0.2039  99823
D      0.2502  0.2042  100148
E      0.2501  0.2041  100133

The Dask dashboard at http://localhost:8787/status shows a real-time task stream, worker utilization bars, and memory consumption per worker. This is invaluable for understanding whether your computation is actually parallelizing efficiently. If you see one worker doing all the work, your partitions may be uneven. If memory climbs unboundedly, you have a repartitioning or persist issue. The dashboard pays for itself in debugging time on the first large job.

To scale to a cluster, replace Client() with Client("scheduler-address:8786") or use dask-jobqueue to launch workers on an HPC cluster, or dask-kubernetes to launch workers in Kubernetes. The computation code doesn’t change at all — only the client initialization does.

Real-Life Example: Parallel Log Analysis Pipeline

This project implements a complete log analysis pipeline for a web server: read multiple log files, parse them, filter by status code, compute aggregated metrics by endpoint, and write the results to Parquet. The pipeline uses Dask Bag for parsing, converts to a Dask DataFrame for aggregation, and writes results for downstream use.

# log_pipeline.py
import dask.bag as db
import dask.dataframe as dd
import json
import os
import tempfile
import re
import pandas as pd

# --- Setup: Create realistic-looking web log files ---
log_dir = tempfile.mkdtemp()
endpoints = ["/api/users", "/api/orders", "/api/products", "/health", "/metrics"]
status_codes = [200, 200, 200, 200, 404, 500, 301]

def generate_log(event_id):
    return json.dumps({
        "id": event_id,
        "method": "GET" if event_id % 5 != 0 else "POST",
        "endpoint": endpoints[event_id % len(endpoints)],
        "status": status_codes[event_id % len(status_codes)],
        "duration_ms": 10 + (event_id % 800),
        "user_id": event_id % 500,
    })

for shard in range(8):
    path = os.path.join(log_dir, f"access_{shard:02d}.jsonl")
    with open(path, "w") as f:
        for i in range(10000):
            f.write(generate_log(shard * 10000 + i) + "\n")

print(f"Log files written to {log_dir}")

# --- Step 1: Load and parse using Dask Bag ---
raw = db.read_text(os.path.join(log_dir, "*.jsonl"))
parsed = raw.map(json.loads)

# --- Step 2: Filter for errors and slow requests ---
errors  = parsed.filter(lambda r: r["status"] >= 400)
slow    = parsed.filter(lambda r: r["duration_ms"] > 500)

# --- Step 3: Convert to Dask DataFrame for aggregation ---
# .to_dataframe() requires all records to have the same keys
df = parsed.to_dataframe()

# Compute endpoint performance summary
endpoint_stats = (
    df.groupby("endpoint")["duration_ms"]
    .agg(["mean", "max", "count"])
    .compute()
    .rename(columns={"mean": "avg_ms", "max": "peak_ms", "count": "requests"})
    .round(1)
    .sort_values("avg_ms", ascending=False)
)

# Compute error rate by endpoint
error_counts = (
    df[df["status"] >= 400]
    .groupby("endpoint")
    .size()
    .compute()
    .rename("errors")
)

summary = endpoint_stats.join(error_counts, how="left").fillna(0)
summary["error_rate_%"] = (summary["errors"] / summary["requests"] * 100).round(2)

print("\nEndpoint Performance Summary:")
print(summary[["requests", "avg_ms", "peak_ms", "error_rate_%"]].to_string())

# --- Step 4: Write aggregated results to Parquet ---
output_path = os.path.join(log_dir, "summary.parquet")
summary.to_parquet(output_path)
print(f"\nSummary written to {output_path}")

# Error count summary
total_errors, total_slow = errors.count().compute(), slow.count().compute()
total_requests = df.shape[0].compute()
print(f"\nTotal requests:  {total_requests:,}")
print(f"Error requests:  {total_errors:,} ({total_errors/total_requests*100:.1f}%)")
print(f"Slow requests:   {total_slow:,} ({total_slow/total_requests*100:.1f}%)")
Log files written to /tmp/tmp_logs_abc123

Endpoint Performance Summary:
             requests  avg_ms  peak_ms  error_rate_%
/api/orders     16000   407.0    809.0         28.57
/api/products   16000   406.0    808.0         28.57
/api/users      16000   405.5    807.0         28.57
/health         16000   400.5    802.0         28.57
/metrics         8000   400.5    800.0          0.00

Summary written to /tmp/tmp_logs_abc123/summary.parquet

Total requests:  80,000
Error requests:  22,857 (28.6%)
Slow requests:   24,800 (31.0%)

This pipeline reads 8 log shards in parallel, filters and aggregates using Dask’s task graph, and writes the output to Parquet for downstream queries. Each step is lazy until .compute() is called, which means Dask can optimize the full pipeline — for example, fusing the parse and filter steps into a single pass over the data. To scale this to a real server environment, replace the db.read_text glob with an S3 path (s3://my-bucket/logs/*.jsonl) and replace Client() with a distributed cluster client. The business logic doesn’t change.

Frequently Asked Questions

When should I use Dask instead of pandas?

Use Dask when your dataset doesn’t fit comfortably in RAM (roughly when it exceeds 50-60% of available memory), when you have multiple large files to process together, or when a computation that works correctly on a sample is too slow on the full dataset. For datasets under a few hundred MB that fit easily in memory, plain pandas is faster because it has less scheduling overhead. The practical rule: profile with pandas first, switch to Dask when you hit a memory or time wall.

How does Dask compare to PySpark?

Dask is lighter, easier to install, and integrates directly with the Python data science ecosystem (pandas, NumPy, scikit-learn). PySpark requires a JVM, has its own API that diverges from pandas, and is more commonly managed by a dedicated infrastructure team. For Python teams running on a single machine or a small cluster and primarily working with pandas-compatible data, Dask is the more productive choice. For petabyte-scale data on a large enterprise Hadoop/Spark cluster, PySpark wins because of its mature scheduler and wider industry support.

What does “lazy evaluation” mean in practice?

Lazy evaluation means that calling df.groupby("col")["val"].sum() on a Dask DataFrame doesn’t actually compute anything — it returns a Dask object representing the computation. Dask builds an internal task graph recording what needs to happen. When you call .compute(), Dask executes the entire graph, often fusing steps to minimize passes over the data. The practical implication: building a pipeline with ten transformations is free; each .compute() call triggers the full chain. Avoid calling .compute() in a loop — instead, collect all your delayed objects and pass them to dask.compute(*delayed_list) to execute the whole batch in one scheduler call.

How do I choose the right number of partitions?

Target partition sizes between 100 MB and 1 GB each. Too many tiny partitions and you spend more time on task scheduling overhead than actual computation. Too few massive partitions and you lose parallelism. A practical starting point: df.repartition(npartitions=num_cpus * 4) — this gives each CPU several partitions to work through while amortizing scheduler overhead. For DataFrames read from files, Dask creates one partition per file by default. If your files are small (e.g., 10 MB each), use dd.read_csv("*.csv", blocksize="256MB") to merge small files into larger partitions automatically.

How do I prevent Dask from running out of memory?

The most common cause of Dask OOM is computing too many things at once. Two key techniques: First, use dask.distributed.Client with the distributed scheduler — it has built-in spill-to-disk when workers approach their memory limit. Second, break your computation into explicit stages with df.persist() between stages — this keeps intermediate results in distributed memory rather than recomputing them, and the distributed scheduler manages eviction to disk under pressure. Third, avoid wide joins and sorts on very large DataFrames when possible — they require shuffling data across all partitions and are expensive both in time and peak memory.

Conclusion

Dask fills the gap between single-machine pandas and full-scale Spark clusters: it lets you scale Python data pipelines to datasets larger than RAM using familiar APIs, deploying across cores or machines with minimal code changes. We covered all four main interfaces — DataFrames for tabular data, Arrays for numerical computing, Bags for unstructured records, and Delayed for arbitrary parallelism — and saw how the choice of scheduler (threads, processes, synchronous, distributed) controls how tasks actually run. The Dask dashboard is your window into what the scheduler is actually doing and where your bottlenecks are.

The next step is to take an existing slow pandas script and profile it with dask.config.set(scheduler='synchronous') to verify correctness, then switch to the threaded or distributed scheduler and check the speedup. Start with real data you’re already processing — the feedback loop of “does this scale?” is the fastest way to build Dask intuition. Once comfortable, explore dask-ml for parallelized machine learning preprocessing, and dask-kubernetes or coiled.io for cloud-native cluster deployment using the exact same Dask code.

For the full API reference, the official Dask documentation covers every scheduler configuration option, best practices for partitioning, and the distributed futures API that wasn’t covered here. The Dask Examples repository has runnable Jupyter notebooks for common workflows including time series, geospatial data, and machine learning.

How To Use Python watchfiles for File Watching

How To Use Python watchfiles for File Watching

Intermediate

You are building a development tool, a config reloader, or a file-processing pipeline. You want it to react instantly when a file changes — no manual restarts, no polling in a tight loop burning CPU. The classic approach was watchdog, but uvicorn, FastAPI’s dev server, and several modern Python tools have moved to watchfiles instead. It is faster, simpler, and backed by a Rust native-event library that uses zero-overhead OS file system notifications.

Python’s watchfiles library wraps the Rust notify crate to deliver real-time change events on Linux (inotify), macOS (FSEvents), and Windows (ReadDirectoryChangesW). It works with both synchronous and async Python, requires a single pip install, and has no system-level dependencies to manage. If you have used watchdog, you will find watchfiles achieves the same result with a much smaller API surface.

This article covers the core watch() and awatch() functions, filtering by file type, working with async code using awatch(), integrating with FastAPI for a live-reload pattern, and a real-life config-watcher that restarts a service when its YAML file changes. By the end you will know how to trigger any Python code the moment a file on disk is modified.

Python watchfiles: Quick Example

Here is the minimal working example — a blocking watcher that prints every change to the current directory. Run this in one terminal, then edit or create a file in another terminal to see events arrive.

# quick_watchfiles.py
from watchfiles import watch

print("Watching current directory. Press Ctrl+C to stop.")

for changes in watch("."):
    for change_type, path in changes:
        print(f"{change_type.name}: {path}")

Output (after editing a file called config.yml):

Watching current directory. Press Ctrl+C to stop.
modified: /home/user/project/config.yml
modified: /home/user/project/config.yml
added: /home/user/project/notes.txt

The watch() function blocks and yields a set of (Change, path) tuples each time any file in the watched path changes. Change is an enum with three values: Change.added, Change.modified, and Change.deleted. The loop body runs once per batch of changes — if 10 files change simultaneously, you get one iteration with 10 tuples, not 10 separate iterations.

The sections below cover filtering, async usage, debouncing, and how to build a production-grade auto-reloader with this foundation.

What Is watchfiles and Why Use It?

watchfiles is a Python wrapper around the Rust notify library, written by Samuel Colvin (the creator of Pydantic). It uses native OS file-system event APIs — inotify on Linux, FSEvents on macOS, and ReadDirectoryChangesW on Windows — instead of polling. This means it reacts in milliseconds without burning CPU in a busy loop.

The key difference from polling-based alternatives: polling wakes up every N milliseconds and checks whether any file modification time has changed. Native events arrive from the OS exactly when a change happens. For high-frequency workflows like hot-reload on every keystroke, the difference in CPU load is significant.

Featurepolling (manual)watchdogwatchfiles
Event sourceos.stat() loopOS native + fallbackRust notify (OS native)
CPU idle loadHighLowVery low
async supportManualVia threadsBuilt-in (awatch)
API complexityHighMedium (handlers)Low (iterator)
DebouncingManualManualBuilt-in (100ms default)
pip installN/Awatchdogwatchfiles

Install before running any of the examples below:

# In your terminal
pip install watchfiles

The package ships pre-compiled Rust binaries for all common platforms, so there is no Rust toolchain needed on your machine. The install is a single .whl download.

Python watchfiles inotify Rust native events
inotify + Rust — zero-poll, zero-guilt file watching.

The watch() Function

The synchronous watch() function is an iterator that blocks until changes occur, then yields a set of change events. It accepts one or more paths and an optional set of keyword arguments for filtering and debouncing.

# watch_basics.py
from watchfiles import watch, Change

# Watch multiple directories simultaneously
for changes in watch("./src", "./config"):
    for change_type, path in changes:
        if change_type == Change.added:
            print(f"[+] New file: {path}")
        elif change_type == Change.modified:
            print(f"[~] Changed:  {path}")
        elif change_type == Change.deleted:
            print(f"[-] Deleted:  {path}")

Output (after creating and modifying a file):

[+] New file: /project/config/new_rule.yml
[~] Changed:  /project/src/main.py
[-] Deleted:  /project/src/old_module.py

Passing multiple paths as separate arguments watches all of them with a single thread — no need to spin up multiple watchers. The yielded set may contain changes from any of the watched paths in a single batch.

Stopping the Watcher Programmatically

By default, watch() runs until interrupted. To stop it after a condition is met, call stop_event — a threading event you pass in and set from another thread when you want the watcher to exit.

# watch_stop.py
import threading
from watchfiles import watch

stop = threading.Event()

def watcher():
    for changes in watch(".", stop_event=stop):
        for change_type, path in changes:
            print(f"{change_type.name}: {path}")
    print("Watcher stopped.")

t = threading.Thread(target=watcher, daemon=True)
t.start()

# Stop after 10 seconds (in a real tool, this would be a signal handler)
import time
time.sleep(10)
stop.set()
t.join()

Output:

modified: /project/data.json
Watcher stopped.

stop_event accepts any object with a .is_set() method — a threading.Event works perfectly. This pattern is essential when the watcher runs in a background thread and needs to be shut down cleanly when the main process exits or receives a signal.

Python watchfiles stop_event threading
stop_event.set() — the polite way to end an infinite loop.

Filtering Changes

In real projects you usually care about specific file types. watchfiles has a built-in filtering mechanism via the watch_filter parameter that accepts a callable returning True to include a change or False to ignore it.

# watch_filter.py
from watchfiles import watch, Change

def python_files_only(change: Change, path: str) -> bool:
    """Only watch .py files, ignore __pycache__ and .pyc."""
    return path.endswith(".py") and "__pycache__" not in path

for changes in watch("./src", watch_filter=python_files_only):
    for change_type, path in changes:
        print(f"{change_type.name}: {path}")

Output:

modified: /project/src/main.py
modified: /project/src/utils.py

The filter receives the Change type and the full file path as a string. This means you can filter on both: ignore deleted events for .log files but react to all events for .py files, for example. Returning False drops the event entirely — it never reaches the outer loop.

Built-In Filter Classes

watchfiles also ships two ready-made filter classes for the most common cases.

# watch_builtin_filters.py
from watchfiles import watch
from watchfiles.filters import PythonFilter, DefaultFilter

# PythonFilter: watches only .py files, skips __pycache__ and test files
for changes in watch("./app", watch_filter=PythonFilter()):
    for _, path in changes:
        print(f"Python file changed: {path}")

# DefaultFilter: ignores .git/, __pycache__/, .pytest_cache/, temp files
# This is the default if you don't set watch_filter at all
for changes in watch(".", watch_filter=DefaultFilter()):
    for change_type, path in changes:
        print(f"{change_type.name}: {path}")

Output:

Python file changed: /project/app/models.py
modified: /project/README.md

PythonFilter is the right choice for hot-reload tools that only care about source code. DefaultFilter is a sensible default for general file watching — it silences the noise from version control, caches, and editor temp files that would otherwise fire constantly as you work.

Async Watching with awatch()

For async code — FastAPI, aiohttp, asyncio scripts — use awatch(). It is the async equivalent of watch() and fits naturally into async for loops without blocking the event loop.

# awatch_basic.py
import asyncio
from watchfiles import awatch

async def watch_configs():
    print("Watching config directory...")
    async for changes in awatch("./config"):
        for change_type, path in changes:
            print(f"Config changed: {change_type.name} -- {path}")
            await reload_config(path)

async def reload_config(path: str):
    """Simulate an async config reload."""
    print(f"  Reloading from {path}...")
    await asyncio.sleep(0.1)  # simulate async I/O
    print(f"  Done.")

asyncio.run(watch_configs())

Output (after editing config/settings.yml):

Watching config directory...
Config changed: modified -- /project/config/settings.yml
  Reloading from /project/config/settings.yml...
  Done.

awatch() accepts the same parameters as watch(), including watch_filter and stop_event. The stop event for async is an asyncio.Event instead of a threading.Event. Inside the loop body, you can await any coroutine — reading a file, reloading a cache, posting a webhook — without blocking other tasks in the event loop.

Python watchfiles awatch async event loop
awatch() — file events in the event loop, no thread required.

Debouncing

Editors often write a file in multiple rapid steps — a save triggers a rename of the temp file, then a write, then a permission update. Without debouncing, you would get 3-5 events for a single conceptual change. watchfiles debounces by default, batching changes that arrive within 100 milliseconds into one set.

# watch_debounce.py
from watchfiles import watch

# Default: debounce_ms=100 (100 ms batching window)
for changes in watch("./src"):
    print(f"Got {len(changes)} change(s) in one batch")
    for change_type, path in changes:
        print(f"  {change_type.name}: {path}")

# Reduce to 50ms for faster reaction (at the cost of more batches)
for changes in watch("./src", debounce=50):
    print(f"Got {len(changes)} change(s)")

# Increase to 500ms to group many rapid changes together
for changes in watch("./src", debounce=500):
    print(f"Got {len(changes)} change(s)")

Output (after one ctrl+S save in VS Code):

Got 3 change(s) in one batch
  modified: /project/src/main.py
  modified: /project/src/main.py
  modified: /project/src/main.py

Even at 100ms debounce, a single save can produce multiple modified events for the same file if the editor writes in stages. The deduplication responsibility is still yours — filter the set to unique paths before acting. A simple paths = {path for _, path in changes} gives you the distinct files without worrying about how many events each generated.

Real-Life Example: YAML Config Auto-Reloader

This example builds a service that loads a YAML configuration file on startup and automatically reloads it whenever the file changes — without restarting the process. This is the pattern used by web servers, task queues, and daemons that need live config updates.

Python watchfiles YAML config auto-reloader
No restart required. The config just… updates itself.
# config_reloader.py
import asyncio
import time
from pathlib import Path
import yaml          # pip install pyyaml
from watchfiles import awatch
from watchfiles.filters import DefaultFilter

CONFIG_PATH = Path("config/settings.yml")

# --- Simulated app state ---
class AppConfig:
    def __init__(self):
        self.data: dict = {}
        self.loaded_at: float = 0.0

    def load(self, path: Path) -> None:
        """Load config from YAML file."""
        try:
            with path.open() as f:
                self.data = yaml.safe_load(f) or {}
            self.loaded_at = time.time()
            print(f"[config] Loaded {len(self.data)} keys from {path.name}")
        except FileNotFoundError:
            print(f"[config] WARNING: {path} not found -- using empty config")
            self.data = {}
        except yaml.YAMLError as exc:
            print(f"[config] ERROR: Invalid YAML -- {exc}")

config = AppConfig()

# --- Watcher coroutine ---
async def watch_config(stop: asyncio.Event) -> None:
    """Watch settings.yml and reload on every change."""
    print(f"[watcher] Watching {CONFIG_PATH} for changes...")
    async for changes in awatch(
        CONFIG_PATH.parent,
        watch_filter=DefaultFilter(),
        stop_event=stop,
    ):
        changed_paths = {path for _, path in changes}
        if str(CONFIG_PATH.resolve()) in {str(Path(p).resolve()) for p in changed_paths}:
            print(f"[watcher] {CONFIG_PATH.name} changed -- reloading...")
            config.load(CONFIG_PATH)
            print(f"[watcher] debug_mode = {config.data.get('debug_mode', False)}")

# --- Main application ---
async def main() -> None:
    # Create a sample config if it doesn't exist
    CONFIG_PATH.parent.mkdir(exist_ok=True)
    if not CONFIG_PATH.exists():
        CONFIG_PATH.write_text("debug_mode: false\nlog_level: info\nmax_workers: 4\n")

    # Load initial config
    config.load(CONFIG_PATH)

    stop = asyncio.Event()
    watcher_task = asyncio.create_task(watch_config(stop))

    print("[app] Running. Edit config/settings.yml to see a live reload.")
    print("[app] Press Ctrl+C to stop.")

    try:
        await asyncio.sleep(60)  # simulate app running
    except asyncio.CancelledError:
        pass
    finally:
        stop.set()
        await watcher_task
        print("[app] Shutdown complete.")

asyncio.run(main())

Output (after editing settings.yml to set debug_mode: true):

[config] Loaded 3 keys from settings.yml
[app] Running. Edit config/settings.yml to see a live reload.
[app] Press Ctrl+C to stop.
[watcher] Watching config/settings.yml for changes...
[watcher] settings.yml changed -- reloading...
[config] Loaded 3 keys from settings.yml
[watcher] debug_mode = True

The key design decisions: the watcher task runs as a background asyncio task alongside the main application coroutine. The asyncio.Event provides a clean shutdown path — when the main coroutine exits, it sets stop, which causes awatch() to exit its loop. To extend this into a production pattern, replace the AppConfig.load() method with your actual config parsing, and replace asyncio.sleep(60) with whatever your application actually does.

Frequently Asked Questions

What is the difference between watchfiles and watchdog?

watchdog is a well-established Python library that wraps OS file-system APIs using a thread-based observer pattern with event handler classes. watchfiles is newer, uses a Rust backend for lower overhead, and exposes a simpler iterator API instead of the observer/handler pattern. Both use native OS events on supported platforms. For async Python code, watchfiles integrates more naturally via awatch(). For projects already using watchdog with complex handler hierarchies, the migration cost may not be worth it — but for new projects, watchfiles requires less boilerplate.

Does watchfiles watch subdirectories recursively?

Yes, by default. Passing "./src" to watch() watches the entire tree under src/, including all subdirectories and their files. There is no recursive=True flag needed — it is always on. To limit to a single directory without recursion, you would need a custom watch_filter that checks the path depth.

watchfiles watches the resolved path of symlinked directories. If ./config is a symlink pointing to /etc/myapp/config, events arrive for changes to the real files at /etc/myapp/config/. However, the paths in the change set will be the real paths, not the symlink path. This can cause a mismatch if your code compares paths to the symlink location — always resolve paths with Path.resolve() before comparing.

How does watchfiles perform on large directory trees?

Since watchfiles uses OS native events rather than polling, adding more files to a watched directory does not increase CPU usage during idle periods. The OS kernel registers watches at the inode level and only wakes the process when an event occurs. In practice, watching a 10,000-file repository behaves the same as watching a 100-file project at rest. The startup cost of registering all watches grows with the number of directories, but after that the overhead is near-zero.

Can I react to changes and run code in the same process vs a subprocess?

Both patterns work. For config reloading or cache invalidation, running the handler code in the same process (as in the real-life example above) is the right choice — no inter-process communication needed, and state is shared directly. For hot-reloading a web server or long-running service, spawning a subprocess and killing/restarting it on changes is safer — it guarantees a clean state with no leftover globals. Tools like uvicorn use watchfiles in process-restart mode: they watch the source tree and os.execv() themselves when Python files change.

Does watchfiles work on Windows?

Yes. On Windows, watchfiles uses the ReadDirectoryChangesW API, which is the same native event source used by most Windows file watchers. The Python API is identical across platforms — you write the same watch() or awatch() code and it works on Linux, macOS, and Windows without any conditional imports or platform-specific configuration.

Conclusion

Python’s watchfiles library makes file watching a one-import, one-loop operation. This article covered the synchronous watch() iterator, stopping watchers cleanly with stop_event, filtering by file type with custom functions and built-in PythonFilter / DefaultFilter classes, the async awatch() function for event-loop-friendly code, debounce configuration, and a complete async YAML config reloader. The library’s Rust backend means near-zero CPU cost at rest, making it appropriate for tools that run in the background for hours or days.

To extend the config reloader: add a schema validation step using Pydantic after each reload, send a notification to a Slack channel when the config changes using httpx, or combine watchfiles with rich‘s live display to show a real-time feed of changes in a terminal dashboard. Each of those extensions is under 20 additional lines.

For the full API reference, including watch_filter signatures, all constructor parameters, and advanced usage with rust_timeout and yield_on_timeout, see the official documentation at watchfiles.helpmanual.io.

How To Use Python Enums and When You Should

Intermediate

You’ve seen it in every codebase: a string constant like "pending" scattered across thirty files, a magic integer like 2 that means “admin role” but is never defined anywhere, a status field that accepts anything and silently breaks when someone types "Pending" with a capital P. These are the symptoms of code that uses raw literals instead of named constants — and Python’s enum module is the standard fix. Enums are not just constants with names. They are types with identity, comparison semantics, iteration support, and first-class integration with type checkers and Python’s match statement.

The enum module ships in the Python standard library — no installation required. It has been available since Python 3.4, and the variants you need most (Enum, IntEnum, StrEnum, Flag) are all in one import. Python 3.11 added StrEnum and improved IntEnum behavior, so the examples in this article require Python 3.11 or later for the StrEnum sections. Everything else works from Python 3.4 onward.

This article covers everything you need to use enums confidently. We’ll start with a quick working example, then walk through the core enum types (Enum, IntEnum, StrEnum, Flag), iteration and comparison, using enums in match statements, and when NOT to use them. There is a real-life project at the end — a task management system — that ties all of it together.

Python Enums: Quick Example

This minimal example shows the core pattern. Define constants once, reference them by name everywhere, and let Python handle comparison and validation.

# enums_quick.py
from enum import Enum, auto

class OrderStatus(Enum):
    PENDING = auto()
    PROCESSING = auto()
    SHIPPED = auto()
    DELIVERED = auto()
    CANCELLED = auto()

def describe_order(status: OrderStatus) -> str:
    if status == OrderStatus.PENDING:
        return "Your order is waiting to be processed."
    elif status == OrderStatus.SHIPPED:
        return "Your order is on the way!"
    elif status == OrderStatus.DELIVERED:
        return "Your order has arrived."
    elif status == OrderStatus.CANCELLED:
        return "Your order was cancelled."
    else:
        return f"Order is {status.name.lower()}."

current = OrderStatus.SHIPPED
print(describe_order(current))
print(f"Status name: {current.name}")
print(f"Status value: {current.value}")
print(f"Is shipped: {current == OrderStatus.SHIPPED}")

Output:

Your order is on the way!
Status name: SHIPPED
Status value: 3
Is shipped: True

auto() assigns integer values automatically — you never have to pick or remember them. Each enum member has a .name (the Python attribute name) and a .value (the assigned value). Comparison with == checks identity, not just value — OrderStatus.SHIPPED == 3 is False unless you use IntEnum. That strictness is usually what you want, because it prevents code that accidentally mixes raw integers with enum values from silently doing the wrong thing.

What Are Enums and When Should You Use Them?

An enum (short for enumeration) is a set of named constants that form a logical group. Python’s Enum class turns these constants into a proper type: members are unique, iterable, comparable, and serializable. You can think of an enum as a named lookup table where each entry is an immutable, singleton object.

The right time to reach for an enum is whenever you have a fixed set of related values that represent distinct states, categories, or options. The wrong time is when the set of values is dynamic (loaded from a database at runtime), unbounded (any string is valid), or purely numeric with no semantic meaning (a count, an ID, a score).

SituationUse enum?Why
Order status: pending, shipped, deliveredYesFixed set, semantic meaning, type-checked
HTTP methods: GET, POST, PUT, DELETEYesFixed set, comparison matters, readable in logs
User role loaded from DB at runtimeNoDynamic set — use a string or int validated at boundaries
Permission flags that combine (read | write)Yes (Flag)Bitwise combination is exactly what Flag is for
Constant pi or gravitational constantNoSingle value — use a module-level constant
Days of the weekYesFixed, ordered, iterable — classic enum use case

The practical test: if you find yourself writing if status == "pending" in more than one place, you should probably be writing if status == OrderStatus.PENDING instead. Enums make typos into AttributeErrors caught at load time rather than silent bugs caught at 3am.

Basic Enum: Named Constants With Identity

The base Enum class is the most common starting point. Values can be anything — integers, strings, tuples — but the members are compared by identity, not by value. This means two enums with the same value but different names are different members (unless they are aliases, which we cover below).

# basic_enum.py
from enum import Enum, auto

class Direction(Enum):
    NORTH = auto()
    SOUTH = auto()
    EAST = auto()
    WEST = auto()

    def opposite(self) -> "Direction":
        opposites = {
            Direction.NORTH: Direction.SOUTH,
            Direction.SOUTH: Direction.NORTH,
            Direction.EAST: Direction.WEST,
            Direction.WEST: Direction.EAST,
        }
        return opposites[self]

# Access members by name or value
print(Direction.NORTH)          # Direction.NORTH
print(Direction.NORTH.name)     # NORTH
print(Direction.NORTH.value)    # 1

# Look up a member by value
d = Direction(2)
print(d)                        # Direction.SOUTH

# Look up by name
d2 = Direction["EAST"]
print(d2)                       # Direction.EAST

# Enum methods work naturally
current = Direction.NORTH
print(f"Heading {current.name}, opposite is {current.opposite().name}")

# Iteration
print("All directions:", [d.name for d in Direction])

# Membership test
print(Direction.WEST in Direction)   # True

Output:

Direction.NORTH
NORTH
1
Direction.SOUTH
Direction.EAST
Heading NORTH, opposite is SOUTH
All directions: ['NORTH', 'SOUTH', 'EAST', 'WEST']
True

A few things to notice. You can add methods to an enum class — opposite() is a real method that returns another enum member. The Direction(2) lookup-by-value is useful when deserializing data from an API or database. The Direction["EAST"] lookup-by-name is useful when parsing config strings. Both raise ValueError or KeyError on invalid input, giving you early validation rather than silent failures downstream.

IntEnum and StrEnum: When Values Matter

The base Enum does not compare equal to its underlying value. This is deliberate — it prevents accidentally passing an integer where an enum is expected. But sometimes you genuinely need the enum value to behave like its underlying type: when writing to a database that stores integers, or when generating JSON that needs plain strings. That is where IntEnum and StrEnum come in.

# intenum_strenum.py
from enum import IntEnum, StrEnum  # StrEnum requires Python 3.11+

class Priority(IntEnum):
    LOW = 1
    MEDIUM = 2
    HIGH = 3
    CRITICAL = 4

class Color(StrEnum):
    RED = "red"
    GREEN = "green"
    BLUE = "blue"

# IntEnum compares with integers directly
task_priority = Priority.HIGH
print(task_priority > 2)              # True  (behaves like int 3)
print(task_priority == 3)             # True
print(sorted([Priority.CRITICAL, Priority.LOW, Priority.MEDIUM]))  # sorted!

# IntEnum values work in numeric contexts
tasks_today = {Priority.LOW: 5, Priority.HIGH: 2}
total_weighted = sum(p * count for p, count in tasks_today.items())
print(f"Weighted load: {total_weighted}")   # 1*5 + 3*2 = 11

# StrEnum compares with strings directly
html_color = Color.RED
print(html_color == "red")            # True
print(f"CSS: color: {html_color};")   # Works directly in f-strings
print(isinstance(html_color, str))    # True

# Use StrEnum in dict lookups without converting
config = {"red": "#FF0000", "green": "#00FF00", "blue": "#0000FF"}
print(config[Color.BLUE])             # #0000FF -- lookup works directly

Output:

True
True
[<Priority.LOW: 1>, <Priority.MEDIUM: 2>, <Priority.HIGH: 3>]
Weighted load: 11
True
CSS: color: red;
True
#0000FF

The tradeoff with IntEnum and StrEnum is that you give up some type safety — code that passes raw integers or strings will no longer be caught by a type checker. Use them when interoperability with external systems (databases, APIs, config files) is more important than strict type boundaries. For internal domain logic, prefer the base Enum.

Flag Enums: Combining Options

The Flag class is for situations where values can be combined — file permissions, feature flags, user capabilities. Each member is a power of two, and you combine them with the | operator. Checking membership uses the in operator or &.

# flag_enum.py
from enum import Flag, auto

class Permission(Flag):
    READ = auto()       # 1
    WRITE = auto()      # 2
    EXECUTE = auto()    # 4
    DELETE = auto()     # 8

    # Convenience aliases
    READ_WRITE = READ | WRITE
    ADMIN = READ | WRITE | EXECUTE | DELETE

def check_access(user_perms: Permission, required: Permission) -> bool:
    return required in user_perms

# Combine permissions with |
guest_perms = Permission.READ
editor_perms = Permission.READ | Permission.WRITE
admin_perms = Permission.ADMIN

print(f"Guest:  {guest_perms}")
print(f"Editor: {editor_perms}")
print(f"Admin:  {admin_perms}")

# Check individual permissions
print(check_access(editor_perms, Permission.WRITE))    # True
print(check_access(guest_perms, Permission.WRITE))     # False
print(check_access(admin_perms, Permission.DELETE))    # True

# Iterate over combined flags to see components
print("Admin permissions:")
for perm in admin_perms:
    print(f"  - {perm.name}")

Output:

Guest:  Permission.READ
Editor: Permission.READ|WRITE
Admin:  Permission.READ|WRITE|EXECUTE|DELETE
True
False
True
Admin permissions:
  - READ
  - WRITE
  - EXECUTE
  - DELETE

Flag is the cleanest way to represent additive permission sets in Python. The in operator checks whether a flag is set in a combination, and iterating over a combined value yields each individual flag that is included. This replaces the old pattern of bitmasking integers manually, and makes the code both readable and type-safe.

Enums in match Statements

Python 3.10’s match statement works naturally with enums and is a cleaner alternative to chains of if/elif when you need to handle multiple enum cases. The exhaustive pattern matching also helps type checkers warn you when you forget a case.

# enums_match.py
from enum import Enum, auto

class TrafficLight(Enum):
    RED = auto()
    YELLOW = auto()
    GREEN = auto()

def get_action(light: TrafficLight) -> str:
    match light:
        case TrafficLight.RED:
            return "Stop completely."
        case TrafficLight.YELLOW:
            return "Prepare to stop."
        case TrafficLight.GREEN:
            return "Proceed when safe."
        case _:
            return "Unknown signal -- stop and assess."

# Cycle through all states
for state in TrafficLight:
    print(f"{state.name}: {get_action(state)}")

Output:

RED: Stop completely.
YELLOW: Prepare to stop.
GREEN: Proceed when safe.

The match statement with enums is especially readable because each case is self-documenting — case TrafficLight.RED is unambiguous in a way that case 1 never is. Type checkers like mypy can detect missing cases when combined with _ fallthrough, giving you a compile-time reminder to handle new enum members you add later.

Real-Life Example: Task Management System

This project uses enums throughout a task management system: task status, priority, and user permissions all use the appropriate enum type. The system routes tasks, checks permissions, and generates a report.

# task_manager.py
from enum import Enum, Flag, StrEnum, auto
from dataclasses import dataclass, field
from datetime import datetime
from typing import Optional

class TaskStatus(Enum):
    TODO = auto()
    IN_PROGRESS = auto()
    BLOCKED = auto()
    REVIEW = auto()
    DONE = auto()

class Priority(StrEnum):
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"
    CRITICAL = "critical"

class TeamPermission(Flag):
    VIEW = auto()
    COMMENT = auto()
    EDIT = auto()
    CLOSE = auto()
    MANAGE = VIEW | COMMENT | EDIT | CLOSE

@dataclass
class Task:
    title: str
    status: TaskStatus = TaskStatus.TODO
    priority: Priority = Priority.MEDIUM
    assignee: Optional[str] = None
    created_at: datetime = field(default_factory=datetime.now)

    def transition(self, new_status: TaskStatus) -> None:
        valid_transitions = {
            TaskStatus.TODO: {TaskStatus.IN_PROGRESS},
            TaskStatus.IN_PROGRESS: {TaskStatus.BLOCKED, TaskStatus.REVIEW, TaskStatus.DONE},
            TaskStatus.BLOCKED: {TaskStatus.IN_PROGRESS},
            TaskStatus.REVIEW: {TaskStatus.IN_PROGRESS, TaskStatus.DONE},
            TaskStatus.DONE: set(),
        }
        if new_status not in valid_transitions[self.status]:
            raise ValueError(
                f"Cannot transition {self.status.name} -> {new_status.name}. "
                f"Valid: {[s.name for s in valid_transitions[self.status]]}"
            )
        self.status = new_status

def can_close_task(perms: TeamPermission) -> bool:
    return TeamPermission.CLOSE in perms

def print_report(tasks: list[Task]) -> None:
    by_status: dict[TaskStatus, list[Task]] = {s: [] for s in TaskStatus}
    for task in tasks:
        by_status[task.status].append(task)

    print("\n=== Task Board Report ===")
    for status, group in by_status.items():
        if group:
            print(f"\n[{status.name}]")
            for task in group:
                assignee = task.assignee or "unassigned"
                print(f"  [{task.priority.upper()}] {task.title} ({assignee})")

# --- Demo ---
tasks = [
    Task("Write API docs", priority=Priority.HIGH, assignee="alice"),
    Task("Fix login bug", priority=Priority.CRITICAL, assignee="bob"),
    Task("Update dependencies", priority=Priority.LOW),
    Task("Code review PR #42", priority=Priority.MEDIUM, assignee="carol"),
]

# Transition tasks through statuses
tasks[0].transition(TaskStatus.IN_PROGRESS)
tasks[1].transition(TaskStatus.IN_PROGRESS)
tasks[1].transition(TaskStatus.REVIEW)
tasks[2].transition(TaskStatus.IN_PROGRESS)
tasks[2].transition(TaskStatus.DONE)

# Permission check
dev_perms = TeamPermission.VIEW | TeamPermission.COMMENT | TeamPermission.EDIT
lead_perms = TeamPermission.MANAGE
print(f"Dev can close tasks: {can_close_task(dev_perms)}")
print(f"Lead can close tasks: {can_close_task(lead_perms)}")

# Report
print_report(tasks)

Output:

Dev can close tasks: False
Lead can close tasks: True

=== Task Board Report ===

[IN_PROGRESS]
  [HIGH] Write API docs (alice)
  [LOW] Update dependencies (unassigned)

[REVIEW]
  [CRITICAL] Fix login bug (bob)

[DONE]
  [LOW] Update dependencies (unassigned)

[TODO]
  [MEDIUM] Code review PR #42 (carol)

This system uses three different enum types for three different purposes: Enum for status (identity matters, no comparison to raw values needed), StrEnum for priority (values are passed directly to display strings and API responses), and Flag for permissions (combinations are the whole point). The transition() method enforces valid state changes using the enum itself as the key in a transition table — a pattern that makes invalid state transitions into immediate ValueErrors rather than silent data corruption. You can extend this by adding a @classmethod from_string to each enum for loading from database rows or API payloads.

Frequently Asked Questions

How do I serialize enums to JSON?

The standard json module does not know how to serialize enums by default — you will get a TypeError. The cleanest fix is a custom encoder: class EnumEncoder(json.JSONEncoder): def default(self, obj): return obj.value if isinstance(obj, Enum) else super().default(obj). Then pass cls=EnumEncoder to json.dumps(). Alternatively, if you are using StrEnum or IntEnum, the values serialize directly since they are plain strings or ints. For loading from JSON, use MyEnum(raw_value) to reconstruct the enum member at the deserialization boundary.

What values does auto() assign?

auto() assigns integers starting from 1 by default, incrementing by 1 for each member. You can override this behavior by defining _generate_next_value_(name, start, count, last_values) in your enum class. For example, returning name.lower() from that method makes auto() produce lowercase string values matching the member names — a shortcut that StrEnum formalized. The value of auto() is that you never have to manually number your enum members or worry about gaps when you reorder or add members.

What are enum aliases and how do I prevent them?

If two enum members have the same value, the second is an alias for the first — accessing it returns the first member. This is intentional for cases like TUESDAY = TUES = 2. To prevent accidental aliases (and get an error if you accidentally duplicate values), use the @unique decorator from the enum module: @enum.unique above the class definition raises a ValueError at class creation time if any duplicate values exist. This is a good default for most enums where each value should be distinct.

When should I use Enum vs Literal?

typing.Literal["pending", "shipped", "done"] is a type annotation for functions that accept a fixed set of string values. It is pure typing with no runtime behavior — you can pass any string and Python will not complain at runtime, only the type checker will. Enum is a runtime construct: invalid values are caught when you try to construct a member (OrderStatus("invalid") raises ValueError). Use Literal when you are annotating function signatures that accept existing string constants you do not control. Use Enum when you own the constants and want runtime validation, iteration, and type-checker integration.

What is the functional Enum API?

Python’s enum module supports a functional syntax for creating enums dynamically: Status = Enum("Status", ["PENDING", "SHIPPED", "DONE"]). This creates the same Status enum class as the class syntax, but is useful when the members are not known until runtime — for example, loading status names from a database or config file. Values are assigned starting from 1. You can also pass a dict to control values: Enum("Color", {"RED": "#FF0000", "GREEN": "#00FF00"}). The functional API produces real enum classes, not just dicts, so all the normal enum features (iteration, comparison, methods) work as expected.

Conclusion

Python enums replace the most common source of silent bugs — magic strings and integers — with named, type-checked constants that are safe to compare, iterate, and serialize. We covered the four most important types: base Enum for named constants with strict comparison, IntEnum and StrEnum for interoperability with external systems, and Flag for combinable permission and option sets. We saw how auto() removes the need to manually assign values, how enums integrate with match statements for exhaustive pattern matching, and how to enforce valid state transitions using enum-keyed transition tables.

The next step is to replace the oldest magic constants in your codebase. Start with any field that stores a status, state, type, or category as a raw string or integer — those are the high-value targets. Add @enum.unique to catch accidental value duplication, and add a classmethod for loading from external sources. For the full reference, the Python enum documentation covers every detail including functional API, mixin classes, and data enums added in Python 3.12.

How To Use Python Click for CLI Applications

How To Use Python Click for CLI Applications

Intermediate

You need to write a Python script that accepts arguments from the command line. You reach for argparse, spend 20 minutes writing boilerplate, and end up with 40 lines of setup code before your actual logic even starts. There is a better way. Click is a Python library that turns regular functions into CLI commands with a single decorator. The same script that took 40 lines of argparse takes 10 lines with Click, and it generates better help text automatically.

Click was created by the Pallets team (the same people who make Flask) and is installed with pip install click. It handles argument parsing, type validation, help text generation, and nested subcommands out of the box. Flask, Black, and dozens of other widely used tools build their CLIs on top of Click.

This article covers installing Click and writing your first command, using options and arguments with type validation, setting default values and prompting users for input, grouping commands into subcommand CLIs, and building a real file-processing tool. By the end you will have the full toolkit to replace argparse in any project.

Click in Python: Quick Example

Here is a complete, runnable Click CLI in under 15 lines. Save it as hello.py and run it from the terminal:

# hello.py
import click

@click.command()
@click.option("--name", default="World", help="Who to greet.")
@click.option("--count", default=1, type=int, help="How many times to greet.")
def greet(name, count):
    """A simple greeting command."""
    for _ in range(count):
        click.echo(f"Hello, {name}!")

if __name__ == "__main__":
    greet()

Output:

$ python hello.py
Hello, World!

$ python hello.py --name Alice --count 3
Hello, Alice!
Hello, Alice!
Hello, Alice!

$ python hello.py --help
Usage: hello.py [OPTIONS]

  A simple greeting command.

Options:
  --name TEXT     Who to greet.
  --count INTEGER How many times to greet.
  --help          Show this message and exit.

Notice that --help is generated automatically from the function docstring and the help= parameters you pass to each option. With argparse, you would have written parser = argparse.ArgumentParser(description=...), then parser.add_argument("--name", ...), then parser.add_argument("--count", ...), then args = parser.parse_args(), then called your function manually. Click collapses all of that into decorator declarations.

What Is Click and How Does It Work?

Click (Command Line Interface Creation Kit) is a decorator-based CLI library. The core concept is simple: you write a normal Python function, add @click.command() to mark it as a CLI entry point, add @click.option() or @click.argument() decorators to declare the inputs, and Click handles everything else — parsing, type conversion, validation, and help text.

The distinction between options and arguments matters in Click. Options are named parameters preceded by --, like --output file.txt. Arguments are positional parameters that appear without a name, like the filename in cat myfile.txt. Options are optional by default (you can set defaults); arguments are required by default.

FeatureargparseClick
Setup boilerplate~10 lines before any logicJust decorators
Help textManual via help=From docstring + help=
Type validationVia type=Via type=, richer built-ins
SubcommandsSubparsers (verbose)@click.group() (clean)
Password promptsManual + getpassBuilt-in with hide_input=True
File handlingManual open/closeBuilt-in click.File type
Progress barsThird-partyBuilt-in click.progressbar
Sudo Sam holding decorator symbol transforming a function
One decorator. Automatic –help. No ArgumentParser boilerplate in sight.

Options and Arguments

Options and arguments are the two building blocks of any CLI. Understanding when to use each one leads to interfaces that feel intuitive to users.

Use arguments for required, positional inputs — the thing the command operates on. Use options for modifiers that change how the command behaves. The Unix convention is: cp source destination uses arguments, cp -r uses an option. Click encourages this same pattern.

# file_tool.py
import click

@click.command()
@click.argument("source")
@click.argument("destination")
@click.option("--verbose", "-v", is_flag=True, help="Enable verbose output.")
@click.option("--dry-run", is_flag=True, help="Show what would happen without doing it.")
def copy_file(source, destination, verbose, dry_run):
    """Copy SOURCE to DESTINATION."""
    if verbose:
        click.echo(f"Copying {source} -> {destination}")
    if dry_run:
        click.echo(f"[dry-run] Would copy {source} to {destination}")
        return
    # Actual copy logic would go here
    click.echo(f"Copied {source} to {destination}")

if __name__ == "__main__":
    copy_file()

Output:

$ python file_tool.py data.csv output/data.csv --verbose
Copying data.csv -> output/data.csv
Copied data.csv to output/data.csv

$ python file_tool.py data.csv output/data.csv --dry-run
[dry-run] Would copy data.csv to output/data.csv

$ python file_tool.py --help
Usage: file_tool.py [OPTIONS] SOURCE DESTINATION

  Copy SOURCE to DESTINATION.

Options:
  -v, --verbose  Enable verbose output.
  --dry-run      Show what would happen without doing it.
  --help         Show this message and exit.

The is_flag=True parameter makes an option a boolean toggle — present means True, absent means False. The short form alias -v is declared alongside --verbose in the same decorator. Click also uppercases argument names in the help text automatically, which is how SOURCE and DESTINATION appear in all caps in the usage line.

Types and Built-In Validation

Click’s type system validates inputs before your function even runs, generating clear error messages automatically. Built-in types include INT, FLOAT, BOOL, STRING, UUID, PATH, and File. There is also click.Choice for enum-style options and click.IntRange for bounded numbers.

# types_demo.py
import click

@click.command()
@click.option(
    "--format",
    type=click.Choice(["json", "csv", "tsv"], case_sensitive=False),
    default="json",
    help="Output format."
)
@click.option(
    "--workers",
    type=click.IntRange(1, 32),
    default=4,
    show_default=True,
    help="Number of worker threads (1-32)."
)
@click.option(
    "--output",
    type=click.Path(dir_okay=False, writable=True),
    required=True,
    help="Output file path."
)
def process(format, workers, output):
    """Process data and write results."""
    click.echo(f"Processing with {workers} workers -> {output} ({format})")

if __name__ == "__main__":
    process()

Output:

$ python types_demo.py --output results.json
Processing with 4 workers -> results.json (json)

$ python types_demo.py --format xml --output results.json
Error: Invalid value for '--format': 'xml' is not one of 'json', 'csv', 'tsv'.

$ python types_demo.py --workers 100 --output results.json
Error: Invalid value for '--workers': 100 is not in the range 1<=x<=32.

$ python types_demo.py --help
Usage: types_demo.py [OPTIONS]

Options:
  --format [json|csv|tsv]   Output format.
  --workers INTEGER RANGE   Number of worker threads (1-32).  [default: 4; 1<=x<=32]
  --output PATH             Output file path.  [required]
  --help                    Show this message and exit.

Notice show_default=True on the workers option — it adds [default: 4] to the help text automatically, saving you from writing it manually. The click.Path type validates that the path is writable and not a directory without requiring any custom validation code in your function.

Debug Dee inspecting type validation with green checkmarks
Type errors caught at the CLI layer. Your function gets clean, validated inputs.

Prompts and Confirmation

Some commands need to ask the user for input at runtime — a password, a confirmation before a destructive operation, or a value when an option was not provided. Click has built-in support for all of these patterns.

# prompts_demo.py
import click

@click.command()
@click.option("--username", prompt="Username", help="Your username.")
@click.option(
    "--password",
    prompt="Password",
    hide_input=True,
    confirmation_prompt=True,
    help="Your password."
)
def login(username, password):
    """Log in to the system."""
    click.echo(f"Logging in as: {username}")
    click.echo(f"Password length: {len(password)} chars")

@click.command()
@click.argument("path")
def delete(path):
    """Delete a file at PATH."""
    confirmed = click.confirm(f"Are you sure you want to delete {path!r}?")
    if confirmed:
        click.echo(f"Deleted {path}")
    else:
        click.echo("Cancelled.")

if __name__ == "__main__":
    login()

Interactive session:

$ python prompts_demo.py login
Username: alice
Password:
Repeat for confirmation:
Logging in as: alice
Password length: 8 chars

When prompt=True (or prompt="some text") is set on an option and the option is not provided on the command line, Click prompts the user interactively. The hide_input=True flag suppresses echo for passwords. confirmation_prompt=True asks the user to type the value twice, a standard UX pattern for password entry. For destructive operations, click.confirm() shows a yes/no prompt and returns a boolean.

Groups and Subcommands

Real CLI tools like git, docker, and pip have subcommands: git commit, docker run, pip install. Click builds these with @click.group(), which turns a function into a container for sub-commands.

# cli_group.py
import click

@click.group()
def db():
    """Database management commands."""
    pass

@db.command()
@click.option("--host", default="localhost", help="Database host.")
@click.option("--port", default=5432, type=int, help="Database port.")
def connect(host, port):
    """Connect to the database."""
    click.echo(f"Connecting to {host}:{port}")

@db.command()
@click.argument("table")
@click.option("--limit", default=10, type=int, help="Max rows to show.")
def query(table, limit):
    """Query TABLE and print results."""
    click.echo(f"SELECT * FROM {table} LIMIT {limit}")

@db.command()
@click.confirmation_option(prompt="Are you sure you want to drop all tables?")
def reset():
    """Drop all tables and reset the database."""
    click.echo("Database reset.")

if __name__ == "__main__":
    db()

Output:

$ python cli_group.py --help
Usage: cli_group.py [OPTIONS] COMMAND [ARGS]...

  Database management commands.

Commands:
  connect  Connect to the database.
  query    Query TABLE and print results.
  reset    Drop all tables and reset the database.

$ python cli_group.py connect --host db.prod.example.com
Connecting to db.prod.example.com:5432

$ python cli_group.py query users --limit 5
SELECT * FROM users LIMIT 5

Sub-commands are defined on the group by using @db.command() instead of @click.command(). Each sub-command has its own options and arguments, independently declared. The @click.confirmation_option() decorator is a shorthand for adding a --yes flag that prompts for confirmation before running — ideal for destructive commands where you want a safety check without writing custom prompt logic.

Real-Life Example: CSV Processor CLI

Loop Larry surrounded by flying file folders
A CLI that actually handles real files: types, progress, and error messages included.

Here is a practical multi-command CLI for processing CSV files, combining everything covered above: typed options, file arguments, a group structure, and error handling with click.echo and click.style:

# csv_tool.py
"""
A CSV processing CLI built with Click.
Install: pip install click
Usage: python csv_tool.py --help
"""
import csv
import sys
import click

@click.group()
@click.version_option("1.0.0")
def cli():
    """CSV file processing toolkit."""
    pass

@cli.command()
@click.argument("csv_file", type=click.Path(exists=True, readable=True))
@click.option("--delimiter", default=",", help="Column delimiter.")
@click.option("--max-rows", default=10, type=int, help="Max rows to preview.")
def preview(csv_file, delimiter, max_rows):
    """Preview the first rows of a CSV file."""
    try:
        with open(csv_file, newline="", encoding="utf-8") as f:
            reader = csv.reader(f, delimiter=delimiter)
            headers = next(reader)
            click.echo(click.style("Headers: ", fg="green") + ", ".join(headers))
            click.echo(f"{'---' * 20}")
            for i, row in enumerate(reader):
                if i >= max_rows:
                    break
                click.echo(" | ".join(row))
    except StopIteration:
        click.echo(click.style("Error: Empty CSV file.", fg="red"), err=True)
        sys.exit(1)

@cli.command()
@click.argument("csv_file", type=click.Path(exists=True, readable=True))
@click.option(
    "--output",
    type=click.Path(dir_okay=False, writable=True),
    required=True,
    help="Output file path."
)
@click.option(
    "--columns",
    multiple=True,
    help="Columns to keep. Repeat for multiple: --columns id --columns name"
)
def extract(csv_file, output, columns):
    """Extract specific columns from a CSV file."""
    with open(csv_file, newline="", encoding="utf-8") as f_in:
        reader = csv.DictReader(f_in)
        fieldnames = columns if columns else reader.fieldnames
        with open(output, "w", newline="", encoding="utf-8") as f_out:
            writer = csv.DictWriter(f_out, fieldnames=fieldnames, extrasaction="ignore")
            writer.writeheader()
            row_count = 0
            for row in reader:
                writer.writerow(row)
                row_count += 1
    click.echo(click.style(f"Extracted {row_count} rows", fg="green") + f" -> {output}")

if __name__ == "__main__":
    cli()

Create a sample CSV and run it:

# First, create a sample file
$ echo "id,name,email,age
1,Alice,alice@example.com,30
2,Bob,bob@example.com,25
3,Charlie,charlie@example.com,35" > people.csv

$ python csv_tool.py preview people.csv
Headers: id, name, email, age
------------------------------------------------------------
1 | Alice | alice@example.com | 30
2 | Bob | bob@example.com | 25
3 | Charlie | charlie@example.com | 35

$ python csv_tool.py extract people.csv --output names.csv --columns id --columns name
Extracted 3 rows -> names.csv

This example shows several real-world Click patterns: click.style() for colored terminal output, multiple=True for options that can be repeated, click.Path(exists=True) for automatic file validation, and err=True in click.echo() to write to stderr instead of stdout. The @click.version_option() decorator adds a free --version flag to the group.

Frequently Asked Questions

When should I use Click instead of argparse?

Use Click for any CLI you are writing from scratch or actively maintaining. The decorator syntax is more readable and generates better help text with less code. Use argparse only if you are adding to an existing codebase that already uses it extensively, or if you are building tooling that ships in environments where adding third-party dependencies is restricted (Click is not in the standard library). For scripts you write for yourself, Click is almost always worth the pip install.

How do I test Click commands?

Click provides a CliRunner class in click.testing that lets you invoke commands programmatically without spawning a subprocess. Call runner.invoke(your_command, args=["--option", "value"]) and inspect result.output and result.exit_code. This makes CLI testing as straightforward as testing any other function, without needing to mock sys.argv or capture stdout manually.

Can Click read values from environment variables?

Yes, via the envvar parameter on @click.option(): @click.option("--api-key", envvar="API_KEY"). When the option is not provided on the command line, Click checks the environment variable. You can also set auto_envvar_prefix="MYAPP" on the group to make all options automatically check for environment variables prefixed with MYAPP_. This is a clean pattern for twelve-factor applications that configure via the environment.

How do I package a Click CLI as a pip-installable command?

In your pyproject.toml, set the entry point under [project.scripts]: my-tool = "mypackage.cli:cli". After pip install -e ., running my-tool invokes your Click group or command directly. This is how Flask exposes its flask CLI, how Black exposes the black command, and how most pip-installed tools expose their entry points.

How do I accept a variable number of values for one option?

Two ways: use multiple=True on an option to allow it to be repeated (--tag foo --tag bar), or use nargs=-1 on an argument to accept any number of positional values (tool.py file1.txt file2.txt file3.txt). With multiple=True, the value in your function is a tuple of all the provided values. With nargs=-1, the argument value is also a tuple. Both patterns are shown in the CSV tool example above (--columns uses multiple=True).

Conclusion

Click covers the full range of CLI development tasks: simple single-command scripts, complex multi-subcommand tools, interactive prompts, file handling, and colored output. The decorator pattern means your code describes the interface rather than constructing a parser object, which makes it easier to read and easier to modify later.

The key functions and decorators to remember are @click.command(), @click.group(), @click.option(), @click.argument(), click.echo(), and click.style() for colored output. From there, the official Click documentation covers advanced topics like context objects, plugin systems, and integrating Click with other frameworks.

How To Use Python more-itertools for Advanced Iteration

How To Use Python more-itertools for Advanced Iteration

Intermediate

You need to process a list in batches of 50. You need a sliding window over a time series. You need to group consecutive items by a category without sorting the entire list first. The standard library’s itertools module gets you partway there, but you end up writing the same 5-line helper function in every project: the one that chunks a list into fixed-size groups. That function already exists, is battle-tested, and has 100+ siblings in a library called more-itertools.

more-itertools is a pure-Python library that extends the standard itertools with practical, production-ready iteration utilities. It is the most downloaded iteration utility library in the Python ecosystem with tens of millions of downloads per month, and it ships with functions that cover the patterns you repeatedly hand-roll: chunked batching, sliding windows, grouping consecutive items, interleaving sequences, and more. Everything returns lazy iterators, so it handles large sequences without loading them into memory.

In this article, we will cover installing more-itertools, the most useful functions by category, how they compare to hand-written alternatives, and a real-world data pipeline that ties them together. By the end, you will recognize at least five patterns in your current codebase that more-itertools can replace with a single, well-named function call.

more-itertools: Quick Example

The most common use case: splitting a list into fixed-size chunks for batch processing.

# quick_example.py
from more_itertools import chunked

data = list(range(1, 22))  # 21 items

for batch in chunked(data, 5):
    print(batch)

Output:

[1, 2, 3, 4, 5]
[6, 7, 8, 9, 10]
[11, 12, 13, 14, 15]
[16, 17, 18, 19, 20]
[21]

chunked(iterable, n) splits any iterable into lists of at most n items. The final batch contains whatever remains — it is not padded. Compare this to writing [data[i:i+5] for i in range(0, len(data), 5)], which only works on sequences with a length; chunked works on any iterable including generators and file objects.

Read on for windowed slices, grouping utilities, interleaving, and a full data pipeline example combining several of these tools.

What Is more-itertools and Why Use It?

more-itertools is a community library maintained as a complement to Python’s built-in itertools module. Where itertools provides the low-level combinatorial primitives (chain, product, combinations), more-itertools focuses on practical, higher-level patterns that appear constantly in real data processing code. Every function returns a lazy iterator compatible with the rest of the itertools ecosystem.

FunctionWhat it doesstdlib alternative
chunked(it, n)Split into n-sized listsManual slice loop
windowed(it, n)Sliding window of size nManual deque loop
grouper(it, n)Fixed-size tuples, pads lastzip_longest + repeat
run_length.encode(it)RLE compression of runsgroupby + len
flatten(it)One level of nesting removedchain.from_iterable
interleave(*its)Interleave multiple iterableszip + chain
peekable(it)Look ahead without consumingNo clean stdlib option
first(it)First item or defaultnext() + StopIteration

The main benefit over writing these yourself is correctness: more-itertools handles edge cases (empty iterables, short final chunks, single-item sequences) that hand-rolled versions often miss. Install once, import by name, never write another chunk-splitter.

Installation

# install.sh
pip install more-itertools
python -c "import more_itertools; print(more_itertools.__version__)"

Output:

10.x.x

No dependencies beyond the Python standard library. The module is imported as more_itertools (underscore, not hyphen). Most production projects import specific functions rather than the whole module to keep imports explicit.

Sliding Windows with windowed()

A sliding window moves one step at a time over a sequence, returning overlapping tuples. This is essential for time series analysis, moving averages, and sequence pattern matching:

# windowed_example.py
from more_itertools import windowed

temperatures = [22, 24, 19, 21, 25, 23, 20, 22]

print("3-day sliding window:")
for window in windowed(temperatures, 3):
    avg = sum(window) / len(window)
    print(f"  {window} -> avg {avg:.1f}")

Output:

3-day sliding window:
  (22, 24, 19) -> avg 21.7
  (24, 19, 21) -> avg 21.3
  (19, 21, 25) -> avg 21.7
  (21, 25, 23) -> avg 23.0
  (25, 23, 20) -> avg 22.7
  (23, 20, 22) -> avg 21.7

The equivalent using only the standard library requires a collections.deque and manual appending — roughly 8 lines. windowed(it, n) reduces this to one line. By default, if the iterable is shorter than the window size, windowed pads missing values with None; pass fillvalue=0 to use a different pad value, or use windowed_complete to skip incomplete windows entirely.

Grouping Consecutive Items

Two functions handle consecutive grouping: run_length.encode() compresses repeated values, and consecutive_groups() finds runs of consecutive integers. Both operate lazily:

# grouping_example.py
from more_itertools import run_length, consecutive_groups

# Run-length encoding: compress repeated values
signals = ['A', 'A', 'A', 'B', 'B', 'A', 'C', 'C', 'C', 'C']
encoded = list(run_length.encode(signals))
print("RLE encoded:", encoded)

decoded = list(run_length.decode(encoded))
print("RLE decoded:", decoded)

print()

# Consecutive groups: group sequential integers
page_numbers = [1, 2, 3, 7, 8, 9, 10, 15, 16]
print("Consecutive page ranges:")
for group in consecutive_groups(page_numbers):
    pages = list(group)
    if len(pages) == 1:
        print(f"  Page {pages[0]}")
    else:
        print(f"  Pages {pages[0]}-{pages[-1]}")

Output:

RLE encoded: [('A', 3), ('B', 2), ('A', 1), ('C', 4)]
RLE decoded: ['A', 'A', 'A', 'B', 'B', 'A', 'C', 'C', 'C', 'C']

Consecutive page ranges:
  Pages 1-3
  Pages 7-10
  Page 15
  Pages 15-16

run_length.encode() returns (value, count) pairs, while run_length.decode() reverses the process. The consecutive_groups() function detects gaps in integer sequences — useful for summarizing page ranges, time slot gaps, or any integer-indexed data where runs matter.

Lookahead with peekable()

Sometimes you need to inspect the next item in an iterator without consuming it — a common pattern in parsers and state machines. peekable wraps any iterator and adds a .peek() method:

# peekable_example.py
from more_itertools import peekable

def process_stream(items):
    it = peekable(items)
    results = []
    while it:
        current = next(it)
        # Peek at next item without consuming it
        try:
            upcoming = it.peek()
            if upcoming > current:
                results.append(f"{current} (next is higher: {upcoming})")
            else:
                results.append(f"{current} (next is same/lower: {upcoming})")
        except StopIteration:
            results.append(f"{current} (last item)")
    return results

output = process_stream([3, 5, 5, 2, 8, 1])
for line in output:
    print(line)

Output:

3 (next is higher: 5)
5 (next is same/lower: 5)
5 (next is higher: 2)
2 (next is higher: 8)
8 (next is same/lower: 1)
1 (last item)

peekable also supports prepending values back with it.prepend(value), which is useful when you have consumed an item and need to “unread” it. The object is truthy when items remain, so while it: works as a clean loop termination condition. Without peekable, implementing lookahead requires maintaining a separate next_item variable and careful StopIteration handling.

Essential Utility Functions

Here are several more functions that cover common one-off patterns:

# utilities_example.py
from more_itertools import (
    flatten, interleave, first, last, one,
    unique_everseen, only
)

# flatten: remove one level of nesting
nested = [[1, 2], [3, 4], [5]]
print("flatten:", list(flatten(nested)))

# interleave: merge multiple iterables element by element
evens = [2, 4, 6]
odds = [1, 3, 5]
print("interleave:", list(interleave(evens, odds)))

# first / last: safe access with defaults
items = [10, 20, 30]
print("first:", first(items, default=0))
print("last:", last(items, default=0))
print("first of empty:", first([], default=-1))

# unique_everseen: deduplicate while preserving order
dupes = [3, 1, 4, 1, 5, 9, 2, 6, 5, 3]
print("unique:", list(unique_everseen(dupes)))

# one: assert exactly one item matches a predicate
numbers = [42]
print("one:", one(numbers))  # Returns 42; raises ValueError if 0 or 2+ items

# only: return the single item from a one-element iterable
print("only:", only([99], default=None))

Output:

flatten: [1, 2, 3, 4, 5]
interleave: [2, 1, 4, 3, 6, 5]
first: 10
last: 30
first of empty: -1
unique: [3, 1, 4, 5, 9, 2, 6]
one: 42
only: 99

unique_everseen is particularly useful as an order-preserving deduplication — it keeps the first occurrence of each value, unlike set() which destroys ordering. first and last cleanly handle empty iterables with a default parameter, avoiding the try/next(iter(...))/except StopIteration pattern. one() is a semantic assertion: “I expect exactly one item here” — it raises a clear error if that invariant is violated.

Real-Life Example: Batch API Processor with Progress Tracking

Here is a practical pipeline that uses chunked, windowed, and peekable together to batch-process API requests with rate limiting and look-ahead logging:

# batch_api_processor.py
import time
from more_itertools import chunked, peekable
import urllib.request
import json

def fetch_post(post_id: int) -> dict:
    """Fetch a single post from a test API."""
    url = f"https://jsonplaceholder.typicode.com/posts/{post_id}"
    with urllib.request.urlopen(url, timeout=5) as resp:
        return json.loads(resp.read())

def process_posts_in_batches(post_ids: list[int], batch_size: int = 5):
    """
    Fetch posts in batches with rate limiting.
    Uses chunked() to group IDs, peekable() to detect the last batch.
    """
    batches = peekable(chunked(post_ids, batch_size))
    batch_num = 0

    while batches:
        batch_ids = next(batches)
        batch_num += 1
        is_last = not batches  # peekable: truthy if more items remain

        print(f"\nBatch {batch_num}: fetching IDs {batch_ids}")
        results = []
        for post_id in batch_ids:
            post = fetch_post(post_id)
            results.append({
                "id": post["id"],
                "title": post["title"][:40] + "..." if len(post["title"]) > 40 else post["title"],
                "user": post["userId"]
            })

        for r in results:
            print(f"  [{r['id']}] user={r['user']}: {r['title']}")

        if not is_last:
            print("  Rate limiting: sleeping 0.5s...")
            time.sleep(0.5)
        else:
            print("  Last batch -- no sleep needed.")

    return batch_num

if __name__ == "__main__":
    ids_to_fetch = list(range(1, 12))  # 11 posts -> 3 batches of 5, 5, 1
    total_batches = process_posts_in_batches(ids_to_fetch, batch_size=5)
    print(f"\nDone. Processed {total_batches} batches.")

Output (truncated for brevity):

Batch 1: fetching IDs [1, 2, 3, 4, 5]
  [1] user=1: sunt aut facere repellat provident oc...
  [2] user=1: qui est esse...
  ...
  Rate limiting: sleeping 0.5s...

Batch 2: fetching IDs [6, 7, 8, 9, 10]
  [6] user=1: dolorem eum magni eos aperiam quia...
  ...
  Rate limiting: sleeping 0.5s...

Batch 3: fetching IDs [11]
  [11] user=1: et ea vero quia laudantium aute...
  Last batch -- no sleep needed.

Done. Processed 3 batches.

The combination of chunked and peekable enables a clean pattern: split work into batches of the right size, then use look-ahead to detect the last batch and skip the unnecessary trailing sleep. The peekable wrapper’s truthiness check (not batches) replaces the awkward “am I on the last iteration?” tracking that normally requires a counter comparison. Swap jsonplaceholder.typicode.com for any real API and adjust the batch size and sleep time for that API’s rate limits.

Frequently Asked Questions

How does more-itertools relate to the stdlib itertools module?

more-itertools is a pure addition, not a replacement. It imports and re-exports everything from itertools, so you can use from more_itertools import chain, chunked to access both standard and extended functions from one import. The library started as a collection of recipes from the official itertools documentation that were commonly hand-written but not included in the standard library, and has grown to 100+ functions over the years.

When should I use chunked() vs grouper()?

Use chunked(it, n) when you want the final partial chunk as a shorter list — the last group might have fewer than n items. Use grouper(it, n, fillvalue=None) when you need all groups to be exactly the same length — the final group is padded with the fill value. For batch database inserts where a short final batch is fine, use chunked. For fixed-width record processing where every row must be the same length, use grouper.

Does more-itertools load everything into memory?

No — like the standard itertools library, most functions return lazy iterators. chunked yields one list at a time; windowed yields one tuple at a time; none of them buffer the entire input. The exception is functions that inherently require buffering, like unique_everseen (which maintains a set of seen values) and last (which must consume the entire iterable). For truly large iterables, stick to the lazy functions and avoid the buffering ones.

Is more-itertools slow compared to hand-written code?

more-itertools is pure Python with no C extensions, so it is not faster than equivalent optimized code. For hot loops processing millions of items, the overhead of Python function calls in the iterator protocol can matter. In those cases, NumPy vectorized operations or Polars expressions will outperform any pure-Python iteration. But for typical batch sizes (hundreds to thousands of items), more-itertools performance is indistinguishable from hand-written loops, and the readability benefit is significant.

Which functions should I learn first?

Start with the ones you hand-write most often: chunked for batching, windowed for sliding analysis, flatten for one level of nesting, first and last for safe endpoint access, and unique_everseen for order-preserving deduplication. Then browse the docs for peekable, one, and consecutive_groups — these cover patterns that are tricky to get right with the standard library and that come up more often than you expect once you know they exist.

Conclusion

In this article, we covered more-itertools from installation through practical use: chunked for batch splitting, windowed for sliding windows, run_length.encode and consecutive_groups for grouping, peekable for look-ahead iteration, and a set of utility functions including flatten, unique_everseen, first, and one. The real-life example combined chunked and peekable in a rate-limited batch API processor.

The consistent theme is that more-itertools names patterns you already use. Once you know chunked exists, you stop writing the slice loop. Once you know peekable exists, you stop maintaining a next_item variable by hand. The library’s value is not raw performance — it is replacing ad-hoc implementations with functions that handle edge cases correctly and that any Python developer can read and understand immediately.

Browse the full function list in the official more-itertools documentation — there are 100+ functions, and a few of them will solve a problem you are currently solving the hard way.

How To Use Pydantic Settings for Configuration Management in Python

How To Use Pydantic Settings for Configuration Management in Python

Last Updated: June 04, 2026

Intermediate

You have a working Python app. It connects to a database, calls an external API, and fires emails — all configured by a scattered mix of os.environ.get() calls, hardcoded defaults, and a README that says “remember to set these env vars before running.” Then a teammate joins the project, misses one variable, and gets a cryptic None error three function calls deep. Sound familiar?

Configuration management is one of those problems that feels solved until your app grows. The standard library gives you os.environ, but it has no validation, no types, and no structure. Enter pydantic-settings: a library built on top of Pydantic V2 that lets you define your configuration as a typed model, load it automatically from environment variables or .env files, and get clear validation errors the moment something is missing or wrong — at startup, not buried inside a request handler.

In this article, we will cover how to install and configure pydantic-settings, how to use BaseSettings to define typed config models, how to load from .env files and environment variables, how to handle nested settings and multiple sources, and how to build a real-world app config system. By the end, you will have a battle-tested configuration pattern you can drop into any Python project.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program.

Pydantic Settings: Quick Example

Here is the simplest working example — a settings model that reads from environment variables and validates types automatically:

# quick_example.py
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    app_name: str = "MyApp"
    debug: bool = False
    port: int = 8000
    database_url: str

settings = Settings(database_url="postgresql://localhost/mydb")

print(settings.app_name)
print(settings.debug)
print(settings.port)
print(settings.database_url)

Output:

MyApp
False
8000
postgresql://localhost/mydb

The Settings class inherits from BaseSettings, which tells Pydantic to look for field values in environment variables first, then fall back to defaults. If you set PORT=9000 in your shell, settings.port will be 9000 — no extra code needed. If a required field like database_url is missing entirely, you get a clear ValidationError at import time, not a silent None somewhere unexpected.

The sections below go deeper: loading from .env files, handling secrets, nested settings, multiple environments, and a production-ready config pattern.

What Is pydantic-settings and Why Use It?

pydantic-settings is an official companion library to Pydantic V2, maintained by the Pydantic team. It extends Pydantic’s data validation model to handle configuration sources: environment variables, .env files, JSON files, secrets directories, and custom providers. The key idea is that your configuration is just a Pydantic model, so all of Pydantic’s validation and type coercion features work automatically.

Compare the main approaches to Python config management:

ApproachType safety.env supportValidation errorsComplexity
os.environ.get()None (all strings)ManualNone (silent None)Low
configparserLimitedNoWeakMedium
python-decouplePartialYesBasicLow
pydantic-settingsFull Pydantic V2YesClear ValidationErrorLow-Medium
dynaconfLimitedYesBasicHigh

The big win with pydantic-settings is that you get the full Pydantic validator ecosystem — field constraints, custom validators, computed fields, aliases — without learning a separate config DSL. If you already use Pydantic for your data models, adding pydantic-settings for config is a natural extension of the same pattern.

Installation and Setup

Install with pip. Note that pydantic-settings is a separate package from pydantic itself — you need both:

# install.sh
pip install pydantic-settings
# This also installs pydantic V2 as a dependency if not already present.
# To confirm:
python -m pip show pydantic-settings

Output:

Name: pydantic-settings
Version: 2.x.x
Requires: pydantic, python-dotenv

Notice that python-dotenv is included as a dependency — pydantic-settings uses it under the hood to load .env files, so you do not need to install or import it separately.

Loading from .env Files

The most common pattern is defining settings that load from both environment variables and a .env file. Configure this by adding a nested model_config using SettingsConfigDict:

# settings_env.py
from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env",         # load from this file
        env_file_encoding="utf-8",
        case_sensitive=False,    # PORT and port both match
        extra="ignore",          # silently ignore unknown env vars
    )

    app_name: str = "MyApp"
    debug: bool = False
    port: int = 8000
    database_url: str
    api_key: str

Create a .env file in the same directory:

# .env
APP_NAME=ProdApp
DEBUG=false
PORT=8080
DATABASE_URL=postgresql://user:pass@localhost/mydb
API_KEY=sk-abc123

Now instantiate settings — no arguments needed:

# run_settings.py
from settings_env import Settings

settings = Settings()
print(settings.app_name)
print(settings.port)
print(settings.debug)

Output:

ProdApp
8080
False

The field types enforce automatic coercion: the string "8080" from the env file becomes the integer 8080, and "false" becomes the boolean False — both happen automatically with no parsing code. If a field value cannot be coerced (say, PORT=not_a_number), you get a ValidationError immediately at startup listing every broken field, which is far better than discovering it when the server tries to bind a port.

Field Constraints and Custom Validators

Because BaseSettings is just a Pydantic model, every Pydantic field constraint works exactly as you’d expect. Use Field() to add defaults, descriptions, and range constraints:

# validated_settings.py
from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict

class AppSettings(BaseSettings):
    model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8")

    port: int = Field(default=8000, ge=1024, le=65535, description="HTTP server port")
    workers: int = Field(default=4, ge=1, le=32, description="Number of worker processes")
    log_level: str = Field(default="INFO", pattern="^(DEBUG|INFO|WARNING|ERROR|CRITICAL)$")
    secret_key: str = Field(min_length=32, description="App secret key, must be 32+ chars")

settings = AppSettings()
print(f"Running on port {settings.port} with {settings.workers} workers")
print(f"Log level: {settings.log_level}")

Output (with valid .env):

Running on port 8080 with 4 workers
Log level: INFO

If PORT=999 (below the ge=1024 minimum), you get:

pydantic_core._pydantic_core.ValidationError: 1 validation error for AppSettings
port
  Input should be greater than or equal to 1024 [type=greater_than_equal, input_value=999]

The pattern constraint on log_level ensures no one accidentally sets LOG_LEVEL=verbose and wonders why logging is not working. These constraints run at startup, so misconfigured deployments fail fast and loudly rather than producing subtle misbehavior in production.

Nested Settings Models

Real apps have groups of related settings — database config, cache config, email config. Nest them as sub-models inside your main BaseSettings. Pydantic handles the prefix-based env var lookup automatically:

# nested_settings.py
from pydantic import BaseModel
from pydantic_settings import BaseSettings, SettingsConfigDict

class DatabaseConfig(BaseModel):
    host: str = "localhost"
    port: int = 5432
    name: str = "mydb"
    user: str = "postgres"
    password: str = ""

class CacheConfig(BaseModel):
    host: str = "localhost"
    port: int = 6379
    ttl: int = 300  # seconds

class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env",
        env_nested_delimiter="__"  # DB__HOST maps to database.host
    )

    app_name: str = "MyApp"
    database: DatabaseConfig = DatabaseConfig()
    cache: CacheConfig = CacheConfig()

settings = Settings()
print(f"DB: {settings.database.host}:{settings.database.port}/{settings.database.name}")
print(f"Cache TTL: {settings.cache.ttl}s")

In the .env file, use double-underscore to set nested values:

# .env (nested example)
DATABASE__HOST=db.example.com
DATABASE__PORT=5432
DATABASE__NAME=production_db
DATABASE__USER=admin
DATABASE__PASSWORD=secretpass
CACHE__HOST=redis.example.com
CACHE__TTL=600

Output:

DB: db.example.com:5432/production_db
Cache TTL: 600s

The env_nested_delimiter="__" setting tells pydantic-settings to split on double-underscores and map to nested model attributes. Sub-models like DatabaseConfig use plain BaseModel, not BaseSettings — only the root settings class needs the BaseSettings inheritance.

Multiple Environments

A practical pattern is loading different .env files based on an environment variable like APP_ENV. This is cleaner than maintaining one large file with commented-out sections:

# multi_env_settings.py
import os
from pydantic_settings import BaseSettings, SettingsConfigDict

APP_ENV = os.getenv("APP_ENV", "development")
ENV_FILE = f".env.{APP_ENV}"  # .env.development, .env.production, .env.test

class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=(".env", ENV_FILE),  # tuple: base .env first, then override
        env_file_encoding="utf-8",
        extra="ignore",
    )

    app_name: str = "MyApp"
    debug: bool = True
    database_url: str = "sqlite:///dev.db"
    log_level: str = "DEBUG"

settings = Settings()
print(f"[{APP_ENV.upper()}] {settings.app_name}")
print(f"Debug: {settings.debug}, Log: {settings.log_level}")

Create environment-specific files:

# .env (shared base)
APP_NAME=MyApp

# .env.production
DEBUG=false
DATABASE_URL=postgresql://prod-server/mydb
LOG_LEVEL=WARNING

Output (with APP_ENV=production):

[PRODUCTION] MyApp
Debug: False, Log: WARNING

When env_file is a tuple, files are loaded in order and later files override earlier ones. This lets .env hold shared defaults while .env.production adds production-specific overrides. The real environment variables (set in the shell or via a secrets manager) always take the highest priority over any file.

Handling Secrets Safely

For production deployments, storing secrets in .env files is not ideal — they can end up in version control. pydantic-settings has a built-in SecretsSettingsSource that reads secrets from a directory of files (the pattern used by Docker secrets and Kubernetes secrets):

# secrets_settings.py
from pathlib import Path
from pydantic_settings import BaseSettings, SettingsConfigDict

class ProductionSettings(BaseSettings):
    model_config = SettingsConfigDict(
        secrets_dir="/run/secrets",  # Docker/K8s secrets mount point
        env_file=".env",
    )

    app_name: str = "MyApp"
    database_url: str      # read from /run/secrets/database_url
    api_key: str           # read from /run/secrets/api_key

# For local development, create a mock secrets directory:
# mkdir -p /tmp/secrets
# echo "postgresql://localhost/mydb" > /tmp/secrets/database_url
# echo "sk-dev-key-abc123" > /tmp/secrets/api_key

dev_settings = ProductionSettings(
    _secrets_dir="/tmp/secrets"  # override for local dev
)
print(dev_settings.app_name)
print(dev_settings.database_url[:30] + "...")

Output:

MyApp
postgresql://localhost/mydb...

Each secret is stored as a separate file in the secrets directory, with the filename matching the field name. This pattern integrates directly with Docker’s --secret flag and Kubernetes Secret volume mounts, making it straightforward to move from local .env files to a proper secrets management system in production without changing any application code.

The Singleton Settings Pattern

In most apps, you want settings loaded once at startup and reused everywhere. The cleanest way to do this with pydantic-settings uses a module-level singleton with lru_cache:

# config.py
from functools import lru_cache
from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8")

    app_name: str = "MyApp"
    debug: bool = False
    database_url: str = "sqlite:///app.db"
    secret_key: str = "change-me-in-production-please"
    api_timeout: int = 30

@lru_cache
def get_settings() -> Settings:
    return Settings()

# In your app modules, import and call:
# from config import get_settings
# settings = get_settings()

Then in any module:

# database.py
from config import get_settings

def get_db_connection():
    settings = get_settings()  # returns the same instance every time
    return settings.database_url

# main.py
from config import get_settings

settings = get_settings()
print(f"Starting {settings.app_name} in {'DEBUG' if settings.debug else 'PRODUCTION'} mode")
print(f"API timeout: {settings.api_timeout}s")

Output:

Starting MyApp in PRODUCTION mode
API timeout: 30s

The @lru_cache decorator ensures Settings() is only constructed once. This means the .env file is only read once at startup, validation only runs once, and every module that calls get_settings() gets the same object. In testing, you can call get_settings.cache_clear() to force a fresh settings load with different environment variables.

Real-Life Example: FastAPI App Config System

Here is a complete configuration system for a FastAPI application, demonstrating multiple settings groups, environment switching, and the singleton pattern all working together:

# app_config.py -- Production-ready pydantic-settings config for a FastAPI app
import os
from functools import lru_cache
from pydantic import Field, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import BaseModel

class DatabaseSettings(BaseModel):
    host: str = "localhost"
    port: int = 5432
    name: str = "app_db"
    user: str = "postgres"
    password: str = ""
    pool_size: int = 5

    @property
    def url(self) -> str:
        return f"postgresql://{self.user}:{self.password}@{self.host}:{self.port}/{self.name}"

class RedisSettings(BaseModel):
    host: str = "localhost"
    port: int = 6379
    db: int = 0

    @property
    def url(self) -> str:
        return f"redis://{self.host}:{self.port}/{self.db}"

class AppSettings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=(".env", f".env.{os.getenv('APP_ENV', 'development')}"),
        env_file_encoding="utf-8",
        env_nested_delimiter="__",
        extra="ignore",
    )

    # Core app settings
    app_name: str = "FastAPI App"
    version: str = "1.0.0"
    debug: bool = False
    secret_key: str = Field(min_length=16)
    allowed_hosts: list[str] = ["localhost", "127.0.0.1"]

    # Nested config groups
    database: DatabaseSettings = DatabaseSettings()
    redis: RedisSettings = RedisSettings()

    # External services
    smtp_host: str = "smtp.example.com"
    smtp_port: int = 587
    from_email: str = "noreply@example.com"

    @field_validator("allowed_hosts", mode="before")
    @classmethod
    def parse_allowed_hosts(cls, v):
        # Allow comma-separated string from env var
        if isinstance(v, str):
            return [h.strip() for h in v.split(",")]
        return v

@lru_cache
def get_settings() -> AppSettings:
    return AppSettings()

# Usage example
if __name__ == "__main__":
    # For testing, pass values directly
    settings = AppSettings(secret_key="my-secret-key-for-testing-only")
    print(f"App: {settings.app_name} v{settings.version}")
    print(f"DB URL: {settings.database.url}")
    print(f"Redis URL: {settings.redis.url}")
    print(f"Allowed hosts: {settings.allowed_hosts}")
    print(f"Debug mode: {settings.debug}")

Output:

App: FastAPI App v1.0.0
DB URL: postgresql://postgres:@localhost:5432/app_db
Redis URL: redis://localhost:6379/0
Allowed hosts: ['localhost', '127.0.0.1']
Debug mode: False

This pattern brings several production benefits together: the @property methods on sub-models construct connection URLs from parts (no hardcoded connection strings), the field_validator lets ALLOWED_HOSTS=api.example.com,admin.example.com work as a comma-separated env var, and the nested DatabaseSettings with DATABASE__HOST-style env vars keeps related settings grouped and namespaced. Extending this for new services is as simple as adding another BaseModel sub-class and a field on AppSettings.

Frequently Asked Questions

What takes priority — environment variables or .env files?

Environment variables always win. The priority order from highest to lowest is: actual environment variables set in the shell or container, then .env file values, then field defaults defined in the model. This means you can commit a .env.example file with safe defaults to your repo and override specific values via real environment variables in production without modifying any files. The env_file setting is just a fallback source.

What happens when a required field is missing?

You get a pydantic_core.ValidationError immediately when Settings() is called, listing every missing field. This happens at startup, before your app serves any requests. The error message clearly names the field and explains that it is required — far better than the KeyError or None-related AttributeError you would get from raw os.environ calls. Use this behavior intentionally: import your settings at module level so the app crashes fast if misconfigured.

How do I override settings in tests?

Two clean approaches work well. First, if you use the @lru_cache singleton pattern, call get_settings.cache_clear() in your test setup and set environment variables before calling get_settings() again. Second, instantiate Settings() directly in tests with keyword arguments — these override all sources: Settings(database_url="sqlite:///test.db", debug=True). Using pytest-mock or monkeypatch.setenv() also works cleanly with pydantic-settings because it respects environment variables in the standard way.

Can I load settings from JSON or TOML instead of .env files?

pydantic-settings has a built-in JsonConfigSettingsSource for JSON files, available since version 2.x. For TOML, you can write a custom settings source by subclassing BaseSettings and overriding settings_customise_sources() to add your own loader. The custom source API is well-documented and lets you chain any number of sources together with explicit priority ordering, so mixing JSON config files with environment variable overrides is straightforward.

My .env file has secrets — should I commit it?

No. Add .env to your .gitignore immediately. Commit a .env.example file instead, with placeholder values and comments explaining what each variable does and how to get the real values. For shared development environments, use a secrets manager like 1Password Secrets Automation, AWS Secrets Manager, or Doppler, which can generate the .env file locally without any secrets being stored in the repo. In production, set environment variables directly in your container orchestration or PaaS platform and skip .env files entirely.

Are environment variable names case-sensitive?

By default, pydantic-settings is case-insensitive when matching environment variable names to field names. PORT, port, and Port will all match a field named port. You can change this by setting case_sensitive=True in SettingsConfigDict. On Windows, environment variables are always case-insensitive at the OS level regardless of this setting. Stick to uppercase by convention to avoid confusion across operating systems.

Conclusion

In this article, we covered pydantic-settings from the ground up: installing the library, defining typed settings models with BaseSettings, loading from .env files and environment variables, adding field constraints for validation, using nested models with env_nested_delimiter, handling secrets safely, and building a singleton config pattern with @lru_cache. The real-life example tied all of these patterns together in a FastAPI-ready configuration system.

The central benefit of this approach is that configuration errors surface at startup as clear ValidationErrors with every broken field listed, rather than failing silently at runtime. That one change — moving from scattered os.environ.get() calls to a single validated Settings class — makes misconfigured deployments immediately obvious and makes your configuration self-documenting through field types, defaults, and descriptions.

To go further, check out the official pydantic-settings documentation for advanced topics like custom settings sources, secrets rotation, and integration with cloud parameter stores.

How To Use Python cachetools for In-Memory Caching

How To Use Python cachetools for In-Memory Caching

Intermediate

Your API call takes 300 milliseconds. You call it 50 times a minute, and the data changes only every few hours. That is 50 redundant round-trips, 50 sets of network overhead, and 50 chances for a transient failure to surface to your users. Caching the result for 10 minutes would cut 99% of those calls while keeping the data fresh enough for any reasonable use case. Python’s standard library offers functools.lru_cache, but it has no expiry time, no maximum memory limit, and no way to cache across multiple function arguments without manual key management. cachetools fills all these gaps.

cachetools is a Python library that provides several ready-made cache classes: LRU (Least Recently Used), TTL (Time to Live), LFU (Least Frequently Used), and more. Each cache is a dictionary-like object with a configurable maximum size and an eviction policy that removes entries when the cache is full. Install it with pip install cachetools. There are no mandatory dependencies — cachetools is pure Python and works in any environment.

This article covers the four most useful cache types (LRU, TTL, LFU, RR), using the @cached and @cachedmethod decorators for automatic function memoization, handling thread safety in concurrent applications, cache invalidation strategies, and a real-world example of caching API responses in a Flask application. By the end you will be able to add intelligent caching to any Python function in under five lines of code.

cachetools Quick Example

The fastest way to cache a function’s results is the @cached decorator with an LRU cache:

# quick_cache.py
import time
import cachetools
from cachetools import cached, LRUCache

@cached(cache=LRUCache(maxsize=128))
def get_user_data(user_id: int) -> dict:
    """Simulate a slow database or API call."""
    time.sleep(0.5)  # Simulate 500ms latency
    return {"id": user_id, "name": f"User {user_id}", "score": user_id * 10}

# First call: slow (cache miss)
start = time.perf_counter()
result = get_user_data(42)
print(f"First call:  {(time.perf_counter()-start)*1000:.0f}ms -- {result}")

# Second call: instant (cache hit)
start = time.perf_counter()
result = get_user_data(42)
print(f"Second call: {(time.perf_counter()-start)*1000:.0f}ms -- {result}")

# Different argument: slow again (cache miss)
start = time.perf_counter()
result = get_user_data(99)
print(f"Third call:  {(time.perf_counter()-start)*1000:.0f}ms -- {result}")

Output:

First call:  502ms -- {'id': 42, 'name': 'User 42', 'score': 420}
Second call:   0ms -- {'id': 42, 'name': 'User 42', 'score': 420}
Third call:  501ms -- {'id': 99, 'name': 'User 99', 'score': 990}

The second call returns in under 1 millisecond because the result is already stored in the LRU cache. The third call is slow again because user_id=99 is a different cache key from user_id=42. The cache key is derived from the function arguments by default — you can customize it with the key parameter. The maxsize=128 means the cache holds up to 128 distinct argument combinations; older entries are evicted when the limit is reached.

What Is cachetools and Which Cache Type Should You Use?

cachetools provides several cache implementations, each with a different eviction policy. The eviction policy determines which entry gets removed when the cache is full and a new item needs to be stored. Choosing the right eviction policy depends on your access patterns.

CacheEviction policyBest forHas TTL?
LRUCacheLeast Recently UsedGeneral purpose, most common patternNo
TTLCacheTime To Live + LRUAPI responses, config data that expiresYes
LFUCacheLeast Frequently UsedNon-uniform access, popular items differNo
RRCacheRandom ReplacementUniform access, simple evictionNo
MRUCacheMost Recently UsedSequential scans, most recent is least usefulNo

LRUCache is the right default for most applications — it keeps the items you accessed most recently and evicts items that have not been used for a while, which aligns with real access patterns (hot items get accessed repeatedly, cold items do not). TTLCache adds an expiry time, which is essential when caching data that changes over time, such as API responses or database records. Use LFUCache when a small subset of items is accessed far more often than others — it prioritizes keeping the most popular items regardless of recency.

Cache Katie organizing cache shelves with LRU and TTL containers
maxsize=128 and a TTL. Your database connection pool says thank you.

LRUCache — Least Recently Used

LRUCache keeps the N most recently accessed entries. When the cache is full and a new entry arrives, the least recently used entry is evicted. This is the right choice for function results that are expensive to compute and accessed repeatedly with the same arguments.

# lru_cache_example.py
from cachetools import LRUCache, cached

# Create a cache that holds at most 3 entries
cache = LRUCache(maxsize=3)

@cached(cache=cache)
def compute_fibonacci(n: int) -> int:
    """Compute Fibonacci number -- expensive for large n."""
    if n < 2:
        return n
    return compute_fibonacci(n - 1) + compute_fibonacci(n - 2)

# Fill the cache: fib(10), fib(11), fib(12) are cached
print(compute_fibonacci(10))  # 55
print(compute_fibonacci(11))  # 89
print(compute_fibonacci(12))  # 144

print(f"Cache size: {len(cache)}")   # 3
print(f"Cache keys: {list(cache)}")  # [(10,), (11,), (12,)]

# Access a new value -- fib(13) evicts the LRU entry (fib(10))
print(compute_fibonacci(13))
print(f"Cache keys after: {list(cache)}")  # [(11,), (12,), (13,)]

# Inspect cache statistics
print(f"Cache info: {compute_fibonacci.cache_info()}")

Output:

55
89
144
Cache size: 3
Cache keys: [(10,), (11,), (12,)]
233
Cache keys after: [(11,), (12,), (13,)]
Cache info: CacheInfo(hits=X, misses=X, maxsize=3, currsize=3)

The cache is a regular dictionary-like object you can inspect, clear, and manipulate directly. cache.clear() invalidates all entries at once, and del cache[(10,)] invalidates a specific entry. This manual control is something functools.lru_cache does not easily support.

TTLCache -- Time To Live

TTLCache combines an LRU eviction policy with an expiry time. Every entry in the cache is considered stale after ttl seconds and will be evicted automatically on the next access or when iterating the cache. This makes TTLCache ideal for caching external data that changes over time.

# ttl_cache_example.py
import time
from cachetools import TTLCache, cached

# Cache holds up to 100 entries, each expires after 10 seconds
ttl_cache = TTLCache(maxsize=100, ttl=10)

@cached(cache=ttl_cache)
def fetch_exchange_rate(currency: str) -> float:
    """Simulate fetching a live exchange rate."""
    print(f"  [API call] Fetching rate for {currency}...")
    # In production, this would call a real API
    rates = {"USD": 1.0, "EUR": 0.92, "GBP": 0.79, "JPY": 149.5}
    return rates.get(currency, 1.0)

print("First fetch (cache miss):")
rate = fetch_exchange_rate("EUR")
print(f"EUR rate: {rate}")

print("\nSecond fetch (cache hit -- no API call):")
rate = fetch_exchange_rate("EUR")
print(f"EUR rate: {rate}")

print(f"\nCache has {len(ttl_cache)} entry, expires in {ttl_cache.timer() - ttl_cache['EUR']:.1f}s" if False else "")

# Simulate cache expiry by using a very short TTL
short_cache = TTLCache(maxsize=10, ttl=1)

@cached(cache=short_cache)
def get_timestamp(key: str) -> float:
    return time.time()

t1 = get_timestamp("a")
time.sleep(0.5)
t2 = get_timestamp("a")  # Hit -- same cached value
time.sleep(0.6)
t3 = get_timestamp("a")  # Miss -- TTL expired, new value

print(f"\nTimestamp test:")
print(f"t1={t1:.3f}, t2={t2:.3f} (same -- cache hit)")
print(f"t3={t3:.3f} (different -- TTL expired, new call)")
print(f"t3 > t1: {t3 > t1}")

Output:

First fetch (cache miss):
  [API call] Fetching rate for EUR...
EUR rate: 0.92

Second fetch (cache hit -- no API call):
EUR rate: 0.92

Timestamp test:
t1=1717300000.123, t2=1717300000.123 (same -- cache hit)
t3=1717300001.234 (different -- TTL expired, new call)
t3 > t1: True

The TTL is measured from when the entry is first inserted, not from when it is last accessed. An entry inserted at t=0 with ttl=60 expires at t=60 regardless of how many times it was read in between. If you need sliding expiry (where an access resets the timer), you must manage that manually by deleting and re-inserting the entry on each access.

Debug Dee watching cached API response bubbles expire on a countdown timer
ttl=300. Five minutes of blissful ignorance before your cache admits it doesn't know the current price.

Thread Safety with cachetools

cachetools cache objects are NOT thread-safe by default. If multiple threads read from and write to the cache concurrently, you will get race conditions and corrupt cache state. cachetools provides a Lock parameter for the @cached decorator to serialize access:

# thread_safe_cache.py
import threading
import time
from cachetools import TTLCache, cached

# Thread-safe cache using a threading lock
cache = TTLCache(maxsize=200, ttl=60)
lock = threading.RLock()

@cached(cache=cache, lock=lock)
def get_config(config_key: str) -> str:
    """Simulate a slow config read."""
    time.sleep(0.1)
    configs = {
        "db_host": "postgres.internal:5432",
        "redis_url": "redis://cache.internal:6379",
        "feature_flags": "new_ui=true,dark_mode=false",
    }
    return configs.get(config_key, "")

# Simulate 10 concurrent threads all requesting the same config key
results = []
errors = []

def worker(key: str):
    try:
        value = get_config(key)
        results.append(value)
    except Exception as e:
        errors.append(str(e))

threads = [threading.Thread(target=worker, args=("db_host",)) for _ in range(10)]
start = time.perf_counter()
for t in threads:
    t.start()
for t in threads:
    t.join()
elapsed = time.perf_counter() - start

print(f"10 threads completed in {elapsed:.2f}s")
print(f"All results identical: {len(set(results)) == 1}")
print(f"Result: {results[0]}")
print(f"Errors: {errors}")
print(f"Cache size: {len(cache)}")

Output:

10 threads completed in 0.10s
All results identical: True
Result: postgres.internal:5432
Cache size: 1

With the lock=threading.RLock() parameter, the first thread to request "db_host" acquires the lock, makes the slow call, and populates the cache. All other threads wait for the lock, then immediately get the cached value without making additional calls. The total time is roughly one slow call (100ms) instead of ten (1000ms). Always pass a lock when your cached function will be called from multiple threads -- this includes web frameworks like Flask and Django where request handlers run concurrently.

Caching Instance Methods with @cachedmethod

The @cached decorator creates a single shared cache for all calls to a function. For class methods, you often want each instance to have its own cache. Use @cachedmethod with a cache accessor function:

# cachedmethod_example.py
import threading
from cachetools import TTLCache, cachedmethod
from cachetools.keys import hashkey

class WeatherService:
    """Fetches weather data with per-instance TTL caching."""

    def __init__(self, api_key: str, cache_ttl: int = 300):
        self.api_key = api_key
        self._cache = TTLCache(maxsize=50, ttl=cache_ttl)
        self._lock = threading.RLock()

    @cachedmethod(cache=lambda self: self._cache, lock=lambda self: self._lock)
    def get_weather(self, city: str) -> dict:
        """Simulate API call -- 200ms delay."""
        import time
        time.sleep(0.2)
        # In production: return requests.get(f"https://api.weather.com/{city}").json()
        return {
            "city": city,
            "temp_c": len(city) * 3,   # predictable fake data
            "humidity": 65,
            "description": "Partly cloudy",
        }

    def clear_cache(self):
        self._cache.clear()

    @property
    def cache_size(self):
        return len(self._cache)

# Each instance gets its own cache
svc1 = WeatherService(api_key="key-abc", cache_ttl=300)
svc2 = WeatherService(api_key="key-xyz", cache_ttl=60)

import time
start = time.perf_counter()
print(svc1.get_weather("Sydney"))
print(f"First call: {(time.perf_counter()-start)*1000:.0f}ms")

start = time.perf_counter()
print(svc1.get_weather("Sydney"))  # cache hit in svc1
print(f"Second call (svc1): {(time.perf_counter()-start)*1000:.0f}ms")

start = time.perf_counter()
print(svc2.get_weather("Sydney"))  # cache miss in svc2 (separate instance)
print(f"First call (svc2): {(time.perf_counter()-start)*1000:.0f}ms")

print(f"\nsvc1 cache size: {svc1.cache_size}")
print(f"svc2 cache size: {svc2.cache_size}")

Output:

{'city': 'Sydney', 'temp_c': 18, 'humidity': 65, 'description': 'Partly cloudy'}
First call: 202ms
{'city': 'Sydney', 'temp_c': 18, 'humidity': 65, 'description': 'Partly cloudy'}
Second call (svc1): 0ms
{'city': 'Sydney', 'temp_c': 18, 'humidity': 65, 'description': 'Partly cloudy'}
First call (svc2): 201ms

svc1 cache size: 1
svc2 cache size: 1

The lambda self: self._cache accessor tells @cachedmethod which cache object to use for each instance. Because each instance stores its own self._cache, the two WeatherService objects have independent caches with independent TTLs. This pattern is especially useful when different instances connect to different backends or have different freshness requirements.

Pyro Pete igniting separate cache containers each with countdown timers
cachedmethod. Because your test instance shouldn't be sharing cache state with production.

Real-Life Example: Caching API Responses in a Flask App

The following Flask application caches responses from a public REST API, demonstrating TTL caching, cache inspection, and manual invalidation:

# flask_cached_api.py
import time
import threading
from flask import Flask, jsonify
from cachetools import TTLCache, cached

app = Flask(__name__)

# Cache up to 200 responses, each valid for 5 minutes
response_cache = TTLCache(maxsize=200, ttl=300)
cache_lock = threading.RLock()

import urllib.request
import json as _json

def _fetch_post(post_id: int) -> dict:
    """Fetch a post from JSONPlaceholder (real public API)."""
    url = f"https://jsonplaceholder.typicode.com/posts/{post_id}"
    with urllib.request.urlopen(url, timeout=5) as resp:
        return _json.loads(resp.read())

@cached(cache=response_cache, lock=cache_lock)
def get_post_cached(post_id: int) -> dict:
    """Return post data, hitting cache if available."""
    return _fetch_post(post_id)

@app.route("/posts/")
def post_detail(post_id: int):
    start = time.perf_counter()
    data = get_post_cached(post_id)
    elapsed_ms = (time.perf_counter() - start) * 1000
    return jsonify({
        "data": data,
        "cache_size": len(response_cache),
        "elapsed_ms": round(elapsed_ms, 1),
    })

@app.route("/posts//invalidate", methods=["DELETE"])
def invalidate_post(post_id: int):
    key = (post_id,)
    with cache_lock:
        if key in response_cache:
            del response_cache[key]
            return jsonify({"invalidated": True, "post_id": post_id})
    return jsonify({"invalidated": False, "reason": "not in cache"})

@app.route("/cache/stats")
def cache_stats():
    return jsonify({
        "size": len(response_cache),
        "maxsize": response_cache.maxsize,
        "ttl_seconds": response_cache.ttl,
    })

if __name__ == "__main__":
    app.run(debug=True, port=5000)

Testing the API:

# Terminal -- run the Flask app first, then:

# First request (cache miss -- ~120ms API round-trip)
curl http://localhost:5000/posts/1
# {"data":{"userId":1,"id":1,"title":"sunt aut facere..."},"cache_size":1,"elapsed_ms":124.3}

# Second request (cache hit -- under 1ms)
curl http://localhost:5000/posts/1
# {"data":{"userId":1,"id":1,"title":"sunt aut facere..."},"cache_size":1,"elapsed_ms":0.1}

# Check cache stats
curl http://localhost:5000/cache/stats
# {"maxsize":200,"size":1,"ttl_seconds":300}

# Invalidate post 1 manually
curl -X DELETE http://localhost:5000/posts/1/invalidate
# {"invalidated":true,"post_id":1}

# Next request fetches fresh data
curl http://localhost:5000/posts/1
# {"data":{...},"cache_size":1,"elapsed_ms":118.7}

This pattern -- TTLCache + threading lock + manual invalidation endpoint -- covers 90% of real API caching scenarios. The TTL handles the common case (data goes stale and we eventually want fresh data), the lock handles concurrent requests (only one thread calls the API for each unique post ID), and the DELETE endpoint handles the uncommon case (we know the data changed and want to force a refresh immediately).

Frequently Asked Questions

What is the difference between cachetools and functools.lru_cache?

The main differences are expiry, size control, and flexibility. functools.lru_cache is built in to Python, never expires its entries, and cannot be shared across instances. cachetools adds TTL-based expiry, several eviction policies beyond LRU, and a cache object you can inspect and manipulate independently of the cached function. If you need expiry or you need to invalidate specific cache entries, use cachetools. For simple memoization with no expiry, functools.lru_cache is simpler.

How do I cache functions with unhashable arguments?

cachetools cache keys must be hashable by default. If your function takes a list, dict, or other unhashable type, use a custom key function. For example, to cache a function that takes a list: @cached(cache=LRUCache(128), key=lambda lst: tuple(sorted(lst))). The key function converts the unhashable argument to a hashable representation. cachetools also provides cachetools.keys.hashkey and cachetools.keys.typedkey for common scenarios.

Does cachetools work with async functions?

The standard @cached decorator does not work with async def functions. For async code, you need to either use a synchronous wrapper (call the async function synchronously inside a cached sync function) or use an async-aware caching library like asyncache, which provides @acached and @acached_method decorators compatible with cachetools cache classes. The cachetools cache objects themselves are compatible with async code as long as you use an appropriate async lock such as asyncio.Lock.

How do I handle cache stampede (thundering herd)?

Cache stampede happens when many concurrent requests arrive for an expired cache entry simultaneously -- they all miss the cache, all call the underlying function at the same time, and all receive the result within milliseconds of each other, flooding your backend. The threading lock pattern shown in this article prevents stampede by serializing cache misses: only one thread calls the underlying function at a time. For async applications with very high concurrency, consider adding a probabilistic early expiry (compute a new value when the TTL is 80% elapsed with some probability) or use a separate in-progress flag to detect and suppress parallel computations.

When should I use Redis instead of cachetools?

Use cachetools when the cached data only needs to survive within a single process. Use Redis when you need to share the cache across multiple processes or machines (such as multiple web server workers), when the cache must survive process restarts, or when the cached data is large enough that storing it in-process would consume too much RAM. A common production architecture uses both: cachetools for a fast per-process L1 cache with a short TTL, and Redis for a shared L2 cache with a longer TTL.

Conclusion

cachetools gives you LRU, TTL, LFU, and random-replacement caches as plain Python objects, plus the @cached and @cachedmethod decorators for zero-friction function memoization. You have seen how to apply LRUCache for general memoization, TTLCache for time-expiring API responses, thread-safe caching with lock=threading.RLock(), per-instance caching with @cachedmethod, and a real Flask application with manual cache invalidation.

The best next step is to profile your application and find the three slowest function calls that are called repeatedly with the same arguments. Wrap each one with a TTLCache and an appropriate TTL -- 60 seconds for data that changes often, 300 seconds for data that changes rarely. Measure the before-and-after response times and cache hit rates. For most applications, this change alone produces a measurable improvement in throughput and latency.

For the full cachetools API reference including MRU, RR caches, and custom key functions, see the official cachetools documentation.

How To Use Python uv for Modern Package Management

How To Use Python uv for Modern Package Management

Last Updated: June 02, 2026

Intermediate

You run pip install -r requirements.txt and wait. Then wait some more. The resolver spins through dependency trees, contacts PyPI repeatedly, and installs packages one at a time. On a fresh environment with 40 packages, this takes two or three minutes — every time. On CI/CD it burns build minutes and developer patience alike. There is a better way.

uv is a Python package manager written in Rust that is 10x to 100x faster than pip. It handles virtual environments, dependency resolution, package installation, and project management in a single tool — without the overhead of pip’s pure-Python resolver. A full pip install -r requirements.txt that takes 3 minutes drops to under 5 seconds with uv. Install it with curl -LsSf https://astral.sh/uv/install.sh | sh on macOS and Linux, or powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" on Windows. No Python install required — uv is a self-contained binary.

This article covers installing and configuring uv, creating and activating virtual environments, installing packages and locking dependencies, managing projects with pyproject.toml, running scripts in isolated environments, and migrating from pip and pip-tools. By the end you will have a complete mental model for using uv in daily Python development and CI/CD pipelines.

Pubs - Python How To Program
Written by Pubs

Python developer and educator with 15+ years building production systems across data engineering, web APIs, and AI tooling. Founder of Python How To Program.

uv Quick Example

The most common task: create a virtual environment and install packages. Here is how it loo