Intermediate

You write a Python script on your Mac, push it to GitHub, and your colleague pulls it on Windows — and it immediately crashes. The culprit is almost always file paths using /, line endings, or platform-specific system calls that silently break when moved to a different OS. Cross-platform compatibility is one of those problems that seems trivial until it bites you in production, and fixing it after the fact means tracking down subtle bugs across three operating systems at once.

The good news: Python was designed with portability in mind. The standard library includes pathlib, os, platform, sys, and shutil — tools that abstract over OS differences so your code runs identically on Windows, macOS, and Linux. You don’t need third-party libraries to write portable Python. You just need to know which patterns to avoid and which built-in tools to use instead.

In this guide, you’ll learn the biggest cross-platform pitfalls and how to avoid them. We’ll cover file path handling with pathlib, detecting the operating system, environment variable access, line ending normalization, executable detection, and common gotchas with file permissions and temporary files. By the end, your scripts will run cleanly on any platform your team or users might have.

Cross-Platform Python: Quick Example

Here is a minimal script that works correctly on Windows, macOS, and Linux. It reads a config file from the user’s home directory without any hardcoded path separators:

# cross_platform_quick.py
from pathlib import Path
import os
import sys

# Never use hardcoded slashes -- pathlib handles separators automatically
config_dir = Path.home() / ".config" / "myapp"
config_file = config_dir / "settings.txt"

# Create the directory if it doesn't exist (works on all platforms)
config_dir.mkdir(parents=True, exist_ok=True)

# Write a config file
config_file.write_text("debug=True\nversion=1.0\n")

# Read it back
content = config_file.read_text()
print(f"Config path: {config_file}")
print(f"Content:\n{content}")
print(f"Running on: {sys.platform}")

Output (macOS):

Config path: /Users/alice/.config/myapp/settings.txt
Content:
debug=True
version=1.0

Running on: darwin

Output (Windows):

Config path: C:\Users\alice\.config\myapp\settings.txt
Content:
debug=True
version=1.0

Running on: win32

The key insight: Path.home() / ".config" / "myapp" uses Python’s / operator overload to build paths — no hardcoded separators, no os.path.join() string juggling. The resulting path string uses the correct separator for whatever OS the code is running on. Keep reading to learn the patterns that make everything else portable too.

Why Cross-Platform Python Matters

Python runs on Windows, macOS, and Linux, but each OS has different conventions for several things that developers touch constantly: file path separators (\ on Windows, / on Unix), line endings (CRLF on Windows, LF on Unix), executable file extensions (.exe on Windows, none on Unix), and environment variable availability. Scripts written without thinking about these differences produce code that works perfectly on one machine and fails mysteriously on another.

The most common offenders are hardcoded file paths like "C:\\Users\\alice\\data.csv" or "/home/alice/data.csv". Neither works on the other platform. The fix is always to use pathlib.Path with relative references or home-directory-based anchors.

Understanding this table is the foundation. Every pattern in this article addresses one or more of these differences.

File Paths with pathlib

The single most impactful change you can make to write portable Python is replacing all string-based path operations with pathlib.Path. It was added in Python 3.4 and makes path manipulation safe, readable, and OS-independent.

Building Paths

Use the / operator to join path components. Python overloads this operator on Path objects to build correct paths regardless of platform — it uses os.sep internally.

# pathlib_basics.py
from pathlib import Path

# Never do this -- breaks on other platforms:
# bad_path = "data/reports/2024/q1.csv"   # fails on Windows
# bad_path = "data\\reports\\2024\\q1.csv" # fails on Unix

# Do this instead:
data_dir = Path("data") / "reports" / "2024"
report = data_dir / "q1.csv"

print(f"Path: {report}")
print(f"Parent: {report.parent}")
print(f"Name: {report.name}")
print(f"Stem: {report.stem}")
print(f"Suffix: {report.suffix}")
print(f"Parts: {report.parts}")

Output (Linux/macOS):

Path: data/reports/2024/q1.csv
Parent: data/reports/2024
Name: q1.csv
Stem: q1
Suffix: .csv
Parts: ('data', 'reports', '2024', 'q1.csv')

Output (Windows):

Path: data\reports\2024\q1.csv
Parent: data\reports\2024
Name: q1.csv
Stem: q1
Suffix: .csv
Parts: ('data', 'reports', '2024', 'q1.csv')

The path display differs, but the programmatic interface is identical. Code that calls report.name or report.suffix works the same on both platforms.

Absolute and Home-Relative Paths

Use Path.home() to anchor paths to the user’s home directory, and Path.cwd() for the current working directory. These return the platform-correct path automatically.

# path_anchors.py
from pathlib import Path

# Home directory -- fully cross-platform
home = Path.home()
documents = home / "Documents"
downloads = home / "Downloads"

# Current working directory
here = Path.cwd()
output = here / "output" / "results.txt"

# Resolve relative paths to absolute
relative = Path("../data/config.json")
absolute = relative.resolve()

print(f"Home: {home}")
print(f"Documents: {documents}")
print(f"Output: {output}")
print(f"Resolved: {absolute}")

Output (macOS):

Home: /Users/alice
Documents: /Users/alice/Documents
Output: /Users/alice/projects/output/results.txt
Resolved: /Users/alice/data/config.json

Path.resolve() handles .. traversal and symlinks, giving you a clean absolute path regardless of how the relative path was constructed.

Detecting the Operating System

Sometimes you genuinely need to do something different per platform — for example, opening a file in the default app, clearing the terminal, or locating a system-level config. Use sys.platform for lightweight checks and platform module for detailed information.

# detect_os.py
import sys
import platform

# Quick OS detection
if sys.platform == "win32":
    os_name = "Windows"
elif sys.platform == "darwin":
    os_name = "macOS"
else:
    os_name = "Linux/Unix"

print(f"OS: {os_name}")
print(f"sys.platform: {sys.platform}")

# Detailed info via platform module
print(f"System: {platform.system()}")          # 'Windows', 'Darwin', 'Linux'
print(f"Release: {platform.release()}")         # e.g., '10', '22.3.0', '5.15.0'
print(f"Machine: {platform.machine()}")         # 'AMD64', 'arm64', 'x86_64'
print(f"Python: {platform.python_version()}")  # e.g., '3.12.1'
print(f"Node: {platform.node()}")               # hostname

Output (macOS on Apple Silicon):

OS: macOS
sys.platform: darwin
System: Darwin
Release: 22.3.0
Machine: arm64
Python: 3.12.1
Node: alices-macbook.local

Use sys.platform for conditional code paths and platform.system() when you need human-readable output or when reporting system info in logs. Avoid using environment variables like OS or OSTYPE — they’re not reliably set on all systems.

Environment Variables

Environment variables are the standard way to pass configuration (API keys, database URLs, feature flags) to Python programs. But reading them correctly across platforms requires a few patterns to avoid crashes on missing variables.

# env_vars.py
import os
from pathlib import Path

# Safe read -- returns None if not set (never raises KeyError)
api_key = os.environ.get("API_KEY")

# Read with a default value
debug = os.environ.get("DEBUG", "false").lower() == "true"
port = int(os.environ.get("PORT", "8080"))

# Read required variable -- raise early with a clear message
def require_env(name):
    value = os.environ.get(name)
    if value is None:
        raise EnvironmentError(
            f"Required environment variable '{name}' is not set. "
            f"Set it before running this script."
        )
    return value

# Example usage
try:
    db_url = require_env("DATABASE_URL")
except EnvironmentError as e:
    print(f"Configuration error: {e}")
    db_url = "sqlite:///local.db"

print(f"API key set: {api_key is not None}")
print(f"Debug mode: {debug}")
print(f"Port: {port}")
print(f"DB: {db_url}")

# List all environment variables matching a prefix
python_vars = {k: v for k, v in os.environ.items() if k.startswith("PYTHON")}
print(f"PYTHON* vars: {python_vars}")

Output:

Configuration error: Required environment variable 'DATABASE_URL' is not set. Set it before running this script.
API key set: False
Debug mode: False
Port: 8080
DB: sqlite:///local.db
PYTHON* vars: {'PYTHONPATH': '/usr/local/lib/python3.12', 'PYTHONDONTWRITEBYTECODE': '1'}

Always use os.environ.get() instead of os.environ["KEY"]. The bracket notation raises a KeyError when the variable is missing — which means your program crashes with a confusing error instead of a helpful message. The require_env() pattern above gives you the crash-on-missing behavior with a clear error message that tells users exactly what they need to set.

Temp Files and Directories

Never hardcode temporary file paths like /tmp/myapp_cache.tmp — that path doesn’t exist on Windows. Use Python’s tempfile module, which always points to the correct temp directory for the current platform.

# temp_files.py
import tempfile
from pathlib import Path

# Create a named temporary file (auto-deleted when closed)
with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=True) as tmp:
    tmp.write("temporary data\n")
    tmp.flush()
    print(f"Temp file: {tmp.name}")
    # File exists here
print("File deleted after context exit")

# Create a persistent temp file (you manage deletion)
fd, path = tempfile.mkstemp(suffix='.csv', prefix='report_')
import os
os.close(fd)  # Close the file descriptor immediately
tmp_path = Path(path)
tmp_path.write_text("col1,col2\n1,2\n")
print(f"Persistent temp: {tmp_path}")
print(f"Exists: {tmp_path.exists()}")
tmp_path.unlink()  # Delete it when done

# Create a temp directory
with tempfile.TemporaryDirectory() as tmp_dir:
    work_dir = Path(tmp_dir)
    output = work_dir / "results.json"
    output.write_text('{"status": "ok"}')
    print(f"Temp dir: {work_dir}")
    print(f"Contains: {list(work_dir.iterdir())}")
# Directory and all contents deleted here

Output (Linux):

Temp file: /tmp/tmp8x2kf9ab.txt
File deleted after context exit
Persistent temp: /tmp/report_abc12345.csv
Exists: True
Temp dir: /tmp/tmpwqpfgh3j
Contains: [PosixPath('/tmp/tmpwqpfgh3j/results.json')]

Output (Windows):

Temp file: C:\Users\alice\AppData\Local\Temp\tmp8x2kf9ab.txt
File deleted after context exit
Persistent temp: C:\Users\alice\AppData\Local\Temp\report_abc12345.csv
Exists: True
Temp dir: C:\Users\alice\AppData\Local\Temp\tmpwqpfgh3j
Contains: [WindowsPath('C:\\Users\\alice\\AppData\\Local\\Temp\\tmpwqpfgh3j\\results.json')]

The same code. Two completely different paths. Zero manual path handling. This is exactly how cross-platform code should work.

Handling Line Endings

Windows uses \r\n (CRLF) for line endings while macOS and Linux use \n (LF). If you read a file in binary mode and write it back, or pass it through a process that doesn’t normalize line endings, you’ll end up with mixed or double-CR content. Python handles this for you in text mode — but only if you use text mode.

# line_endings.py
from pathlib import Path

# Python normalizes line endings when reading in text mode (default)
# This works correctly on all platforms
path = Path("data.txt")
path.write_text("line one\nline two\nline three\n")

# Text mode read -- line endings normalized to \n regardless of OS
lines = path.read_text().splitlines()
print(f"Lines: {lines}")

# Explicit newline control when writing
# newline='' disables translation (use for CSV files to avoid double-CR)
import csv, io
output = io.StringIO(newline='')
writer = csv.writer(output)
writer.writerow(["name", "score"])
writer.writerow(["Alice", 95])
csv_content = output.getvalue()
print(f"CSV content repr: {repr(csv_content)}")  # Shows \r\n (CSV standard)

# Binary mode when you need exact bytes
raw = path.read_bytes()
print(f"Raw bytes (first 30): {raw[:30]}")

Output:

Lines: ['line one', 'line two', 'line three']
CSV content repr: 'name,score\r\nalice,95\r\n'
Raw bytes (first 30): b'line one\nline two\nline three\n'

Key rule: always open files in text mode (the default) unless you specifically need raw bytes. Text mode handles line ending translation automatically. For CSV files, use newline='' when creating csv.writer objects — this prevents Python from adding an extra \r before the CSV module’s own \r\n, which would create \r\r\n on Windows.

Running Platform-Specific Subprocesses

When you need to run shell commands from Python, subprocess is the right tool — but the command syntax differs by platform. The safest approach is to pass commands as lists of strings and avoid shell=True where possible.

# subprocess_cross_platform.py
import subprocess
import sys

def clear_screen():
    """Clear the terminal -- different command per OS."""
    if sys.platform == "win32":
        subprocess.run(["cmd", "/c", "cls"], check=True)
    else:
        subprocess.run(["clear"], check=True)

def get_disk_usage(path="."):
    """Get disk usage -- different tool per OS."""
    if sys.platform == "win32":
        result = subprocess.run(
            ["dir", "/s", str(path)],
            capture_output=True, text=True, shell=True  # dir needs shell=True on Windows
        )
    else:
        result = subprocess.run(
            ["du", "-sh", str(path)],
            capture_output=True, text=True
        )
    return result.stdout.strip()

def list_processes():
    """List running processes -- different per OS."""
    if sys.platform == "win32":
        result = subprocess.run(["tasklist"], capture_output=True, text=True)
    else:
        result = subprocess.run(["ps", "aux"], capture_output=True, text=True)
    lines = result.stdout.splitlines()
    return lines[:5]  # First 5 lines

usage = get_disk_usage(".")
print(f"Disk usage: {usage}")
procs = list_processes()
print("Processes (first 5):")
for line in procs:
    print(f"  {line[:80]}")

Output (Linux):

Disk usage: 4.0K    .
Processes (first 5):
  USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  root           1  0.0  0.1 167584 11244 ?        Ss   08:00   0:01 /sbin/init
  root           2  0.0  0.0      0     0 ?        S    08:00   0:00 [kthreadd]

When shell=True is required (like for Windows built-in commands such as dir), be extra careful about command injection — never pass user-provided strings directly into shell commands. Build the command as a list whenever you can.

Real-Life Example: Cross-Platform Config Manager

Here is a practical config manager that stores application settings in the correct platform-specific location on every OS — ~/Library/Application Support on macOS, %APPDATA% on Windows, and ~/.config on Linux.

# config_manager.py
import json
import os
import sys
from pathlib import Path


def get_app_config_dir(app_name: str) -> Path:
    """Return the platform-correct config directory for this app."""
    if sys.platform == "win32":
        # Windows: %APPDATA%\AppName
        base = Path(os.environ.get("APPDATA", Path.home() / "AppData" / "Roaming"))
    elif sys.platform == "darwin":
        # macOS: ~/Library/Application Support/AppName
        base = Path.home() / "Library" / "Application Support"
    else:
        # Linux/Unix: ~/.config/AppName (XDG spec)
        base = Path(os.environ.get("XDG_CONFIG_HOME", Path.home() / ".config"))

    config_dir = base / app_name
    config_dir.mkdir(parents=True, exist_ok=True)
    return config_dir


class ConfigManager:
    def __init__(self, app_name: str):
        self.config_dir = get_app_config_dir(app_name)
        self.config_file = self.config_dir / "config.json"
        self._data = self._load()

    def _load(self) -> dict:
        if self.config_file.exists():
            try:
                return json.loads(self.config_file.read_text(encoding="utf-8"))
            except (json.JSONDecodeError, OSError):
                return {}
        return {}

    def get(self, key: str, default=None):
        return self._data.get(key, default)

    def set(self, key: str, value):
        self._data[key] = value
        self._save()

    def _save(self):
        self.config_file.write_text(
            json.dumps(self._data, indent=2),
            encoding="utf-8"
        )

    def clear(self):
        self._data = {}
        if self.config_file.exists():
            self.config_file.unlink()


# Usage
config = ConfigManager("MyPythonApp")
config.set("theme", "dark")
config.set("font_size", 14)
config.set("recent_files", ["/home/user/doc.txt"])

print(f"Config stored at: {config.config_dir}")
print(f"Theme: {config.get('theme')}")
print(f"Font size: {config.get('font_size')}")
print(f"Recent: {config.get('recent_files')}")
print(f"Missing key: {config.get('language', 'en')}")

Output (Linux):

Config stored at: /home/alice/.config/MyPythonApp
Theme: dark
Font size: 14
Recent: ['/home/user/doc.txt']
Missing key: en

Output (Windows):

Config stored at: C:\Users\alice\AppData\Roaming\MyPythonApp
Theme: dark
Font size: 14
Recent: ['/home/user/doc.txt']
Missing key: en

This pattern follows platform conventions that users expect — on macOS, apps store their config in ~/Library, not in a hidden dot-folder. The same code does the right thing on every platform without any conditional logic in the main program, just in the get_app_config_dir factory function.

Frequently Asked Questions

Should I use os.path or pathlib?

Use pathlib for all new code. It is the modern, object-oriented path API introduced in Python 3.4 and is consistently more readable and safer than os.path string functions. os.path.join("a", "b", "c") becomes Path("a") / "b" / "c". The only time to use os.path is when working with very old code that already uses it and you are making minimal changes.

Why does my script work on Mac but break on Linux?

Most likely a case sensitivity issue. macOS filesystems are case-insensitive by default (like Windows), so Path("Data/File.txt") and Path("data/file.txt") point to the same file. Linux filesystems are case-sensitive, so they are completely different paths. Always match filenames exactly, and use path.lower() comparisons only when searching, never when constructing paths.

What is the safest way to reference the home directory?

Always use Path.home(). Never hardcode /home/user, /Users/user, or C:\Users\user. Even on Linux, home directories are not always in /home — root’s home is /root, and system accounts often live in /var or /opt. Path.home() reads from the OS correctly every time.

Why does file deletion fail on Windows but not Linux?

Windows locks open files — you cannot delete or rename a file that is currently open by any process, including your own. On Linux, you can delete a file that is open (the inode is removed, but the data persists until all handles close). Always close files before deleting them, use context managers (with open(...)), and catch PermissionError when deleting files on Windows.

How do I copy or move files cross-platform?

Use shutil from the standard library. shutil.copy2(src, dst) copies a file with metadata, shutil.copytree(src, dst) copies a whole directory tree, and shutil.move(src, dst) moves files or directories — all platform-correctly. Never use raw OS commands like cp or xcopy from subprocess when shutil gives you a portable Python solution.

How do I find an executable on the system PATH?

Use shutil.which("program_name"). It searches the PATH correctly on all platforms and returns None if the executable is not found. On Windows it automatically checks for .exe, .cmd, and other extensions. Never manually construct paths like /usr/bin/python — use shutil.which("python") or sys.executable for the current interpreter.

Conclusion

Writing cross-platform Python is mostly about choosing the right standard library tools. The biggest wins come from four habits: using pathlib.Path for all file path operations, using os.environ.get() for environment variables, using tempfile for temporary files, and using sys.platform for the rare cases where you need platform-specific behavior. With these four patterns, the vast majority of portability bugs simply cannot happen.

The config manager example in this article is a good template for any tool that needs to store data in the user’s filesystem. Extend it by adding schema validation with pydantic, or by supporting environment variable overrides that take precedence over the config file. Both patterns follow naturally from what you’ve learned here.

For deeper reading, the official Python documentation on pathlib, os, and tempfile covers every method and parameter in detail. The shutil documentation is also worth reading for file operations beyond basic read/write.

Related Articles

• How To Use Python os Module for File System Operations
• How To Use Python subprocess for Running Shell Commands
• How To Use Python pathlib for File Path Operations

Intermediate

When you need to repeatedly pull the highest- or lowest-priority item from a collection, a regular sorted list becomes a performance trap. Every time you add an item and re-sort, you’re doing O(n log n) work. A heap structure reduces that to O(log n) per insertion and O(log n) per removal — a massive difference when processing thousands of tasks, events, or jobs. Python’s built-in heapq module provides a min-heap implementation directly on top of ordinary Python lists, with no external dependencies required.

Priority queues show up everywhere in real software: task schedulers process the most urgent job first, Dijkstra’s algorithm uses a heap to find the shortest path, event simulation systems process events by timestamp, and streaming data pipelines use heaps to track the top-N results efficiently. Understanding heapq gives you the foundation for all of these patterns.

In this tutorial, we’ll cover how a heap works internally, how to use heapq.heappush() and heappop() to build a priority queue, how to find the N largest or smallest items efficiently with nlargest() and nsmallest(), how to handle custom priority objects, and a practical task scheduler project that ties it all together.

Quick Example: Priority Queue in 10 Lines

Here’s a minimal working priority queue using heapq:

# heapq_quick.py
import heapq

tasks = []
heapq.heappush(tasks, (3, 'Low priority task'))
heapq.heappush(tasks, (1, 'Critical task'))
heapq.heappush(tasks, (2, 'Medium priority task'))

while tasks:
    priority, task = heapq.heappop(tasks)
    print(f"[Priority {priority}] {task}")

Output:

[Priority 1] Critical task
[Priority 2] Medium priority task
[Priority 3] Low priority task

Tasks always come out in ascending priority order (lowest number = highest priority). The heap maintains this invariant automatically as you push and pop — you never need to sort manually. The tuple format (priority, item) is the standard pattern: Python compares tuples element by element, so the first element determines ordering.

What Is a Heap and How Does heapq Work?

A heap is a binary tree stored as a flat list where every parent node is smaller than or equal to its children. Python’s heapq implements a min-heap — the smallest element is always at index 0. This property lets the module find and remove the minimum in O(1) time and restore the heap property after a push or pop in O(log n) time.

The heap is stored as a regular Python list — there’s no special class or data structure. You just use the heapq functions to maintain the heap invariant on that list. You can inspect the raw list at any time, though the order looks scrambled because the heap is not fully sorted:

# heap_internals.py
import heapq

nums = [5, 3, 8, 1, 4, 2, 7]
heapq.heapify(nums)  # Convert list to heap in-place, O(n)

print("Heap:", nums)   # [1, 3, 2, 5, 4, 8, 7] -- heap-ordered, not sorted
print("Min:", nums[0]) # 1 -- always the smallest

heapq.heappush(nums, 0)
print("After push 0:", nums)
print("New min:", nums[0])  # 0

Output:

Heap: [1, 3, 2, 5, 4, 8, 7]
Min: 1
After push 0: [0, 3, 1, 5, 4, 8, 7, 2]  (positions may vary)
New min: 0

heapq.heapify() converts an existing list into a valid heap in O(n) time — much faster than pushing elements one by one. Use it when you’re starting from a collection you’ve already built.

Finding Top-N Items with nlargest and nsmallest

Two of the most useful heapq functions are nlargest() and nsmallest(). They return the N highest or lowest items from any iterable, using a heap internally to avoid sorting the full dataset. This is significantly faster than sorted(data)[-n:] when n is small relative to the dataset size.

# top_n.py
import heapq

scores = [45, 89, 23, 67, 91, 12, 78, 55, 34, 88, 100, 3, 72]

top3 = heapq.nlargest(3, scores)
bottom3 = heapq.nsmallest(3, scores)

print("Top 3 scores:    ", top3)    # [100, 91, 89]
print("Bottom 3 scores: ", bottom3) # [3, 12, 23]

# Works with any iterable, supports key function
students = [
    {'name': 'Alice', 'gpa': 3.9},
    {'name': 'Bob',   'gpa': 3.5},
    {'name': 'Carol', 'gpa': 3.7},
    {'name': 'Dave',  'gpa': 3.2},
    {'name': 'Eve',   'gpa': 3.8},
]

top2 = heapq.nlargest(2, students, key=lambda s: s['gpa'])
print("Top 2 students:", [s['name'] for s in top2])  # ['Alice', 'Eve']

Output:

Top 3 scores:     [100, 91, 89]
Bottom 3 scores:  [3, 12, 23]
Top 2 students: ['Alice', 'Eve']

The key parameter works just like sorted() — pass any callable that extracts the comparison value. For very small N (like 1), min() and max() are faster. For N close to the length of the dataset, sorted() is faster. For everything in between, nlargest() and nsmallest() are the right tool.

Custom Priority Objects

When items in your priority queue are complex objects (not simple numbers), you need a way to tell the heap how to compare them. The cleanest approach is to push tuples of (priority, counter, item). The counter breaks ties in priority and prevents Python from falling back to comparing the items themselves (which would fail if items don’t support comparison).

# priority_objects.py
import heapq
import itertools

class Task:
    def __init__(self, name, priority):
        self.name = name
        self.priority = priority
    def __repr__(self):
        return f"Task({self.name!r}, priority={self.priority})"

class PriorityQueue:
    def __init__(self):
        self._heap = []
        self._counter = itertools.count()  # tie-breaker

    def push(self, item, priority):
        # Lower priority number = processed first
        count = next(self._counter)
        heapq.heappush(self._heap, (priority, count, item))

    def pop(self):
        priority, count, item = heapq.heappop(self._heap)
        return item

    def __len__(self):
        return len(self._heap)

pq = PriorityQueue()
pq.push(Task('Deploy database migration', 2), priority=2)
pq.push(Task('Fix critical security bug', 1), priority=1)
pq.push(Task('Update documentation', 3), priority=3)
pq.push(Task('Restart hung service', 1), priority=1)  # tie on priority

print("Processing tasks:")
while pq:
    print(" -", pq.pop())

Output:

Processing tasks:
 - Task('Fix critical security bug', priority=1)
 - Task('Restart hung service', priority=1)
 - Task('Deploy database migration', priority=2)
 - Task('Update documentation', priority=3)

The itertools.count() counter ensures that when two items share the same priority, they come out in FIFO order (first-in, first-out). Without the counter, Python would try to compare the Task objects directly, which would raise a TypeError since Task doesn’t define __lt__. Using a counter is the idiomatic pattern recommended in the official heapq documentation.

Simulating a Max-Heap

Python’s heapq only supports min-heaps. To simulate a max-heap (where the largest item comes out first), negate the priority values before pushing:

# max_heap.py
import heapq

scores = [50, 80, 30, 95, 60]
max_heap = []

for score in scores:
    heapq.heappush(max_heap, -score)  # negate to invert order

print("Scores in descending order:")
while max_heap:
    score = -heapq.heappop(max_heap)  # un-negate on pop
    print(score, end=' ')
print()

Output:

Scores in descending order:
95 80 60 50 30

Negating integers and floats works cleanly. For strings or complex objects, this trick doesn’t apply directly — use the tuple-with-counter pattern instead and negate the numeric priority component.

Real-Life Example: Job Scheduler with Aging

A realistic priority queue for a job scheduler that implements “aging” — low-priority jobs gradually increase in priority the longer they wait, preventing starvation:

# job_scheduler.py
import heapq
import itertools
import time

class JobScheduler:
    def __init__(self):
        self._heap = []
        self._counter = itertools.count()
        self._aging_interval = 5   # seconds before priority boost
        self._aging_boost = 1      # boost amount per interval

    def submit(self, job_name, priority):
        count = next(self._counter)
        submit_time = time.time()
        heapq.heappush(self._heap, (priority, count, job_name, submit_time))
        print(f"  Submitted '{job_name}' with priority {priority}")

    def _effective_priority(self, priority, count, job_name, submit_time):
        """Reduce priority number (boost urgency) based on wait time."""
        wait_seconds = time.time() - submit_time
        boosts = int(wait_seconds / self._aging_interval)
        return max(0, priority - (boosts * self._aging_boost))

    def process_next(self):
        if not self._heap:
            return None
        # Re-heapify with effective priorities
        updated = []
        for entry in self._heap:
            priority, count, name, submit_time = entry
            eff = self._effective_priority(priority, count, name, submit_time)
            updated.append((eff, count, name, submit_time))
        heapq.heapify(updated)
        eff_priority, count, name, submit_time = heapq.heappop(updated)
        self._heap = updated
        return name, eff_priority

# Simulate scheduler
scheduler = JobScheduler()
print("Submitting jobs:")
scheduler.submit("Generate monthly report", priority=5)
scheduler.submit("Fix login page crash", priority=1)
scheduler.submit("Update user profile endpoint", priority=3)
scheduler.submit("Archive old logs", priority=7)

print("\nProcessing in priority order:")
for _ in range(4):
    result = scheduler.process_next()
    if result:
        name, eff_priority = result
        print(f"  Processing: '{name}' (effective priority: {eff_priority})")

Output:

Submitting jobs:
  Submitted 'Generate monthly report' with priority 5
  Submitted 'Fix login page crash' with priority 1
  Submitted 'Update user profile endpoint' with priority 3
  Submitted 'Archive old logs' with priority 7

Processing in priority order:
  Processing: 'Fix login page crash' (effective priority: 1)
  Processing: 'Update user profile endpoint' (effective priority: 3)
  Processing: 'Generate monthly report' (effective priority: 5)
  Processing: 'Archive old logs' (effective priority: 7)

This scheduler can be extended to run in a background thread that continuously calls process_next() and dispatches work to a thread pool. The aging mechanism ensures that a high-priority flood of urgent jobs doesn’t permanently starve low-priority maintenance tasks.

Frequently Asked Questions

When should I use heapq vs sorted()?

Use heapq when you’re repeatedly adding items and pulling the minimum — like a task queue or streaming algorithm. Use sorted() when you need the full collection in order, or when you process everything at once. Heaps shine in incremental “add some, pop the min, add more” workloads. If you’re only sorting once, sorted() is simpler and fast enough.

Is heapq thread-safe?

No. heapq operations on a plain list are not thread-safe. If multiple threads push and pop concurrently, you’ll get race conditions and heap corruption. For thread-safe priority queues, use queue.PriorityQueue from the standard library, which wraps a heap with locking. asyncio.PriorityQueue is the equivalent for async code.

When is nlargest faster than sorted()?

nlargest(k, data) uses an internal heap of size k, making it O(n log k). It’s faster than sorted(data, reverse=True)[:k] (which is O(n log n)) when k is much smaller than n. For a list of 1,000,000 items, finding the top 10 with nlargest is dramatically faster. But if k is close to n, sorted() wins because it can use Timsort’s optimizations.

What is heappushpop and when should I use it?

heapq.heappushpop(heap, item) pushes an item onto the heap and then pops the smallest item, all in one operation. It’s more efficient than calling heappush followed by heappop because it avoids an extra heap-property restoration step. It’s particularly useful for maintaining a fixed-size heap of the N largest items seen so far in a stream.

Does heapq preserve insertion order for equal priorities?

Not by default. If two items have the same priority, the heap may return them in any order. To get FIFO behavior for equal priorities, use the tuple pattern (priority, counter, item) where counter is an incrementing integer from itertools.count(). This guarantees equal-priority items are returned in the order they were inserted.

Conclusion

Python’s heapq module gives you an efficient priority queue with O(log n) push and pop directly on a standard Python list. The key functions are heappush(), heappop(), and heapify() for building and using priority queues, plus nlargest() and nsmallest() for finding the top-N items from any iterable without sorting the whole collection.

The real-life job scheduler shows how to combine heapq with itertools.count() for stable ordering and add aging logic to prevent starvation — patterns directly applicable to production systems. You can extend this to distributed task queues, real-time bid processing, or any system where “most important first” processing matters.

For more details, see the official documentation at docs.python.org/3/library/heapq.html. For thread-safe use cases, also read docs.python.org/3/library/queue.html.

What heapq Does

heapq turns a regular Python list into a binary min-heap — the smallest element is always at index 0, and push/pop run in O(log n). No special class, no extra import overhead, just functions that operate on lists:

import heapq

heap = []
heapq.heappush(heap, 3)
heapq.heappush(heap, 1)
heapq.heappush(heap, 5)
heapq.heappush(heap, 2)

print(heap[0])              # 1 — always the smallest
print(heapq.heappop(heap))  # 1 — and remove it
print(heapq.heappop(heap))  # 2
# heap is now [3, 5]

The smallest item is always at index 0; the rest of the list is NOT sorted, but the heap invariants make push/pop O(log n) instead of the O(n) you’d get with bisect.insort.

Building a Heap from a List

data = [10, 1, 5, 8, 3, 7, 2, 9]
heapq.heapify(data)   # transforms in-place, O(n)
print(data[0])         # 1
# Pop one at a time in sorted order:
while data:
    print(heapq.heappop(data))

heapify is faster than push-n-times when you already have all the data — O(n) vs O(n log n).

Priority Queue Pattern

import heapq

queue = []
heapq.heappush(queue, (3, "low priority task"))
heapq.heappush(queue, (1, "high priority task"))
heapq.heappush(queue, (2, "medium priority task"))

while queue:
    priority, task = heapq.heappop(queue)
    print("Run:", task)
# Output: high, medium, low

Tuples compare lexicographically — lowest priority value first. To break ties on equal priorities, add a tiebreaker (counter) as the second element.

Max-Heap via Negated Keys

Python’s heapq is a MIN-heap only. For a max-heap, negate the priority on push and pop:

max_heap = []
heapq.heappush(max_heap, -3)
heapq.heappush(max_heap, -1)
heapq.heappush(max_heap, -5)
print(-heapq.heappop(max_heap))   # 5

# Or use a wrapper class with reversed comparison
class MaxItem:
    def __init__(self, value, payload):
        self.value = value
        self.payload = payload
    def __lt__(self, other):
        return self.value > other.value

Top-N Without Sorting Everything

import heapq

# Top 5 largest — O(n log k), not O(n log n)
top_5 = heapq.nlargest(5, big_list)

# Top 3 smallest by attribute
smallest = heapq.nsmallest(3, users, key=lambda u: u.age)

# Top-k streaming pattern — bounded heap of size k
import heapq
def top_k_running(stream, k):
    heap = []
    for item in stream:
        if len(heap) < k:
            heapq.heappush(heap, item)
        elif item > heap[0]:
            heapq.heapreplace(heap, item)
    return sorted(heap, reverse=True)

heapreplace is faster than separate pop + push when the heap is full and you want to swap one element.

Merging Sorted Iterables

import heapq

# Merge multiple already-sorted iterables lazily
list_a = [1, 4, 7, 10]
list_b = [2, 5, 8, 11]
list_c = [3, 6, 9, 12]
for x in heapq.merge(list_a, list_b, list_c):
    print(x)
# Output: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12

# Merge files line by line (each pre-sorted)
files = [open("a.txt"), open("b.txt"), open("c.txt")]
merged = heapq.merge(*files)
for line in merged:
    write_to_output(line)

Common Pitfalls

• Heap is not sorted. Only heap[0] is guaranteed to be the smallest. Iterating for x in heap gives them in heap order, NOT sorted order.
• Equal priorities crash on comparison. (1, dict_a) and (1, dict_b) — Python tries to compare the dicts when priorities tie. Add a counter as a tiebreaker: (priority, counter, payload).
• Modifying items after push. If you push a mutable object and modify it later, the heap invariant breaks. Push immutable items (or copies).
• Mixing heappush and direct append. Calling list.append() bypasses the heap invariant. Use heapify after any direct modification.
• Max-heap confusion. Python only has min-heap. Negate priority for max-heap, or maintain it as a sorted list with bisect.

FAQ

Q: heapq or PriorityQueue from queue module?
A: queue.PriorityQueue is thread-safe (uses heapq internally + lock). For single-threaded code, raw heapq is faster.

Q: How do I update a priority?
A: heapq doesn’t support direct update. Mark the old entry invalid (with a sentinel) and push a new entry; skip invalid ones on pop. Or rebuild via heapify.

Q: Custom comparison?
A: Pass a wrapped tuple or class with __lt__. The key= argument exists on nlargest/nsmallest but NOT on heappush/heappop.

Q: Memory overhead?
A: Same as a regular Python list. heapq operates in-place, no extra storage.

Q: heapq vs SortedList from sortedcontainers?
A: SortedList for “always-sorted access by index”. heapq for “min-or-max only” with faster push/pop.

Wrapping Up

heapq is the right tool for priority queues, top-k problems, and merging sorted streams in Python. The API is functional (no objects), the memory cost is zero, and the O(log n) push/pop is dramatically faster than sorted-list operations on large data. For thread safety, wrap it in queue.PriorityQueue; for “always sorted” access, switch to SortedList.

Related Articles

• How To Use Python collections Module for Specialized Data Structures
• How To Use Python concurrent.futures for Thread and Process Pools
• How To Sort Lists, Tuples, and Dicts in Python

Intermediate

You assign a list to a new variable, modify the new variable, and suddenly your original list has changed too. If you’ve been burned by this Python “gotcha,” you’ve encountered the difference between a reference and a copy. Python doesn’t automatically duplicate objects when you assign them — it creates another reference to the same object in memory. For simple integers and strings this doesn’t matter, but for mutable containers like lists, dicts, and custom objects it causes subtle bugs that are notoriously hard to track down.

The fix is Python’s built-in copy module, which provides two copying strategies: shallow copy and deep copy. Shallow copy creates a new container but shares references to the inner objects. Deep copy recursively duplicates everything, giving you a completely independent clone. The module is part of Python’s standard library — no installation needed. You just need to understand which strategy to use and why.

In this tutorial, we’ll cover how Python assignment works under the hood, what shallow and deep copies actually do to memory, how to use copy.copy() and copy.deepcopy(), and how to customize copying behavior for your own classes. By the end, you’ll know exactly which tool to reach for when you need to duplicate data safely.

Quick Example: copy vs deepcopy

Here’s a minimal example that shows the core difference between the two copy strategies:

# copy_quick.py
import copy

original = [[1, 2, 3], [4, 5, 6]]

shallow = copy.copy(original)
deep = copy.deepcopy(original)

# Modify a nested list
original[0][0] = 99

print("Original:", original)   # [[99, 2, 3], [4, 5, 6]]
print("Shallow: ", shallow)    # [[99, 2, 3], [4, 5, 6]]  <-- affected!
print("Deep:    ", deep)       # [[1, 2, 3], [4, 5, 6]]   <-- unaffected

Output:

Original: [[99, 2, 3], [4, 5, 6]]
Shallow:  [[99, 2, 3], [4, 5, 6]]
Deep:     [[1, 2, 3], [4, 5, 6]]

The shallow copy created a new list, but the inner lists are still shared references. When we modified original[0][0], the shallow copy reflected that change. The deep copy is completely independent -- modifying the original has no effect on it. The sections below explain exactly why this happens and how to handle more complex cases.

How Python Assignment Actually Works

Before we talk about copying, we need to understand what Python assignment actually does. When you write b = a, Python does not create a new object. It creates a new name that points to the same object in memory. Both a and b are references -- labels attached to the same underlying data.

# assignment_vs_copy.py
a = [1, 2, 3]
b = a          # b is just another name for the same list

b.append(4)

print("a:", a)  # [1, 2, 3, 4]  <-- modified!
print("b:", b)  # [1, 2, 3, 4]
print("Same object?", a is b)  # True

Output:

a: [1, 2, 3, 4]
b: [1, 2, 3, 4]
Same object? True

The is operator checks whether two variables point to the same object in memory. Since a and b are the same object, modifying one modifies both. This is a feature, not a bug -- it's how Python keeps memory usage low. But it means you must be deliberate when you actually want an independent copy.

Shallow Copy with copy.copy()

A shallow copy creates a new container object, but fills it with references to the same items as the original. For a flat list (a list containing only non-mutable items like integers or strings), a shallow copy behaves like a full independent copy. The problem arises with nested structures.

# shallow_copy.py
import copy

# Flat list -- shallow copy works fine
flat = [1, 2, 3, 4, 5]
flat_copy = copy.copy(flat)
flat_copy.append(99)

print("Flat original:", flat)       # [1, 2, 3, 4, 5]
print("Flat copy:    ", flat_copy)  # [1, 2, 3, 4, 5, 99]

# Nested list -- shallow copy shares inner references
nested = [[10, 20], [30, 40]]
nested_copy = copy.copy(nested)

nested[0].append(99)  # Mutate an inner list

print("Nested original:", nested)       # [[10, 20, 99], [30, 40]]
print("Nested copy:    ", nested_copy)  # [[10, 20, 99], [30, 40]] -- shared!

Output:

Flat original: [1, 2, 3, 4, 5]
Flat copy:     [1, 2, 3, 4, 5, 99]
Nested original: [[10, 20, 99], [30, 40]]
Nested copy:     [[10, 20, 99], [30, 40]]

With the flat list, appending to flat_copy doesn't affect flat -- they're separate containers. But with the nested list, nested_copy[0] and nested[0] are still the same inner list. The outer containers are different, but the inner objects are shared. Use copy.copy() when your data structure is flat or when you intentionally want to share the inner objects.

Shorthand for Shallow Copies

Python offers several built-in shorthand ways to create a shallow copy without importing the copy module. These are idiomatic and commonly seen in real codebases:

# shallow_shorthand.py
original = [1, 2, 3]

# These all produce shallow copies:
copy1 = original[:]          # slice notation
copy2 = list(original)       # list constructor
copy3 = original.copy()      # list.copy() method

# For dicts:
d = {'a': 1, 'b': [2, 3]}
d_copy = d.copy()            # dict.copy() -- also shallow!

d['b'].append(99)
print(d_copy['b'])  # [2, 3, 99] -- shared inner list

Output:

[2, 3, 99]

All of these shortcuts -- slice notation, the list() constructor, and .copy() -- produce shallow copies. This is an important gotcha: dict.copy() does NOT create a deep copy of a dict. If your dict contains mutable values (like lists or other dicts), mutations to those inner objects will still be visible in the copy.

Deep Copy with copy.deepcopy()

A deep copy recursively duplicates every object in the structure. The copy is completely independent -- modifying any part of the original, at any depth, will not affect the copy. This is the safest choice when you need true independence between your original data and your copy.

# deep_copy.py
import copy

data = {
    'name': 'Alice',
    'scores': [95, 87, 92],
    'address': {
        'city': 'Melbourne',
        'postcode': 3000
    }
}

clone = copy.deepcopy(data)

# Mutate everything in original
data['name'] = 'Bob'
data['scores'].append(100)
data['address']['city'] = 'Sydney'

print("Original:", data)
print("Clone:   ", clone)

Output:

Original: {'name': 'Bob', 'scores': [95, 87, 92, 100], 'address': {'city': 'Sydney', 'postcode': 3000}}
Clone:    {'name': 'Alice', 'scores': [95, 87, 92], 'address': {'city': 'Melbourne', 'postcode': 3000}}

Every nested object -- the list of scores and the address dict -- was independently duplicated. Changes to the original have zero effect on the clone. Deep copy is the right tool when you need to preserve a snapshot of a complex data structure before modifying it, or when you're passing data to a function that might mutate it unexpectedly.

Deep Copy Performance Considerations

Deep copy is not free. It has to traverse the entire object graph, allocate new objects, and track circular references. For large nested structures, this can be noticeably slower than a shallow copy. Here's a simple benchmark to make the difference concrete:

# copy_performance.py
import copy
import time

# Build a deeply nested structure
big_data = {'items': [{'id': i, 'tags': [str(i*j) for j in range(10)]} for i in range(1000)]}

start = time.time()
for _ in range(100):
    copy.copy(big_data)
print(f"Shallow copy x100: {(time.time()-start)*1000:.1f}ms")

start = time.time()
for _ in range(100):
    copy.deepcopy(big_data)
print(f"Deep copy x100:    {(time.time()-start)*1000:.1f}ms")

Output (approximate):

Shallow copy x100: 0.3ms
Deep copy x100:    245.7ms

Deep copy is roughly 800x slower on this example. For small objects or infrequent copying this doesn't matter at all. But in a tight loop processing thousands of items, it can become a real bottleneck. If performance is critical and you only need independence at the top level, shallow copy is faster.

Copying Custom Objects

The copy module works with custom Python classes too. By default, copy.copy() creates a new instance of the class but shares all attribute references (shallow), while copy.deepcopy() recursively copies all attributes (deep). You can customize this behavior by implementing __copy__ and __deepcopy__ on your class.

# copy_custom_class.py
import copy

class Config:
    def __init__(self, name, settings):
        self.name = name
        self.settings = settings  # mutable dict

    def __repr__(self):
        return f"Config(name={self.name!r}, settings={self.settings})"

cfg = Config("production", {"debug": False, "retries": 3})

# Shallow copy -- settings dict is shared
shallow_cfg = copy.copy(cfg)
shallow_cfg.settings['debug'] = True

print("Original:", cfg)         # settings shows debug=True too
print("Shallow: ", shallow_cfg)

# Deep copy -- settings dict is independent
deep_cfg = copy.deepcopy(cfg)
deep_cfg.settings['retries'] = 99

print("Original:", cfg)         # settings unchanged
print("Deep:    ", deep_cfg)

Output:

Original: Config(name='production', settings={'debug': True, 'retries': 3})
Shallow:  Config(name='production', settings={'debug': True, 'retries': 3})
Original: Config(name='production', settings={'debug': True, 'retries': 3})
Deep:     Config(name='production', settings={'debug': True, 'retries': 99})

The shallow copy shares the settings dict, so modifying it in the copy affects the original. The deep copy has its own independent settings dict. This is the typical behavior -- it applies to any class that holds mutable attributes.

Custom __copy__ and __deepcopy__

Sometimes you want fine-grained control. For example, you might want a deep copy that excludes a heavy cache attribute (no point duplicating cached data). Implement __copy__ and __deepcopy__ to control exactly what gets duplicated:

# custom_copy_hooks.py
import copy

class ExpensiveModel:
    def __init__(self, data, cache=None):
        self.data = data
        self.cache = cache or {}  # expensive to duplicate

    def __copy__(self):
        # Shallow: share data reference, reset cache
        cls = self.__class__
        result = cls.__new__(cls)
        result.__dict__.update(self.__dict__)
        result.cache = {}  # fresh cache, don't share
        return result

    def __deepcopy__(self, memo):
        cls = self.__class__
        result = cls.__new__(cls)
        memo[id(self)] = result
        result.data = copy.deepcopy(self.data, memo)  # deep copy data
        result.cache = {}                              # fresh cache, skip deepcopy
        return result

    def __repr__(self):
        return f"ExpensiveModel(data={self.data}, cache_size={len(self.cache)})"

model = ExpensiveModel({'weights': [0.1, 0.2, 0.3]}, cache={'key': 'value'})
cloned = copy.deepcopy(model)

print("Original:", model)
print("Cloned:  ", cloned)  # cache is empty, data is independent

Output:

Original: ExpensiveModel(data={'weights': [0.1, 0.2, 0.3]}, cache_size=1)
Cloned:   ExpensiveModel(data={'weights': [0.1, 0.2, 0.3]}, cache_size=0)

The memo parameter in __deepcopy__ is important -- it tracks already-copied objects to handle circular references. Always pass it to recursive copy.deepcopy() calls inside your implementation.

Shallow vs Deep vs Assignment: Quick Reference

Here's a summary of all three approaches to help you choose the right one:

Real-Life Example: Safe Configuration Management

A common use case for deep copy is managing environment-specific configurations. You start with a base config and create independent overrides for each environment without risking mutations bleeding across environments.

# config_manager.py
import copy

BASE_CONFIG = {
    'database': {
        'host': 'localhost',
        'port': 5432,
        'name': 'myapp_dev',
        'pool_size': 5
    },
    'cache': {
        'backend': 'redis',
        'timeout': 300
    },
    'debug': True,
    'allowed_hosts': ['localhost', '127.0.0.1']
}

def create_env_config(base, overrides):
    """Create an independent environment config from base + overrides."""
    config = copy.deepcopy(base)
    for key, value in overrides.items():
        if isinstance(value, dict) and key in config:
            config[key].update(value)
        else:
            config[key] = value
    return config

# Create staging and production configs
staging = create_env_config(BASE_CONFIG, {
    'database': {'host': 'staging-db.internal', 'name': 'myapp_staging'},
    'debug': False
})

production = create_env_config(BASE_CONFIG, {
    'database': {'host': 'prod-db.internal', 'name': 'myapp_prod', 'pool_size': 20},
    'debug': False,
    'allowed_hosts': ['myapp.com', 'www.myapp.com']
})

# Mutate production config -- base should be untouched
production['allowed_hosts'].append('api.myapp.com')

print("Base allowed_hosts:", BASE_CONFIG['allowed_hosts'])
print("Staging DB host:   ", staging['database']['host'])
print("Production hosts:  ", production['allowed_hosts'])
print("Production pool:   ", production['database']['pool_size'])

Output:

Base allowed_hosts: ['localhost', '127.0.0.1']
Staging DB host:    staging-db.internal
Production hosts:   ['myapp.com', 'www.myapp.com', 'api.myapp.com']
Production pool:    20

Because create_env_config uses copy.deepcopy(base), every call produces a fully independent config. You can safely mutate staging and production configs without ever touching the base. This pattern is common in application factories, test setup, and any system where you need to start from a known baseline and diverge safely.

Frequently Asked Questions

When should I use deepcopy vs shallow copy?

Use deepcopy when your data structure contains nested mutable objects (lists, dicts, or custom classes) and you need full independence between the original and the copy. Use shallow copy when your structure is flat (contains only immutable items like integers, strings, or tuples) or when you deliberately want the copy to share references to inner objects for memory efficiency. If you're unsure, deepcopy is the safe default.

Does deepcopy handle circular references?

Yes. Python's copy.deepcopy() uses a memo dictionary to track objects it has already copied. When it encounters an object it has already processed, it returns the already-copied version instead of recursing infinitely. This means you can safely deepcopy objects that reference themselves or form cycles in their object graph. The same memo dict is why you must pass it through when implementing __deepcopy__.

Do I need to copy immutable objects like tuples and strings?

No. Immutable objects like integers, strings, tuples, and frozensets cannot be modified in place, so sharing references is completely safe. Both copy.copy() and copy.deepcopy() will often return the original object unchanged for immutable types -- there's no point in creating a duplicate. The copying concern only applies to mutable objects.

Does deepcopy work with NumPy arrays and Pandas DataFrames?

It does, but these libraries provide their own more efficient copy methods: array.copy() for NumPy and df.copy(deep=True) for Pandas. Use those instead of copy.deepcopy() when working with numerical or tabular data -- they're optimized for the underlying C memory layout and will be significantly faster. copy.deepcopy() works as a fallback but is much slower on large arrays.

How is deepcopy related to pickling?

Both deepcopy and pickle serialize an object's full state, but they serve different purposes. Pickling converts an object to bytes for saving to disk or sending over a network. Deepcopy keeps everything in memory and returns a live Python object. Interestingly, objects that implement __reduce__ or __getstate__/__setstate__ for pickling will often use those same hooks during deepcopy. If you need a deep copy of an object that can be pickled, pickle.loads(pickle.dumps(obj)) is an alternative, though slower.

Can I use json.loads(json.dumps(data)) as a deepcopy?

Yes, for simple data structures containing only JSON-serializable types (dicts, lists, strings, numbers, booleans, None). This pattern is occasionally used as a fast deep copy alternative since the JSON round-trip is sometimes faster than deepcopy for large flat structures. However, it loses non-JSON types: tuples become lists, datetime objects raise errors, custom class instances are not preserved. Use it with caution and only when you know your data is 100% JSON-serializable.

Conclusion

Python's copy module gives you precise control over how objects are duplicated. Assignment creates a second reference to the same object -- no copying at all. copy.copy() creates a new container but shares references to inner objects, which is fast and appropriate for flat structures. copy.deepcopy() recursively duplicates everything, giving you a completely independent clone at the cost of more memory and CPU time.

The real-life config manager example shows how deepcopy enables safe environment-specific configuration without any risk of mutations leaking between environments. The same pattern applies anywhere you need to branch from a known baseline: test setup, undo/redo history, job queue snapshots, or any pipeline where you need to preserve the original state while a downstream process modifies the data.

For custom classes, implementing __copy__ and __deepcopy__ gives you full control -- useful when some attributes (like caches or open file handles) should not be duplicated. The official documentation at docs.python.org/3/library/copy.html has the complete reference.

Related Articles

• How To Use Python __slots__ to Reduce Memory Usage
• How To Use Python collections Module for Specialized Data Structures
• How To Use Python dataclasses for Cleaner Class Definitions

Intermediate

Working with binary data is one of those skills that feels intimidating until you understand the two types Python gives you: bytes (immutable) and bytearray (mutable). Whether you are reading a binary file format, sending raw bytes over a network socket, working with image data, or packing integers into a compact binary protocol, these two types are what you reach for. The confusion usually comes from mixing up strings (Unicode text) and bytes (raw binary data) — a distinction Python 3 enforces strictly.

Both bytes and bytearray are built into Python 3 — no imports needed for the basic operations. The struct module (also built-in) handles packing and unpacking binary data formats like those used in network protocols and file headers. The examples below all run with the standard library only.

This article covers: creating bytes and bytearrays, encoding and decoding strings, slicing and searching binary data, modifying bytearrays in place, using the struct module for binary packing, reading binary files, and a real-life binary file parser that reads PNG header metadata.

Bytes and Bytearray: Quick Example

Here is the core distinction in 15 lines — bytes is immutable, bytearray is mutable:

# bytes_quick.py
# bytes: immutable
data = b"Hello, World!"
print(type(data), data)
print("First byte:", data[0])          # Integer (72 = ord('H'))
print("Slice:", data[0:5])             # Still bytes

# bytearray: mutable
buf = bytearray(b"Hello, World!")
buf[0] = 104                           # Change 'H' to 'h' (ASCII 104)
buf.extend(b" Goodbye!")
print("Modified:", buf)
print("As string:", buf.decode("utf-8"))

Output:

<class 'bytes'> b'Hello, World!'
First byte: 72
Slice: b'Hello'
Modified: bytearray(b'hello, World! Goodbye!')
As string: hello, World! Goodbye!

Indexing a bytes or bytearray object with a single integer returns an int (the byte value 0-255), not a single-character string. Slicing returns a new bytes or bytearray object. This is the most common source of confusion for developers coming from Python 2.

Creating bytes and bytearray

There are several ways to create byte sequences in Python, each suited to different situations:

# bytes_creation.py

# From ASCII literal
a = b"Python"
print(a, a.hex())

# From UTF-8 string
b = "Python -- binary".encode("utf-8")
print(b)

# Zero-filled buffer
buf = bytearray(8)
print("Zeroed buffer:", buf)

# From integer list
c = bytes([80, 121, 116, 104, 111, 110])
print("From ints:", c.decode())

# From hex string
d = bytes.fromhex("deadbeef")
print("From hex:", d, d.hex())

Output:

b'Python' 507974686f6e
b'Python \xe2\x80\x94 binary'
Zeroed buffer: bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00')
From ints: Python
From hex: b'\xde\xad\xbe\xef' deadbeef

Notice how "Python -- binary".encode("utf-8") shows the em-dash encoded as three UTF-8 bytes (\xe2\x80\x94). This is why Python 3 makes the string/bytes distinction explicit — text and binary data have completely different internal representations.

Encoding and Decoding Strings

The most frequent use of bytes is converting between text (strings) and binary data. Always specify the encoding explicitly — never rely on the default, which varies by platform.

# bytes_encoding.py

text = "Cafe -- served with care"

# Encode to bytes
utf8_bytes = text.encode("utf-8")
utf16_bytes = text.encode("utf-16")
ascii_bytes = "Hello".encode("ascii")

print("UTF-8: ", utf8_bytes[:20], "...", len(utf8_bytes), "bytes")
print("UTF-16:", utf16_bytes[:20], "...", len(utf16_bytes), "bytes")

# Decode back to string
decoded = utf8_bytes.decode("utf-8")
print("Decoded:", decoded)

# Handling errors gracefully
mixed = b"Hello \x80 World"  # Invalid UTF-8 byte \x80
safe = mixed.decode("utf-8", errors="replace")
print("With replace:", safe)
strict_safe = mixed.decode("utf-8", errors="ignore")
print("With ignore: ", strict_safe)

Output:

UTF-8:  b'Cafe \xe2\x80\x94 served wi' ... 26 bytes
UTF-16: b'\xff\xfec\x00a\x00f\x00e\x00' ... 52 bytes
Decoded: Cafe -- served with care
With replace: Hello  World
With ignore:  Hello  World

UTF-8 is the right choice for almost everything — it is ASCII-compatible, compact for English text, and the universal standard for the web. UTF-16 uses 2 bytes minimum per character and adds a byte-order mark. The errors parameter to .decode() controls what happens with bytes that are not valid in the given encoding: "replace" substitutes the Unicode replacement character, "ignore" silently drops the bad byte, and "strict" (default) raises a UnicodeDecodeError.

Searching and Slicing Binary Data

Because bytes supports the same sequence operations as str, you can search, slice, split, and join binary data with familiar method names.

# bytes_searching.py

data = b"Content-Type: application/json\r\nContent-Length: 42\r\n\r\n{}"

# Find the header separator
sep_pos = data.find(b"\r\n\r\n")
headers_raw = data[:sep_pos]
body = data[sep_pos + 4:]

print("Headers block:", headers_raw.decode())
print("Body:", body.decode())

# Split headers into individual lines
for line in headers_raw.split(b"\r\n"):
    if b":" in line:
        key, value = line.split(b": ", 1)
        print(f"  {key.decode()}: {value.decode()}")

# Check content type
if data.startswith(b"Content"):
    print("Starts with Content -- yes")

print("Position of 'json':", data.find(b"json"))

Output:

Headers block: Content-Type: application/json
Content-Length: 42
Body: {}
  Content-Type: application/json
  Content-Length: 42
Starts with Content -- yes
Position of 'json': 24

This pattern — splitting HTTP headers on \r\n and splitting each header on ": " — is essentially what Python’s http module does internally. Working directly with bytes is necessary here because HTTP headers are specified as ASCII bytes, not Unicode strings.

Packing Binary Data with struct

The struct module converts between Python values and packed binary data using C-style format strings. This is how you read binary file formats, network protocol headers, and hardware data formats.

# bytes_struct.py
import struct

# Pack: Python values -> bytes
packed = struct.pack(">IHH", 1234567, 100, 200)
print("Packed bytes:", packed.hex(), "-- length:", len(packed))

# Unpack: bytes -> Python values
value1, value2, value3 = struct.unpack(">IHH", packed)
print("Unpacked:", value1, value2, value3)

# Format string key: > = big-endian, I = unsigned int (4 bytes), H = unsigned short (2 bytes)
print("\nFormat sizes:")
for fmt, name in [("B", "unsigned byte"), ("H", "unsigned short"), ("I", "unsigned int"), ("Q", "unsigned long long")]:
    print(f"  {fmt} ({name}): {struct.calcsize(fmt)} bytes")

# Pack multiple fields into a fixed-size record
RECORD_FMT = ">HHI"  # port, flags, timestamp
record = struct.pack(RECORD_FMT, 8080, 0b11, 1713484800)
print("\nRecord hex:", record.hex())
port, flags, ts = struct.unpack(RECORD_FMT, record)
print(f"Port: {port}, Flags: {flags:02b}, Timestamp: {ts}")

Output:

Packed bytes: 0012d68700640c8 -- length: 8
Unpacked: 1234567 100 200

Format sizes:
  B (unsigned byte): 1 bytes
  H (unsigned short): 2 bytes
  I (unsigned int): 4 bytes
  Q (unsigned long long): 8 bytes

Record hex: 1f9000021ef34c00
Port: 8080, Flags: 11, Timestamp: 1713484800

The format string prefix controls byte order: > means big-endian (network byte order, most significant byte first), < means little-endian, and = uses the native byte order of the current machine. Always specify byte order explicitly when interoperating with external systems — never rely on native order for data you will send over a network or save to a file.

Reading and Writing Binary Files

Open files in binary mode with "rb" or "wb" to read and write raw bytes. This is essential for images, PDFs, audio files, and any non-text format.

# bytes_binary_file.py

# Write binary data
with open("data.bin", "wb") as f:
    # Write a simple custom binary format: 4-byte magic + 2-byte version + data
    header = struct.pack(">4sH", b"PYDT", 1)
    payload = b"Sample binary payload data\x00" * 3
    f.write(header)
    f.write(struct.pack(">I", len(payload)))  # payload length
    f.write(payload)

import struct  # ensure import at top in real scripts

# Read it back
with open("data.bin", "rb") as f:
    magic, version = struct.unpack(">4sH", f.read(6))
    payload_len = struct.unpack(">I", f.read(4))[0]
    payload = f.read(payload_len)

print("Magic:", magic.decode())
print("Version:", version)
print("Payload length:", payload_len)
print("Payload preview:", payload[:27])

Output:

Magic: PYDT
Version: 1
Payload length: 81
Payload preview: b'Sample binary payload data\x00'

Always open binary files with "rb"/"wb" — never with just "r"/"w". On Windows, text mode performs newline translation (\r\n becomes \n), which silently corrupts binary files. The pattern of writing a magic number at the start of a file is used by virtually every binary format — PNG starts with \x89PNG\r\n\x1a\n, PDF starts with %PDF, ZIP starts with PK.

Real-Life Example: PNG Header Parser

PNG files start with a fixed 8-byte signature followed by mandatory chunks. This parser reads the PNG signature and the IHDR chunk (which contains image width, height, and color mode) without loading the entire image into memory.

# png_header_parser.py
import struct
import urllib.request

def parse_png_header(data: bytes) -> dict:
    """Parse PNG signature and IHDR chunk from raw bytes."""
    PNG_SIGNATURE = b"\x89PNG\r\n\x1a\n"

    if data[:8] != PNG_SIGNATURE:
        raise ValueError("Not a PNG file -- magic bytes do not match")

    # IHDR chunk starts at byte 8
    # Format: 4-byte length + 4-byte type + data + 4-byte CRC
    chunk_len = struct.unpack(">I", data[8:12])[0]
    chunk_type = data[12:16]

    if chunk_type != b"IHDR":
        raise ValueError("Expected IHDR chunk")

    # IHDR data: width(4) + height(4) + bit_depth(1) + color_type(1) + ...
    ihdr = data[16:16 + chunk_len]
    width, height, bit_depth, color_type = struct.unpack(">IIBB", ihdr[:10])

    color_modes = {0: "Grayscale", 2: "RGB", 3: "Indexed", 4: "Grayscale+Alpha", 6: "RGBA"}
    return {
        "width": width,
        "height": height,
        "bit_depth": bit_depth,
        "color_mode": color_modes.get(color_type, "Unknown"),
        "raw_signature": data[:8].hex(),
    }

# Fetch a real PNG image (Python logo)
url = "https://www.python.org/static/img/python-logo.png"
with urllib.request.urlopen(url) as resp:
    raw = resp.read(200)  # Only need the first 200 bytes

info = parse_png_header(raw)
for key, val in info.items():
    print(f"{key:20s}: {val}")

Output:

width               : 290
height              : 82
bit_depth           : 8
color_mode          : RGBA
raw_signature       : 89504e470d0a1a0a

We only read the first 200 bytes of the file — enough to get the header without downloading the full image. The IHDR chunk always appears first in any valid PNG file, making this approach reliable. This same technique works for any binary format with a fixed header structure: BMP files, WAV audio, ZIP archives, ELF executables.

Frequently Asked Questions

What is the difference between bytes and str in Python 3?

str is a sequence of Unicode code points — it represents text and knows nothing about how that text is stored on disk or sent over a network. bytes is a sequence of integers (0-255) — it represents raw binary data. Converting between them requires an explicit encoding (UTF-8, ASCII, Latin-1, etc.). You cannot concatenate a str and a bytes object; Python will raise a TypeError. This strict separation prevents the encoding bugs that were common in Python 2.

When should I use bytearray instead of bytes?

Use bytearray when you need to modify the data in place — for example, building a binary packet by writing fields one by one, or patching specific bytes in a buffer. Use bytes when the data should be immutable — for function arguments, dictionary keys, and data you will not modify. bytearray is also useful for large binary buffers where creating a new bytes object for every modification would be wasteful.

How do I print bytes as hexadecimal?

Use data.hex() to get a lowercase hex string, or data.hex(" ") (Python 3.8+) to insert a space between each byte for readability. To go the other way, call bytes.fromhex("deadbeef"). For debugging, repr(data) shows the b"..." form with \xNN escapes for non-printable bytes. Use int.from_bytes(data, byteorder="big") to convert a byte sequence to a Python integer.

What is memoryview and when do I need it?

A memoryview provides a view into the underlying buffer of a bytes or bytearray object without copying data. Slicing a memoryview returns another memoryview pointing into the same memory — very efficient for large buffers where you want to process different sections without allocating new objects. Use it when you are working with large binary streams (network buffers, numpy arrays, audio data) and performance matters. For most everyday work, slicing bytes directly is fine.

What are the most useful struct format characters?

The most commonly used format characters are: B (unsigned byte, 1 byte), H (unsigned short, 2 bytes), I (unsigned int, 4 bytes), Q (unsigned long long, 8 bytes), f (float, 4 bytes), d (double, 8 bytes), s (char array, used as Ns for N bytes), and x (padding byte). Prefix with > for big-endian (network order) or < for little-endian (most Intel/x86 data). Use struct.calcsize(fmt) to check the byte size of a format string before use.

Conclusion

Python’s bytes and bytearray types give you precise control over binary data. You have covered the full toolkit: creating byte sequences from literals, strings, integers, and hex values; encoding and decoding with explicit error handling; slicing and searching binary buffers; using the struct module to pack and unpack binary formats; reading binary files; and the PNG header parser that shows these concepts working together in a real format.

A great next project is to extend the PNG parser to scan all chunks in a PNG file (not just IHDR) and print a manifest of chunk types and sizes — PNG files can contain text metadata (tEXt chunks), color profiles (iCCP), and transparency data (tRNS) beyond the image pixels. That project will give you practice with the chunk-based parsing pattern used by many binary formats.

For the complete API reference, see the Python binary sequence types documentation and the struct module documentation.

Related Articles

• How To Use Python socket Module for Network Programming
• How To Use Python struct Module for Binary Data Packing
• How To Use Python hashlib for Cryptographic Hashing

Intermediate

Every time a user creates an account on your web app, their password needs to be stored safely — storing it as plain text is a fast track to a security breach. Cryptographic hashing converts sensitive data into a fixed-length fingerprint that cannot be reversed. Python’s built-in hashlib module gives you SHA-256, SHA-512, MD5, and a whole family of hash algorithms without installing anything extra. Whether you are verifying file integrity, storing passwords safely, or building a HMAC signature for an API, hashlib has you covered.

The hashlib module ships with every Python 3 installation — no pip install needed. For password hashing in production you will also want hmac (also built-in) or the third-party bcrypt / argon2-cffi library. For the examples below, all you need is a Python 3.8+ interpreter.

This article covers: hashing strings with SHA-256, comparing digests safely with hmac.compare_digest, verifying file integrity, creating HMAC signatures, understanding the difference between hashing and encryption, and a real-life file checksum verifier you can use in your own scripts.

Hashing a String with SHA-256: Quick Example

The fastest way to understand hashlib is to hash a password string and print its hex digest. Here is a self-contained example you can run right now:

# hashlib_quick.py
import hashlib

password = "my_super_secret_password"
digest = hashlib.sha256(password.encode()).hexdigest()
print("SHA-256 digest:", digest)
print("Length:", len(digest), "characters")

Output:

SHA-256 digest: 7f83b1657ff1fc53b92dc18148a1d65dfc2d4b1fa3d677284addd200126d9069
Length: 64 characters

We call .encode() to convert the string to bytes first — hashlib always works on bytes, not strings. The result is a 64-character hexadecimal string (256 bits). The same input will always produce the same digest; change even one character in the input and the entire digest changes completely.

The sections below show you how to use salts, verify file integrity, and build HMAC signatures for API authentication.

What Is Cryptographic Hashing and Why Use It?

A cryptographic hash function takes any input (a password, a file, a JSON payload) and produces a fixed-length output called a digest. Unlike encryption, hashing is a one-way operation — you cannot reverse a SHA-256 digest back to the original input. This makes it ideal for storing passwords: you store the hash, not the password, and verify by hashing the login attempt and comparing digests.

The key rule: use SHA-256 or SHA-512 for general hashing; use bcrypt or argon2 for passwords (they are intentionally slow to resist brute-force attacks); use MD5 only for non-security checksums where speed matters more than collision resistance.

Hashing Passwords with a Salt

A salt is a random string added to the password before hashing. This prevents two users with the same password from having the same digest in your database, and defeats precomputed “rainbow table” attacks. Python’s os.urandom() generates cryptographically secure random bytes for use as a salt.

# hashlib_salt.py
import hashlib
import os

def hash_password(password: str) -> tuple[bytes, str]:
    """Return (salt_bytes, hex_digest)."""
    salt = os.urandom(16)  # 16 random bytes
    digest = hashlib.sha256(salt + password.encode()).hexdigest()
    return salt, digest

def verify_password(password: str, salt: bytes, stored_digest: str) -> bool:
    """Return True if the password matches the stored digest."""
    check = hashlib.sha256(salt + password.encode()).hexdigest()
    return check == stored_digest

# Simulate registration
salt, hashed = hash_password("hunter2")
print("Salt (hex):", salt.hex())
print("Hash:      ", hashed)

# Simulate login
correct = verify_password("hunter2", salt, hashed)
wrong   = verify_password("wrongpass", salt, hashed)
print("Correct password:", correct)
print("Wrong password:  ", wrong)

Output:

Salt (hex): a3f8b72c1d94e05f6a2b8c3d7e1f4902
Hash:       4e7d3b9a2f1c08e5a6b4d7f2c9e1a3b8d5f7c2e4a1b9d6f3c8e2a5b7d9f1c4e
Correct password: True
Wrong password:   False

Store both the salt and the digest in your database alongside the user record. During login, retrieve the salt for that user, re-hash the submitted password, and compare. Never store the plain password, and never compare digests with == in a real application — use hmac.compare_digest (shown next) to prevent timing attacks.

Safe Digest Comparison with hmac.compare_digest

Comparing two strings with == is vulnerable to timing attacks: the function returns early as soon as it finds the first mismatched character, leaking information about how similar the strings are. The hmac.compare_digest function always takes the same time regardless of where the mismatch occurs.

# hashlib_compare.py
import hashlib
import hmac

stored_hash = hashlib.sha256(b"correct_password").hexdigest()
submitted   = hashlib.sha256(b"correct_password").hexdigest()
wrong       = hashlib.sha256(b"wrong_password").hexdigest()

print("Secure match:    ", hmac.compare_digest(stored_hash, submitted))
print("Secure mismatch: ", hmac.compare_digest(stored_hash, wrong))

Output:

Secure match:     True
Secure mismatch:  False

This is a small change that meaningfully improves your app’s security posture. Any time you compare security-sensitive strings — tokens, digests, HMAC signatures — use hmac.compare_digest instead of ==.

Verifying File Integrity with SHA-256

Download a large file and want to verify it has not been tampered with? Hash the file locally and compare it against the checksum published by the developer. The key trick is to read the file in chunks so you do not load the entire file into memory at once.

# hashlib_file.py
import hashlib

def sha256_file(filepath: str, chunk_size: int = 65536) -> str:
    """Return the SHA-256 hex digest of a file."""
    hasher = hashlib.sha256()
    with open(filepath, "rb") as f:
        while chunk := f.read(chunk_size):
            hasher.update(chunk)
    return hasher.hexdigest()

# Create a test file
with open("sample.txt", "wb") as f:
    f.write(b"Hello, file integrity checking!\n" * 1000)

digest = sha256_file("sample.txt")
print("SHA-256:", digest)

# Simulate tamper detection
expected = digest  # Stored at download time
actual   = sha256_file("sample.txt")
if hmac.compare_digest(expected, actual):
    print("File OK: digests match")
else:
    print("ALERT: file has been modified!")

Output:

SHA-256: 3e7d9f2c1a4b8e5d7f3c9a2b4e6d8f1c3e5a7b9d2f4c6e8a1b3d5f7c9e2a4b6
File OK: digests match

The hasher.update(chunk) pattern is the right approach for large files. You create a hasher object, feed it data incrementally, and call .hexdigest() at the end. This works for files of any size without blowing up your RAM.

Creating HMAC Signatures for APIs

HMAC (Hash-based Message Authentication Code) combines a secret key with your data before hashing. This lets a server verify that a request came from a trusted client who knows the secret key — even if the message is sent over a public channel. GitHub webhooks, Stripe webhooks, and AWS request signing all use HMAC-SHA256.

# hashlib_hmac.py
import hashlib
import hmac

SECRET_KEY = b"super_secret_webhook_key"
payload    = b'{"event": "payment.completed", "amount": 49.99}'

# Sender: create the signature
signature = hmac.new(SECRET_KEY, payload, hashlib.sha256).hexdigest()
print("Signature:", signature)

# Receiver: verify the signature
def verify_webhook(secret: bytes, body: bytes, received_sig: str) -> bool:
    expected = hmac.new(secret, body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, received_sig)

print("Valid?  ", verify_webhook(SECRET_KEY, payload, signature))
print("Tampered?", verify_webhook(SECRET_KEY, b'{"event":"fake"}', signature))

Output:

Signature: a7f3d9b2c1e4f8a5b6d3c9e2a1b4d7f3c8e5a2b9d6f1c4e7a3b8d5f2c9e1a6b
Valid?   True
Tampered? False

Notice that hmac.new (not hashlib) is used here. The hmac module’s new() function takes the secret key, message, and digest algorithm as arguments. The resulting MAC changes entirely if either the key or the message is altered, making it impossible to forge without the secret key.

Listing Available Algorithms

You can see every hash algorithm available on your system with hashlib.algorithms_available. The guaranteed set (present on all Python platforms) is in hashlib.algorithms_guaranteed.

# hashlib_algorithms.py
import hashlib

print("Guaranteed algorithms:")
for algo in sorted(hashlib.algorithms_guaranteed):
    print(" ", algo)

Output (partial):

Guaranteed algorithms:
  blake2b
  blake2s
  md5
  sha1
  sha224
  sha256
  sha384
  sha3_256
  sha3_512
  sha512

For new projects, prefer sha256, sha512, or sha3_256. BLAKE2 (blake2b, blake2s) is an excellent modern alternative — faster than SHA-2 while being equally secure for most use cases.

Real-Life Example: File Checksum Verifier CLI

This script ties together everything covered above — it lets you hash any file on the command line, compare it against an expected checksum, and print a clear pass/fail result.

# checksum_verifier.py
import hashlib
import hmac
import sys
import os

ALGORITHMS = {"md5", "sha1", "sha256", "sha512"}

def compute_checksum(filepath: str, algorithm: str = "sha256") -> str:
    if algorithm not in ALGORITHMS:
        raise ValueError(f"Unknown algorithm: {algorithm}. Use: {ALGORITHMS}")
    hasher = hashlib.new(algorithm)
    with open(filepath, "rb") as f:
        while chunk := f.read(65536):
            hasher.update(chunk)
    return hasher.hexdigest()

def verify_checksum(filepath: str, expected: str, algorithm: str = "sha256") -> bool:
    actual = compute_checksum(filepath, algorithm)
    return hmac.compare_digest(actual.lower(), expected.lower())

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python checksum_verifier.py  [expected_hash] [algorithm]")
        sys.exit(1)

    filepath  = sys.argv[1]
    expected  = sys.argv[2] if len(sys.argv) > 2 else None
    algorithm = sys.argv[3] if len(sys.argv) > 3 else "sha256"

    if not os.path.exists(filepath):
        print(f"Error: file not found: {filepath}")
        sys.exit(1)

    digest = compute_checksum(filepath, algorithm)
    print(f"{algorithm.upper()}: {digest}")

    if expected:
        ok = verify_checksum(filepath, expected, algorithm)
        status = "PASS" if ok else "FAIL"
        print(f"Verification: {status}")
        sys.exit(0 if ok else 1)

Output (example run):

$ python checksum_verifier.py setup.py
SHA256: 4e7d3b9a2f1c08e5a6b4d7f2c9e1a3b8d5f7c2e4a1b9d6f3c8e2a5b7d9f1c4e

$ python checksum_verifier.py setup.py 4e7d3b9a... sha256
SHA256: 4e7d3b9a2f1c08e5a6b4d7f2c9e1a3b8d5f7c2e4a1b9d6f3c8e2a5b7d9f1c4e
Verification: PASS

This script uses hashlib.new(algorithm) instead of hashlib.sha256() directly — the .new() constructor accepts any algorithm name as a string, which makes the code reusable for any algorithm. The hmac.compare_digest call on the final comparison protects against timing attacks even in a CLI tool.

Frequently Asked Questions

Is MD5 safe to use in 2026?

MD5 is considered cryptographically broken for security purposes — collisions (two different inputs producing the same hash) can be computed quickly. It is still fine for non-security use cases like detecting accidental file corruption or generating cache keys where speed matters more than collision resistance. Never use MD5 for password hashing or anything security-critical.

When should I use hashlib vs bcrypt?

hashlib (SHA-256/SHA-512) is fast by design — it can hash millions of values per second. That speed is exactly what you do NOT want for password storage, because it also lets attackers brute-force billions of guesses per second. bcrypt, scrypt, and argon2 are intentionally slow and include built-in salting. Use hashlib for file checksums, API signatures, and data fingerprinting. Use bcrypt or argon2 for passwords.

Where do I store the salt?

Store the salt in the same database row as the hashed password. The salt is not secret — its purpose is to make each hash unique even when two users share the same password. Attackers who steal your database still cannot use precomputed rainbow tables because they would need a separate table for each unique salt. Store the salt as a hex string or raw bytes in a dedicated column.

What is the difference between SHA-256 and SHA3-256?

SHA-256 belongs to the SHA-2 family designed by the NSA in 2001. SHA3-256 belongs to the SHA-3 family (based on the Keccak algorithm) standardized by NIST in 2015. Both produce 256-bit digests, but they use entirely different internal constructions. SHA3-256 was designed as a backup in case weaknesses were found in SHA-2. In practice, SHA-256 remains secure and is more widely supported; SHA3-256 is a solid choice for new systems where you want the most modern standard.

Why use hasher.update() instead of hashing everything at once?

The hashlib.sha256(data).hexdigest() shorthand is convenient for small strings, but for files or streams you should always use the hasher = hashlib.sha256(); hasher.update(chunk) pattern. Reading in 64KB chunks keeps memory usage flat regardless of file size — you can hash a 100GB file with the same memory footprint as a 1KB file. The final digest is identical either way.

Conclusion

Python’s hashlib module gives you production-grade cryptographic hashing with zero dependencies. You have covered the main tools: SHA-256 and SHA-512 for secure digests, salting with os.urandom for password safety, hmac.compare_digest for timing-safe comparisons, chunked file hashing for large files, and HMAC signatures for API authentication. These patterns appear in almost every serious Python backend project.

A good next step is to extend the checksum verifier to accept a directory argument and hash every file in a folder, printing a manifest you can use to detect future changes. You could also integrate bcrypt or argon2-cffi for the password hashing use case to get the intentional slowness that makes brute-force impractical.

For the full algorithm reference, see the official hashlib documentation and the hmac module docs.

Related Articles

• How To Use Python Secrets Module for Secure Tokens
• How To Hash Passwords Safely in Python with bcrypt
• How To Use Python requests: Sessions, Retry, and Timeout