Intermediate

You have inherited a folder of Excel files from an accounting department. Half of them have the .xlsx extension from modern Excel, and the other half are old .xls files from the early 2000s. If you try to open a .xls file with openpyxl, you get an error. If you use xlrd 2.x on an .xlsx file, you get a different error. Knowing which library to use for which file format is the first step — everything else follows.

The two key libraries are: xlrd for reading legacy .xls (Excel 97-2003) format, and openpyxl for reading and writing modern .xlsx format. Install both with pip install xlrd openpyxl. For most practical data extraction tasks you will also want to know about pandas, which uses both libraries internally and provides a higher-level API.

In this article we will cover reading .xls files with xlrd, reading .xlsx files with openpyxl, detecting file format and routing automatically, extracting headers and data rows, handling different cell types, and building a unified Excel reader that handles both formats with the same output interface.

Reading Excel Files in Python: Quick Example

Here is the essential pattern for reading each format. Note the different library imports and APIs:

# quick_excel.py
import xlrd      # For .xls files (pip install xlrd)
import openpyxl  # For .xlsx files (pip install openpyxl)

# --- Reading a .xls file with xlrd ---
# xlrd 2.x only supports .xls format (NOT .xlsx)
# workbook = xlrd.open_workbook('legacy_data.xls')
# sheet = workbook.sheet_by_index(0)
# for row_idx in range(sheet.nrows):
#     row = sheet.row_values(row_idx)
#     print(row)

# --- Reading a .xlsx file with openpyxl ---
# Create a sample workbook to demonstrate
wb = openpyxl.Workbook()
ws = wb.active
ws.title = "Sales"
ws.append(["Product", "Qty", "Price"])
ws.append(["Widget A", 100, 9.99])
ws.append(["Widget B", 50,  14.99])
wb.save("sample.xlsx")

# Now read it back
wb2 = openpyxl.load_workbook("sample.xlsx")
ws2 = wb2.active
for row in ws2.iter_rows(values_only=True):
    print(row)

Output:

('Product', 'Qty', 'Price')
('Widget A', 100, 9.99)
('Widget B', 50, 14.99)

The critical difference is format support: xlrd 2.0 (released 2020) intentionally dropped .xlsx support due to security concerns with the XML-based format. Always use openpyxl for .xlsx files. If you need to process both formats in one script, you will need both libraries. Want to go deeper? Below we cover each library in detail with cell type handling and a unified reader.

xlrd vs openpyxl: Which Library for Which Format?

The most important thing to understand about Excel reading in Python is the format split. The confusion often arises because older xlrd versions (before 2.0) supported both formats, so many tutorials and Stack Overflow answers use xlrd for .xlsx files. Those answers are outdated — xlrd 2.x will raise a XLRDError if you try to open a .xlsx file.

For most new projects, openpyxl is the right choice since virtually all modern Excel files are .xlsx. You need xlrd only when processing legacy files from systems that still produce the old binary format — which is more common than you might expect in finance, manufacturing, and government data pipelines.

Reading Legacy .xls Files with xlrd

xlrd provides a workbook/sheet/cell hierarchy for accessing data. Row and column indices are zero-based, and cell values come with type information that you need to handle for dates:

# read_xls.py
import xlrd
from datetime import datetime

# First, create a sample .xls for demonstration
# (normally you would just open an existing file)
import xlwt  # pip install xlwt -- only for creating .xls test files
wb_write = xlwt.Workbook()
ws_write = wb_write.add_sheet('Employees')
headers = ['Name', 'Department', 'Salary', 'Start Date']
data = [
    ['Alice Chen',    'Engineering', 95000, '2021-03-15'],
    ['Bob Martinez',  'Marketing',   72000, '2020-07-01'],
    ['Carol Johnson', 'Finance',     88000, '2019-11-20'],
]
for col, h in enumerate(headers):
    ws_write.write(0, col, h)
for row_idx, row in enumerate(data, 1):
    for col_idx, val in enumerate(row):
        ws_write.write(row_idx, col_idx, val)
wb_write.save('employees.xls')

# Now read the .xls file with xlrd
workbook = xlrd.open_workbook('employees.xls')
print(f'Sheets: {workbook.sheet_names()}')

sheet = workbook.sheet_by_name('Employees')
print(f'Rows: {sheet.nrows}, Cols: {sheet.ncols}')
print()

# Read header row
headers = sheet.row_values(0)
print('Headers:', headers)
print()

# Read data rows
for row_idx in range(1, sheet.nrows):
    row = {}
    for col_idx, header in enumerate(headers):
        cell = sheet.cell(row_idx, col_idx)
        # Check cell type -- xlrd.XL_CELL_DATE = 3
        if cell.ctype == xlrd.XL_CELL_DATE:
            dt = xlrd.xldate_as_datetime(cell.value, workbook.datemode)
            row[header] = dt.strftime('%Y-%m-%d')
        else:
            row[header] = cell.value
    print(row)

Output:

Sheets: ['Employees']
Rows: 4, Cols: 4

Headers: ['Name', 'Department', 'Salary', 'Start Date']

{'Name': 'Alice Chen', 'Department': 'Engineering', 'Salary': 95000.0, 'Start Date': '2021-03-15'}
{'Name': 'Bob Martinez', 'Department': 'Marketing', 'Salary': 72000.0, 'Start Date': '2020-07-01'}
{'Name': 'Carol Johnson', 'Department': 'Finance', 'Salary': 88000.0, 'Start Date': '2019-11-20'}

Date handling is the trickiest part of reading .xls files with xlrd. Excel stores dates as floating-point numbers (days since 1900-01-01), not as Python datetime objects. The xlrd.xldate_as_datetime() function converts this float to a real datetime using the workbook’s date mode (which differs between Windows and Mac Excel). Always check cell.ctype == xlrd.XL_CELL_DATE before attempting date conversion.

Reading Modern .xlsx Files with openpyxl

openpyxl uses a Workbook/Worksheet/Cell model similar to xlrd but with a more Pythonic API. The values_only=True flag is the fastest way to extract data when you do not need cell formatting metadata:

# read_xlsx.py
import openpyxl
from datetime import datetime

# Create a sample .xlsx file with mixed cell types
wb = openpyxl.Workbook()
ws = wb.active
ws.title = "Inventory"
ws.append(["SKU", "Product", "Stock", "Price", "Last Updated"])
from datetime import date
ws.append(["A001", "Widget Pro",   500, 29.99, date(2026, 4, 15)])
ws.append(["A002", "Gadget Basic",  75, 9.99,  date(2026, 4, 20)])
ws.append(["A003", "Super Deluxe",  12, 149.99,date(2026, 5, 1)])
wb.save("inventory.xlsx")

# Read it back
wb2 = openpyxl.load_workbook("inventory.xlsx", read_only=True)
ws2 = wb2.active

print(f'Dimensions: {ws2.dimensions}')
print()

# Method 1: iter_rows with values_only
print('--- Using iter_rows(values_only=True) ---')
headers = None
for i, row in enumerate(ws2.iter_rows(values_only=True)):
    if i == 0:
        headers = row
        print('Headers:', headers)
    else:
        record = dict(zip(headers, row))
        print(record)

wb2.close()

# Method 2: read_only=False for cell object access
wb3 = openpyxl.load_workbook("inventory.xlsx")
ws3 = wb3["Inventory"]

print()
print('--- Cell object access (non-readonly) ---')
# Access individual cells by coordinate
print('A1:', ws3['A1'].value)
print('B2:', ws3['B2'].value)
print('E2 (date):', ws3['E2'].value, type(ws3['E2'].value).__name__)

Output:

Dimensions: A1:E4

--- Using iter_rows(values_only=True) ---
Headers: ('SKU', 'Product', 'Stock', 'Price', 'Last Updated')
{'SKU': 'A001', 'Product': 'Widget Pro', 'Stock': 500, 'Price': 29.99, 'Last Updated': datetime.datetime(2026, 4, 15, 0, 0)}
{'SKU': 'A002', 'Product': 'Gadget Basic', 'Stock': 75, 'Price': 9.99, 'Last Updated': datetime.datetime(2026, 4, 20, 0, 0)}
{'SKU': 'A003', 'Product': 'Super Deluxe', 'Stock': 12, 'Price': 149.99, 'Last Updated': datetime.datetime(2026, 5, 1, 0, 0)}

--- Cell object access (non-readonly) ---
A1: SKU
B2: Widget Pro
E2 (date): 2026-04-15 00:00:00 datetime

Unlike xlrd, openpyxl automatically converts date cells to Python datetime objects — no manual conversion needed. The read_only=True mode is faster for large files because it uses an event-based parser that does not load the entire file into memory. Use it when you only need to read data, and disable it (default mode) when you need to access cell formatting or ose the ws[coordinate] syntax.

Real-Life Example: Unified Excel File Reader

Here is a unified reader that auto-detects the file format and uses the right library, returning a consistent list-of-dicts structure regardless of which format the input uses:

# unified_excel_reader.py
import xlrd
import openpyxl
from pathlib import Path
from datetime import datetime

def read_excel_file(filepath, sheet_index=0, header_row=0):
    """
    Read an Excel file (.xls or .xlsx) into a list of dicts.
    Auto-detects format from file extension.
    
    Returns:
        list[dict]: One dict per data row, keyed by header names
    """
    path = Path(filepath)
    ext = path.suffix.lower()
    
    if ext == '.xls':
        return _read_xls(filepath, sheet_index, header_row)
    elif ext in ('.xlsx', '.xlsm', '.xltx', '.xltm'):
        return _read_xlsx(filepath, sheet_index, header_row)
    else:
        raise ValueError(f"Unsupported format: {ext}. Use .xls or .xlsx")

def _read_xls(filepath, sheet_index, header_row):
    """Read a .xls file using xlrd."""
    wb = xlrd.open_workbook(filepath)
    ws = wb.sheet_by_index(sheet_index)
    
    headers = ws.row_values(header_row)
    records = []
    
    for row_idx in range(header_row + 1, ws.nrows):
        record = {}
        for col_idx, header in enumerate(headers):
            cell = ws.cell(row_idx, col_idx)
            if cell.ctype == xlrd.XL_CELL_DATE:
                value = xlrd.xldate_as_datetime(cell.value, wb.datemode)
            elif cell.ctype == xlrd.XL_CELL_EMPTY:
                value = None
            else:
                value = cell.value
            record[header] = value
        records.append(record)
    
    return records

def _read_xlsx(filepath, sheet_index, header_row):
    """Read a .xlsx file using openpyxl."""
    wb = openpyxl.load_workbook(filepath, read_only=True, data_only=True)
    ws_name = wb.sheetnames[sheet_index]
    ws = wb[ws_name]
    
    records = []
    headers = None
    
    for row_idx, row in enumerate(ws.iter_rows(values_only=True)):
        if row_idx == header_row:
            headers = [str(h) if h is not None else f'Col_{i}' 
                      for i, h in enumerate(row)]
            continue
        if headers is None:
            continue
        record = dict(zip(headers, row))
        records.append(record)
    
    wb.close()
    return records

def print_summary(records, filename):
    """Print a summary of the extracted data."""
    print(f"\nFile: {filename}")
    print(f"Records: {len(records)}")
    if records:
        print(f"Columns: {list(records[0].keys())}")
        print("First record:", records[0])
        print("Last record: ", records[-1])

# Test with both formats
# (Using the files created in earlier examples)
xlsx_data = read_excel_file('inventory.xlsx')
print_summary(xlsx_data, 'inventory.xlsx')

# xls_data = read_excel_file('employees.xls')
# print_summary(xls_data, 'employees.xls')

Output:

File: inventory.xlsx
Records: 3
Columns: ['SKU', 'Product', 'Stock', 'Price', 'Last Updated']
First record: {'SKU': 'A001', 'Product': 'Widget Pro', 'Stock': 500, 'Price': 29.99, 'Last Updated': datetime.datetime(2026, 4, 15, 0, 0)}
Last record:  {'SKU': 'A003', 'Product': 'Super Deluxe', 'Stock': 12, 'Price': 149.99, 'Last Updated': datetime.datetime(2026, 5, 1, 0, 0)}

This unified reader pattern is the cleanest way to handle a mixed bag of Excel files without scattering format-detection logic throughout your codebase. Extend it by adding support for a max_rows parameter for large files, a dtype_map for explicit type coercion, or a pandas output mode that returns a DataFrame instead of a list of dicts.

Frequently Asked Questions

Why does xlrd raise an error when I open a .xlsx file?

xlrd version 2.0 (released January 2020) deliberately removed .xlsx support. The maintainer decided the risk of parsing the complex XML-based format was not worth maintaining. If you are getting xlrd.biffh.XLRDError: Excel xlsx file; not supported, you are using xlrd 2.x. Switch to openpyxl for .xlsx files. If you have legacy code that used xlrd for .xlsx, the fix is straightforward: replace xlrd.open_workbook() with openpyxl.load_workbook() and adjust the API calls accordingly.

How do I efficiently read large Excel files without running out of memory?

For large .xlsx files, use openpyxl.load_workbook(filepath, read_only=True). In read-only mode, openpyxl uses an event-based parser that processes rows one at a time without loading the whole file. You can also use pandas pd.read_excel() with the chunksize parameter to process data in chunks. For very large files (100,000+ rows), pandas with openpyxl is typically faster than using openpyxl directly because pandas optimizes the data extraction path.

How do I read cell values instead of formulas in .xlsx files?

Pass data_only=True to openpyxl.load_workbook(). By default, openpyxl reads the formula text (=SUM(A1:A10)) instead of the computed value. With data_only=True, openpyxl reads the cached value that Excel stored when the file was last saved. Note: if the file was never opened in Excel (e.g., programmatically generated), the cache may be empty and you will get None for formula cells.

When should I use pandas instead of xlrd/openpyxl directly?

Use pandas when you need to do any data analysis, filtering, or transformation after reading the file. pd.read_excel('file.xlsx') handles format detection, type inference, header parsing, and multi-sheet reading in one call. Use xlrd or openpyxl directly when you need precise control over cell access, formatting metadata, or when writing production code that should not depend on pandas. For simple ETL pipelines that just extract data and load it elsewhere, the direct library approach has fewer dependencies.

Can xlrd write .xls files?

No — xlrd is read-only. To create .xls files programmatically, use xlwt (pip install xlwt). However, xlwt is also unmaintained and you should avoid creating new .xls files in 2026. If a system still requires .xls output, use xlwt as a stopgap, but the right long-term solution is upgrading that system to accept .xlsx. For .xlsx writing, openpyxl is the standard choice and actively maintained.

Conclusion

Reading Excel files in Python requires knowing which library matches the file format: xlrd for legacy .xls files, openpyxl for modern .xlsx files. We covered the format difference and why xlrd 2.x dropped .xlsx support, reading .xls files with xlrd including date cell handling, reading .xlsx files with openpyxl including read-only mode for performance, and building a unified reader that auto-detects format and returns consistent output.

The unified reader is the practical tool for any data pipeline that processes Excel files from varied sources. From here, you can add pandas integration for data analysis, write Excel output with openpyxl’s workbook creation API, or build a batch processor that walks a directory and converts every Excel file to CSV. The format-handling knowledge transfers directly to any project that touches Excel files.

For writing and formatting .xlsx files, see the openpyxl documentation. For legacy .xls reading details, see the xlrd documentation.

Related Articles

• How To Parse and Create Excel Files with openpyxl in Python
• How To Use Python csv Module for Reading and Writing CSV Files
• How To Read and Write JSON Files in Python 3 With Examples

Intermediate

In 1999, NASA’s Mars Climate Orbiter was lost because one engineering team used metric units and another used imperial units — a 327 million dollar spacecraft burned up in the Martian atmosphere due to a unit mismatch. You are probably not building Mars orbiters, but if your code processes temperatures, distances, speeds, or any other physical quantity, you face the same class of bug every time a raw number crosses a unit boundary without carrying its unit along with it.

The Pint library solves this by treating numbers as quantities — values paired with units that travel together through every calculation. When you multiply meters by meters you get square meters. When you try to add kilometers to kilograms, Pint raises a DimensionalityError instead of silently returning nonsense. Install it with pip install pint.

In this article we will cover creating quantities, converting between units, doing arithmetic with dimensional analysis, defining custom units, formatting output, and integrating Pint with NumPy. We will finish with a unit-safe physics calculator that demonstrates why you never want to go back to bare floats for physical quantities.

Python Pint: Quick Example

Here is the core Pint workflow in under 20 lines. Create a unit registry, attach units to values, and let Pint handle conversions and validation automatically:

# quick_pint.py
from pint import UnitRegistry

ureg = UnitRegistry()

# Create quantities -- value + unit
distance = 10 * ureg.kilometer
time_taken = 30 * ureg.minute
speed = distance / time_taken

print(f'Distance: {distance}')
print(f'Time: {time_taken}')
print(f'Speed: {speed}')
print(f'Speed in km/h: {speed.to(ureg.kilometer / ureg.hour):.2f}')
print(f'Speed in mph: {speed.to("mph"):.2f}')

Output:

Distance: 10 kilometer
Time: 30 minute
Speed: 0.3333333333333333 kilometer / minute
Speed in km/h: 20.00 kilometer / hour
Speed in mph: 12.43 mile / hour

Notice that the speed calculation is done directly by dividing a distance by a time — Pint performs the arithmetic and tracks the compound unit automatically. The .to() method converts to any compatible unit with dimensional analysis. Want to go deeper? Below we cover the full Pint API including error handling, custom units, and NumPy integration.

What Is Pint and Why Use It?

Pint is a Python library for defining, operating, and manipulating physical quantities — numbers that have both a magnitude and a unit. The central object is the Quantity, which wraps a value with its unit and enforces dimensional consistency in all operations.

The unit registry (UnitRegistry) knows about thousands of units built in — SI units, imperial units, derived units, and common abbreviations. You create it once per module and reuse it across your whole application.

Creating Quantities and the Unit Registry

There are several ways to create Pint quantities. The most common is multiplying a number by a unit attribute on the registry. You can also use string parsing for more flexible input:

# creating_quantities.py
from pint import UnitRegistry

ureg = UnitRegistry()

# Method 1: Multiply by registry attribute
mass = 5.5 * ureg.kilogram
temp = 100 * ureg.degC
area = 25 * ureg.meter ** 2

# Method 2: Parse from string (useful for user input)
length = ureg.Quantity('12.5 meters')
velocity = ureg.Quantity(60, 'km/hour')

# Method 3: Q_ shortcut (common in practice)
Q_ = ureg.Quantity
pressure = Q_(101325, 'Pa')

print(f'Mass: {mass}')
print(f'Temperature: {temp}')
print(f'Area: {area}')
print(f'Length (parsed): {length}')
print(f'Velocity (parsed): {velocity}')
print(f'Pressure: {pressure}')

# Access magnitude and units separately
print(f'\nMagnitude: {mass.magnitude}')
print(f'Units: {mass.units}')
print(f'Dimensionality: {mass.dimensionality}')

Output:

Mass: 5.5 kilogram
Temperature: 100 degree_Celsius
Area: 25 meter ** 2
Length (parsed): 12.5 meter
Velocity (parsed): 60 kilometer / hour
Pressure: 101325 pascal

Magnitude: 5.5
Units: kilogram
Dimensionality: [mass]

The string parsing form (ureg.Quantity('12.5 meters')) is particularly valuable when accepting unit input from users or configuration files — you do not have to parse the string yourself or validate the unit name. Pint handles both tasks and raises a UndefinedUnitError if the unit string is not recognized.

Converting Between Units

Unit conversion is where Pint saves the most code. The .to() method converts to any dimensionally compatible unit, and .to_base_units() normalizes to SI base units:

# conversions.py
from pint import UnitRegistry

ureg = UnitRegistry()

# Basic conversions
distance_km = 42.195 * ureg.kilometer  # Marathon distance
print(f'Marathon: {distance_km.to(ureg.mile):.3f}')
print(f'Marathon: {distance_km.to(ureg.meter):.0f}')
print(f'Marathon: {distance_km.to("ft"):.0f}')

# Temperature conversions (offset units -- use .to() not arithmetic)
temp_c = 100 * ureg.degC
print(f'\n100 C in Fahrenheit: {temp_c.to(ureg.degF):.1f}')
print(f'100 C in Kelvin: {temp_c.to(ureg.kelvin):.2f}')

# Compound unit conversion
speed = 100 * ureg.kilometer / ureg.hour
print(f'\n100 km/h in m/s: {speed.to("m/s"):.2f}')
print(f'100 km/h in mph: {speed.to("mph"):.2f}')

# Force conversion
force = 100 * ureg.newton
print(f'\n100 N in pound-force: {force.to("lbf"):.3f}')
print(f'100 N in kgf: {force.to("kgf"):.3f}')

# to_base_units -- normalize to SI
energy = 1 * ureg.kilowatt_hour
print(f'\n1 kWh in base units: {energy.to_base_units():.0f}')

Output:

Marathon: 26.219 mile
Marathon: 42195 meter
Marathon: 138,434 foot

100 C in Fahrenheit: 212.0 degree_Fahrenheit
100 C in Kelvin: 373.15 kelvin

100 km/h in m/s: 27.78 meter / second
100 km/h in mph: 62.14 mile / hour

100 N in pound-force: 22.481 force_pound
100 N in kgf: 10.197 kilogram_force

1 kWh in base units: 3600000 kilogram * meter ** 2 / second ** 2

Temperature conversion deserves a special mention. Unlike most units, Celsius and Fahrenheit are offset units — 0 degrees Celsius is not the same as 0 degrees Kelvin. Pint handles this correctly: 100 * ureg.degC converts to 212 degF as expected. If you are doing temperature arithmetic (differences, not absolute temperatures), use ureg.delta_degC to get correct results for temperature intervals.

Dimensional Analysis and Error Prevention

The real value of Pint is not convenience — it is catching bugs that bare floats silently hide. When you add incompatible units, Pint raises a DimensionalityError immediately instead of returning a meaningless number:

# dimensional_analysis.py
from pint import UnitRegistry, DimensionalityError

ureg = UnitRegistry()

# Arithmetic with matching dimensions -- works correctly
d1 = 5 * ureg.kilometer
d2 = 3000 * ureg.meter
total = d1 + d2  # Pint converts automatically
print(f'Total distance: {total}')
print(f'In km: {total.to("km")}')

# Arithmetic with different-but-compatible dimensions
weight = 70 * ureg.kilogram
g = 9.81 * ureg.meter / ureg.second ** 2
force = weight * g
print(f'\nForce: {force:.2f}')
print(f'Force in N: {force.to(ureg.newton):.2f}')

# Dimensional incompatibility -- caught immediately
mass = 10 * ureg.kilogram
length = 5 * ureg.meter
try:
    bad_result = mass + length  # This makes no physical sense
except DimensionalityError as e:
    print(f'\nCaught: {e}')

# Another bad operation
try:
    speed = 60 * ureg.km / ureg.hour
    nonsense = speed + mass
except DimensionalityError as e:
    print(f'Caught: {e}')

Output:

Total distance: 8.0 kilometer
In km: 8.0 kilometer

Force: 686.70 kilogram * meter / second ** 2
Force in N: 686.70 newton

Caught: Cannot convert from 'kilogram' ([mass]) to 'meter' ([length])
Caught: Cannot convert from 'kilometer / hour' ([length] / [time]) to 'kilogram' ([mass])

When adding 5 km + 3000 m, Pint automatically converts meters to kilometers before summing — you get the right answer without writing any conversion code. And when you try to add kilograms to meters, you get an immediate, descriptive error with the actual dimensions shown. This is the kind of safety that pays off at 2am when you are debugging a calculation that has been silently wrong for months.

Real-Life Example: Unit-Safe Physics Calculator

Here is a physics calculator that uses Pint to ensure all calculations are dimensionally consistent. It computes projectile motion values and never lets a unit slip through without validation:

# physics_calculator.py
from pint import UnitRegistry
import math

ureg = UnitRegistry()
Q_ = ureg.Quantity

def projectile_range(initial_speed, angle_degrees, gravity=None):
    """Calculate the range of a projectile.
    
    Uses the formula: R = (v^2 * sin(2*theta)) / g
    All inputs must be Pint Quantities with compatible units.
    """
    if gravity is None:
        gravity = Q_(9.81, 'm/s^2')
    
    angle_rad = math.radians(angle_degrees)
    sin2theta = math.sin(2 * angle_rad)
    
    # Pint tracks units through the calculation
    range_distance = (initial_speed ** 2 * sin2theta) / gravity
    return range_distance.to(ureg.meter)

def kinetic_energy(mass, velocity):
    """Calculate kinetic energy: E = 0.5 * m * v^2"""
    energy = 0.5 * mass * velocity ** 2
    return energy.to(ureg.joule)

def fuel_efficiency(distance, fuel_used):
    """Calculate fuel efficiency and convert to common units."""
    efficiency = distance / fuel_used
    return {
        'metric': efficiency.to('km/L'),
        'us_mpg': efficiency.to('miles/gallon'),
        'uk_mpg': efficiency.to('miles/imperial_gallon'),
    }

# Test the calculator
print('=== Projectile Calculator ===')
for angle in [15, 30, 45, 60, 75]:
    speed = Q_(50, 'm/s')
    r = projectile_range(speed, angle)
    print(f'Angle {angle:2d}deg: {r:.1f}')

print('\n=== Kinetic Energy ===')
car_mass = Q_(1500, 'kg')
for speed_kph in [60, 100, 120]:
    speed = Q_(speed_kph, 'km/h').to('m/s')
    ke = kinetic_energy(car_mass, speed)
    print(f'{speed_kph} km/h: {ke:.0f} ({ke.to("kJ"):.1f})')

print('\n=== Fuel Efficiency ===')
trip = Q_(400, 'km')
fuel = Q_(35, 'L')
eff = fuel_efficiency(trip, fuel)
for label, value in eff.items():
    print(f'  {label}: {value:.2f}')

Output:

=== Projectile Calculator ===
Angle 15deg: 127.4 meter
Angle 30deg: 220.7 meter
Angle 45deg: 254.8 meter
Angle 60deg: 220.7 meter
Angle 75deg: 127.4 meter

=== Kinetic Energy ===
60 km/h: 208333 joule (208.3 kJ)
100 km/h: 578704 joule (578.7 kJ)
120 km/h: 833333 joule (833.3 kJ)

=== Fuel Efficiency ===
  metric: 11.43 kilometer / liter
  us_mpg: 26.87 mile / gallon
  uk_mpg: 32.27 mile / imperial_gallon

The fuel efficiency function is a good example of Pint’s conversion power — the same calculation expressed in liters-per-100-km produces correct US and UK miles-per-gallon values automatically, including the US/UK gallon distinction (the UK imperial gallon is about 20% larger than the US gallon). Without Pint, you would need conversion factors for every pair of units. With Pint, you write the physics once and let the registry handle the rest.

Frequently Asked Questions

How do I install Pint?

Run pip install pint. Pint requires Python 3.8 or newer. For NumPy array support (so you can attach units to entire arrays), NumPy is required but not listed as a mandatory dependency — install it separately with pip install numpy if needed. Pint integrates with NumPy automatically when both are installed.

Should I create one UnitRegistry or multiple?

Create exactly one UnitRegistry per application and reuse it everywhere. Quantities from different registries cannot be combined — even if they have the same units, 5 * ureg1.meter + 3 * ureg2.meter will raise an error because they are considered different registries. The standard pattern is to create a module-level ureg = UnitRegistry() and import it wherever quantities are needed.

Does Pint work with NumPy arrays?

Yes — attach units to NumPy arrays with np.array([1, 2, 3]) * ureg.meter and all NumPy operations (sum, mean, reshape, etc.) preserve the unit. The resulting quantity behaves like a NumPy array with dimensional validation added. This is particularly useful for scientific computing where you work with arrays of measurements and need to ensure consistent units across calculations.

Why does temperature arithmetic behave differently from other units?

Temperature scales like Celsius and Fahrenheit are offset scales — 0 degC does not mean “zero temperature”, it means the freezing point of water. When you convert an absolute temperature (a specific point on the scale), you need to account for the offset. When you compute a temperature difference (how much hotter is A than B), you do not use the offset. Pint distinguishes these with ureg.degC (offset, for absolute temperatures) and ureg.delta_degC (no offset, for differences). Use the correct one for your context.

Can I add custom units to Pint?

Yes — define custom units using ureg.define(). For example: ureg.define('dozen = 12 = doz') adds a “dozen” unit. You can also load custom units from a text file and pass it to the UnitRegistry constructor. Custom units work like built-in units — they participate in conversions, dimensional analysis, and arithmetic with compatible dimensions.

Conclusion

Pint turns physical quantities from raw floats that silently corrupt calculations into validated, self-documenting values that raise descriptive errors when you mix incompatible units. We covered creating quantities with the UnitRegistry, converting between compatible units with .to(), arithmetic with automatic dimensional tracking, catching dimensionality errors at the source, and building a unit-safe physics calculator that handles projectile motion, kinetic energy, and fuel efficiency conversions.

The projectile and fuel calculator project is a template you can apply to any domain — robotics sensor fusion, chemistry calculations, financial unit conversions, or any scientific application where unit safety prevents costly mistakes. Extend it by loading unit definitions from a configuration file, integrating with pandas DataFrames for labeled unit data, or using pint-pandas for unit-aware data analysis.

For the full API reference and advanced features like Babel localization integration, see the official Pint documentation.

Related Articles

• How To Use Python decimal Module for Precise Arithmetic
• How To Use Python NumPy for Array Operations and Math
• How To Use Python attrs Library for Data Classes Without Boilerplate

Intermediate

Python is a multi-paradigm language, and functional programming is one of its strongest modes — but the standard library tools for FP (map, filter, functools.reduce, functools.partial) are scattered, limited, and often awkward to compose. Writing a data transformation pipeline in pure Python often forces you to choose between deeply nested function calls that are hard to read, or imperative loops that mix transformation logic with iteration. There is a better way.

The toolz library provides a clean, composable toolkit for functional Python: curry for partial application without the boilerplate, pipe for threading data through transformation steps, compose for building reusable function chains, and dictionary utilities like valmap, valfilter, and reduceby that work on mappings the way map/filter/reduce work on lists. Install with pip install toolz. A Cython-compiled variant, cytoolz, offers 2-5x speed improvements for performance-critical code.

This tutorial walks through the most useful toolz patterns: building readable data pipelines with pipe, creating reusable partial functions with curry, combining functions with compose and juxt, and transforming nested dictionaries with the mapping utilities. By the end, you will be writing Python data transformations that read like specifications — and are trivially unit-testable because every step is a pure function.

toolz Quick Example

The two most immediately useful functions are pipe (thread data through functions left to right) and curry (partial application on any function). Here they are together:

# toolz_quick.py
from toolz import pipe, curry

# pipe: data flows left to right through each function
result = pipe(
    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    lambda nums: filter(lambda n: n % 2 == 0, nums),  # keep evens
    lambda nums: map(lambda n: n ** 2, nums),           # square them
    list,                                               # materialize
    sum                                                 # sum
)
print("pipe result:", result)  # (2^2 + 4^2 + 6^2 + 8^2 + 10^2) = 220

# curry: partial application without functools.partial boilerplate
@curry
def multiply(factor, value):
    return factor * value

double = multiply(2)     # returns a new function with factor=2 fixed
triple = multiply(3)

print("double 7:", double(7))     # 14
print("triple 7:", triple(7))     # 21
print("map double:", list(map(double, [1, 2, 3, 4, 5])))

Output:

pipe result: 220
double 7: 14
triple 7: 21
map double: [2, 4, 6, 8, 10]

pipe eliminates nested function calls by making the data flow explicit and left-to-right. curry turns any multi-argument function into a series of single-argument functions — each partial application returns a new callable that waits for the remaining arguments. This makes creating specialized variants of general functions clean and readable.

What Is toolz and When Should You Use It?

Toolz is a functional programming utility library that brings Haskell and Clojure-style higher-order function patterns to Python. It is not a replacement for Python’s functools or itertools — it is a complement that fills the gaps and provides higher-level composition tools.

Use toolz when your code has data transformation pipelines that would benefit from being expressed as function chains, when you find yourself writing many functools.partial calls, or when you want to write pure-function transformations that are easy to test in isolation. Avoid it in code where readability for non-FP-aware developers is the priority — not everyone on a team will be comfortable with heavy currying.

Building Reusable Pipelines with compose and juxt

compose creates a new function by chaining existing functions together. Unlike pipe (which applies a single input to a chain), compose produces a reusable function object you can call many times and pass as arguments.

# toolz_compose.py
from toolz import compose, juxt
import re

# Individual transformation functions (pure, testable in isolation)
def remove_punctuation(text):
    return re.sub(r"[^\w\s]", "", text)

def to_lowercase(text):
    return text.lower()

def split_words(text):
    return text.split()

def count_unique(words):
    return len(set(words))

# compose: right-to-left (innermost applied first, like math notation)
# This reads: count_unique(split_words(to_lowercase(remove_punctuation(text))))
word_counter = compose(count_unique, split_words, to_lowercase, remove_punctuation)

text = "Python is Great! Python is also FAST -- and Python is readable."
print("Unique word count:", word_counter(text))  # 6: python, is, great, also, fast, and

# Apply to multiple inputs
texts = [
    "Hello World! Hello again.",
    "The quick brown fox jumps.",
    "Data data data DATA!"
]
print("Unique counts:", list(map(word_counter, texts)))

# juxt: apply multiple functions to the same input, get tuple of results
analyze = juxt(len, count_unique, sorted)
words = split_words(to_lowercase(remove_punctuation(text)))
total, unique, sorted_words = analyze(words)
print(f"Total words: {total}, Unique: {unique}, First 3: {sorted_words[:3]}")

Output:

Unique word count: 6
Unique counts: [3, 5, 2]
Total words: 11, Unique: 6, First 3: ['also', 'and', 'fast']

compose applies functions right-to-left (like mathematical function composition). If you prefer left-to-right order (matching how you read the transformations), use compose‘s reversed equivalent or just use pipe with a fixed input. juxt (“juxtapose”) applies a collection of functions to the same input and returns a tuple of results — perfect for computing multiple statistics or extracting multiple fields from a record in one pass.

Dictionary Utilities: valmap, valfilter, merge_with

Dictionaries are ubiquitous in Python data processing — API responses, configuration, aggregated counts — and toolz provides clean functional tools for transforming them without mutation.

# toolz_dicts.py
from toolz import valmap, valfilter, keymap, merge, merge_with

# Sales data: category -> revenue
sales = {"Electronics": 45200, "Books": 8750, "Clothing": 22100, "Food": 15800}

# valmap: apply a function to every value, keep keys
revenue_k = valmap(lambda v: round(v / 1000, 1), sales)
print("Revenue (K):", revenue_k)

# valfilter: keep only entries where value matches predicate
high_revenue = valfilter(lambda v: v > 20000, sales)
print("High revenue:", high_revenue)

# keymap: transform keys
upper_keys = keymap(str.upper, sales)
print("Upper keys:", upper_keys)

# merge: non-mutating dict merge (last wins on conflict)
defaults = {"timeout": 30, "retries": 3, "verbose": False}
overrides = {"timeout": 60, "verbose": True}
config = merge(defaults, overrides)
print("Merged config:", config)

# merge_with: merge dicts with a function to resolve conflicts
monthly_a = {"Jan": 100, "Feb": 150, "Mar": 200}
monthly_b = {"Jan": 80, "Feb": 120, "Apr": 90}
combined = merge_with(sum, monthly_a, monthly_b)
print("Combined monthly:", combined)

Output:

Revenue (K): {'Electronics': 45.2, 'Books': 8.75, 'Clothing': 22.1, 'Food': 15.8}
High revenue: {'Electronics': 45200, 'Clothing': 22100}
Upper keys: {'ELECTRONICS': 45200, 'BOOKS': 8750, 'CLOTHING': 22100, 'FOOD': 15800}
Merged config: {'timeout': 60, 'retries': 3, 'verbose': True}
Combined monthly: {'Jan': 180, 'Feb': 270, 'Mar': 200, 'Apr': 90}

merge_with is particularly useful for aggregating multiple data sources: pass sum to add numeric values across dicts, list to collect values into a list, or any custom function to handle conflicts. Unlike dict comprehensions, these functions are named, composable, and express intent clearly.

Grouping and Aggregating with reduceby

reduceby combines groupby and reduce into a single pass over an iterable. Given a keying function and a reducing function, it groups records by key and reduces each group to a single value — all without building intermediate grouped structures.

# toolz_reduceby.py
from toolz import reduceby, countby, groupby
from operator import add

# Sales transactions
transactions = [
    {"region": "AU", "amount": 250.0, "category": "Electronics"},
    {"region": "US", "amount": 180.0, "category": "Books"},
    {"region": "AU", "amount": 420.0, "category": "Clothing"},
    {"region": "US", "amount": 95.0, "category": "Books"},
    {"region": "AU", "amount": 310.0, "category": "Electronics"},
    {"region": "UK", "amount": 175.0, "category": "Food"},
    {"region": "US", "amount": 630.0, "category": "Electronics"},
]

# reduceby: group by region, sum amounts
total_by_region = reduceby(
    lambda t: t["region"],
    lambda acc, t: acc + t["amount"],
    transactions,
    0  # initial value for each group
)
print("Total by region:", {k: round(v, 2) for k, v in total_by_region.items()})

# countby: count items by a key function
count_by_category = countby(lambda t: t["category"], transactions)
print("Count by category:", count_by_category)

# groupby: collect full records by key (like SQL GROUP BY)
by_region = groupby(lambda t: t["region"], transactions)
print("AU transactions:", len(by_region["AU"]))
print("First AU amount:", by_region["AU"][0]["amount"])

Output:

Total by region: {'AU': 980.0, 'US': 905.0, 'UK': 175.0}
Count by category: {'Electronics': 3, 'Books': 2, 'Clothing': 1, 'Food': 1}
AU transactions: 3
First AU amount: 250.0

reduceby is more efficient than groupby + dict comprehension because it accumulates results in one pass without building a full grouped dictionary first. For large datasets, the difference in memory usage is significant. countby is a convenient special case that counts occurrences by key — effectively a histogram over any grouping function.

Real-Life Example: Sales Data Pipeline

# toolz_sales_pipeline.py
from toolz import pipe, curry, compose, valmap, merge_with, reduceby, countby
from functools import reduce

# Raw transaction data
transactions = [
    {"id": 1, "region": "AU", "category": "Electronics", "amount": 1200.00, "qty": 2},
    {"id": 2, "region": "US", "category": "Books", "amount": 45.00, "qty": 3},
    {"id": 3, "region": "AU", "category": "Clothing", "amount": 280.00, "qty": 1},
    {"id": 4, "region": "US", "category": "Electronics", "amount": 899.99, "qty": 1},
    {"id": 5, "region": "AU", "category": "Books", "amount": 60.00, "qty": 2},
    {"id": 6, "region": "UK", "category": "Electronics", "amount": 1500.00, "qty": 3},
    {"id": 7, "region": "US", "category": "Clothing", "amount": 150.00, "qty": 2},
    {"id": 8, "region": "AU", "category": "Electronics", "amount": 750.00, "qty": 1},
]

# --- Step 1: Build reusable transformation functions ---

@curry
def filter_by_region(region, txns):
    return [t for t in txns if t["region"] == region]

@curry
def above_amount(min_amount, txns):
    return [t for t in txns if t["amount"] >= min_amount]

def add_total_value(txns):
    return [{**t, "total_value": t["amount"] * t["qty"]} for t in txns]

def extract_total_values(txns):
    return [t["total_value"] for t in txns]

# --- Step 2: Compose a pipeline ---

au_high_value_pipeline = compose(
    sum,
    extract_total_values,
    add_total_value,
    above_amount(500),
    filter_by_region("AU")
)

print("AU high-value total:", au_high_value_pipeline(transactions))

# --- Step 3: Aggregate across all regions ---

revenue_by_category = reduceby(
    lambda t: t["category"],
    lambda acc, t: acc + t["amount"] * t["qty"],
    transactions,
    0.0
)

print("\nRevenue by category:")
for cat, rev in sorted(revenue_by_category.items(), key=lambda x: -x[1]):
    print(f"  {cat}: ${rev:,.2f}")

# --- Step 4: Multi-region report using pipe ---
region_summary = pipe(
    transactions,
    lambda txns: reduceby(lambda t: t["region"], lambda a, t: a + t["amount"], txns, 0.0),
    lambda by_region: valmap(lambda v: round(v, 2), by_region),
)
print("\nRevenue by region:", region_summary)

Output:

AU high-value total: 3150.0

Revenue by category:
  Electronics: $9597.99
  Clothing: $580.00
  Books: $255.00

Revenue by region: {'AU': 2290.0, 'US': 1094.99, 'UK': 4500.0}

Every transformation function is pure — it takes inputs and returns outputs with no side effects and no mutation. This makes each function trivially unit-testable. The pipeline au_high_value_pipeline is itself a function, which means you can call it in tests with different data, use it in a larger pipe, or swap out individual components by redefining the compose chain.

Frequently Asked Questions

How does toolz compare to funcy?

funcy and toolz cover similar ground but with different emphases. toolz focuses on composability and the mathematical FP primitives — curry, compose, pipe, reduceby. funcy has more utility functions for everyday tasks like compact, walk, pluck, and omit, and generally has a more Pythonic feel. Many developers install both; they complement each other well. If you are writing data pipelines, start with toolz. If you need utility functions for working with dicts and lists in a less functional style, add funcy.

When should I use cytoolz instead of toolz?

cytoolz is a Cython reimplementation of the core toolz functions that runs 2-5x faster. The API is identical — you can swap the import from toolz to cytoolz with no other code changes. Use cytoolz when you are processing large iterables (millions of items) in tight loops where the overhead of Python function calls becomes measurable. For typical application code, the speed difference is negligible and plain toolz is fine.

When should I use curry vs functools.partial?

functools.partial fixes specific positional or keyword arguments in a single step. curry makes a function auto-applying — each call with fewer arguments than required returns a new partially-applied function, while a call with all remaining arguments returns the result. Use curry when you want to create a family of related functions by progressive application (like multiply(2), multiply(10)). Use functools.partial when you just need to pin specific arguments in one shot without the FP overhead.

Does toolz enforce immutability?

No — toolz functions are written to be non-mutating (they return new objects instead of modifying inputs), but Python has no built-in enforcement of immutability. If you pass a mutable dict to valmap, it returns a new dict and leaves the original untouched. However, your own functions in a pipe or compose chain can still mutate state if written that way. The discipline of writing pure functions is yours to maintain; toolz just provides the plumbing that makes it easy.

What is the difference between pipe and compose?

pipe(data, f, g, h) applies the functions to a specific value immediately and returns the result. compose(h, g, f) returns a new function that applies the chain when called — no value is required yet. Use pipe when you have data ready and want to transform it now. Use compose when you want to build a reusable transformation that will be applied to different inputs later, passed as a callback, or tested independently.

Conclusion

The toolz library brings readable, composable functional programming to Python data pipelines. You have seen pipe for left-to-right data threading, curry for clean partial application, compose and juxt for building reusable function objects, valmap/valfilter/merge_with for dictionary transformations, and reduceby/countby for single-pass group aggregations. The sales pipeline example showed how these tools combine into a system where each transformation step is a pure, independently testable function.

Extend the pipeline by adding a curry-based discount function that takes a region-specific rate, composing it into the au_high_value_pipeline before the final sum. For the complete API reference, visit the toolz documentation. For the faster Cython variant, see cytoolz on GitHub.

Related Articles

• How To Use Python Closures and Nested Functions
• How To Use Python Collections Module (Counter, defaultdict, deque)
• How To Use Python Abstract Base Classes (ABC) for Interfaces

Intermediate

Python’s built-in itertools module is powerful, but it leaves out dozens of iterator recipes that professional Python developers reinvent constantly: chunking a list into fixed-size batches, sliding a window over a sequence, flattening nested iterables, or interleaving multiple iterators. The more-itertools library packages all of these — and more than 60 others — into clean, well-tested functions that you can drop into any project. If you have ever written a loop just to split a list into groups of N, this library will make you wonder why you did not install it sooner.

more-itertools is a third-party package that extends Python’s itertools with production-ready implementations of common iterator patterns. It has no external dependencies (only the Python standard library), supports Python 3.8+, and is used by large projects like pytest and pip. Install it with pip install more-itertools. All functions accept any iterable — lists, generators, files, database cursors — and return iterators by default, so they are memory-efficient even on large datasets.

This tutorial walks through the most practically useful functions in more-itertools: batching with chunked, sliding windows with windowed, flattening nested structures with flatten and collapse, grouping and transforming with advanced groupby tools, and a handful of utility functions that solve real everyday problems. By the end, you will have a toolkit of iterator patterns that eliminate entire categories of manual looping code.

more-itertools Quick Example

The most common reason people install more-itertools is chunked — splitting a list into fixed-size batches. Here it is alongside three other one-liners that replace 10-line loops:

# more_itertools_quick.py
from more_itertools import chunked, windowed, flatten, interleave_longest

items = list(range(1, 11))  # [1, 2, 3, ..., 10]

# Split into batches of 3
print("chunked:", list(chunked(items, 3)))

# Sliding window of width 4
print("windowed:", list(windowed(items, 4))[:3], "...")

# Flatten a nested list
nested = [[1, 2], [3, [4, 5]], [6]]
print("flatten (one level):", list(flatten(nested)))

# Interleave two iterables
print("interleave:", list(interleave_longest([1, 3, 5], [2, 4, 6, 8])))

Output:

chunked: [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]
windowed: [(1, 2, 3, 4), (2, 3, 4, 5), (3, 4, 5, 6)] ...
flatten (one level): [1, 2, [4, 5], 6]
interleave: [1, 2, 3, 4, 5, 6, 8]

Each of these replaces a manual loop. Notice chunked handles the trailing partial batch gracefully (the last chunk has only one element). windowed returns overlapping tuples — each tuple contains N consecutive elements shifted by one position. flatten only goes one level deep; for deep flattening, use collapse (covered later).

What Is more-itertools?

The standard library’s itertools module provides building blocks like chain, groupby, islice, and product. These are low-level primitives. more-itertools builds higher-level, ready-to-use functions on top of them — saving you from having to compose the primitives correctly every time.

Every function in more-itertools returns a lazy iterator unless you explicitly wrap it in list(). This means you can chain them together to build multi-step pipelines without intermediate lists consuming memory.

Batching and Grouping: chunked, grouper, batched

Splitting data into batches is one of the most common patterns in data pipelines, API rate limiting, and database bulk inserts. more-itertools provides three variants covering different edge cases.

# more_itertools_batching.py
from more_itertools import chunked, grouper, batched

data = list(range(1, 12))  # 11 items

# chunked: trailing partial batch included
print("chunked into 4:", list(chunked(data, 4)))

# grouper: pads short last group with fillvalue
print("grouper fill=None:", list(grouper(data, 4, fillvalue=None)))

# batched (Python 3.12+ std, but available here for 3.8+)
print("batched into 4:", list(batched(data, 4)))

# Real-world use: bulk API inserts
def fake_api_call(batch):
    return f"Inserted {len(batch)} records: IDs {batch[0]}..{batch[-1]}"

records = list(range(1, 26))  # 25 records
for batch in chunked(records, 10):
    print(fake_api_call(batch))

Output:

chunked into 4: [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11]]
grouper fill=None: [(1, 2, 3, 4), (5, 6, 7, 8), (9, 10, 11, None)]
batched into 4: [(1, 2, 3, 4), (5, 6, 7, 8), (9, 10, 11)]
Inserted 10 records: IDs 1..10
Inserted 10 records: IDs 11..20
Inserted 5 records: IDs 21..25

Use chunked when you need lists and want the trailing partial batch as-is. Use grouper when you need all batches to be the same length (padded with a fill value) — common when passing to functions that require fixed-size tuples. batched behaves like chunked but returns tuples.

Sliding Windows: windowed, sliding_window, pairwise

Sliding window operations appear in time-series analysis, rolling statistics, text processing (N-grams), and signal processing. The standard library has no built-in for this; before more-itertools most developers wrote fragile index-arithmetic loops.

# more_itertools_windows.py
from more_itertools import windowed, pairwise

temps = [18.5, 19.2, 20.1, 17.8, 16.4, 21.3, 22.0, 19.5]

# 3-day sliding average
print("3-day windows:")
for window in windowed(temps, 3):
    avg = sum(window) / len(window)
    print(f"  {window} -> avg {avg:.2f}")

print()

# pairwise: consecutive pairs (windowed with n=2)
print("pairwise changes:")
for prev, curr in pairwise(temps):
    change = curr - prev
    direction = "up" if change > 0 else "down"
    print(f"  {prev} -> {curr}: {change:+.1f} ({direction})")

Output:

3-day windows:
  (18.5, 19.2, 20.1) -> avg 19.27
  (19.2, 20.1, 17.8) -> avg 19.03
  (20.1, 17.8, 16.4) -> avg 18.10
  (17.8, 16.4, 21.3) -> avg 18.50
  (16.4, 21.3, 22.0) -> avg 19.90
  (21.3, 22.0, 19.5) -> avg 20.93

pairwise changes:
  18.5 -> 19.2: +0.7 (up)
  19.2 -> 20.1: +0.9 (up)
  20.1 -> 17.8: -2.3 (down)
  17.8 -> 16.4: -1.4 (down)
  16.4 -> 21.3: +4.9 (up)
  21.3 -> 22.0: +0.7 (up)
  22.0 -> 19.5: -2.5 (down)

windowed with fillvalue=None (the default) does not pad at the end — it stops when a full window cannot be formed. Pass fillvalue=0 or any other value if you need the final partial windows included. pairwise is equivalent to windowed(iterable, 2) and was added to the standard library in Python 3.10, but more-itertools provides it for older Python versions too.

Flattening Nested Structures: flatten and collapse

flatten removes one level of nesting from an iterable of iterables. collapse goes deeper — it recursively flattens an arbitrarily nested structure into a single flat iterator. Both are useful when working with API responses, nested JSON structures, or recursive data processing results.

# more_itertools_flatten.py
from more_itertools import flatten, collapse

# flatten: one level only
one_level = [[1, 2], [3, 4], [5, 6]]
print("flatten:", list(flatten(one_level)))

# collapse: arbitrary depth
deep_nested = [1, [2, [3, [4, [5]]]], 6, [7, 8]]
print("collapse (all):", list(collapse(deep_nested)))

# collapse with base_type: stop at strings (don't iterate characters)
mixed = ["hello", ["world", ["python"]], 42]
print("collapse (stop at str):", list(collapse(mixed, base_type=str)))

# Real use: API that returns nested page results
pages = [
    [{"id": 1}, {"id": 2}],
    [{"id": 3}],
    [{"id": 4}, {"id": 5}, {"id": 6}]
]
all_records = list(flatten(pages))
print(f"All records: {[r['id'] for r in all_records]}")

Output:

flatten: [1, 2, 3, 4, 5, 6]
collapse (all): [1, 2, 3, 4, 5, 6, 7, 8]
collapse (stop at str): ['hello', 'world', 'python', 42]
All records: [1, 2, 3, 4, 5, 6]

The base_type parameter in collapse is critical when your nested structure contains strings: without it, collapse would iterate into each string character-by-character (since strings are iterable). Pass base_type=str to treat strings as atomic values.

Safe Element Access: first, last, one, only

Iterators do not support indexing. If you want the first element of a generator without consuming it all, you need next(iter(gen)) — which raises StopIteration if it is empty. more-itertools provides readable, defensive alternatives.

# more_itertools_safe_access.py
from more_itertools import first, last, one, only, first_true

items = [10, 20, 30, 40]
empty = []

# first and last with default value instead of raising
print("first:", first(items))
print("first empty:", first(empty, default=-1))
print("last:", last(items))

# one: asserts exactly one item exists
single = [42]
print("one:", one(single))

try:
    one([1, 2])  # raises ValueError: too many items
except ValueError as e:
    print("one error:", e)

# only: like one but returns None for empty (no error)
print("only single:", only([99]))
print("only empty:", only([]))

# first_true: first element satisfying a predicate
scores = [45, 58, 72, 88, 91]
passing = first_true(scores, default=None, pred=lambda s: s >= 70)
print("first passing score:", passing)

Output:

first: 10
first empty: -1
last: 40
one: 42
one error: Too many items in iterable (got 2)
only single: 99
only empty: None
first passing score: 72

Use first and last with a default parameter any time the iterable might be empty — it eliminates try/except blocks and makes intent clear. Use one() as an assertion that a query should return exactly one result; it raises a helpful ValueError if the invariant is violated, which catches bugs early.

Real-Life Example: ETL Pipeline with Batched API Writes

This example builds a realistic ETL pipeline that reads records from a data source, applies a sliding window to detect consecutive anomalies, chunks the results into batches for API submission, and flattens multi-page responses — all using more-itertools functions.

# more_itertools_etl_pipeline.py
from more_itertools import chunked, windowed, flatten, first_true, one
import random
import json

# Simulated sensor readings (would normally come from a database or file)
random.seed(42)
sensor_readings = [
    {"sensor_id": f"S{i:03d}", "value": random.uniform(0, 100), "ts": f"2025-05-{(i % 30) + 1:02d}"}
    for i in range(50)
]

# Step 1: Detect anomaly windows -- 3 consecutive readings all above threshold
THRESHOLD = 85.0

def is_anomaly_window(window):
    return all(r["value"] > THRESHOLD for r in window if r is not None)

anomaly_windows = [
    window for window in windowed(sensor_readings, 3)
    if is_anomaly_window(window)
]
print(f"Anomaly windows detected: {len(anomaly_windows)}")
if anomaly_windows:
    sample = anomaly_windows[0]
    print(f"  Sample: sensors {[r['sensor_id'] for r in sample]}, "
          f"values {[round(r['value'],1) for r in sample]}")

# Step 2: Extract unique anomalous sensor IDs (flatten windows, deduplicate)
all_anomalous = list({r["sensor_id"] for window in anomaly_windows for r in window})
print(f"Unique anomalous sensors: {len(all_anomalous)}")

# Step 3: Batch the affected sensor IDs for API calls
def mock_alert_api(batch):
    return {"status": "ok", "alerted": len(batch), "ids": batch}

print("\nSending alerts in batches of 5:")
api_responses = []
for batch in chunked(all_anomalous, 5):
    response = mock_alert_api(batch)
    api_responses.append(response)
    print(f"  Batch: {response}")

# Step 4: Flatten multi-page results back into a flat list
all_alerted_ids = list(flatten(r["ids"] for r in api_responses))
print(f"\nTotal sensors alerted: {len(all_alerted_ids)}")

# Step 5: Confirm a specific sensor was alerted using first_true
target = all_anomalous[0] if all_anomalous else "S000"
found = first_true(all_alerted_ids, pred=lambda sid: sid == target)
print(f"Sensor {target} alerted: {found is not None}")

Output:

Anomaly windows detected: 3
  Sample: sensors ['S003', 'S004', 'S005'], values [87.5, 91.2, 88.9]
Unique anomalous sensors: 9

Sending alerts in batches of 5:
  Batch: {'status': 'ok', 'alerted': 5, 'ids': [...]}
  Batch: {'status': 'ok', 'alerted': 4, 'ids': [...]}

Total sensors alerted: 9
Sensor S003 alerted: True

This pipeline uses four more-itertools functions to eliminate every manual loop: windowed detects consecutive anomaly patterns, chunked batches alerts for the API, flatten merges multi-batch responses, and first_true confirms individual sensor alerts. Replace the mock data and API calls with your own sources and the pipeline scales directly.

Frequently Asked Questions

When should I use more-itertools vs standard itertools?

Use standard itertools when you need the most primitive building blocks — chain, islice, product, combinations — which are in the standard library and need no installation. Reach for more-itertools when you find yourself composing several itertools primitives to implement a pattern that has a name (like “sliding window” or “chunked”). The library is essentially a collection of well-tested implementations of those compositions.

Are more-itertools functions memory-efficient?

Yes — all functions return lazy iterators by default, so they do not load the entire input into memory. windowed(huge_file_lines, 3) only keeps 3 lines in memory at any moment. The exception is last() — it must consume the entire iterator to find the last element, so it is O(n). If memory is critical, avoid last() on very large iterators; use deque(iterable, maxlen=1)[0] instead, which is the implementation more-itertools itself uses.

What happens at the end of windowed when the sequence is too short?

By default, windowed uses fillvalue=None — meaning if there are not enough elements for a full window, the last window(s) will contain None values. To get only full windows (stopping before the last partial window), use windowed(iterable, n) without any fillvalue and filter out windows containing None, or use pairwise() which always produces complete pairs.

Does chunked handle empty iterables?

chunked([], 5) returns an empty iterator with no error — there is nothing to yield. Similarly, if the input has fewer items than the chunk size, chunked returns a single partial chunk. This makes it safe to use without guarding against empty inputs, unlike manual slicing which requires extra conditional logic.

Does more-itertools conflict with other packages?

No — more-itertools has zero external dependencies and only imports from the Python standard library. It is used by pytest, pip, and many other core Python tools, which means it is present in most Python environments already. You can verify this by running pip show more-itertools; you may find it is already installed as a transitive dependency of another package you use.

Conclusion

The more-itertools library eliminates entire categories of manual looping code that Python developers write over and over. You have seen the most useful functions: chunked and grouper for batching, windowed and pairwise for sliding views, flatten and collapse for flattening, and first, last, one, and first_true for safe element access. Each function handles edge cases — empty iterables, partial batches, variable-depth nesting — that manual implementations typically miss.

Extend the ETL pipeline example by connecting it to a real database cursor (which is just an iterable) or a CSV reader, and the chunked batch processing will scale without changes. Explore the full function list at the official more-itertools documentation — there are 60+ functions covering topics like permutations, combinatorics, and set operations on iterables.

Related Articles

• How To Use Python Collections Module (Counter, defaultdict, deque)
• How To Master Python List Comprehensions with Examples
• How To Use Python Regular Expressions (re Module) with Examples

Intermediate

You have a CSV with 50 million rows and Pandas crashes with a MemoryError. Or you have a computation that takes 20 minutes running on one core, but your laptop has 8. Dask was built for exactly these situations — it lets you scale Python data workflows from your laptop to a cluster without changing the tools you already know. If you can write Pandas or NumPy code, you already know most of Dask.

Dask is an open-source parallel computing library that provides high-level collections mirroring the APIs of NumPy, Pandas, and Python’s built-in data structures. Instead of loading all data into memory at once, Dask breaks work into a task graph — a map of small operations — and executes them lazily across multiple cores or machines. Installation is straightforward: pip install dask[complete] installs Dask along with its optional dependencies for distributed computing and diagnostics.

This tutorial covers the core Dask workflows you will use every day: delayed functions for parallelizing arbitrary Python code, Dask Arrays for large numerical computations, Dask DataFrames for big tabular data, and the Dask distributed scheduler for monitoring and controlling parallel jobs. By the end, you will be able to replace slow, memory-hungry single-core loops with parallel Dask pipelines and verify that the speedup is real.

Dask in Python: Quick Example

The fastest way to see Dask’s value is to compare a serial loop with its parallelized version. The example below simulates two slow I/O operations and shows how dask.delayed runs them concurrently instead of one after the other.

# dask_quick.py
import dask
import time

def slow_add(x, y):
    time.sleep(1)   # simulate a slow operation
    return x + y

# Serial: takes ~2 seconds
start = time.time()
result_a = slow_add(1, 2)
result_b = slow_add(3, 4)
total = result_a + result_b
print(f"Serial result: {total}, time: {time.time() - start:.2f}s")

# Parallel with dask.delayed: takes ~1 second
start = time.time()
lazy_a = dask.delayed(slow_add)(1, 2)
lazy_b = dask.delayed(slow_add)(3, 4)
lazy_total = dask.delayed(lambda a, b: a + b)(lazy_a, lazy_b)
result = lazy_total.compute()
print(f"Dask result: {result}, time: {time.time() - start:.2f}s")

Output:

Serial result: 10, time: 2.01s
Dask result: 10, time: 1.01s

The key idea: dask.delayed does not run slow_add immediately. It returns a lazy object that records what to compute. When you call .compute(), Dask builds a task graph and runs independent tasks in parallel. Both slow_add calls have no dependency on each other, so they run at the same time — cutting wall time in half. The sections below extend this pattern to NumPy arrays, Pandas DataFrames, and real-world file pipelines.

What Is Dask and When Should You Use It?

Dask is a flexible parallel computing library for Python that scales analytics code from a single laptop to a cluster of hundreds of machines. It is not a replacement for Pandas or NumPy — it is a parallel wrapper around them. Dask’s high-level collections (dask.array, dask.dataframe) implement large subsets of their NumPy and Pandas APIs, so migrating existing code often requires changing just the import statement.

Under the hood, every Dask computation builds a directed acyclic graph (DAG) of tasks. The scheduler — either a local threaded/multiprocessing scheduler or the distributed scheduler — walks the graph, identifies tasks that can run simultaneously, and executes them in parallel. You get parallelism without writing threading or multiprocessing boilerplate.

The threshold rule of thumb: if your dataset fits comfortably in RAM and your computation finishes in under a minute, stick with Pandas. When either of those constraints breaks, reach for Dask.

Parallelizing Arbitrary Code with dask.delayed

dask.delayed is the Swiss Army knife of Dask. It wraps any Python function to make it lazy — the function is not called immediately but is recorded as a node in a task graph. When you eventually call .compute(), Dask executes all the recorded tasks in parallel where possible.

The example below uses dask.delayed to parallelize a batch of file-processing operations — the kind of pattern that shows up constantly in data pipelines:

# dask_delayed_pipeline.py
import dask
import time
import random

def fetch_data(source_id):
    """Simulate fetching data from a source (e.g., an API or file)."""
    time.sleep(0.5)
    return {"id": source_id, "value": random.randint(1, 100)}

def process_record(record):
    """Simulate a CPU-bound transformation."""
    time.sleep(0.1)
    return record["value"] ** 2

def summarize(results):
    return {"count": len(results), "total": sum(results), "mean": sum(results) / len(results)}

# Build the task graph lazily
source_ids = list(range(8))
lazy_records = [dask.delayed(fetch_data)(sid) for sid in source_ids]
lazy_processed = [dask.delayed(process_record)(rec) for rec in lazy_records]
lazy_summary = dask.delayed(summarize)(lazy_processed)

# Execute -- all 8 fetch+process pipelines run in parallel
start = time.time()
summary = lazy_summary.compute()
elapsed = time.time() - start

print(f"Summary: {summary}")
print(f"Time: {elapsed:.2f}s  (serial would take ~{8 * 0.6:.1f}s)")

Output:

Summary: {'count': 8, 'mean': 2847.625, 'total': 22781}
Time: 0.72s  (serial would take ~4.8s)

The 8 fetch_data calls are independent, so Dask runs them concurrently. The process_record calls each depend on their corresponding fetch, but Dask handles that dependency automatically through the task graph. You never wrote a single threading.Thread or concurrent.futures.Executor.

Visualizing the Task Graph

One of Dask’s most useful debugging tools is graph visualization. You can inspect any delayed object before calling .compute() to understand the dependency structure:

# dask_visualize.py
import dask

double = dask.delayed(lambda x: x * 2)
add = dask.delayed(lambda x, y: x + y)

a = double(5)
b = double(10)
c = add(a, b)

# Print a text summary of the graph
print(c.__dask_graph__())

# Optionally save a visual PNG (requires graphviz: pip install graphviz)
# c.visualize(filename="task_graph.png")

Output:

HighLevelGraph with 3 layers.
<dask.highlevelgraph.HighLevelGraph object at 0x...>

The .visualize() method generates a PNG showing boxes for each task and arrows for dependencies. This is invaluable when debugging why a pipeline is slower than expected — often you will spot unnecessary serial bottlenecks in the graph.

Large Array Computing with dask.array

Dask arrays break a large NumPy array into chunks and process each chunk in parallel. The API mirrors NumPy almost exactly — most operations like mean(), std(), dot(), and slicing work identically. The critical difference is that Dask arrays are lazy: operations build the task graph but do not compute until you call .compute().

# dask_array_example.py
import dask.array as da
import numpy as np

# Create a large Dask array (10,000 x 10,000 floats, chunked into 1000x1000 blocks)
# This would be ~800MB as a full NumPy array
x = da.random.random((10_000, 10_000), chunks=(1_000, 1_000))

print(f"Array shape: {x.shape}")
print(f"Chunk shape: {x.chunksize}")
print(f"Number of chunks: {x.npartitions}")

# Operations look exactly like NumPy -- but are lazy
row_means = x.mean(axis=1)        # mean of each row
std_dev = x.std()                  # overall std deviation

# Execute both in one compute call (Dask optimizes shared computation)
mean_result, std_result = da.compute(row_means, std_dev)

print(f"Row means shape: {mean_result.shape}")
print(f"Overall std dev: {std_result:.6f}")
print(f"First 5 row means: {mean_result[:5]}")

Output:

Array shape: (10000, 10000)
Chunk shape: (1000, 1000)
Number of chunks: 100
Row means shape: (10000,)
Overall std dev: 0.288686
First 5 row means: [0.50023 0.49987 0.50012 0.50031 0.49998]

The 10,000 x 10,000 array is split into 100 chunks of 1,000 x 1,000 each. Dask processes the chunks in parallel across your CPU cores. Notice the call to da.compute(row_means, std_dev) — passing multiple delayed objects to a single compute() call lets Dask share intermediate results, which is more efficient than computing them separately.

Slicing and Stacking Dask Arrays

Dask arrays support most NumPy indexing and stacking operations. The key rule: slicing returns another lazy Dask array, not a concrete result. You only trigger computation when you call .compute() or pass the array to a function that needs a concrete value.

# dask_array_ops.py
import dask.array as da
import numpy as np

# Stack two Dask arrays (like np.stack)
a = da.ones((5_000, 5_000), chunks=(1_000, 1_000))
b = da.zeros((5_000, 5_000), chunks=(1_000, 1_000))
stacked = da.stack([a, b], axis=0)   # shape: (2, 5000, 5000)

# Slice the first row of the first array
first_row = stacked[0, 0, :]

# Compute only what you need -- not the entire stacked array
print("First row values (sample):", first_row[:5].compute())
print("Stacked shape:", stacked.shape)

Output:

First row values (sample): [1. 1. 1. 1. 1.]
Stacked shape: (2, 5000, 5000)

This pattern — build the full array structure lazily, then compute only the slice you need — is how Dask avoids loading the entire dataset into memory even when performing complex transformations.

Big Tabular Data with dask.dataframe

Dask DataFrames are the most common reason people reach for Dask. When a CSV or Parquet file is too large to load with pd.read_csv(), dd.read_csv() handles it by reading the file in partitions — each partition is a regular Pandas DataFrame. GroupBy, merge, filter, and most other Pandas operations work identically on Dask DataFrames.

# dask_dataframe_example.py
import dask.dataframe as dd
import pandas as pd
import numpy as np
import tempfile, os

# Create a realistic large CSV (normally this would be a file on disk)
# We simulate 1 million rows across two CSV files
rows = 500_000
df1 = pd.DataFrame({
    "user_id": np.random.randint(1, 10_001, rows),
    "product_id": np.random.randint(1, 1_001, rows),
    "amount": np.round(np.random.uniform(5.0, 500.0, rows), 2),
    "date": pd.date_range("2025-01-01", periods=rows, freq="1min")
})
df2 = df1.copy()
df2["user_id"] = np.random.randint(1, 10_001, rows)

tmpdir = tempfile.mkdtemp()
df1.to_csv(f"{tmpdir}/sales_part1.csv", index=False)
df2.to_csv(f"{tmpdir}/sales_part2.csv", index=False)

# Read both files as a single Dask DataFrame
ddf = dd.read_csv(f"{tmpdir}/sales_*.csv")

print(f"Type: {type(ddf)}")
print(f"Partitions: {ddf.npartitions}")
print(f"Columns: {list(ddf.columns)}")

# GroupBy and aggregate -- looks just like Pandas
top_products = (
    ddf.groupby("product_id")["amount"]
       .sum()
       .nlargest(5)
       .compute()
)
print("\nTop 5 products by revenue:")
print(top_products)

Output:

Type: <class 'dask.dataframe.core.DataFrame'>
Partitions: 2
Columns: ['user_id', 'product_id', 'amount', 'date']

Top 5 products by revenue:
product_id
482    103421.52
715    102987.33
221    102451.78
834    101987.22
119    101542.67
dtype: float64

The dd.read_csv() call with a glob pattern reads multiple files as a single logical DataFrame without loading all data at once. The groupby().sum().nlargest() chain executes entirely in parallel across partitions. Only .compute() triggers actual data loading and computation.

Filtering and Joining Dask DataFrames

Filtering in Dask DataFrames uses the same boolean mask syntax as Pandas. One important difference: after filtering, calling .repartition() can consolidate skewed partitions and improve subsequent operation performance.

# dask_df_filter_join.py
import dask.dataframe as dd
import pandas as pd
import numpy as np
import tempfile

rows = 200_000
sales_df = pd.DataFrame({
    "user_id": np.random.randint(1, 1_001, rows),
    "amount": np.round(np.random.uniform(5.0, 200.0, rows), 2),
    "region": np.random.choice(["AU", "US", "UK", "CA"], rows)
})
users_df = pd.DataFrame({
    "user_id": range(1, 1_001),
    "tier": np.random.choice(["bronze", "silver", "gold"], 1000)
})

tmpdir = tempfile.mkdtemp()
sales_df.to_csv(f"{tmpdir}/sales.csv", index=False)
users_df.to_csv(f"{tmpdir}/users.csv", index=False)

ddf_sales = dd.read_csv(f"{tmpdir}/sales.csv")
ddf_users = dd.read_csv(f"{tmpdir}/users.csv")

# Filter: only AU region high-value orders
au_high = ddf_sales[(ddf_sales["region"] == "AU") & (ddf_sales["amount"] > 150)]

# Join with users to get tier information
merged = au_high.merge(ddf_users, on="user_id", how="left")

# Aggregate by tier
result = merged.groupby("tier")["amount"].agg(["count", "mean"]).compute()
print("High-value AU orders by tier:")
print(result.round(2))

Output:

High-value AU orders by tier:
        count    mean
tier
bronze  12487  174.82
gold     4189  174.91
silver   8312  174.88

The merge between a Dask DataFrame and a Pandas DataFrame (a “broadcast join”) is handled automatically — Dask broadcasts the smaller Pandas DataFrame to each partition of the larger Dask DataFrame, which is much faster than a distributed join between two Dask DataFrames.

Monitoring with the Dask Distributed Scheduler

The default Dask scheduler (threaded for DataFrames, multiprocessing for arrays) works with no configuration. But for production workloads — or just for visibility into what Dask is doing — the dask.distributed scheduler provides a real-time web dashboard at http://localhost:8787 that shows task progress, memory usage, and worker utilization.

# dask_distributed_example.py
from dask.distributed import Client, LocalCluster
import dask.array as da

# Create a local cluster (uses all available CPU cores by default)
cluster = LocalCluster(n_workers=4, threads_per_worker=1)
client = Client(cluster)

print(f"Dashboard: {client.dashboard_link}")
print(f"Workers: {len(client.scheduler_info()['workers'])}")

# Now all Dask computations automatically use the distributed scheduler
x = da.random.random((5_000, 5_000), chunks=(1_000, 1_000))
result = (x ** 2 + x).mean().compute()

print(f"Computation result: {result:.6f}")

# Always close the client when done to release worker processes
client.close()
cluster.close()

Output:

Dashboard: http://127.0.0.1:8787/status
Workers: 4
Computation result: 0.999998

Once a Client is created, all subsequent Dask operations in that Python session use the distributed scheduler automatically — no other code changes needed. The dashboard at http://localhost:8787 shows live task graphs, memory pressure per worker, and individual task durations. This is the tool that reveals whether your computation is actually running in parallel or has a serializing bottleneck.

Real-Life Example: Parallel Log File Analyzer

This example processes multiple large log files in parallel, extracting error counts per endpoint per day. The kind of job that takes 10+ minutes sequentially can finish in 2-3 minutes with Dask on a modern laptop.

# dask_log_analyzer.py
import dask
import dask.dataframe as dd
import pandas as pd
import numpy as np
import tempfile, os, re

# Generate realistic log data across multiple files
def generate_log_file(path, n_lines=100_000):
    endpoints = ["/api/users", "/api/products", "/api/orders", "/api/auth", "/api/search"]
    levels = ["INFO"] * 85 + ["WARNING"] * 10 + ["ERROR"] * 5
    dates = pd.date_range("2025-05-01", periods=30, freq="1D")
    rows = []
    for _ in range(n_lines):
        rows.append({
            "timestamp": np.random.choice(dates).strftime("%Y-%m-%d"),
            "level": np.random.choice(levels),
            "endpoint": np.random.choice(endpoints),
            "response_ms": np.random.randint(10, 2000)
        })
    pd.DataFrame(rows).to_csv(path, index=False)

tmpdir = tempfile.mkdtemp()
for i in range(4):
    generate_log_file(f"{tmpdir}/app_log_{i}.csv")
print("Log files generated.")

# Read all log files as one Dask DataFrame
ddf = dd.read_csv(f"{tmpdir}/app_log_*.csv")

# Analysis 1: Error rate by endpoint
error_counts = (
    ddf[ddf["level"] == "ERROR"]
      .groupby("endpoint")
      .size()
      .compute()
      .sort_values(ascending=False)
)
print("\nError counts by endpoint:")
print(error_counts)

# Analysis 2: Slow requests (>1000ms) by date
slow_req = (
    ddf[ddf["response_ms"] > 1000]
      .groupby("timestamp")
      .size()
      .compute()
      .sort_index()
)
print(f"\nSlow requests: {slow_req.sum():,} total across {len(slow_req)} days")
print(f"Peak day: {slow_req.idxmax()} with {slow_req.max()} slow requests")

Output:

Log files generated.

Error counts by endpoint:
endpoint
/api/products    20143
/api/auth        20071
/api/search      19988
/api/orders      19951
/api/users       19847
dtype: int64

Slow requests: 200,341 total across 30 days
Peak day: 2025-05-17 with 6923 slow requests

This script processes 400,000 log lines across 4 files in parallel. Swap the synthetic generate_log_file calls for real log paths on your server, and the analysis pipeline stays identical. Extend it by adding a groupby(["endpoint", "timestamp"]) for per-endpoint daily trends, or filter by response time percentile to identify outlier days.

Frequently Asked Questions

How does Dask compare to Apache Spark?

Dask is Python-native and integrates directly with NumPy, Pandas, and scikit-learn — no JVM required. Spark is more mature for truly massive distributed workloads (petabyte scale) with built-in fault tolerance and many managed cloud offerings. For most Python data science workflows at the terabyte scale or below, Dask is simpler to set up and faster to iterate with. Spark’s Python API (PySpark) has more overhead and a less Pythonic feel than native Dask code.

How do I choose the right chunk size?

The rule of thumb: chunks should be between 100MB and 1GB in memory after loading. Too small (e.g., 1MB chunks) and you spend more time on scheduler overhead than computation. Too large (e.g., 10GB chunks) and you lose parallelism because only one chunk fits in memory at a time. For Dask DataFrames, aim for partitions that take 1-5 seconds to process. Use ddf.memory_usage(deep=True).sum().compute() divided by ddf.npartitions to measure actual partition size.

Why is my Dask code slower than Pandas?

Dask has overhead: building the task graph, serializing data between workers, and coordinating the scheduler all take time. For datasets that fit in RAM and operations that complete in seconds, Pandas is faster. Dask pays off when data exceeds RAM, when you have many independent computations that can run in parallel, or when processing takes more than a few minutes serially. Always benchmark before and after to confirm Dask actually helps your specific workload.

What does ddf.persist() do?

.persist() triggers computation and keeps the result in distributed memory across workers — it does not return a concrete result like .compute(), but subsequent operations on the persisted object are much faster because they skip re-reading from disk. Use .persist() when you will perform multiple downstream operations on the same intermediate result, like filtering a large DataFrame and then running several different aggregations on the filtered result.

Can I use Dask with scikit-learn?

Yes, via the dask-ml package (pip install dask-ml). It provides parallel implementations of common preprocessing steps like StandardScaler and MinMaxScaler that work on Dask DataFrames, plus a ParallelPostFit wrapper that parallelizes prediction (but not training) of any scikit-learn estimator. For full distributed model training, dask-ml integrates with XGBoost and LightGBM which have native Dask support.

Conclusion

Dask brings parallel and out-of-core computing to Python without requiring you to abandon NumPy, Pandas, or your existing code patterns. You have seen the four core tools: dask.delayed for parallelizing arbitrary Python functions, dask.array for chunked numerical computation, dask.dataframe for big tabular data, and the distributed scheduler for production visibility. The consistent theme is laziness — build the computation graph first, execute later, compute only what you need.

Extend the real-life log analyzer by connecting it to a real Parquet file dataset on S3 using dd.read_parquet("s3://your-bucket/*.parquet") and deploying a multi-worker cluster with LocalCluster(n_workers=8). The code changes are minimal because Dask’s API is designed to scale transparently from your laptop to a cloud cluster.

For complete API documentation and distributed deployment guides, visit the official Dask documentation.

Related Articles

• How To Use NumPy Arrays for Scientific Computing in Python
• How To Create Plots and Charts with Matplotlib in Python
• How To Use Celery for Background Tasks in Python

Intermediate

Your Python service processes thousands of JSON messages per second — API responses, message queue events, webhook payloads. Every message needs to be decoded, validated against a schema, and encoded again. You are using Pydantic, which is excellent, but its validation overhead starts to show at scale. You need something faster. msgspec is a C-extension library for Python that encodes and decodes JSON (and MessagePack and other formats) while validating against typed Python structs — and it is typically 5-10x faster than Pydantic V2, which is itself already fast.

msgspec defines data models using its Struct class — similar to dataclasses but implemented in C for maximum performance. It handles encoding, decoding, and type validation in a single pass, with no intermediate Python dictionaries. The result is encode/decode cycles that are often faster than pure JSON parsing with the built-in json module, because msgspec validates and structures the data during parsing rather than after it.

In this article you will learn how to install msgspec, define Struct models, encode and decode JSON, handle nested structures and optional fields, validate incoming data with error handling, and benchmark msgspec against the standard json module. By the end you will know when msgspec is the right choice and how to integrate it into a real application.

msgspec Quick Example

Here is the minimal setup — define a Struct, encode it to JSON, and decode it back:

# msgspec_quick.py
import msgspec
import msgspec.json

class User(msgspec.Struct):
    id: int
    name: str
    email: str
    active: bool = True

# Create an instance
user = User(id=1, name="Alice Smith", email="alice@example.com")
print(user)

# Encode to JSON bytes
encoded = msgspec.json.encode(user)
print(encoded)

# Decode back from JSON bytes
decoded = msgspec.json.decode(encoded, type=User)
print(decoded)
print(decoded.name)

Output:

User(id=1, name='Alice Smith', email='alice@example.com', active=True)
b'{"id":1,"name":"Alice Smith","email":"alice@example.com","active":true}'
User(id=1, name='Alice Smith', email='alice@example.com', active=True)
Alice Smith

The Struct class uses standard Python type annotations. msgspec.json.encode() returns bytes (not a string) for maximum efficiency. msgspec.json.decode() takes the type to decode into as its second argument — this is what enables the simultaneous parsing and validation in one pass. Keep reading to see how to handle optional fields, nested structs, and validation errors.

What Is msgspec and When To Use It?

msgspec is a C extension library that implements its own JSON encoder/decoder optimized specifically for Python type-annotated structs. It skips the intermediate Python dict that the standard json module creates and instead constructs the target type directly during parsing. This is why it is faster than even the fastest pure-Python alternatives.

Use msgspec when you need type-safe JSON handling and performance is a concern — high-traffic APIs, message queue consumers, data pipeline ingestion. Use Pydantic when you need complex validation rules, custom validators, or FastAPI’s dependency injection integration.

Installing msgspec

# terminal
pip install msgspec

Verify:

# verify_msgspec.py
import msgspec
print(msgspec.__version__)

Output:

0.18.6

Defining Structs

msgspec Struct supports all standard Python type annotations including Optional, list, dict, Union, Literal, and nested Structs:

# msgspec_structs.py
from typing import Optional
import msgspec

class Address(msgspec.Struct):
    street: str
    city: str
    country: str = "AU"

class Product(msgspec.Struct):
    sku: str
    name: str
    price: float
    tags: list[str] = []
    metadata: dict[str, str] = {}

class Order(msgspec.Struct):
    id: int
    customer_email: str
    items: list[Product]
    shipping_address: Address
    notes: Optional[str] = None
    discount_pct: float = 0.0

# Build a nested structure
order = Order(
    id=1001,
    customer_email="alice@example.com",
    items=[
        Product(sku="PY-001", name="Python Course", price=49.99, tags=["education", "tech"]),
        Product(sku="PY-002", name="VS Code Theme", price=9.99),
    ],
    shipping_address=Address(street="42 George Street", city="Sydney"),
    discount_pct=10.0,
)

encoded = msgspec.json.encode(order)
print(encoded.decode())

Output:

{"id":1001,"customer_email":"alice@example.com","items":[{"sku":"PY-001","name":"Python Course","price":49.99,"tags":["education","tech"],"metadata":{}},{"sku":"PY-002","name":"VS Code Theme","price":9.99,"tags":[],"metadata":{}}],"shipping_address":{"street":"42 George Street","city":"Sydney","country":"AU"},"notes":null,"discount_pct":10.0}

Default values work just like dataclasses. Optional[str] means the field can be a string or None. Nested Structs are encoded and decoded recursively. Lists of Structs (list[Product]) are fully supported and validated — each item in the JSON array must be a valid Product.

Decoding and Validation

The real power of msgspec is simultaneous decoding and validation. When a JSON input does not match the expected schema, msgspec raises a ValidationError with a precise error message:

# msgspec_validation.py
import msgspec
import msgspec.json
from typing import Optional

class Event(msgspec.Struct):
    event_type: str
    user_id: int
    payload: dict[str, str]
    version: int = 1

valid_json = b'{"event_type":"login","user_id":42,"payload":{"ip":"1.2.3.4"}}'
invalid_json_1 = b'{"event_type":"login","user_id":"not_an_int","payload":{}}'
invalid_json_2 = b'{"event_type":"login","payload":{}}'  # missing user_id

# Successful decode
event = msgspec.json.decode(valid_json, type=Event)
print(f"Decoded: {event}")

# Type mismatch
try:
    msgspec.json.decode(invalid_json_1, type=Event)
except msgspec.ValidationError as e:
    print(f"Type error: {e}")

# Missing required field
try:
    msgspec.json.decode(invalid_json_2, type=Event)
except msgspec.ValidationError as e:
    print(f"Missing field: {e}")

Output:

Decoded: Event(event_type='login', user_id=42, payload={'ip': '1.2.3.4'}, version=1)
Type error: Expected `int`, got `str` - at `$.user_id`
Missing field: Object missing required field `user_id` - at `$`

The error message includes a JSONPath-like location ($.user_id) that tells you exactly where the validation failed. This is critical for debugging and for returning meaningful error responses in API handlers. The ValidationError is lightweight and catches both type errors and structural errors (missing required fields, extra fields if you use forbid_unknown_fields=True).

Using the Encoder and Decoder for Performance

For high-throughput code, create a reusable Encoder and Decoder instead of calling module-level functions. This avoids recreating type lookup tables on every call:

# msgspec_encoder_decoder.py
import msgspec
import msgspec.json
from typing import Optional
import time

class Metric(msgspec.Struct):
    name: str
    value: float
    timestamp: int
    labels: dict[str, str] = {}

# Create reusable encoder/decoder (do this once at module level)
encoder = msgspec.json.Encoder()
decoder = msgspec.json.Decoder(Metric)

# Sample JSON payloads
sample = b'{"name":"cpu_usage","value":72.5,"timestamp":1746201600,"labels":{"host":"web-01"}}'

# Benchmark: 100,000 decode+encode cycles
N = 100_000
start = time.perf_counter()

for _ in range(N):
    metric = decoder.decode(sample)
    encoded = encoder.encode(metric)

elapsed = time.perf_counter() - start
print(f"msgspec: {N:,} decode+encode cycles in {elapsed:.3f}s")
print(f"  Rate: {N/elapsed:,.0f} ops/sec")
print(f"  Per op: {elapsed/N*1000:.4f}ms")
print(f"\nSample output: {encoded[:80]}")

Output:

msgspec: 100,000 decode+encode cycles in 0.187s
  Rate: 534,759 ops/sec
  Per op: 0.0019ms

Sample output: b'{"name":"cpu_usage","value":72.5,"timestamp":1746201600,"labels":{"host":"web-01"}}'

At 500,000+ operations per second, a single Python process can handle extremely high message throughput. Using module-level Encoder and Decoder instances instead of calling msgspec.json.encode()/decode() directly gives another 10-20% speedup on hot paths. Always initialize them at module level, not inside request handlers.

Real-Life Example: API Event Processor

Here is a complete webhook event processor that uses msgspec for fast, type-safe event handling:

# event_processor.py
import msgspec
import msgspec.json
from typing import Optional, Literal
from datetime import datetime

class WebhookUser(msgspec.Struct):
    id: int
    email: str
    name: Optional[str] = None

class WebhookEvent(msgspec.Struct):
    event: Literal["user.created", "user.deleted", "order.placed", "order.refunded"]
    data: WebhookUser
    timestamp: int
    webhook_id: str
    api_version: str = "2026-01"

# Typed event handlers
def handle_user_created(event: WebhookEvent):
    print(f"New user: {event.data.name or 'Unknown'} <{event.data.email}>")

def handle_order_placed(event: WebhookEvent):
    ts = datetime.fromtimestamp(event.timestamp).strftime("%Y-%m-%d %H:%M")
    print(f"Order placed by user {event.data.id} at {ts}")

HANDLERS = {
    "user.created": handle_user_created,
    "user.deleted": lambda e: print(f"User {e.data.id} deleted"),
    "order.placed": handle_order_placed,
    "order.refunded": lambda e: print(f"Order refunded for {e.data.email}"),
}

decoder = msgspec.json.Decoder(WebhookEvent)

def process_webhook(raw_json: bytes) -> bool:
    try:
        event = decoder.decode(raw_json)
        handler = HANDLERS.get(event.event)
        if handler:
            handler(event)
            return True
        else:
            print(f"Unknown event type: {event.event}")
            return False
    except msgspec.ValidationError as e:
        print(f"Invalid webhook payload: {e}")
        return False
    except Exception as e:
        print(f"Processing error: {e}")
        return False

# Test with sample payloads
payloads = [
    b'{"event":"user.created","data":{"id":1,"email":"alice@example.com","name":"Alice"},"timestamp":1746201600,"webhook_id":"wh_abc123"}',
    b'{"event":"order.placed","data":{"id":1,"email":"alice@example.com"},"timestamp":1746201700,"webhook_id":"wh_def456"}',
    b'{"event":"user.created","data":{"id":"not_int","email":"bad@example.com"},"timestamp":1746201800,"webhook_id":"wh_err"}',
]

print("Processing webhooks:\n")
for payload in payloads:
    success = process_webhook(payload)
    print(f"  -> {'OK' if success else 'FAILED'}\n")

Output:

Processing webhooks:

New user: Alice <alice@example.com>
  -> OK

Order placed by user 1 at 2026-05-02 16:00
  -> OK

Invalid webhook payload: Expected `int`, got `str` - at `$.data.id`
  -> FAILED

The Literal["user.created", "user.deleted", ...] annotation restricts the event field to exactly those four strings — msgspec validates this and raises a ValidationError for any other value. This eliminates a whole class of bugs where unexpected event types silently pass through to the wrong handler. The pattern — module-level decoder, try/except ValidationError, dispatch to typed handlers — is the production-ready approach for high-throughput webhook processing.

Frequently Asked Questions

When should I use msgspec instead of Pydantic?

Use msgspec when performance is a primary concern and you do not need Pydantic’s advanced features: custom validators with @validator, field aliases, computed fields, FastAPI’s native Pydantic integration, or JSON Schema generation. msgspec is 5-10x faster than Pydantic V2 for encode/decode, but Pydantic has a richer ecosystem and more flexible validation. For most FastAPI projects, Pydantic is the right choice. For high-throughput background services, msgspec often wins.

Does msgspec support formats other than JSON?

Yes — msgspec supports MessagePack (msgspec.msgpack) which is a binary format that is both faster and more compact than JSON. It also supports YAML and TOML via optional extras. MessagePack is the right choice for internal service-to-service communication where human readability is not needed and you want maximum throughput and minimum bandwidth.

Can I use msgspec with FastAPI?

Yes, but with some effort. FastAPI uses Pydantic models natively for request/response validation and OpenAPI schema generation. You can bypass this by accepting Request objects directly and decoding manually with msgspec. The msgspec package provides a convert() function that works with dicts (from await request.json()). For new FastAPI projects where performance is critical, consider using the litestar framework which has native msgspec support.

How do I handle nullable fields and missing values?

Use Optional[Type] (equivalent to Type | None in Python 3.10+) for fields that can be null. The JSON value null maps to Python None. For fields that can be absent from the JSON entirely, set a default value: field: str = "" or field: Optional[str] = None. Fields without defaults are required — msgspec raises ValidationError if they are missing from the input.

How do I handle custom types like datetime?

msgspec has built-in support for datetime.datetime (encoded as ISO 8601 strings), uuid.UUID (encoded as strings), decimal.Decimal, and enum.Enum. For truly custom types, use msgspec.json.Encoder(enc_hook=...)) and Decoder(dec_hook=...) to provide custom encoding/decoding hooks. These hooks receive the unrecognized type and must return a serializable value.

Conclusion

msgspec delivers JSON encoding and decoding with type validation at speeds that make it practical for the highest-throughput Python services. The key patterns: define your schema with msgspec.Struct using standard type annotations; use Optional[T] for nullable fields and defaults for optional ones; create module-level Encoder and Decoder instances for hot paths; and always wrap decoder.decode() in a try/except for msgspec.ValidationError when handling external input.

The webhook processor example shows how to combine Literal type constraints, nested Structs, and a dispatch table for a clean, type-safe event processing pipeline. Extend it by adding a dead-letter queue for failed payloads and a metrics counter for validation error rates — those two additions turn it from an example into a production-ready processor.

For the full API reference, supported types, and MessagePack documentation, see the msgspec documentation.

Related Articles

• How To Use Pydantic V2 for Data Validation in Python
• How To Read and Write JSON Files in Python
• How To Use Python attrs Library for Data Classes Without Boilerplate

Intermediate

You have a list of dictionaries, a query result, or a DataFrame you want to print to the terminal — and the default Python output looks like a wall of brackets and commas. Every developer has been there. Debugging data pipelines, writing CLI tools, generating quick reports: at some point you need output that a human can actually read without squinting. Python’s tabulate library solves this in one line.

The tabulate package converts Python data structures into formatted ASCII tables with customizable styles, alignment, and number formatting. It handles lists of lists, lists of dicts, NumPy arrays, Pandas DataFrames, and more. Installation is a single pip command, and it has no mandatory dependencies — making it one of the easiest wins in your CLI toolkit.

In this article you will learn how to install and use tabulate for common data structures, explore the most useful table formats, control alignment and number precision, handle missing values gracefully, and combine tabulate with real-world data in a practical CLI report. By the end you will know exactly when and how to reach for tabulate instead of print.

Python tabulate: Quick Example

Here is the simplest possible tabulate usage — a list of dictionaries printed as a neat grid:

# tabulate_quick.py
from tabulate import tabulate

data = [
    {"name": "Alice", "score": 95, "grade": "A"},
    {"name": "Bob",   "score": 82, "grade": "B"},
    {"name": "Carol", "score": 74, "grade": "C"},
]

print(tabulate(data, headers="keys"))

Output:

name      score  grade
------  -------  -------
Alice        95  A
Bob          82  B
Carol        74  C

The headers="keys" argument tells tabulate to use dictionary keys as column headers. The library automatically right-aligns numeric columns and left-aligns strings — sensible defaults that make the table immediately readable. Keep reading to see how to customize styles, alignment, and formatting for more demanding use cases.

What Is tabulate and Why Use It?

The tabulate library is a pure-Python utility that transforms sequences of records into formatted text tables. Think of it as str.format() for tabular data: you hand it a list of rows and a header, and it figures out column widths, alignment, and separators automatically.

Without tabulate, you would write custom formatting code every time: calculate max column widths, pad strings, build separator lines. That is tedious and error-prone. Tabulate does all of that and adds support for 30+ table styles including plain text, GitHub Markdown, HTML, LaTeX, and more.

Installing tabulate

Install from PyPI with pip:

# terminal
pip install tabulate

Verify the installation:

# verify_tabulate.py
import tabulate
print(tabulate.__version__)

Output:

0.9.0

Choosing a Table Format

The tablefmt parameter controls the visual style. Here are the most useful formats side by side:

# tabulate_formats.py
from tabulate import tabulate

rows = [["Python", 3.12, "Interpreted"], ["Rust", 1.75, "Compiled"], ["Go", 1.21, "Compiled"]]
headers = ["Language", "Version", "Type"]

for fmt in ["simple", "grid", "github", "plain"]:
    print(f"\n--- {fmt} ---")
    print(tabulate(rows, headers=headers, tablefmt=fmt))

Output (selected formats):

--- simple ---
Language      Version  Type
----------  ---------  -----------
Python           3.12  Interpreted
Rust             1.75  Compiled
Go               1.21  Compiled

--- grid ---
+------------+-----------+-------------+
| Language   |   Version | Type        |
+============+===========+=============+
| Python     |      3.12 | Interpreted |
+------------+-----------+-------------+
| Rust       |      1.75 | Compiled    |
+------------+-----------+-------------+
| Go         |      1.21 | Compiled    |
+------------+-----------+-------------+

--- github ---
| Language   |   Version | Type        |
|:-----------|----------:|:------------|
| Python     |      3.12 | Interpreted |
| Rust       |      1.75 | Compiled    |
| Go         |      1.21 | Compiled    |

Use simple for most terminal output. Use github when generating Markdown for READMEs or PR descriptions. Use grid when you want clear visual cell boundaries. For HTML output, use html — tabulate generates proper <table> markup with <th> and <td> tags.

Column Alignment and Number Precision

Tabulate auto-aligns columns but you can override this with the colalign parameter. Number formatting uses floatfmt and intfmt:

# tabulate_alignment.py
from tabulate import tabulate

portfolio = [
    ["AAPL",  182.63,  1500000.25,  "+2.3%"],
    ["GOOGL", 140.12, 14000000.00,  "-0.8%"],
    ["MSFT",  378.85,  9500000.50,  "+1.1%"],
]
headers = ["Ticker", "Price", "Market Cap", "Change"]

print(tabulate(
    portfolio,
    headers=headers,
    tablefmt="simple",
    colalign=("left", "right", "right", "center"),
    floatfmt=("", ".2f", ",.2f", ""),
))

Output:

Ticker      Price     Market Cap    Change
--------  -------  -------------  --------
AAPL       182.63  1,500,000.25   +2.3%
GOOGL      140.12  14,000,000.00  -0.8%
MSFT       378.85  9,500,000.50   +1.1%

The colalign tuple takes one value per column: "left", "right", or "center". The floatfmt tuple follows Python’s format spec mini-language — ".2f" for two decimal places, ",.2f" adds thousands separator. Pass an empty string to skip formatting for that column.

Handling Missing Values

Real-world data is messy — rows may have different lengths or None values. Tabulate handles both gracefully with the missingval parameter:

# tabulate_missing.py
from tabulate import tabulate

rows = [
    ["Alice",  "Engineering", 95000],
    ["Bob",    "Marketing",   None],
    ["Carol",  None,          72000],
    ["Dave",   "Engineering"],
]

headers = ["Name", "Department", "Salary"]

print(tabulate(rows, headers=headers, tablefmt="simple", missingval="N/A"))

Output:

Name    Department      Salary
------  ------------  --------
Alice   Engineering      95000
Bob     Marketing          N/A
Carol   N/A              72000
Dave    Engineering        N/A

The missingval parameter fills in any None or missing cell with the string you provide. This makes tabulate safe to use with real database queries where columns can be NULL — no need to pre-clean the data before display.

Working with Different Data Sources

Tabulate accepts multiple input formats. Here is how it works with each:

# tabulate_sources.py
from tabulate import tabulate

# 1. List of lists
rows_lol = [[1, "alpha", 3.14], [2, "beta", 2.71]]
print("List of lists:")
print(tabulate(rows_lol, headers=["ID", "Name", "Value"], tablefmt="simple"))

# 2. List of dicts
rows_dicts = [
    {"city": "Sydney", "pop": 5300000, "country": "AU"},
    {"city": "London", "pop": 9000000, "country": "UK"},
]
print("\nList of dicts:")
print(tabulate(rows_dicts, headers="keys", tablefmt="simple"))

# 3. Dict of lists (column-oriented)
rows_cols = {"x": [1, 2, 3], "y": [10, 20, 30], "z": [100, 200, 300]}
print("\nDict of lists:")
print(tabulate(rows_cols, headers="keys", tablefmt="simple"))

Output:

List of lists:
  ID  Name      Value
----  ------  -------
   1  alpha      3.14
   2  beta       2.71

List of dicts:
city        pop  country
------  -------  ---------
Sydney  5300000  AU
London  9000000  UK

Dict of lists:
  x    y    z
---  ---  ---
  1   10  100
  2   20  200
  3   30  300

For headers="keys" with dicts, tabulate uses the dictionary keys in insertion order (Python 3.7+). For column-oriented data (a dict of lists), tabulate handles it correctly without any conversion on your part.

Real-Life Example: CLI Package Dependency Report

Here is a practical script that reads installed packages and generates a formatted report — useful for auditing environments before deployment:

# package_report.py
import subprocess
import sys
from tabulate import tabulate

def get_installed_packages():
    result = subprocess.run(
        [sys.executable, "-m", "pip", "list", "--format=json"],
        capture_output=True, text=True
    )
    import json
    packages = json.loads(result.stdout)
    return packages

def get_outdated_packages():
    result = subprocess.run(
        [sys.executable, "-m", "pip", "list", "--outdated", "--format=json"],
        capture_output=True, text=True
    )
    import json
    if result.stdout.strip():
        return {p["name"].lower(): p["latest_version"] for p in json.loads(result.stdout)}
    return {}

packages = get_installed_packages()
outdated = get_outdated_packages()

rows = []
for pkg in sorted(packages, key=lambda p: p["name"].lower()):
    name = pkg["name"]
    version = pkg["version"]
    latest = outdated.get(name.lower(), "")
    status = "OUTDATED" if latest else "ok"
    rows.append([name, version, latest or "--", status])

print("\nInstalled Packages Report")
print(f"Total: {len(rows)} packages, {len(outdated)} outdated\n")

print(tabulate(
    rows,
    headers=["Package", "Installed", "Latest", "Status"],
    tablefmt="simple",
    colalign=("left", "right", "right", "center"),
    missingval="--",
))

Output (example):

Installed Packages Report
Total: 47 packages, 3 outdated

Package      Installed    Latest    Status
---------  -----------  --------  --------
certifi         2023.11  2024.02   OUTDATED
pip              23.3.2   24.0.0   OUTDATED
requests          2.31.0  --          ok
tabulate          0.9.0   --          ok
urllib3           2.1.0   2.2.0    OUTDATED

This script shows how to combine tabulate with subprocess output for a genuinely useful DevOps tool. Extend it by adding a --html flag that changes tablefmt="html" and writes to a file, or filter to show only outdated packages by slicing the rows list before passing to tabulate.

Frequently Asked Questions

Can I use tabulate directly with a Pandas DataFrame?

Yes — pass the DataFrame and headers="keys": tabulate(df, headers="keys", tablefmt="grid"). DataFrames also have a built-in df.to_markdown() method that wraps tabulate internally. For Jupyter notebooks, the built-in HTML rendering is usually more appropriate than tabulate.

How do I handle very wide tables that wrap in the terminal?

Use maxcolwidths (added in tabulate 0.9.0) to truncate long values: tabulate(data, headers="keys", maxcolwidths=30) caps each column at 30 characters. For very wide tables, consider using tablefmt="plain" (no borders) or transpose the data so columns become rows.

How do I add a row index to my table?

Pass showindex=True for an auto-incrementing index, or pass a list for custom indices: tabulate(rows, headers=headers, showindex=range(1, len(rows)+1)). You can also pass showindex="always" which works for any input type including DataFrames.

Is tabulate fast enough for large datasets?

Tabulate is optimized for display, not batch processing — it is fine for hundreds to a few thousand rows. For tables with 10,000+ rows you will notice slowdown because it must scan all values to compute column widths. Display only a sample (data[:100]) or use a fixed column width you specify manually.

Can I add colors or bold text to tabulate output?

Tabulate itself does not add ANSI color codes, but you can add them to your data strings before passing to tabulate. Libraries like colorama work alongside tabulate. Note that ANSI codes add invisible characters that can throw off alignment — test carefully. If you want colors plus rich formatting, consider using the rich library’s built-in Table class as an alternative.

Conclusion

The tabulate library turns the chore of formatting tabular data into a single function call. The key patterns: pass lists of dicts with headers="keys" for the quickest readable output; use tablefmt="github" for Markdown output; use colalign and floatfmt for precise numeric formatting; and missingval for handling None values from real-world data sources.

The real-life example — a package dependency reporter — shows how to combine tabulate with subprocess to build a useful DevOps tool in about 40 lines. Extend it with a --html output option and automated email delivery for a lightweight dependency audit system.

For the full list of format names and options, see the tabulate documentation on PyPI and the GitHub repository which includes format screenshots for every style.

Related Articles

• How To Use the Rich Library for Beautiful Terminal Output in Python
• How To Read and Write CSV Files in Python
• How To Use Python argparse for Command-Line Arguments