Intermediate
Python is a multi-paradigm language, and functional programming is one of its strongest modes — but the standard library tools for FP (map, filter, functools.reduce, functools.partial) are scattered, limited, and often awkward to compose. Writing a data transformation pipeline in pure Python often forces you to choose between deeply nested function calls that are hard to read, or imperative loops that mix transformation logic with iteration. There is a better way.
The toolz library provides a clean, composable toolkit for functional Python: curry for partial application without the boilerplate, pipe for threading data through transformation steps, compose for building reusable function chains, and dictionary utilities like valmap, valfilter, and reduceby that work on mappings the way map/filter/reduce work on lists. Install with pip install toolz. A Cython-compiled variant, cytoolz, offers 2-5x speed improvements for performance-critical code.
This tutorial walks through the most useful toolz patterns: building readable data pipelines with pipe, creating reusable partial functions with curry, combining functions with compose and juxt, and transforming nested dictionaries with the mapping utilities. By the end, you will be writing Python data transformations that read like specifications — and are trivially unit-testable because every step is a pure function.
toolz Quick Example
The two most immediately useful functions are pipe (thread data through functions left to right) and curry (partial application on any function). Here they are together:
# toolz_quick.py
from toolz import pipe, curry
# pipe: data flows left to right through each function
result = pipe(
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
lambda nums: filter(lambda n: n % 2 == 0, nums), # keep evens
lambda nums: map(lambda n: n ** 2, nums), # square them
list, # materialize
sum # sum
)
print("pipe result:", result) # (2^2 + 4^2 + 6^2 + 8^2 + 10^2) = 220
# curry: partial application without functools.partial boilerplate
@curry
def multiply(factor, value):
return factor * value
double = multiply(2) # returns a new function with factor=2 fixed
triple = multiply(3)
print("double 7:", double(7)) # 14
print("triple 7:", triple(7)) # 21
print("map double:", list(map(double, [1, 2, 3, 4, 5])))
Output:
pipe result: 220
double 7: 14
triple 7: 21
map double: [2, 4, 6, 8, 10]
pipe eliminates nested function calls by making the data flow explicit and left-to-right. curry turns any multi-argument function into a series of single-argument functions — each partial application returns a new callable that waits for the remaining arguments. This makes creating specialized variants of general functions clean and readable.
What Is toolz and When Should You Use It?
Toolz is a functional programming utility library that brings Haskell and Clojure-style higher-order function patterns to Python. It is not a replacement for Python’s functools or itertools — it is a complement that fills the gaps and provides higher-level composition tools.
| toolz function | Standard library equivalent | What toolz adds |
|---|---|---|
| curry | functools.partial | Auto-currying on call, not just one application |
| pipe | Nested function calls | Left-to-right readable chaining |
| compose | lambda x: f(g(x)) | Named, reusable function compositions |
| juxt | Multiple lambdas | Apply N functions to same input, return tuple |
| reduceby | itertools.groupby + reduce | GroupBy and aggregate in one pass |
| valmap/valfilter | Dict comprehension | Map/filter over dict values with key preserved |
| merge/merge_with | dict.update | Non-mutating dict merge with conflict resolution |
Use toolz when your code has data transformation pipelines that would benefit from being expressed as function chains, when you find yourself writing many functools.partial calls, or when you want to write pure-function transformations that are easy to test in isolation. Avoid it in code where readability for non-FP-aware developers is the priority — not everyone on a team will be comfortable with heavy currying.
Building Reusable Pipelines with compose and juxt
compose creates a new function by chaining existing functions together. Unlike pipe (which applies a single input to a chain), compose produces a reusable function object you can call many times and pass as arguments.
# toolz_compose.py
from toolz import compose, juxt
import re
# Individual transformation functions (pure, testable in isolation)
def remove_punctuation(text):
return re.sub(r"[^\w\s]", "", text)
def to_lowercase(text):
return text.lower()
def split_words(text):
return text.split()
def count_unique(words):
return len(set(words))
# compose: right-to-left (innermost applied first, like math notation)
# This reads: count_unique(split_words(to_lowercase(remove_punctuation(text))))
word_counter = compose(count_unique, split_words, to_lowercase, remove_punctuation)
text = "Python is Great! Python is also FAST -- and Python is readable."
print("Unique word count:", word_counter(text)) # 6: python, is, great, also, fast, and
# Apply to multiple inputs
texts = [
"Hello World! Hello again.",
"The quick brown fox jumps.",
"Data data data DATA!"
]
print("Unique counts:", list(map(word_counter, texts)))
# juxt: apply multiple functions to the same input, get tuple of results
analyze = juxt(len, count_unique, sorted)
words = split_words(to_lowercase(remove_punctuation(text)))
total, unique, sorted_words = analyze(words)
print(f"Total words: {total}, Unique: {unique}, First 3: {sorted_words[:3]}")
Output:
Unique word count: 6
Unique counts: [3, 5, 2]
Total words: 11, Unique: 6, First 3: ['also', 'and', 'fast']
compose applies functions right-to-left (like mathematical function composition). If you prefer left-to-right order (matching how you read the transformations), use compose‘s reversed equivalent or just use pipe with a fixed input. juxt (“juxtapose”) applies a collection of functions to the same input and returns a tuple of results — perfect for computing multiple statistics or extracting multiple fields from a record in one pass.
Dictionary Utilities: valmap, valfilter, merge_with
Dictionaries are ubiquitous in Python data processing — API responses, configuration, aggregated counts — and toolz provides clean functional tools for transforming them without mutation.
# toolz_dicts.py
from toolz import valmap, valfilter, keymap, merge, merge_with
# Sales data: category -> revenue
sales = {"Electronics": 45200, "Books": 8750, "Clothing": 22100, "Food": 15800}
# valmap: apply a function to every value, keep keys
revenue_k = valmap(lambda v: round(v / 1000, 1), sales)
print("Revenue (K):", revenue_k)
# valfilter: keep only entries where value matches predicate
high_revenue = valfilter(lambda v: v > 20000, sales)
print("High revenue:", high_revenue)
# keymap: transform keys
upper_keys = keymap(str.upper, sales)
print("Upper keys:", upper_keys)
# merge: non-mutating dict merge (last wins on conflict)
defaults = {"timeout": 30, "retries": 3, "verbose": False}
overrides = {"timeout": 60, "verbose": True}
config = merge(defaults, overrides)
print("Merged config:", config)
# merge_with: merge dicts with a function to resolve conflicts
monthly_a = {"Jan": 100, "Feb": 150, "Mar": 200}
monthly_b = {"Jan": 80, "Feb": 120, "Apr": 90}
combined = merge_with(sum, monthly_a, monthly_b)
print("Combined monthly:", combined)
Output:
Revenue (K): {'Electronics': 45.2, 'Books': 8.75, 'Clothing': 22.1, 'Food': 15.8}
High revenue: {'Electronics': 45200, 'Clothing': 22100}
Upper keys: {'ELECTRONICS': 45200, 'BOOKS': 8750, 'CLOTHING': 22100, 'FOOD': 15800}
Merged config: {'timeout': 60, 'retries': 3, 'verbose': True}
Combined monthly: {'Jan': 180, 'Feb': 270, 'Mar': 200, 'Apr': 90}
merge_with is particularly useful for aggregating multiple data sources: pass sum to add numeric values across dicts, list to collect values into a list, or any custom function to handle conflicts. Unlike dict comprehensions, these functions are named, composable, and express intent clearly.
Grouping and Aggregating with reduceby
reduceby combines groupby and reduce into a single pass over an iterable. Given a keying function and a reducing function, it groups records by key and reduces each group to a single value — all without building intermediate grouped structures.
# toolz_reduceby.py
from toolz import reduceby, countby, groupby
from operator import add
# Sales transactions
transactions = [
{"region": "AU", "amount": 250.0, "category": "Electronics"},
{"region": "US", "amount": 180.0, "category": "Books"},
{"region": "AU", "amount": 420.0, "category": "Clothing"},
{"region": "US", "amount": 95.0, "category": "Books"},
{"region": "AU", "amount": 310.0, "category": "Electronics"},
{"region": "UK", "amount": 175.0, "category": "Food"},
{"region": "US", "amount": 630.0, "category": "Electronics"},
]
# reduceby: group by region, sum amounts
total_by_region = reduceby(
lambda t: t["region"],
lambda acc, t: acc + t["amount"],
transactions,
0 # initial value for each group
)
print("Total by region:", {k: round(v, 2) for k, v in total_by_region.items()})
# countby: count items by a key function
count_by_category = countby(lambda t: t["category"], transactions)
print("Count by category:", count_by_category)
# groupby: collect full records by key (like SQL GROUP BY)
by_region = groupby(lambda t: t["region"], transactions)
print("AU transactions:", len(by_region["AU"]))
print("First AU amount:", by_region["AU"][0]["amount"])
Output:
Total by region: {'AU': 980.0, 'US': 905.0, 'UK': 175.0}
Count by category: {'Electronics': 3, 'Books': 2, 'Clothing': 1, 'Food': 1}
AU transactions: 3
First AU amount: 250.0
reduceby is more efficient than groupby + dict comprehension because it accumulates results in one pass without building a full grouped dictionary first. For large datasets, the difference in memory usage is significant. countby is a convenient special case that counts occurrences by key — effectively a histogram over any grouping function.
Real-Life Example: Sales Data Pipeline
# toolz_sales_pipeline.py
from toolz import pipe, curry, compose, valmap, merge_with, reduceby, countby
from functools import reduce
# Raw transaction data
transactions = [
{"id": 1, "region": "AU", "category": "Electronics", "amount": 1200.00, "qty": 2},
{"id": 2, "region": "US", "category": "Books", "amount": 45.00, "qty": 3},
{"id": 3, "region": "AU", "category": "Clothing", "amount": 280.00, "qty": 1},
{"id": 4, "region": "US", "category": "Electronics", "amount": 899.99, "qty": 1},
{"id": 5, "region": "AU", "category": "Books", "amount": 60.00, "qty": 2},
{"id": 6, "region": "UK", "category": "Electronics", "amount": 1500.00, "qty": 3},
{"id": 7, "region": "US", "category": "Clothing", "amount": 150.00, "qty": 2},
{"id": 8, "region": "AU", "category": "Electronics", "amount": 750.00, "qty": 1},
]
# --- Step 1: Build reusable transformation functions ---
@curry
def filter_by_region(region, txns):
return [t for t in txns if t["region"] == region]
@curry
def above_amount(min_amount, txns):
return [t for t in txns if t["amount"] >= min_amount]
def add_total_value(txns):
return [{**t, "total_value": t["amount"] * t["qty"]} for t in txns]
def extract_total_values(txns):
return [t["total_value"] for t in txns]
# --- Step 2: Compose a pipeline ---
au_high_value_pipeline = compose(
sum,
extract_total_values,
add_total_value,
above_amount(500),
filter_by_region("AU")
)
print("AU high-value total:", au_high_value_pipeline(transactions))
# --- Step 3: Aggregate across all regions ---
revenue_by_category = reduceby(
lambda t: t["category"],
lambda acc, t: acc + t["amount"] * t["qty"],
transactions,
0.0
)
print("\nRevenue by category:")
for cat, rev in sorted(revenue_by_category.items(), key=lambda x: -x[1]):
print(f" {cat}: ${rev:,.2f}")
# --- Step 4: Multi-region report using pipe ---
region_summary = pipe(
transactions,
lambda txns: reduceby(lambda t: t["region"], lambda a, t: a + t["amount"], txns, 0.0),
lambda by_region: valmap(lambda v: round(v, 2), by_region),
)
print("\nRevenue by region:", region_summary)
Output:
AU high-value total: 3150.0
Revenue by category:
Electronics: $9597.99
Clothing: $580.00
Books: $255.00
Revenue by region: {'AU': 2290.0, 'US': 1094.99, 'UK': 4500.0}
Every transformation function is pure — it takes inputs and returns outputs with no side effects and no mutation. This makes each function trivially unit-testable. The pipeline au_high_value_pipeline is itself a function, which means you can call it in tests with different data, use it in a larger pipe, or swap out individual components by redefining the compose chain.
Frequently Asked Questions
How does toolz compare to funcy?
funcy and toolz cover similar ground but with different emphases. toolz focuses on composability and the mathematical FP primitives — curry, compose, pipe, reduceby. funcy has more utility functions for everyday tasks like compact, walk, pluck, and omit, and generally has a more Pythonic feel. Many developers install both; they complement each other well. If you are writing data pipelines, start with toolz. If you need utility functions for working with dicts and lists in a less functional style, add funcy.
When should I use cytoolz instead of toolz?
cytoolz is a Cython reimplementation of the core toolz functions that runs 2-5x faster. The API is identical — you can swap the import from toolz to cytoolz with no other code changes. Use cytoolz when you are processing large iterables (millions of items) in tight loops where the overhead of Python function calls becomes measurable. For typical application code, the speed difference is negligible and plain toolz is fine.
When should I use curry vs functools.partial?
functools.partial fixes specific positional or keyword arguments in a single step. curry makes a function auto-applying — each call with fewer arguments than required returns a new partially-applied function, while a call with all remaining arguments returns the result. Use curry when you want to create a family of related functions by progressive application (like multiply(2), multiply(10)). Use functools.partial when you just need to pin specific arguments in one shot without the FP overhead.
Does toolz enforce immutability?
No — toolz functions are written to be non-mutating (they return new objects instead of modifying inputs), but Python has no built-in enforcement of immutability. If you pass a mutable dict to valmap, it returns a new dict and leaves the original untouched. However, your own functions in a pipe or compose chain can still mutate state if written that way. The discipline of writing pure functions is yours to maintain; toolz just provides the plumbing that makes it easy.
What is the difference between pipe and compose?
pipe(data, f, g, h) applies the functions to a specific value immediately and returns the result. compose(h, g, f) returns a new function that applies the chain when called — no value is required yet. Use pipe when you have data ready and want to transform it now. Use compose when you want to build a reusable transformation that will be applied to different inputs later, passed as a callback, or tested independently.
Conclusion
The toolz library brings readable, composable functional programming to Python data pipelines. You have seen pipe for left-to-right data threading, curry for clean partial application, compose and juxt for building reusable function objects, valmap/valfilter/merge_with for dictionary transformations, and reduceby/countby for single-pass group aggregations. The sales pipeline example showed how these tools combine into a system where each transformation step is a pure, independently testable function.
Extend the pipeline by adding a curry-based discount function that takes a region-specific rate, composing it into the au_high_value_pipeline before the final sum. For the complete API reference, visit the toolz documentation. For the faster Cython variant, see cytoolz on GitHub.