Intermediate
Your data processing script finishes in 47 seconds. You know it is slow, but you have no idea which part is the culprit — is it the CSV parsing, the nested loop, or the NumPy computation at the end? You run cProfile, get a 200-line dump of function call counts, and spend the next hour squinting at aggregate numbers that tell you which function was slow but not which line inside it. You need a better tool.
scalene is a high-performance Python profiler that works at the line level, splits CPU time into Python time and native (C extension) time, and tracks memory allocations — all with statistical sampling that adds almost no overhead to your program. You install it with one pip install and prefix your command with scalene. No decorators, no code changes, no instrumentation.
This article covers everything you need to get useful profiles out of scalene: the quick-start workflow, how to read its output, CPU and memory profiling separately, HTML reports, filtering noise with thresholds, and a real-life profiling session where we find and fix two bottlenecks in a data processing script. By the end you will know exactly how to answer “why is my Python code slow?”
Python scalene: Quick Example
Here is the fastest way to see scalene in action. First, install it, then point it at any Python file — no changes to the file required.
# terminal -- installation
pip install scalene
Output:
Successfully installed scalene-1.5.51 ...
Now create a small script that has a visible bottleneck:
# slow_demo.py
import time
def fast_work():
return sum(range(1_000_000))
def slow_work():
# Artificially slow: repeated string concatenation
result = ""
for i in range(5_000):
result += str(i)
return result
def main():
for _ in range(3):
fast_work()
for _ in range(10):
slow_work()
main()
Run it under scalene:
# terminal
scalene slow_demo.py
Output (terminal table — abbreviated):
slow_demo.py: % of time = 100.00% out of 1.23s.
% |% |% | Memory (MB) | slow_demo.py
CPU Python native | alloc peak | line
─────────────────────────────────────────────────────────────────────────────
5.00% 5.00% 0.00% | 0.0 2.1 | 5: def fast_work():
6.00% 6.00% 0.00% | 0.0 2.1 | 6: return sum(range(1_000_000))
84.00% 84.00% 0.00% | 9.8 9.8 | 11: result += str(i)
5.00% 5.00% 0.00% | 0.0 2.1 | 14: def main():
Line 11 is consuming 84% of CPU time and allocating 9.8 MB — immediately obvious. The Python column shows this is pure Python overhead, not a C extension, which means there is a better Python algorithm waiting to replace it. That is the value of scalene in a single glance.
What Is scalene and Why Use It?
scalene is a statistical profiler created by Emery Berger at UMass Amherst. Instead of hooking into every function call the way cProfile does, it uses sampling: it interrupts your program hundreds of times per second, records what line is executing, and builds a statistical picture of where time is spent. Because it only samples, the overhead is low — typically 10-20% added runtime, compared to 2-10x slowdown with cProfile.
The most important thing scalene adds that other profilers do not is the CPU time breakdown. Every line shows three numbers: total CPU%, Python%, and native%. Native time is time spent inside C extensions like NumPy, Pandas, or OpenCV. If a line shows 80% total but 2% Python and 78% native, you know the bottleneck is in the C library — rewriting that line in pure Python will make it slower, not faster. You need to find a different NumPy operation, not replace NumPy with a loop.
| Profiler | Granularity | Python vs native | Memory tracking | Overhead |
|---|---|---|---|---|
cProfile | Function level | No | No | Medium (2-5x) |
line_profiler | Line level | No | No | High (requires decoration) |
memory_profiler | Line level | No | Yes | Very high (10-20x) |
scalene | Line level | Yes | Yes | Low (1.1-1.2x) |
The combination of line-level granularity, the Python/native split, and memory tracking in a single low-overhead tool is what makes scalene worth reaching for first when you need to understand why code is slow.
Reading scalene Output
The terminal table that scalene prints uses a consistent column layout. Understanding each column lets you immediately identify the right fix.
# reading_output_reference.py
# (This is a reference -- not runnable standalone. See slow_demo.py above.)
# Column layout scalene prints:
#
# %CPU | %Python | %native | Memory alloc | Memory peak | line# | source line
#
# %CPU -- percentage of total wall-clock time attributed to this line
# %Python -- of that CPU time, how much ran in the Python interpreter itself
# %native -- of that CPU time, how much ran inside a C extension (NumPy, etc.)
# alloc -- net memory allocated on this line (MB), averaged over samples
# peak -- peak memory in use when this line was running (MB)
The key relationship is: %Python + %native = %CPU (approximately — small rounding differences are normal). When you see a high %native on a line calling a NumPy function, the time is inside the C library and is already optimized. Look elsewhere. When you see high %Python on a loop or string operation, that is where your rewrite effort belongs.
GPU Column
If you have an NVIDIA GPU and your code uses libraries like PyTorch or CuPy, scalene adds a %GPU column automatically. This column shows what percentage of time that line was waiting on GPU operations. No extra flags needed — scalene detects GPU usage automatically when CUDA is available.
CPU Profiling in Depth
Let us build a realistic script with multiple functions and see how scalene pinpoints the slow paths. This script downloads user data from a public API and processes it in two ways — one efficient, one not.
# cpu_profile_demo.py
import urllib.request
import json
def fetch_users():
"""Fetch 10 fake users from jsonplaceholder."""
url = "https://jsonplaceholder.typicode.com/users"
with urllib.request.urlopen(url) as resp:
return json.loads(resp.read())
def process_slow(users):
"""Build a name index with slow string operations."""
index = ""
for user in users:
# String concatenation in a loop -- O(n^2) due to immutability
index += user["name"].lower().replace(" ", "_") + ","
return index.split(",")
def process_fast(users):
"""Build the same index with a list comprehension."""
return [
user["name"].lower().replace(" ", "_")
for user in users
]
def analyze(users):
"""Run both processors 500 times each to amplify the difference."""
slow_results = []
for _ in range(500):
slow_results = process_slow(users)
fast_results = []
for _ in range(500):
fast_results = process_fast(users)
return slow_results, fast_results
def main():
users = fetch_users()
slow, fast = analyze(users)
print(f"Slow result count: {len(slow)}")
print(f"Fast result count: {len(fast)}")
main()
Output (without profiling):
Slow result count: 11
Fast result count: 10
Now run it under scalene:
# terminal
scalene cpu_profile_demo.py
Output (scalene terminal — key lines):
% |% |%
CPU Python native | cpu_profile_demo.py
──────────────────────────────────────────────────────────
68.00% 68.00% 0.00% | 12: index += user["name"].lower()...
14.00% 14.00% 0.00% | 17: return [
9.00% 0.00% 9.00% | 6: with urllib.request.urlopen(url)...
Line 12 (the concatenation loop) takes 68% of total runtime even though it does the same logical work as line 17 (the list comprehension at 14%). Both show 0% native time — this is a pure Python performance difference. The fix is already in the script: use the list comprehension pattern from process_fast.
Memory Profiling
CPU time is only half the story. Memory allocations slow programs down too — the garbage collector has to clean up everything you allocate, and excessive allocation creates cache pressure. scalene tracks memory per line by default, but you can focus on memory analysis with the --memory flag to see only lines where allocations happened.
# memory_demo.py
def allocate_badly():
"""Accumulate large lists inside a loop -- common leak pattern."""
cache = []
for i in range(1000):
# Creating a new list each iteration instead of appending
row = list(range(1000)) # 1000 ints per iteration
cache.append(row)
return cache
def allocate_well():
"""Pre-allocate and fill -- same result, lower peak memory."""
cache = [None] * 1000
for i in range(1000):
cache[i] = list(range(1000))
return cache
def main():
result_a = allocate_badly()
result_b = allocate_well()
print(f"Both results: {len(result_a)} rows, {len(result_b)} rows")
main()
Run with the --memory flag to filter output to only memory-heavy lines:
# terminal
scalene --memory memory_demo.py
Output (memory columns highlighted):
Memory (MB) | memory_demo.py
alloc peak | line
─────────────────────────────────
7.8 7.8 | 7: row = list(range(1000))
0.0 7.8 | 8: cache.append(row)
0.0 15.6 | 15: cache[i] = list(range(1000))
The alloc column shows how much memory is freshly allocated on each sample of that line. Line 7 allocates 7.8 MB — those are the intermediate list objects being created and added to the cache. The peak column shows the highest memory in use while that line was running. By line 15, peak is 15.6 MB because both caches exist simultaneously. Knowing the per-line allocation lets you decide where to add object reuse, streaming, or generators.
HTML Reports for Deeper Analysis
For complex programs, the terminal output is useful but the HTML report is better. It shows the full source file annotated with profiling data inline, color-coded by heat, and lets you sort and filter interactively.
# terminal
scalene --html --outfile profile_report.html cpu_profile_demo.py
Output:
Out: profile_report.html
Open profile_report.html in any browser. You will see the source file on the left and bar charts next to each line showing CPU%, Python%, native%, and memory allocation. Lines with high CPU are highlighted in red; lines with high memory allocation are highlighted in blue. You can click column headers to sort functions by their total contribution, which makes it easy to find where the most time accumulates across many call sites.
The HTML report is especially useful when sharing profiling results with teammates — it is self-contained (no dependencies) and shows the source code and data together, so the reader does not need to run anything to understand the bottleneck.
Filtering Output with Thresholds
For large programs, scalene‘s output can get long. Use threshold flags to suppress lines that are below a meaningful contribution — this focuses the report on what actually matters.
# terminal -- only show lines contributing more than 1% CPU
scalene --cpu-percent-threshold 1 cpu_profile_demo.py
# terminal -- only show lines allocating more than 1 MB
scalene --memory-percent-threshold 1 memory_demo.py
# terminal -- combine both thresholds
scalene --cpu-percent-threshold 1 --memory-percent-threshold 1 cpu_profile_demo.py
Output (with threshold — only hot lines shown):
%CPU %Python %native | Memory alloc | line
────────────────────────────────────────────────────
68.00% 68.00% 0.00% | 0.0 2.1 | 12: index += ...
14.00% 14.00% 0.00% | 0.0 2.1 | 17: return [
Setting thresholds to 1-5% is a good starting point for medium-sized programs. For programs with many functions and long runtimes, a 5% threshold cuts out the noise while keeping anything you should care about. The --reduced-profile flag does something similar — it prints only functions and lines that exceed an internal significance threshold, which scalene calculates automatically based on the total runtime.
Profiling Without the Command Line
If you want to profile only a specific section of code rather than an entire script, use the scalene context manager. This is useful when startup and teardown time are irrelevant and you only want to measure the core computation.
# targeted_profile.py
from scalene import scalene_profiler
def heavy_computation(n):
"""Simulate heavy work."""
total = 0
for i in range(n):
total += i * i
return total
def light_setup():
"""Simulate setup we do not want to profile."""
return list(range(100))
def main():
data = light_setup() # Not profiled
# Only the block below is profiled
with scalene_profiler.enable_profiling():
result = heavy_computation(5_000_000)
processed = [x * 2 for x in data]
print(f"Result: {result}")
print(f"Processed: {len(processed)} items")
main()
Run this file normally (not under the scalene command) and scalene will print its report only for the code inside the with block:
# terminal
python targeted_profile.py
Output:
Result: 41666666791666675000
Processed: 100 items
targeted_profile.py: % of time = 100.00% out of 2.14s.
%CPU %Python %native | Memory | line
─────────────────────────────────────────────────────
98.00% 98.00% 0.00% | 0.0 | 10: total += i * i
2.00% 2.00% 0.00% | 0.0 | 20: processed = [x * 2 ...
The context manager approach is cleaner than decorating individual functions with @profile (a pattern from line_profiler that scalene does not require). You get targeted data without changing function signatures or adding class-level decorators.
Real-Life Example: Profiling and Fixing a Data Pipeline
Here is a realistic data processing pipeline that has two deliberate bottlenecks. We will profile it with scalene, identify both issues, fix them, and verify the improvement.
# data_pipeline.py
import urllib.request
import json
def fetch_posts():
"""Fetch 100 posts from a public REST API."""
url = "https://jsonplaceholder.typicode.com/posts"
with urllib.request.urlopen(url) as resp:
return json.loads(resp.read())
def build_word_index(posts):
"""
Build a word frequency index across all post bodies.
Bottleneck 1: String concatenation instead of list join.
"""
combined = ""
for post in posts:
combined += post["body"] + " " # O(n^2) memory pattern
words = combined.lower().split()
freq = {}
for word in words:
freq[word] = freq.get(word, 0) + 1
return freq
def find_top_authors(posts, word_index):
"""
Find which user IDs wrote posts containing the top-10 words.
Bottleneck 2: Repeated substring search instead of set lookup.
"""
top_words = sorted(word_index, key=word_index.get, reverse=True)[:10]
results = {}
for post in posts:
for word in top_words:
# Slow: rebuilding a lowercase string 10x per post
if word in post["body"].lower():
uid = post["userId"]
results.setdefault(uid, set()).add(word)
return results
def main():
posts = fetch_posts()
# Run each function 100x to make profiling differences visible
for _ in range(100):
index = build_word_index(posts)
for _ in range(100):
authors = find_top_authors(posts, index)
print(f"Unique words: {len(index)}")
print(f"Active authors: {len(authors)}")
main()
Output (unprofiled):
Unique words: 638
Active authors: 10
Profile it:
# terminal
scalene --cpu-percent-threshold 2 data_pipeline.py
scalene output (key lines):
74.00% Python | 14: combined += post["body"] + " "
19.00% Python | 28: if word in post["body"].lower():
Two clear targets. Here is the fixed version:
# data_pipeline_fixed.py
import urllib.request
import json
def fetch_posts():
url = "https://jsonplaceholder.typicode.com/posts"
with urllib.request.urlopen(url) as resp:
return json.loads(resp.read())
def build_word_index(posts):
"""Fix 1: Use str.join() instead of concatenation."""
combined = " ".join(post["body"] for post in posts)
words = combined.lower().split()
freq = {}
for word in words:
freq[word] = freq.get(word, 0) + 1
return freq
def find_top_authors(posts, word_index):
"""Fix 2: Pre-lowercase each body once, then check all words."""
top_words = set(
sorted(word_index, key=word_index.get, reverse=True)[:10]
)
results = {}
for post in posts:
body_lower = post["body"].lower() # compute once per post
body_words = set(body_lower.split())
matched = top_words & body_words # set intersection -- O(min(m,n))
if matched:
uid = post["userId"]
results.setdefault(uid, set()).update(matched)
return results
def main():
posts = fetch_posts()
for _ in range(100):
index = build_word_index(posts)
for _ in range(100):
authors = find_top_authors(posts, index)
print(f"Unique words: {len(index)}")
print(f"Active authors: {len(authors)}")
main()
Output:
Unique words: 638
Active authors: 10
Same results. Profile the fixed version to confirm the bottlenecks are gone:
# terminal
scalene --cpu-percent-threshold 2 data_pipeline_fixed.py
scalene output (fixed):
52.00% native | 5: with urllib.request.urlopen(url)...
31.00% Python | 11: words = combined.lower().split()
11.00% Python | 21: body_words = set(body_lower.split())
The two Python bottlenecks are gone. The dominant cost is now the network call (52% native — inside the C HTTP library), which is expected. The word-splitting operations at 31% and 11% are the remaining algorithmic work and are much harder to optimize further without caching. This is what a clean profiling result looks like: the remaining hotspots are either I/O-bound or genuinely algorithmic, not avoidable inefficiency.
Frequently Asked Questions
How much overhead does scalene add to my program?
scalene uses statistical sampling, which means it only interrupts the program periodically to check what is running. In practice this adds 10-20% to wall-clock time for most programs. This compares to 2-10x overhead for cProfile and even higher for memory_profiler. For very short scripts (under one second of runtime), the profiling overhead is proportionally larger, so scalene works best on programs that run for at least a few seconds.
What does “native” CPU time mean and what should I do about it?
Native time is time spent executing C, C++, or Fortran code inside a compiled extension like NumPy, Pandas, PIL, or a similar library. When you see high native time on a line, the bottleneck is inside the library’s compiled code — already highly optimized. The right response is not to rewrite that line in Python (that will make it slower). Instead, look for a different vectorized operation, check if you can reduce the data size passed in, or investigate whether you are calling the library function more times than necessary.
Why does the memory “alloc” column show 0 for many lines?
Memory allocation is also tracked via sampling, so lines that allocate very small amounts or allocate only occasionally between samples will show 0. The alloc column is an average over the samples taken when that line was running — not an exact byte count. If you need exact allocation tracking, combine scalene with tracemalloc (built into the standard library) for a two-tool view. tracemalloc gives exact counts at a higher overhead cost; scalene gives you the approximate picture quickly.
Can scalene profile multiprocessing and multithreaded programs?
scalene can profile multithreaded programs and will show per-thread CPU data. For multiprocessing programs that use multiprocessing.Process, scalene can profile child processes too — pass the --profile-all flag to include all spawned processes in the report. The HTML report then shows data aggregated across all threads and processes, which is useful for spotting work-imbalance issues in parallel code.
When should I use cProfile instead of scalene?
cProfile is deterministic — it records every single function call and is part of Python’s standard library, so there is nothing to install. Use it when you need an exact call count (for algorithmic analysis) or when you are on an environment where you cannot install packages. Use scalene when you want line-level data, memory tracking, or the Python/native time split. For most real-world optimization work, scalene‘s output is more immediately actionable because it points to the exact line rather than the function that contained the slow line.
Conclusion
You now have a complete scalene workflow: install it once, prefix any script to profile it, read the CPU/Python/native columns to distinguish fixable Python overhead from expected C-library time, use the memory columns to find excessive allocations, and export HTML reports when you need to share findings or explore a large codebase interactively. The context manager API gives you surgical profiling of specific code sections when you already know roughly where to look.
The real-life pipeline example shows the full cycle: profile, identify two bottlenecks (string concatenation and repeated .lower() calls), fix both with idiomatic Python, and confirm with a second profile pass that the dominant cost has shifted from avoidable Python overhead to genuine I/O and algorithmic work. That is the end state you are aiming for with any optimization effort.
For more on scalene‘s advanced features — GPU profiling, AI-assisted optimization suggestions, and the web-based UI — see the official scalene repository on GitHub and the PyPI package page for the latest release notes.
Related Articles
- How To Use Python memray for Memory Profiling
- How To Use Python cachetools for In-Memory Caching
- How To Use Python __slots__ to Reduce Memory Usage
- How To Use Python Generators and yield for Memory-Efficient Code
- Understanding Python Memory Management and Garbage Collection
- How To Use Python whenever for Correct Date and Time Handling