Large language models know a lot, but they don’t know your stuff. They don’t know your company’s internal documentation, your product’s support tickets, last quarter’s meeting notes, or the custom knowledge base your team spent three years building. Retrieval-Augmented Generation (RAG) is the engineering pattern that solves this: instead of retraining the model on your data (expensive, slow, quickly outdated), you retrieve the most relevant pieces of your data at query time and inject them into the model’s context window. The model reasons over real information rather than hallucinating from training data.
LangChain is the Python library that makes building RAG pipelines significantly less painful. It provides composable abstractions for document loaders, text splitters, embedding models, vector stores, and retrieval chains — all the components you’d otherwise have to wire together yourself. The abstraction layer also means you can swap OpenAI embeddings for a local model, or swap ChromaDB for Pinecone, without rewriting your pipeline.
In this tutorial you’ll build a complete RAG system from scratch: loading and chunking documents, creating embeddings, storing them in a vector database, and building a question-answering chain that retrieves relevant context and generates grounded answers. By the end you’ll have a working system you can point at your own documents.
Quick Answer RAG = load documents -> split into chunks -> embed chunks -> store in vector DB -> at query time: embed query -> similarity search -> retrieve top-K chunks -> stuff into LLM prompt -> generate answer. LangChain’s RetrievalQA chain handles the retrieval-and-generation step. Use FAISS or ChromaDB for the vector store, OpenAIEmbeddings or a local model for embeddings.
Retrieval-Augmented: because hallucinations are so last quarter
What Is RAG and Why Does It Work?
LLMs generate text by predicting the most likely next token given their training distribution. That training distribution is frozen at the training cutoff date and contains only what was publicly available on the internet. When you ask about your internal documentation, the model has never seen it — so it either says “I don’t know” or, more troublingly, makes something up that sounds plausible.
RAG short-circuits this problem. When a user asks a question, you first search your document database for the most relevant chunks of text. You then include those chunks in the prompt sent to the LLM: “Here is relevant context from our documentation. Using only this context, answer the following question.” The model reasons over real information you’ve provided rather than its training data.
The key technology enabling fast document search is vector embeddings. An embedding model converts text into a dense vector — a list of hundreds or thousands of numbers that encodes the semantic meaning of the text. Two texts that mean similar things will have vectors close to each other in high-dimensional space. You embed all your documents once, store the vectors in a vector database, and at query time embed the question and find the nearest document vectors. This semantic search finds relevant content even when the exact words don’t match.
Our RAG system needs several libraries working together: langchain and langchain-openai for the LLM orchestration layer, langchain-community for document loaders, faiss-cpu for the local vector store, tiktoken for token counting, and pypdf for reading PDF files. Install them all with:
The langchain core package provides the abstractions. langchain-openai adds OpenAI-specific integrations (ChatGPT, embeddings). langchain-community adds community-maintained integrations for document loaders, vector stores, and other tools. faiss-cpu is Facebook’s fast similarity search library for vector storage. tiktoken is OpenAI’s tokenizer library, used internally for accurate chunk sizing. pypdf enables PDF loading.
You’ll need an OpenAI API key. Set it as an environment variable:
export OPENAI_API_KEY="sk-your-key-here" # Linux/Mac
set OPENAI_API_KEY=sk-your-key-here # Windows Command Prompt
Step 1: Loading Documents
LangChain’s document loaders convert files into a standard Document object with page_content (the text) and metadata (source file, page number, etc.):
# rag_system.py
from langchain_community.document_loaders import (
TextLoader,
PyPDFLoader,
DirectoryLoader,
WebBaseLoader,
)
# Load a single text file
loader = TextLoader("company_handbook.txt", encoding="utf-8")
docs = loader.load()
print(f"Loaded {len(docs)} document(s)")
print(f"Content preview: {docs[0].page_content[:200]}")
print(f"Metadata: {docs[0].metadata}")
# Load a PDF (splits by page automatically)
pdf_loader = PyPDFLoader("annual_report.pdf")
pdf_docs = pdf_loader.load()
print(f"PDF has {len(pdf_docs)} pages")
# Load all .txt files from a directory
dir_loader = DirectoryLoader(
path="./documents/",
glob="**/*.txt",
loader_cls=TextLoader,
loader_kwargs={"encoding": "utf-8"}
)
all_docs = dir_loader.load()
print(f"Loaded {len(all_docs)} documents from directory")
# Load web pages
web_loader = WebBaseLoader([
"https://docs.python.org/3/library/functions.html",
"https://docs.python.org/3/library/exceptions.html"
])
web_docs = web_loader.load()
The metadata attached to each document is important — when you retrieve a chunk later, you want to know which document it came from so you can cite your sources. LangChain’s loaders populate source in the metadata automatically for file-based loaders.
Chunk it. Encode it. Retrieve it when it matters.
Step 2: Splitting Documents into Chunks
LLMs have context window limits. You can’t stuff an entire 200-page manual into a prompt — you need to split documents into chunks and retrieve only the relevant ones. Good chunking is more important than most people realize: chunks that are too small lack context; chunks that are too large waste the context window on irrelevant information.
# rag_system.py
from langchain.text_splitter import RecursiveCharacterTextSplitter
# RecursiveCharacterTextSplitter: tries to split on natural boundaries
# (paragraphs, then sentences, then words, then characters)
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000, # Characters per chunk
chunk_overlap=200, # Overlap prevents losing context at boundaries
length_function=len,
separators=["\n\n", "\n", ". ", " ", ""] # Priority order for splitting
)
# Split all loaded documents
chunks = text_splitter.split_documents(all_docs)
print(f"Split {len(all_docs)} documents into {len(chunks)} chunks")
print(f"Average chunk size: {sum(len(c.page_content) for c in chunks) / len(chunks):.0f} chars")
# Inspect a chunk
sample = chunks[5]
print(f"\nChunk content:\n{sample.page_content}")
print(f"\nChunk metadata: {sample.metadata}")
The chunk_overlap parameter is important. When you split at a boundary, you risk losing the context that connects two adjacent chunks. A 200-character overlap ensures each chunk includes the end of the previous chunk, so sentences that span boundaries aren’t orphaned. The tradeoff is slightly more storage and some redundancy in retrieved chunks — worth it for coherence.
Step 3: Creating and Storing Embeddings
Now the important part: convert each chunk into a vector and store them in a vector database. This is the one-time indexing step — you run it when you load new documents, not on every query.
# embeddings.py
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
import os
# Initialize the embedding model
embeddings = OpenAIEmbeddings(
model="text-embedding-3-small", # Cheaper than ada-002, better quality
openai_api_key=os.environ["OPENAI_API_KEY"]
)
# Test the embedding
test_vector = embeddings.embed_query("What is Python used for?")
print(f"Embedding dimensions: {len(test_vector)}") # 1536
# Create the vector store from our chunks
vector_store = FAISS.from_documents(
documents=chunks,
embedding=embeddings
)
# Save to disk so you don't re-embed every time
vector_store.save_local("faiss_index")
print(f"Vector store created with {vector_store.index.ntotal} vectors")
The FAISS.from_documents() call sends each chunk to the OpenAI embeddings API and builds the FAISS index. This is the most API-expensive step — you pay per token for embeddings. For a 100-page PDF (~50,000 tokens), the cost is about $0.001. After saving to disk, you reload it for every subsequent query without re-embedding.
# Loading a saved vector store on subsequent runs
vector_store = FAISS.load_local(
"faiss_index",
embeddings,
allow_dangerous_deserialization=True # Required flag in newer LangChain
)
Your documents, vectorized and ready for interrogation
Step 4: Building the Retriever
A retriever takes a query, embeds it, and returns the most similar document chunks:
# rag_system.py
# Create retriever from vector store
retriever = vector_store.as_retriever(
search_type="similarity",
search_kwargs={
"k": 4, # Return top 4 most relevant chunks
}
)
# Test retrieval directly
query = "How do I request time off?"
relevant_docs = retriever.invoke(query)
print(f"Retrieved {len(relevant_docs)} chunks for query: '{query}'")
for i, doc in enumerate(relevant_docs):
print(f"\n--- Chunk {i+1} (source: {doc.metadata.get('source', 'unknown')}) ---")
print(doc.page_content[:300] + "...")
The search_type="mmr" option uses Maximal Marginal Relevance — it balances relevance with diversity, preventing the retriever from returning four nearly-identical chunks when your question matches a repeated section of the document. For knowledge bases with redundant content, MMR produces better results.
Step 5: Building the RAG Chain
The retrieval chain ties everything together: it retrieves relevant chunks for a query and passes them to the LLM with an appropriate prompt:
# rag_system.py
from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate
# Initialize the LLM
llm = ChatOpenAI(
model="gpt-4o-mini",
temperature=0, # Deterministic answers for factual QA
openai_api_key=os.environ["OPENAI_API_KEY"]
)
# Custom prompt that instructs the model to use only the provided context
qa_prompt = PromptTemplate(
input_variables=["context", "question"],
template="""You are a helpful assistant that answers questions based on the provided context.
If the answer is not contained in the context below, say "I don't have information about that."
Do not make up information.
Context:
{context}
Question: {question}
Answer:"""
)
# Build the QA chain
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff", # "stuff" = put all chunks in one prompt
retriever=retriever,
return_source_documents=True,
chain_type_kwargs={"prompt": qa_prompt}
)
# Ask a question
result = qa_chain.invoke({"query": "What is the policy for remote work?"})
print("Answer:")
print(result["result"])
print("\nSources:")
for doc in result["source_documents"]:
print(f" - {doc.metadata.get('source', 'unknown')}: {doc.page_content[:100]}...")
The chain_type="stuff" approach puts all retrieved chunks into a single prompt. This is simple and works well for small-to-medium retrievals. For larger retrievals or when chunks exceed context limits, use "map_reduce" (summarizes each chunk separately then combines) or "refine" (iteratively refines the answer with each chunk).
The Modern Approach: LCEL Chains
LangChain’s newer “Expression Language” (LCEL) provides a more composable way to build the same pipeline using the pipe operator:
# rag_system.py
from langchain.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
# Define prompt
prompt = ChatPromptTemplate.from_template("""Answer the question based only on the following context.
If you cannot answer from the context, say so clearly.
Context:
{context}
Question: {question}
Answer:""")
def format_docs(docs):
"""Format retrieved documents for injection into prompt."""
return "\n\n".join(
f"[Source: {doc.metadata.get('source', 'unknown')}]\n{doc.page_content}"
for doc in docs
)
# Compose the chain with | operator
rag_chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
# Invoke the chain
answer = rag_chain.invoke("How does the annual review process work?")
print(answer)
LCEL’s pipe syntax makes the data flow explicit: the question goes to both the retriever (to find context) and passthrough (to reach the prompt unchanged), both get formatted into the prompt, the prompt goes to the LLM, and the LLM’s response is parsed to a string. Each | is a step in the pipeline.
LCEL: the pipeline syntax that finally clicked
Adding Conversation History
RAG chains that don’t remember previous questions in a conversation frustrate users. Here’s a conversational RAG chain that maintains chat history:
# rag_system.py
from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory
# Memory stores the conversation history
memory = ConversationBufferMemory(
memory_key="chat_history",
return_messages=True,
output_key="answer"
)
# Conversational retrieval chain
conv_chain = ConversationalRetrievalChain.from_llm(
llm=llm,
retriever=retriever,
memory=memory,
return_source_documents=True,
verbose=False
)
# Multi-turn conversation
questions = [
"What are the company's core values?",
"How do those values apply to customer service?", # References "those values" from above
"What's an example of living one of those values at work?"
]
for question in questions:
result = conv_chain.invoke({"question": question})
print(f"Q: {question}")
print(f"A: {result['answer']}\n")
The memory object accumulates conversation turns and the chain automatically reformulates follow-up questions to be self-contained before retrieval. “How do those values apply?” becomes something like “How do the company’s core values apply to customer service?” before the retriever searches — because “those values” alone wouldn’t find relevant chunks.
Real-Life Example: A Python Documentation Assistant
Here’s a complete, runnable RAG system built over Python’s official documentation:
# real_life_project.py
import os
from pathlib import Path
from langchain_community.document_loaders import WebBaseLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import FAISS
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate
def build_python_docs_assistant():
"""Build a RAG assistant over Python documentation pages."""
print("Loading Python documentation...")
urls = [
"https://docs.python.org/3/library/functions.html",
"https://docs.python.org/3/library/exceptions.html",
"https://docs.python.org/3/library/stdtypes.html",
"https://docs.python.org/3/library/itertools.html",
]
loader = WebBaseLoader(urls)
docs = loader.load()
print(f"Loaded {len(docs)} pages")
splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=150)
chunks = splitter.split_documents(docs)
print(f"Created {len(chunks)} chunks")
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
index_path = "python_docs_index"
if Path(f"{index_path}.faiss").exists():
print("Loading existing index...")
vector_store = FAISS.load_local(index_path, embeddings,
allow_dangerous_deserialization=True)
else:
print("Creating new index...")
vector_store = FAISS.from_documents(chunks, embeddings)
vector_store.save_local(index_path)
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
retriever = vector_store.as_retriever(search_kwargs={"k": 5})
prompt = PromptTemplate(
input_variables=["context", "question"],
template="""You are a Python expert assistant. Answer questions about Python using
only the documentation excerpts provided below. Include relevant function signatures when available.
If the answer isn't in the context, say so.
Documentation excerpts:
{context}
Question: {question}
Answer:"""
)
chain = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=retriever,
return_source_documents=True,
chain_type_kwargs={"prompt": prompt}
)
return chain
def ask(chain, question: str):
"""Ask a question and display the answer with sources."""
print(f"\nQ: {question}")
result = chain.invoke({"query": question})
print(f"A: {result['result']}")
sources = {doc.metadata.get("source", "unknown") for doc in result["source_documents"]}
print(f"Sources: {', '.join(sources)}")
# Build and use the assistant
assistant = build_python_docs_assistant()
ask(assistant, "What does the sorted() function return and how do I use the key parameter?")
ask(assistant, "What is the difference between StopIteration and GeneratorExit exceptions?")
Citations included. Your AI just learned accountability.
Frequently Asked Questions
How much does it cost to build a RAG system with OpenAI?
The main cost is embedding your documents. text-embedding-3-small costs $0.02 per million tokens. A 100-page PDF (~50,000 tokens) costs about $0.001 to embed. Querying costs are minimal — embedding a query is a few hundred tokens. For most internal knowledge bases, the total indexing cost is under $1.
Can I use a local LLM instead of OpenAI?
Yes — swap ChatOpenAI for OllamaLLM (with Ollama running locally) or HuggingFacePipeline. Similarly, swap OpenAIEmbeddings for OllamaEmbeddings or HuggingFaceEmbeddings. The rest of the pipeline stays identical. This is LangChain’s main value proposition — swappable components.
How do I handle documents that update frequently?
Use a vector store that supports upsert operations (ChromaDB, Pinecone). Track a hash or last-modified timestamp for each source document, and re-embed only changed files. For frequently updating data, consider a refresh schedule rather than real-time updates.
What chunk size should I use?
It depends on your content and model context window. A common starting point is 512-1000 characters with 10-20% overlap. For technical documentation with dense information, smaller chunks (256-512) work better. For narrative text, larger chunks (1000-2000) preserve context. Always test with representative queries and adjust based on answer quality.
Why does my RAG system give wrong answers even when the right document is in the database?
The most common causes are: chunks too small (losing context), poor chunk boundaries (splitting mid-sentence), the embedding model not capturing domain-specific terminology well, or not retrieving enough chunks (increase k). Check the retrieved chunks for a failing query — if the right content isn’t being retrieved, the problem is in chunking or retrieval. If it is retrieved but the answer is wrong, the problem is in the prompt or LLM.
What’s the difference between FAISS and ChromaDB?
FAISS is a pure similarity-search library — fast, in-memory, no persistence overhead. ChromaDB is a full vector database with built-in persistence, metadata filtering, and a server mode for multi-process access. FAISS is better for prototyping and read-heavy workloads. ChromaDB is better for production use cases where you need metadata filtering or document updates.
Summary
You’ve built a complete RAG system: document loading, chunking, embedding, vector storage, retrieval, and LLM-powered answer generation. The pipeline turns any document collection into a queryable knowledge base that gives grounded, source-cited answers instead of hallucinations. The LangChain abstractions mean you can swap any component — different embedding models, vector stores, or LLMs — without rewriting the pipeline.
The next level is improving retrieval quality with hybrid search (combining vector search with BM25 keyword search), implementing reranking to improve chunk ordering, and adding metadata filtering to target specific document subsets. For related tutorials, see How To Build a Chatbot with Python and Ollama (local LLMs) and Pydantic V2 Data Validation for structuring the outputs of your RAG chain.
You have probably been there before. You find the perfect website full of data you need, whether it is product prices, job listings, real estate data, or sports statistics. You fire up requests and BeautifulSoup, write a quick script, and run it. The result? An empty page. No data. The HTML source contains nothing but a single <div id="root"></div> and a bunch of JavaScript files. The data you can see in the browser simply does not exist in the raw HTML. Welcome to the world of dynamic websites.
The good news is that Python has a powerful combination of tools to handle exactly this problem. Selenium automates a real web browser, letting it execute JavaScript and render the page just like a human visitor would. Once the content is loaded, BeautifulSoup steps in to parse and extract the data you need. Together, they can scrape virtually any website, no matter how much JavaScript it uses. Both libraries are well-documented, widely used in the industry, and easy to learn.
In this article we will cover everything you need to scrape dynamic websites with confidence. We will start with a quick working example so you can see results in 30 seconds. Then we will walk through the difference between static and dynamic websites, when to use Selenium versus simpler tools, how to install and configure everything, how to wait for content to load properly, how to handle pagination and user interaction, and finally we will build a complete real-life job scraper that exports results to CSV. By the end, you will have a reusable pattern you can adapt to scrape almost any dynamic site.
Scraping a Dynamic Website: Quick Example
Let us start with a complete working example you can copy and run right now. We will scrape quotes.toscrape.com/js/, a practice site that loads famous quotes entirely through JavaScript. If you tried to scrape this page with requests, you would get an empty page because the quotes are injected into the DOM by a script after the page loads. Selenium handles this by running a real browser that executes the JavaScript first.
# quick_scrape.py
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
# Start browser (Chrome required)
driver = webdriver.Chrome()
try:
# Navigate to the JS-rendered quotes page
driver.get("https://quotes.toscrape.com/js/")
# Wait up to 10 seconds for JavaScript to render the quotes
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.CLASS_NAME, "quote")))
# Hand the fully-rendered HTML to BeautifulSoup for parsing
soup = BeautifulSoup(driver.page_source, "html.parser")
# Extract each quote's text and author
quotes = soup.find_all("div", class_="quote")
for quote in quotes[:5]:
text = quote.find("span", class_="text").text
author = quote.find("small", class_="author").text
print(f"{author}: {text[:60]}...")
finally:
driver.quit()
Output:
Albert Einstein: “The world as we have created it is a process of our ...
J.K. Rowling: “It is our choices, Harry, that show what we truly are...
Albert Einstein: “There are only two ways to live your life. One is a...
Jane Austen: “The person, be it gentleman or lady, who has not pleas...
Marilyn Monroe: “Imperfection is beauty, madness is genius and it's ...
Here is the HTML structure of each quote on that page, so you can see exactly what the code is targeting. Keep this snippet as a reference in case the site ever changes its layout:
<!-- HTML structure of each quote on quotes.toscrape.com/js/ -->
<div class="quote" itemscope itemtype="http://schema.org/CreativeWork">
<span class="text" itemprop="text">"The world as we have..."</span>
<span>by
<small class="author" itemprop="author">Albert Einstein</small>
<a href="/author/Albert-Einstein">(about)</a>
</span>
<div class="tags">
<a class="tag" href="/tag/change/page/1/">change</a>
<a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>
</div>
</div>
There are three key things happening in this example. First, Selenium opens a real Chrome browser and navigates to the page, which triggers all the JavaScript to execute. Second, WebDriverWait pauses the script until the quote elements actually appear in the DOM, which is critical because the data is injected asynchronously by JavaScript. Third, once the page is fully rendered, we pass driver.page_source (the complete HTML after JavaScript has run) to BeautifulSoup, which gives us the familiar find_all and find methods for extracting exactly what we need. This three-step pattern of load, wait, and parse is the foundation of every dynamic scraper.
Want to go deeper? Below we cover when you actually need Selenium versus simpler tools, how to configure headless mode for speed, advanced waiting strategies, handling pagination, and a complete real-life project.
What Are Dynamic Websites and Why Do They Need Selenium?
To understand why some websites need Selenium, it helps to know the difference between static and dynamic content. A static website sends all its HTML content in the initial server response. When you fetch the page with requests.get(), you get back the complete page with all the text, links, and data already embedded in the HTML. Most older websites and many simple blogs work this way.
A dynamic website, on the other hand, sends back a mostly empty HTML shell along with JavaScript files. Your browser executes that JavaScript, which then makes additional API calls, processes the responses, and builds the page content on the fly. Modern frameworks like React, Angular, and Vue.js all work this way. When you try to scrape a dynamic site with requests, you get back the empty shell because requests does not execute JavaScript.
Here is a simple way to tell the difference. Open the website in Chrome, right-click, and select “View Page Source.” If you can see all the data you want in the source code, it is a static site and you can use requests and BeautifulSoup alone. If the source code is mostly JavaScript and the data is missing, it is a dynamic site and you need Selenium to render the page first.
Selenium solves this by automating a real browser. It launches Chrome (or Firefox, or Edge), navigates to the URL, and lets the browser do what browsers do: execute JavaScript, make API calls, render the DOM, and display the content. Once the page is fully rendered, Selenium gives you access to the final HTML, which you can then parse with BeautifulSoup just like any static page.
Static sites hand you the data on a silver platter. Dynamic sites make you work for it.
Selenium vs Requests: When to Use Each
Not every scraping job needs Selenium. In fact, using Selenium when you do not need it is a common beginner mistake that makes your scraper 10-50x slower than necessary. The table below will help you choose the right tool for the job.
Feature
requests + BeautifulSoup
Selenium + BeautifulSoup
Speed
Very fast (milliseconds per page)
Slow (seconds per page)
JavaScript support
None
Full browser JavaScript engine
Resource usage
Minimal (no browser)
Heavy (launches full browser)
User interaction
Cannot click, scroll, or type
Full interaction (click, scroll, type, drag)
Best for
Static HTML pages, APIs, RSS feeds
SPAs, JavaScript-rendered content, login walls
Setup complexity
pip install only
Needs browser + WebDriver installed
The best way to see this difference is to try both approaches on the same data. The site quotes.toscrape.com has two versions: a static version where all quotes are in the HTML, and a JavaScript version where they are injected by a script. Let us try scraping the static version with requests first.
# compare_static.py
# APPROACH 1: requests (fast, for static sites)
import requests
from bs4 import BeautifulSoup
response = requests.get("https://quotes.toscrape.com/")
soup = BeautifulSoup(response.text, "html.parser")
quotes = soup.find_all("div", class_="quote")
print(f"[requests] Found {len(quotes)} quotes on static page")
for q in quotes[:3]:
author = q.find("small", class_="author").text
print(f" - {author}")
Output:
[requests] Found 10 quotes on static page
- Albert Einstein
- J.K. Rowling
- Albert Einstein
That took a fraction of a second. Now try the same approach on the JavaScript version, where the quotes are loaded dynamically:
Zero quotes found because requests fetched the raw HTML before JavaScript ran. Now compare with Selenium, which lets the browser execute the JavaScript first:
# compare_dynamic_success.py
# APPROACH 2: Selenium (slower, but handles JavaScript)
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
try:
driver.get("https://quotes.toscrape.com/js/")
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.CLASS_NAME, "quote")))
quotes = driver.find_elements(By.CLASS_NAME, "quote")
print(f"[Selenium] Found {len(quotes)} quotes on JS page")
for q in quotes[:3]:
author = q.find_element(By.CLASS_NAME, "author").text
print(f" - {author}")
finally:
driver.quit()
Output:
[Selenium] Found 10 quotes on JS page
- Albert Einstein
- J.K. Rowling
- Albert Einstein
Both approaches found the same 10 quotes, but the requests version only works on the static page, while Selenium works on both. The trade-off is speed: requests ran in about 0.2 seconds while Selenium took 3-4 seconds because it had to launch Chrome, navigate to the page, and wait for JavaScript to execute. Always try requests first and only reach for Selenium when you confirm the content is loaded dynamically.
requests: 0.2 seconds. Selenium: “hold on, I’m launching an entire browser real quick.”
Installing Selenium and ChromeDriver
Before you can start scraping dynamic websites, you need to install two things: the Selenium Python library and a WebDriver that matches your browser. The WebDriver is a separate executable that Selenium uses to communicate with the browser. We will use Chrome and ChromeDriver since Chrome is the most popular choice, but Selenium also supports Firefox (geckodriver), Edge (msedgedriver), and Safari.
Starting with Selenium 4.6+, you no longer need to manually download ChromeDriver. Selenium Manager handles it automatically. Here is how to get everything set up and verify it works.
# install_and_verify.py
# Step 1: Install the required packages
# Run this in your terminal:
# pip install selenium beautifulsoup4
# Step 2: Verify the installation
from selenium import webdriver
print(f"Selenium version: {webdriver.__version__}")
# Step 3: Test that ChromeDriver works
try:
driver = webdriver.Chrome() # Selenium Manager downloads ChromeDriver automatically
print("Chrome WebDriver is working!")
print(f"Browser version: {driver.capabilities['browserVersion']}")
driver.quit()
except Exception as e:
print(f"Error: {e}")
print("If ChromeDriver is not found, install Chrome browser first.")
Output:
Selenium version: 4.18.0
Chrome WebDriver is working!
Browser version: 122.0.6261.94
If you see the success message, you are ready to go. If you get an error about ChromeDriver not being found, make sure you have Google Chrome installed on your system. Selenium Manager will handle the rest. For older versions of Selenium (before 4.6), you would need to manually download ChromeDriver from the ChromeDriver website and either place it in your system PATH or specify the path in your code.
pip install selenium — the two most exciting words in web scraping.
Loading Dynamic Pages With Selenium
Once Selenium is installed, the next step is learning how to load pages and configure the browser for scraping. The most important configuration option is headless mode, which runs Chrome without opening a visible window. This is faster, uses less memory, and is essential for running scrapers on servers or in automated pipelines.
The code below demonstrates a complete setup with headless mode, proper error handling, and the key techniques for loading dynamic content. We will use quotes.toscrape.com/js/ again since it is a reliable, publicly available dynamic site that anyone can scrape without restrictions.
# load_dynamic_page.py
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
import time
# Configure Chrome for headless mode (no visible window)
chrome_options = Options()
chrome_options.add_argument("--headless") # Run without GUI
chrome_options.add_argument("--no-sandbox") # Required for some Linux environments
chrome_options.add_argument("--disable-dev-shm-usage") # Prevent memory issues
driver = webdriver.Chrome(options=chrome_options)
try:
# Navigate to the JS-rendered quotes page
driver.get("https://quotes.toscrape.com/js/")
print(f"Page title: {driver.title}")
# Create a reusable wait object (max 10 seconds)
wait = WebDriverWait(driver, 10)
# Wait for the quote containers to appear in the DOM
quotes = wait.until(
EC.presence_of_all_elements_located((By.CLASS_NAME, "quote"))
)
print(f"Found {len(quotes)} quotes after JS rendering")
# Scroll to bottom to check for lazy-loaded content
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(1) # Give lazy-loaded content time to appear
# Grab the fully rendered HTML for parsing
html = driver.page_source
print(f"Page source retrieved ({len(html):,} bytes)")
# Quick preview of what we got
first_quote = quotes[0].find_element(By.CLASS_NAME, "text").text
print(f"First quote: {first_quote[:50]}...")
finally:
driver.quit()
Output:
Page title: Quotes to Scrape
Found 10 quotes after JS rendering
Page source retrieved (11,254 bytes)
First quote: “The world as we have created it is a process...
There are a few important things to notice here. The --headless argument is what makes Chrome run invisibly in the background. The WebDriverWait object is reusable and takes a maximum timeout in seconds. If the element does not appear within that time, Selenium raises a TimeoutException, which is much better than guessing with time.sleep(). The execute_script call at the end scrolls the page to the bottom, which triggers lazy-loaded content on many modern websites. After scrolling, a brief sleep gives the new content time to render before we grab page_source.
Headless mode: all the power of a real browser, none of the window dressing.
Parsing Rendered HTML With BeautifulSoup
Once Selenium has loaded and rendered the page, you need to extract the specific data points you care about. This is where BeautifulSoup shines. You pass driver.page_source (the fully rendered HTML) to BeautifulSoup, and then use its familiar find, find_all, and CSS selector methods to navigate the DOM tree and pull out text, attributes, and links.
The key skill here is defensive parsing. Real websites are messy. Elements might be missing on some items, classes might change, or content might be empty. Always check that an element exists before calling .text on it, or you will get AttributeError crashes in production. The example below shows how to safely extract multiple fields from each quote, including tags that may not exist on every entry.
# parse_quotes.py
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
driver = webdriver.Chrome()
try:
driver.get("https://quotes.toscrape.com/js/")
# Wait for quote elements to load
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.CLASS_NAME, "quote")))
# Parse the fully rendered page with BeautifulSoup
soup = BeautifulSoup(driver.page_source, "html.parser")
# Find every quote container on the page
quotes = soup.find_all("div", class_="quote")
print(f"Found {len(quotes)} quotes\n")
# Extract data from each quote with defensive checks
for quote in quotes:
text_elem = quote.find("span", class_="text")
author_elem = quote.find("small", class_="author")
tags_container = quote.find("div", class_="tags")
# Use conditional expressions to handle missing elements gracefully
text = text_elem.text.strip() if text_elem else "Unknown"
author = author_elem.text.strip() if author_elem else "Unknown"
# Extract all tag links, default to empty list if container missing
if tags_container:
tags = [tag.text for tag in tags_container.find_all("a", class_="tag")]
else:
tags = []
print(f"{author}")
print(f" {text[:70]}...")
print(f" Tags: {', '.join(tags) if tags else 'none'}")
print()
finally:
driver.quit()
Output:
Found 10 quotes
Albert Einstein
“The world as we have created it is a process of our thinking. It cann...
Tags: change, deep-thoughts, thinking, world
J.K. Rowling
“It is our choices, Harry, that show what we truly are, far more than...
Tags: abilities, choices
Albert Einstein
“There are only two ways to live your life. One is as though nothing ...
Tags: inspirational, life, live, miracle, miracles
Jane Austen
“The person, be it gentleman or lady, who has not pleasure in a good ...
Tags: aliteracy, books, classic, humor
Notice the defensive pattern on each field: text_elem.text.strip() if text_elem else "Unknown". This ensures your scraper keeps running even when individual items are missing a field. The tags extraction shows a more advanced pattern where we first check if the container exists, then extract all child links from it. In real-world scraping, you will encounter incomplete data constantly, and defensive parsing is what separates a scraper that crashes on page 3 from one that runs reliably across thousands of pages.
Defensive parsing: because real-world HTML is a construction site, not a museum.
Advanced: Waiting Strategies for Dynamic Content
The single biggest source of bugs in web scraping is timing. Your script runs faster than the browser can render content, so you need to explicitly tell Selenium to wait for specific conditions before proceeding. Selenium provides several built-in wait conditions through the expected_conditions module (commonly imported as EC). Understanding which condition to use and when is essential for writing reliable scrapers.
There are three types of waits you should know about. Implicit waits set a global timeout that applies to every element lookup. Explicit waits (using WebDriverWait) wait for a specific condition on a specific element. time.sleep() is the brute-force approach that pauses for a fixed number of seconds regardless of whether the element loaded instantly or not. You should almost always prefer explicit waits because they are both faster (they return as soon as the condition is met) and more reliable (they fail clearly with a timeout error if something goes wrong).
# wait_strategies.py
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
try:
driver.get("https://quotes.toscrape.com/js/")
wait = WebDriverWait(driver, 10)
# Wait for element to exist in the DOM (even if hidden)
element = wait.until(
EC.presence_of_element_located((By.CLASS_NAME, "quote"))
)
print("Quote element is in DOM")
# Wait for element to be visible on screen
element = wait.until(
EC.visibility_of_element_located((By.CLASS_NAME, "quote"))
)
print("Quote element is visible")
# Wait for the "Next" link to be clickable (visible + enabled)
next_link = wait.until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "li.next a"))
)
print("Next link is clickable")
# Click to go to page 2 and wait for new quotes to load
next_link.click()
new_quotes = wait.until(
EC.presence_of_all_elements_located((By.CLASS_NAME, "quote"))
)
print(f"Page 2 loaded with {len(new_quotes)} quotes")
# Verify we are on page 2 by checking the first author
first_author = new_quotes[0].find_element(By.CLASS_NAME, "author").text
print(f"First author on page 2: {first_author}")
finally:
driver.quit()
Output:
Quote element is in DOM
Quote element is visible
Next link is clickable
Page 2 loaded with 10 quotes
First author on page 2: Dr. Seuss
The difference between presence_of_element_located and visibility_of_element_located is subtle but important. Presence means the element exists in the HTML DOM, even if it is hidden with CSS (display: none). Visibility means the element is both present AND visible on screen. For scraping, you usually want presence since you care about the data being in the DOM, not whether it is visually displayed. For interaction (like clicking buttons), use element_to_be_clickable which ensures the element is both visible and enabled.
Handling Pagination and Page Interaction
Many websites split their content across multiple pages. To scrape all the data, you need to navigate through each page, extract the content, and move to the next one. This is where Selenium really shines over requests, because you can click “Next” buttons, scroll through infinite-scroll pages, and interact with filters and search forms just like a human user would.
The site quotes.toscrape.com/js/ has 10 pages of quotes with a “Next” link at the bottom. The example below demonstrates a paginated scraper that clicks through the first three pages, collecting all the quotes from each one into a single list. Pay attention to the error handling on the “next page” link, which gracefully handles the case where there are no more pages to navigate.
# paginated_scraper.py
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
driver = webdriver.Chrome()
all_quotes = []
try:
driver.get("https://quotes.toscrape.com/js/")
wait = WebDriverWait(driver, 10)
page_num = 1
while page_num <= 3:
print(f"Scraping page {page_num}...")
# Wait for the quote elements to load on the current page
wait.until(
EC.presence_of_all_elements_located((By.CLASS_NAME, "quote"))
)
# Parse the current page with BeautifulSoup
soup = BeautifulSoup(driver.page_source, "html.parser")
quotes = soup.find_all("div", class_="quote")
# Extract data from each quote
for quote in quotes:
text = quote.find("span", class_="text").text
author = quote.find("small", class_="author").text
all_quotes.append({"text": text, "author": author})
print(f" Found {len(quotes)} quotes on page {page_num}")
# Try to click the "Next" link to go to the next page
try:
next_link = driver.find_element(By.CSS_SELECTOR, "li.next a")
next_link.click()
# Wait for the page to reload with new quotes
wait.until(EC.staleness_of(
driver.find_element(By.CLASS_NAME, "quote")
))
wait.until(
EC.presence_of_element_located((By.CLASS_NAME, "quote"))
)
page_num += 1
except Exception:
print(" No more pages available")
break
# Print summary
print(f"\nTotal quotes collected: {len(all_quotes)}")
for item in all_quotes[:3]:
print(f" {item['author']}: {item['text'][:50]}...")
finally:
driver.quit()
Output:
Scraping page 1...
Found 10 quotes on page 1
Scraping page 2...
Found 10 quotes on page 2
Scraping page 3...
Found 10 quotes on page 3
Total quotes collected: 30
Albert Einstein: “The world as we have created it is a process of...
J.K. Rowling: “It is our choices, Harry, that show what we truly...
Albert Einstein: “There are only two ways to live your life. One ...
The try/except block around the "next page" link is crucial. When you reach the last page, the "Next" link disappears, and find_element will raise a NoSuchElementException. By catching this exception, the scraper gracefully exits the loop instead of crashing. Notice the staleness_of wait after clicking: this waits until the old quote element goes stale (meaning the page has started reloading), and then we wait for new quotes to appear. This two-step wait pattern is more reliable than a simple time.sleep() because it handles both fast and slow page loads correctly.
Page 1... page 2... page 47... this is fine.
Real-Life Example: Scraping Job Listings to CSV
Now let us put everything together into a production-quality scraper. We will scrape realpython.github.io/fake-jobs/, a static job board created by Real Python for learning purposes. While this particular site does not require Selenium (it is static HTML), the scraper below is written with the full Selenium pattern so you can adapt it to any real dynamic job board like Indeed, LinkedIn, or Glassdoor by changing the URL and CSS selectors. The techniques, class structure, and CSV export are all production-ready.
From chaos to CSV: turning 100 job listings into structured data, one scrape at a time.
Here is the HTML structure of each job card on that page, so you know what we are targeting and can adapt the selectors if the site changes:
<!-- HTML structure of each job card on realpython.github.io/fake-jobs/ -->
<div class="card">
<div class="card-content">
<div class="media-content">
<h2 class="title is-5">Senior Python Developer</h2>
<h3 class="subtitle is-6 company">Payne, Roberts and Davis</h3>
</div>
<div class="content">
<p class="location">Stewartbury, AA</p>
<footer>
<a class="card-footer-item" href="...">Apply</a>
</footer>
</div>
</div>
</div>
Notice how the class encapsulates all the browser setup, scraping logic, and export functionality into clean methods. The scrape_jobs method handles the Selenium interaction, while save_to_csv handles the data export. This separation makes the code easy to extend and reuse for any job board.
# job_scraper.py
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup
from datetime import datetime
import csv
class JobScraper:
"""Scrapes job listings from a job board using Selenium."""
def __init__(self, url):
# Configure headless Chrome for silent operation
options = Options()
options.add_argument("--headless")
options.add_argument("--no-sandbox")
self.driver = webdriver.Chrome(options=options)
self.url = url
self.jobs = []
def scrape_jobs(self, keyword=""):
"""Navigate to job board, extract listings, optionally filter by keyword."""
try:
self.driver.get(self.url)
print(f"Navigating to {self.url}...")
wait = WebDriverWait(self.driver, 15)
# Wait for the job cards to render
wait.until(
EC.presence_of_all_elements_located((By.CLASS_NAME, "card-content"))
)
# Parse the fully loaded page
soup = BeautifulSoup(self.driver.page_source, "html.parser")
job_cards = soup.find_all("div", class_="card-content")
print(f"Found {len(job_cards)} job listings...")
# Extract structured data from each job card
for card in job_cards:
title_elem = card.find("h2", class_="title")
company_elem = card.find("h3", class_="company")
location_elem = card.find("p", class_="location")
link_elem = card.find("a", string="Apply")
title = title_elem.text.strip() if title_elem else "N/A"
company = company_elem.text.strip() if company_elem else "N/A"
location = location_elem.text.strip() if location_elem else "N/A"
apply_url = link_elem.get("href") if link_elem else "#"
# Filter by keyword if provided
if keyword and keyword.lower() not in title.lower():
continue
self.jobs.append({
"title": title,
"company": company,
"location": location,
"apply_url": apply_url,
"scraped_at": datetime.now().isoformat()
})
return self.jobs
finally:
self.driver.quit()
def save_to_csv(self, filename="jobs.csv"):
"""Export scraped jobs to a CSV file."""
if not self.jobs:
print("No jobs to save")
return
with open(filename, "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(
f,
fieldnames=["title", "company", "location", "apply_url", "scraped_at"]
)
writer.writeheader()
writer.writerows(self.jobs)
print(f"Saved {len(self.jobs)} jobs to {filename}")
# Run the scraper
scraper = JobScraper("https://realpython.github.io/fake-jobs/")
jobs = scraper.scrape_jobs(keyword="Python")
print(f"\n=== Found {len(jobs)} Python Jobs ===")
for job in jobs[:5]:
print(f"\n{job['title']}")
print(f" Company: {job['company']}")
print(f" Location: {job['location']}")
scraper.save_to_csv("python_jobs.csv")
Output:
Navigating to https://realpython.github.io/fake-jobs/...
Found 100 job listings...
=== Found 10 Python Jobs ===
Senior Python Developer
Company: Payne, Roberts and Davis
Location: Stewartbury, AA
Python Programmer (Entry-Level)
Company: Richards, Bates and Johnson
Location: North Tylermouth, AA
Python Developer
Company: Wright, Patterson and Thomas
Location: Lake Marytown, AA
Python Programmer
Company: Garcia PLC
Location: Katherineberg, AA
Software Developer (Python)
Company: Villanueva, Sanders and Black
Location: Browntown, AA
Saved 10 jobs to python_jobs.csv
This scraper demonstrates several production patterns. The class-based design makes it easy to reuse and extend. The keyword filter shows how to narrow results without relying on the site having a search function. The defensive parsing with if elem else default handles missing data gracefully. The timestamp field lets you track when each listing was scraped, which is useful for monitoring job markets over time. To adapt this scraper for a real job board like Indeed or LinkedIn, you would change the URL, update the CSS selectors to match that site's HTML structure, and possibly add pagination logic from the previous section.
Frequently Asked Questions
Why is my Selenium scraper so slow?
Selenium is inherently slower than requests because it launches a full web browser for every scraping session. However, there are several ways to speed it up significantly. First, always use headless mode (--headless) since rendering a visible window is unnecessary overhead. Second, disable image loading with --blink-settings=imagesEnabled=false to skip downloading large image files. Third, replace time.sleep() calls with explicit WebDriverWait conditions, which return as soon as the element appears rather than waiting a fixed amount of time. Finally, check whether the data you need is actually loaded via a hidden API call. You can inspect the browser's Network tab to find JSON API endpoints that return the data directly, which would let you use requests instead of Selenium entirely.
How do I handle JavaScript alerts and pop-ups?
JavaScript alerts, confirm dialogs, and cookie consent banners are common obstacles when scraping. For native browser alerts (the ones that pause JavaScript execution), Selenium provides the Alert class. You call Alert(driver) to get a reference to the alert, then .accept() to click OK or .dismiss() to click Cancel. For cookie consent banners and other overlay pop-ups that are just HTML elements, you can use regular Selenium selectors to find the "Accept" or "Close" button and click it. If a pop-up is blocking your scraper, wrapping the dismissal code in a try/except block lets you handle cases where the pop-up does not appear.
How do I avoid getting blocked while scraping?
Websites use several techniques to detect and block scrapers. The most effective countermeasure is to behave like a real user. Add random delays between requests using time.sleep(random.uniform(2, 5)) instead of fixed intervals. Rotate your User-Agent header to mimic different browsers. If you are scraping at volume, consider using a proxy rotation service to distribute requests across different IP addresses. Most importantly, always check the website's robots.txt file and terms of service. Respecting rate limits and scraping policies is not just ethical, it also prevents your IP from getting permanently banned.
How do I scrape pages that require logging in?
Selenium can automate the login process just like a human user. Navigate to the login page, find the username and password input fields using find_element, type your credentials with send_keys(), and submit the form by pressing Enter or clicking the submit button. After login, the browser session maintains your cookies and authentication state, so subsequent page navigations will be authenticated. For sites with two-factor authentication, you may need to add a manual pause (input("Press Enter after completing 2FA...")) so you can handle the verification step yourself.
What is the difference between find_element and find_elements?
find_element (singular) returns the first matching element on the page. If no match exists, it immediately raises a NoSuchElementException. find_elements (plural) returns a Python list of all matching elements. If no matches exist, it returns an empty list instead of raising an error. Use find_element when you expect exactly one match (like a page title or a specific button). Use find_elements when you expect multiple matches (like all the products on a page) or when you want to check whether an element exists without triggering an exception.
Conclusion
Scraping dynamic websites does not have to be intimidating. The core pattern is the same every time: use Selenium to load the page and execute JavaScript, wait for the content you need to appear, then hand the rendered HTML to BeautifulSoup for parsing. We covered how to distinguish static from dynamic websites, when to use requests versus Selenium, how to configure headless Chrome, how to wait for elements reliably with WebDriverWait and expected_conditions, how to navigate paginated content, and how to build a complete job scraper with CSV export.
The job scraper example is a solid starting point that you can adapt for your own projects. Try extending it to scrape a different website, add database storage with SQLite, or build a scheduled scraper that runs daily and alerts you when new listings match your criteria. The techniques in this article apply to any dynamic website, from e-commerce platforms to social media dashboards to real estate listings.
You write a function that takes a user dictionary and returns their email. It works perfectly — until three months later when someone passes a list of users instead of a single user, and the function silently returns garbage instead of crashing with a helpful error. This is the kind of bug that type hints prevent. They let you declare exactly what types your functions expect and return, so tools like mypy can catch mistakes before your code ever runs.
The best part is that type hints are built into Python 3.5+ and require zero extra installation for basic use. They are completely optional — Python does not enforce them at runtime — but they serve as living documentation that IDEs and type checkers can validate automatically. If you use VS Code or PyCharm, you are already getting type hint benefits through autocomplete and inline error detection.
In this article we will start with a quick example so you can see the value immediately. Then we will cover basic type hint syntax for variables and functions, collection types like lists and dictionaries, Optional and Union types for flexible parameters, type hints for classes, and how to run mypy to catch type errors statically. We will finish with a real-life project that refactors untyped code into a fully type-safe inventory management system.
Python Type Hints: Quick Example
Here is a simple function with type hints that calculates a discounted price. The hints tell you (and your tools) exactly what goes in and what comes out, with no guessing required.
The price: float annotation says the first argument should be a float, quantity: int says the second should be an integer, and -> float after the parentheses says the function returns a float. If someone tries to call calculate_total("free", 3), a type checker like mypy will flag it as an error before the code runs. Notice that Python itself will not raise an error — type hints are advisory, not enforced — but the tooling catches it for you.
Want to go deeper? Below we cover every type hint pattern you will need in real projects, from basic annotations to generics and TypedDict.
What Are Type Hints and Why Use Them?
Type hints (also called type annotations) are a way to declare the expected types of variables, function parameters, and return values in Python. They were introduced in PEP 484 (Python 3.5) and have been expanded in every Python release since. Think of them as labels on boxes — the label says “contains integers” but Python does not actually check whether you put strings in the box. External tools like mypy, pyright, and your IDE do the checking for you.
Here is why type hints matter in practice:
Benefit
Without Type Hints
With Type Hints
Reading code
Guess what data contains from context
data: dict[str, list[int]] tells you exactly
IDE support
Limited autocomplete, no inline errors
Full autocomplete, real-time error detection
Bug detection
Bugs found at runtime (or in production)
Bugs caught before code runs via mypy
Refactoring
Change a function signature, hope nothing breaks
mypy shows every caller that needs updating
Documentation
Write docstrings that go stale
Types are always accurate (enforced by tools)
Type hints do not affect performance — Python ignores them at runtime. They also do not make Python a statically typed language. You can still write untyped code, and typed and untyped code can coexist in the same project. The value comes from the tooling ecosystem that reads and validates your annotations. Let us start with the basic syntax.
Basic Type Hint Syntax
The fundamental pattern is simple: add a colon and a type after variable names or parameters, and use -> to annotate return types. Here are the most common basic types you will use every day.
# basic_types.py
# Variable annotations
name: str = "Alice"
age: int = 30
price: float = 19.99
is_active: bool = True
# Function with type hints
def greet(name: str, excited: bool = False) -> str:
"""Return a greeting message."""
if excited:
return f"Hello, {name}! Welcome!"
return f"Hello, {name}."
# Function that returns nothing
def log_message(message: str) -> None:
"""Print a log message. Returns nothing."""
print(f"[LOG] {message}")
# Test the functions
print(greet("Bob"))
print(greet("Charlie", excited=True))
log_message("Server started")
Output:
Hello, Bob.
Hello, Charlie! Welcome!
[LOG] Server started
The four basic types — str, int, float, and bool — cover most simple cases. Use -> None for functions that do not return a value (like functions that only print or write to a file). Default values work normally alongside type hints — excited: bool = False means the parameter is a boolean that defaults to False. One important note: int is compatible with float in type checking, so a function annotated with float will accept integers without complaint.
list[str] instead of just data. Your future self will send a thank-you card.
Collection Types
Real code rarely works with single values — you pass lists, dictionaries, sets, and tuples everywhere. Type hints for collections tell you not just that something is a list, but what the list contains. Since Python 3.9, you can use the built-in collection types directly (lowercase list, dict, set, tuple). For Python 3.8 and earlier, import the capitalized versions from the typing module.
# collection_types.py
# Lists — specify what's inside
names: list[str] = ["Alice", "Bob", "Charlie"]
scores: list[int] = [95, 87, 92, 78]
# Dictionaries — specify key and value types
user_ages: dict[str, int] = {"Alice": 30, "Bob": 25}
config: dict[str, str | int | bool] = {
"host": "localhost",
"port": 8080,
"debug": True
}
# Sets — specify element type
unique_tags: set[str] = {"python", "tutorial", "beginner"}
# Tuples — specify each position's type
coordinates: tuple[float, float] = (40.7128, -74.0060)
rgb_color: tuple[int, int, int] = (255, 128, 0)
# Function using collection types
def get_top_students(
grades: dict[str, float],
threshold: float = 90.0
) -> list[str]:
"""Return names of students scoring above the threshold."""
return [name for name, grade in grades.items() if grade >= threshold]
# Test
class_grades = {"Alice": 95.5, "Bob": 82.0, "Charlie": 91.3, "Diana": 88.7}
top = get_top_students(class_grades)
print(f"Top students: {top}")
Output:
Top students: ['Alice', 'Charlie']
The key insight is that list[str] is much more informative than just list. When your IDE sees list[str], it knows that iterating over the list yields strings, so it can offer string methods in autocomplete. The dict[str, float] annotation tells tools that keys are strings and values are floats — so grades["Alice"] is a float and grades.keys() returns strings. For tuples, you specify the type of each position because tuples are often used as fixed-size records (like coordinates or RGB values). If you want a variable-length tuple of one type, use tuple[int, ...] with an ellipsis.
Optional and Union Types
Sometimes a value can be one of several types, or it might be None. Python 3.10 introduced the | (pipe) operator for union types, which is the cleanest syntax. For earlier versions, use Union and Optional from the typing module.
# optional_union.py
from typing import Optional
# Union type: can be str or int (Python 3.10+ syntax)
user_id: str | int = "user_123"
user_id = 456 # Also valid
# Optional: can be the type or None
# Optional[str] is exactly the same as str | None
middle_name: Optional[str] = None
def find_user(user_id: str | int) -> dict[str, str] | None:
"""Look up a user by string or numeric ID. Returns None if not found."""
users = {
"user_123": {"name": "Alice", "email": "alice@mail.com"},
456: {"name": "Bob", "email": "bob@mail.com"},
}
return users.get(user_id)
def format_name(first: str, last: str, middle: Optional[str] = None) -> str:
"""Format a full name, optionally including a middle name."""
if middle:
return f"{first} {middle} {last}"
return f"{first} {last}"
# Test
print(find_user("user_123"))
print(find_user(999))
print(format_name("John", "Doe"))
print(format_name("John", "Doe", middle="Michael"))
Output:
{'name': 'Alice', 'email': 'alice@mail.com'}
None
John Doe
John Michael Doe
The str | int syntax means the value can be either a string or an integer. The Optional[str] type is shorthand for str | None — use it when a parameter or return value might be None. This is one of the most important patterns in type-hinted code because None is the source of countless AttributeError exceptions. When mypy sees that find_user returns dict | None, it will force you to check for None before accessing dictionary keys — catching potential crashes at analysis time instead of runtime.
Optional[str] means it could be a string or None. Welcome to the guessing game that type hints eliminate.
Type Hints for Classes
Type hints work seamlessly with your own classes. You annotate instance attributes, method parameters, and return types just like regular functions. The dataclasses module makes this especially clean because it uses type hints as the primary way to define fields.
# class_types.py
from dataclasses import dataclass
from datetime import datetime
@dataclass
class Product:
"""A product in an inventory system."""
name: str
price: float
quantity: int
category: str = "General"
def total_value(self) -> float:
"""Calculate the total inventory value for this product."""
return round(self.price * self.quantity, 2)
def apply_discount(self, percent: float) -> float:
"""Return the discounted price without modifying the original."""
return round(self.price * (1 - percent / 100), 2)
@dataclass
class Order:
"""A customer order containing multiple products."""
order_id: str
items: list[Product]
created_at: datetime
def grand_total(self) -> float:
"""Calculate the total cost of all items in the order."""
return round(sum(item.price * item.quantity for item in self.items), 2)
def item_count(self) -> int:
"""Return the total number of items across all products."""
return sum(item.quantity for item in self.items)
# Test
laptop = Product("Laptop", 999.99, 5, "Electronics")
mouse = Product("Mouse", 29.99, 20)
order = Order(
order_id="ORD-001",
items=[laptop, mouse],
created_at=datetime.now()
)
print(f"Laptop value: ${laptop.total_value()}")
print(f"Laptop at 10% off: ${laptop.apply_discount(10)}")
print(f"Order total: ${order.grand_total()}")
print(f"Order items: {order.item_count()}")
Output:
Laptop value: $4999.95
Laptop at 10% off: $899.99
Order total: $5599.75
Order items: 25
The @dataclass decorator reads your type annotations and automatically generates __init__, __repr__, and __eq__ methods. This means the type hints are not just documentation — they directly define your class structure. When you annotate items: list[Product] on the Order class, your IDE knows that iterating over self.items yields Product objects, so it can autocomplete item.price, item.quantity, and all other Product attributes. Without the type hint, your IDE would have no idea what item is inside the loop.
Running mypy to Catch Type Errors
Type hints only reach their full potential when paired with a type checker. mypy is the most popular one — it reads your annotations and reports any inconsistencies without running your code. Install it with pip install mypy, then run it on your Python files.
# mypy_demo.py
# Save this file and run: mypy mypy_demo.py
def add_numbers(a: int, b: int) -> int:
"""Add two integers together."""
return a + b
def get_username(user: dict[str, str]) -> str:
"""Extract the username from a user dictionary."""
return user["username"]
# These lines have type errors that mypy will catch:
result = add_numbers(10, "20") # Error: str is not int
total = add_numbers(10, 20) + "!" # Error: can't add int and str
name = get_username(["Alice"]) # Error: list is not dict
Output (from running mypy mypy_demo.py):
mypy_demo.py:12: error: Argument 2 to "add_numbers" has incompatible type "str"; expected "int" [arg-type]
mypy_demo.py:13: error: Unsupported operand types for + ("int" and "str") [operator]
mypy_demo.py:14: error: Argument 1 to "get_username" has incompatible type "list[str]"; expected "dict[str, str]" [arg-type]
Found 3 errors in 1 file (checked 1 source file)
Every error message includes the file name, line number, a clear explanation of what is wrong, and the error code in brackets. The [arg-type] errors mean you passed the wrong type to a function parameter, and [operator] means you used an operator with incompatible types. These are bugs that would have crashed at runtime — mypy catches them instantly. You can add mypy to your CI/CD pipeline so that type errors block pull requests, or configure your IDE to run it on every save. For large existing codebases, you can adopt type hints gradually — mypy only checks files that have annotations and ignores untyped code by default.
mypy catches bugs before your code even runs. That’s not a test — it’s a time machine.
Real-Life Example: Type-Safe Inventory System
Type hints are like a golden shield for your code — they protect you from bugs before they even happen!
Let us build a practical project that shows how type hints improve a real codebase. This inventory management system tracks products, processes orders, and generates reports — all with complete type safety. Every function clearly declares what it expects and returns, so mypy can verify the entire system is consistent.
# inventory_system.py
from dataclasses import dataclass, field
from datetime import datetime
from typing import Optional
@dataclass
class Product:
sku: str
name: str
price: float
stock: int
category: str
@dataclass
class OrderItem:
product: Product
quantity: int
def subtotal(self) -> float:
return round(self.product.price * self.quantity, 2)
@dataclass
class Inventory:
products: dict[str, Product] = field(default_factory=dict)
order_history: list[list[OrderItem]] = field(default_factory=list)
def add_product(self, product: Product) -> None:
self.products[product.sku] = product
def find_product(self, sku: str) -> Optional[Product]:
return self.products.get(sku)
def process_order(self, items: list[OrderItem]) -> float | None:
# Verify stock for all items first
for item in items:
if item.product.stock < item.quantity:
print(f"Insufficient stock for {item.product.name}")
return None
# Deduct stock and calculate total
total: float = 0.0
for item in items:
item.product.stock -= item.quantity
total += item.subtotal()
self.order_history.append(items)
return round(total, 2)
def low_stock_report(self, threshold: int = 10) -> list[tuple[str, str, int]]:
return [
(p.sku, p.name, p.stock)
for p in self.products.values()
if p.stock <= threshold
]
def sales_summary(self) -> dict[str, float]:
summary: dict[str, float] = {}
for order in self.order_history:
for item in order:
category = item.product.category
summary[category] = summary.get(category, 0.0) + item.subtotal()
return {k: round(v, 2) for k, v in summary.items()}
# --- Demo ---
inv = Inventory()
inv.add_product(Product("LAP-001", "Laptop Pro", 1299.99, 15, "Electronics"))
inv.add_product(Product("MOU-001", "Wireless Mouse", 24.99, 50, "Accessories"))
inv.add_product(Product("KEY-001", "Mechanical Keyboard", 89.99, 8, "Accessories"))
inv.add_product(Product("MON-001", "4K Monitor", 449.99, 3, "Electronics"))
# Process an order
laptop = inv.find_product("LAP-001")
mouse = inv.find_product("MOU-001")
if laptop and mouse:
order_items = [OrderItem(laptop, 2), OrderItem(mouse, 5)]
total = inv.process_order(order_items)
print(f"Order total: ${total}")
# Low stock report
print(f"\nLow stock items:")
for sku, name, stock in inv.low_stock_report():
print(f" {sku}: {name} ({stock} remaining)")
# Sales summary
print(f"\nSales by category: {inv.sales_summary()}")
This project demonstrates several important type hint patterns working together. The find_product method returns Optional[Product], which forces callers to check for None before using the result — notice the if laptop and mouse: guard before processing the order. The process_order method returns float | None, signaling that it can fail (returning None when stock is insufficient). The low_stock_report returns list[tuple[str, str, int]], which lets you unpack each tuple directly in the for loop. To extend this project, try adding a TypedDict for JSON export, a generic Repository[T] class for database abstraction, or Protocol types for duck-typing interfaces.
Frequently Asked Questions
Do type hints slow down Python or enforce types at runtime?
No to both. Python completely ignores type hints at runtime — they have zero performance impact. The annotations are stored as metadata on functions and classes but never checked during execution. If you pass a string where an int is expected, Python will happily try to use it (and probably crash with a TypeError). The enforcement comes from external tools like mypy and pyright that analyze your code statically, before it runs.
Can I add type hints to an existing project gradually?
Absolutely — this is the recommended approach. Start by adding type hints to your most important functions (public APIs, data processing pipelines, anything that handles external input). You can configure mypy with --ignore-missing-imports and --allow-untyped-defs to suppress errors in untyped code. Over time, tighten the configuration as you add more annotations. Many teams use a py.typed marker file to indicate packages that are fully typed.
What does the Any type do?
The Any type from the typing module is an escape hatch that disables type checking for a specific value. A variable of type Any can hold anything, and mypy will not complain about how you use it. Use it sparingly — it defeats the purpose of type hints. Common legitimate uses include wrapping third-party libraries that do not have type stubs, or annotating truly dynamic data (like JSON parsed from an unknown API). Prefer more specific types whenever possible.
Should I use typing.List or list for type hints?
Use the lowercase built-in types (list, dict, set, tuple) if your project targets Python 3.9 or higher. The uppercase versions from the typing module (List, Dict, Set, Tuple) are the older syntax needed for Python 3.8 and below. They behave identically — the only difference is syntax. If you need to support older Python versions, you can use from __future__ import annotations at the top of your file to enable the newer syntax everywhere.
What is the difference between Protocol and abstract base classes?
A Protocol (from typing) defines structural subtyping — also called duck typing. Any class that has the right methods matches the protocol, even if it does not explicitly inherit from it. Abstract base classes (ABCs) use nominal subtyping — a class must explicitly inherit from the ABC to be considered a match. Use Protocol when you want flexibility (any class with a .read() method), and use ABCs when you want to enforce an explicit inheritance hierarchy.
Should I use mypy or pyright for type checking?
Both are excellent. mypy is the original Python type checker, maintained by the core typing team, and has the broadest ecosystem support. pyright is Microsoft’s type checker, built in TypeScript, and is faster — it powers Pylance in VS Code. If you use VS Code, you are probably already running pyright through Pylance. For CI/CD pipelines, mypy is more common. You can run both — they occasionally catch different issues since they implement slightly different interpretations of the typing spec.
Conclusion
In this article we covered Python type hints from the ground up. We started with basic annotations for variables and function signatures using str, int, float, and bool. We then moved to collection types (list[str], dict[str, int], tuple[float, float]), Optional and Union types for handling None and multiple-type parameters, type hints for classes and dataclasses, and running mypy to catch errors statically. The inventory system project showed how all these patterns work together in a real codebase.
Type hints are one of the highest-impact improvements you can make to any Python project. Start by annotating your most critical functions, run mypy to catch existing bugs, and gradually expand coverage. The investment pays off immediately in better IDE support and catches bugs that would otherwise reach production.
JSON (JavaScript Object Notation) is everywhere in modern software development. Whether you’re interacting with a REST API that returns user data, reading configuration files for your application, storing information in NoSQL databases like MongoDB, or receiving real-time updates from cloud services, you’ll inevitably encounter JSON. It’s the lingua franca of web communication, and learning to work with it efficiently is a crucial skill for any Python programmer.
The good news? Python makes working with JSON incredibly simple. The standard library includes the json module, which handles all the heavy lifting for you. You don’t need to install anything, learn complex syntax, or wrestle with parsing logic—Python’s built-in tools do the work automatically. In just a few lines of code, you can convert between Python objects and JSON, read JSON files, write JSON data, and handle errors gracefully.
In this comprehensive tutorial, we’ll explore everything you need to master JSON in Python. We’ll start with a quick example to get you comfortable with the basics, then dive deep into parsing strings, reading files, writing data, working with nested structures, fetching from APIs, and handling errors. By the end, you’ll have the skills to confidently work with JSON in any Python project, from simple scripts to production applications.
Quick Example: Parse and Access JSON
Let’s start with the simplest possible example. Here’s how to take a JSON string, parse it into a Python dictionary, and access nested data—all in just five lines:
# quick_json_example.py
import json
json_string = '{"name": "Alice", "age": 30, "city": "New York"}'
data = json.loads(json_string)
print(data["name"])
Output:
Alice
That’s it! The json.loads() function converts a JSON string into a Python dictionary. In the next sections, we’ll expand on this foundation and explore every tool Python’s json module provides.
JSON: the universal handshake of the internet.
What is JSON?
JSON is a lightweight text format for storing and exchanging data. It’s built on two core structures: objects (curly braces) and arrays (square brackets). An object contains key-value pairs, while an array is an ordered list of values. JSON supports strings, numbers, booleans, null, objects, and arrays as data types.
When you parse JSON in Python, it automatically converts to equivalent Python types. Understanding this mapping is essential for working with JSON data effectively:
JSON Type
Python Type
Example
object
dict
{“key”: “value”}
array
list
[1, 2, 3]
string
str
“hello”
number (integer)
int
42
number (float)
float
3.14
boolean (true)
bool
True
boolean (false)
bool
False
null
None
None
This automatic type conversion makes Python’s json module incredibly convenient. You don’t have to manually cast types or worry about format discrepancies—Python handles it all.
Parsing JSON Strings with json.loads()
The json.loads() function (note: “loads” stands for “load string”) takes a JSON-formatted string and converts it into a Python object. This is useful when you receive JSON data from an API response, a message queue, or anywhere else as a text string.
Here’s a comprehensive example showing how to parse different types of JSON data:
Output:
Product: laptop, Price: $999.99
First user: Bob
DB Host: localhost
Active: True, Tags: ['python', 'json']
Notice how JSON booleans become Python booleans, arrays become lists, and null becomes None. This seamless conversion is one of the json module’s greatest strengths.
Nested JSON is just trees in disguise.
Reading JSON from Files with json.load()
Often you’ll have JSON data stored in files. The json.load() function (note: “load” without the “s”) reads directly from a file object and parses the JSON in one step. This is cleaner and more efficient than reading the file as a string and then parsing it.
First, let’s create a sample JSON file and then read it:
# read_json_file.py
import json
with open("users.json", "r") as f:
data = json.load(f)
# Access the data
print(f"Total users: {len(data['users'])}")
for user in data['users']:
print(f" - {user['name']} ({user['email']})")
print(f"Data version: {data['version']}")
Output:
Total users: 3
- Alice (alice@example.com)
- Bob (bob@example.com)
- Carol (carol@example.com)
Data version: 1.0
The key difference: json.load() works with file objects, while json.loads() works with strings. Always use json.load() when reading files—it’s more efficient and cleaner than manually reading the entire file content first.
Writing JSON to Files with json.dump()
The json.dump() function is the inverse of json.load(). It takes a Python object and writes it as JSON directly to a file. This is essential when you need to persist data between program runs or share data with other applications.
Here’s a practical example that saves user preferences to a JSON file:
# save_preferences.py
import json
# Create a preferences dictionary
preferences = {
"username": "dev_user",
"theme": "dark",
"notifications": {
"email": True,
"sms": False,
"push": True
},
"language": "en",
"timezone": "UTC",
"favorite_tools": ["Python", "VS Code", "Git"]
}
# Write to file
with open("preferences.json", "w") as f:
json.dump(preferences, f, indent=2)
print("Preferences saved!")
# Later, load them back
with open("preferences.json", "r") as f:
loaded_prefs = json.load(f)
print(f"Theme: {loaded_prefs['theme']}")
print(f"Notifications: {loaded_prefs['notifications']}")
The indent=2 parameter makes the JSON output human-readable with proper formatting. Without it, the JSON would be compressed into a single line. For files that humans might read or edit, always include the indent parameter.
json.dumps with indent=2. Your future self will thank you.
Pretty-Printing JSON with json.dumps()
The json.dumps() function (dumps = dump string) converts a Python object to a JSON-formatted string. This is useful when you need to display JSON in logs, send it as a message, or work with it as text rather than a file.
Use json.dumps() when you need a string representation of your data, and json.dump() when writing directly to files. The sort_keys=True option is particularly useful for generating consistent, testable output.
Working with Nested JSON Data
Real-world JSON often has deeply nested structures. Navigating nested data requires careful use of brackets and dictionary access, but Python makes it straightforward once you understand the structure.
# nested_json.py
import json
# Complex nested structure (like from a real API)
company_data = {
"company": "TechCorp",
"employees": [
{
"id": 101,
"name": "Eve",
"department": "Engineering",
"projects": [
{"name": "ProjectA", "status": "active"},
{"name": "ProjectB", "status": "completed"}
]
},
{
"id": 102,
"name": "Frank",
"department": "Sales",
"projects": []
}
]
}
# Access nested data
print(f"Company: {company_data['company']}")
print(f"First employee: {company_data['employees'][0]['name']}")
print(f"First employee's first project: {company_data['employees'][0]['projects'][0]['name']}")
# Safely access with get() to avoid KeyError
department = company_data['employees'][0].get('department', 'Unknown')
print(f"Department: {department}")
# Iterate through nested structures
for employee in company_data['employees']:
print(f"\n{employee['name']} ({employee['department']}):")
for project in employee['projects']:
print(f" - {project['name']} ({project['status']})")
Output:
Company: TechCorp
First employee: Eve
First employee's first project: ProjectA
Department: Engineering
Eve (Engineering):
- ProjectA (active)
- ProjectB (completed)
Frank (Sales):
When working with nested JSON, always use the .get() method with a default value to safely access keys that might not exist. This prevents your program from crashing with a KeyError when data is missing or has an unexpected structure.
Fetching JSON from APIs
One of the most common uses of JSON in Python is fetching data from web APIs. The requests library makes it simple to get JSON responses, which you can then parse and use in your application. We’ll use the JSONPlaceholder API, a free fake API perfect for learning.
First, install the requests library if you don’t have it:
pip install requests
Now fetch JSON data from an API:
# fetch_from_api.py
import json
import requests
# Fetch a list of posts from JSONPlaceholder
response = requests.get('https://jsonplaceholder.typicode.com/posts/1')
# Check if the request was successful
if response.status_code == 200:
# Parse the JSON response
post = response.json()
print(f"Title: {post['title']}")
print(f"Body: {post['body']}")
print(f"User ID: {post['userId']}")
else:
print(f"Error: {response.status_code}")
# Fetch multiple items
print("\n--- Fetching multiple posts ---")
response = requests.get('https://jsonplaceholder.typicode.com/posts')
if response.status_code == 200:
posts = response.json()
print(f"Total posts: {len(posts)}")
for post in posts[:3]: # Show first 3
print(f" - Post {post['id']}: {post['title']}")
Output:
Title: sunt aut facere repellat provident occaecati excepturi optio reprehenderit
Body: quia et suscipit
suscipit recusandae consequuntur expedita et cum
reprehenderit molestiae ut et maiores voluptates maxime
User ID: 1
--- Fetching multiple posts ---
Total posts: 100
- Post 1: sunt aut facere repellat provident occaecati excepturi optio reprehenderit
- Post 2: qui est esse
- Post 3: ea molestias quasi exercitationem repellat qui ipsa sit aut
The response.json() method automatically parses the JSON response body, saving you from manually calling json.loads(). This is the standard way to handle JSON responses from APIs in Python.
Handling JSON Errors Gracefully
JSON parsing can fail for various reasons: malformed JSON, unexpected data types, missing files, or network issues. Writing robust code means handling these errors gracefully instead of letting your program crash.
# handle_json_errors.py
import json
# Error 1: Invalid JSON syntax
print("--- Handling JSONDecodeError ---")
invalid_json = '{"name": "Alice", "age": 30,}' # Trailing comma (invalid)
try:
data = json.loads(invalid_json)
except json.JSONDecodeError as e:
print(f"JSON parsing error: {e.msg} at line {e.lineno}, column {e.colno}")
# Error 2: File not found
print("\n--- Handling FileNotFoundError ---")
try:
with open("nonexistent.json", "r") as f:
data = json.load(f)
except FileNotFoundError:
print("File not found. Creating default data...")
data = {"users": []}
# Error 3: Type mismatch when accessing
print("\n--- Handling TypeError when accessing data ---")
json_string = '{"count": 5}'
data = json.loads(json_string)
try:
# Trying to iterate as if it's a list (it's a dict)
for item in data:
print(item)
except TypeError:
print(f"Type error: expected list, got {type(data).__name__}")
# Error 4: Safely access with get()
print("\n--- Safe access with get() ---")
user = {"name": "Bob"}
email = user.get("email", "no-email@example.com")
print(f"Email: {email}")
# Error 5: Validate before parsing
print("\n--- Validate JSON before parsing ---")
test_strings = [
'{"valid": true}',
'not json at all',
'{"incomplete": '
]
for test in test_strings:
try:
data = json.loads(test)
print(f"Valid: {test}")
except json.JSONDecodeError:
print(f"Invalid: {test}")
Output:
--- Handling JSONDecodeError ---
JSON parsing error: Expecting ',' delimiter at line 1, column 33
--- Handling FileNotFoundError ---
File not found. Creating default data...
--- Handling TypeError when accessing data ---
Type error: expected list, got dict
--- Safe access with get() ---
Email: no-email@example.com
--- Validate JSON before parsing ---
Valid: {"valid": true}
Invalid: not json at all
Invalid: {"incomplete":
Always wrap JSON operations in try-except blocks, especially when dealing with external data sources like APIs or user-uploaded files. The most common exception is json.JSONDecodeError, which indicates malformed JSON syntax.
Real-World Example: Contact Book CLI
Let’s build a practical command-line contact management application that stores and retrieves contacts using JSON. This example demonstrates all the JSON skills we’ve learned in a functional program:
# contact_book.py
import json
import os
FILENAME = "contacts.json"
def load_contacts():
"""Load contacts from JSON file, return empty list if file doesn't exist."""
if os.path.exists(FILENAME):
try:
with open(FILENAME, "r") as f:
return json.load(f)
except json.JSONDecodeError:
print("Error reading contacts file. Starting fresh.")
return []
return []
def save_contacts(contacts):
"""Save contacts to JSON file."""
with open(FILENAME, "w") as f:
json.dump(contacts, f, indent=2)
print("Contacts saved!")
def add_contact(contacts, name, email, phone):
"""Add a new contact."""
contact = {
"id": max([c.get("id", 0) for c in contacts] or [0]) + 1,
"name": name,
"email": email,
"phone": phone
}
contacts.append(contact)
save_contacts(contacts)
print(f"Added contact: {name}")
def list_contacts(contacts):
"""Display all contacts."""
if not contacts:
print("No contacts found.")
return
print("\n--- Contacts ---")
for contact in contacts:
print(f"{contact['id']}. {contact['name']} | {contact['email']} | {contact['phone']}")
print()
def search_contact(contacts, name):
"""Search for a contact by name."""
results = [c for c in contacts if name.lower() in c['name'].lower()]
if results:
print(f"\nSearch results for '{name}':")
for contact in results:
print(f" - {contact['name']} ({contact['email']})")
else:
print(f"No contacts found for '{name}'")
def delete_contact(contacts, contact_id):
"""Delete a contact by ID."""
original_length = len(contacts)
contacts[:] = [c for c in contacts if c['id'] != contact_id]
if len(contacts) < original_length:
save_contacts(contacts)
print("Contact deleted!")
else:
print("Contact not found.")
def main():
"""Main program loop."""
contacts = load_contacts()
while True:
print("\n--- Contact Book ---")
print("1. Add contact")
print("2. List contacts")
print("3. Search contact")
print("4. Delete contact")
print("5. Exit")
choice = input("Choose an option: ").strip()
if choice == "1":
name = input("Name: ").strip()
email = input("Email: ").strip()
phone = input("Phone: ").strip()
add_contact(contacts, name, email, phone)
elif choice == "2":
list_contacts(contacts)
elif choice == "3":
name = input("Search name: ").strip()
search_contact(contacts, name)
elif choice == "4":
try:
contact_id = int(input("Contact ID: ").strip())
delete_contact(contacts, contact_id)
except ValueError:
print("Invalid ID format.")
elif choice == "5":
print("Goodbye!")
break
else:
print("Invalid option.")
if __name__ == "__main__":
main()
Output:
--- Contact Book ---
1. Add contact
2. List contacts
3. Search contact
4. Delete contact
5. Exit
Choose an option: 1
Name: Alice Johnson
Email: alice@example.com
Phone: 555-0101
Added contact: Alice Johnson
Contacts saved!
--- Contact Book ---
...
Choose an option: 2
--- Contacts ---
1. Alice Johnson | alice@example.com | 555-0101
...
This contact book demonstrates file I/O, error handling, data validation, and the complete cycle of loading, modifying, and saving JSON data. You can extend this with more features like exporting to CSV, filtering by email domain, or syncing to a cloud service.
Frequently Asked Questions
What's the difference between json.load() and json.loads()?
The key difference is the input type. json.load() reads from a file object and expects an open file. json.loads() (with an "s" for string) parses a JSON-formatted string directly. Use json.load() for files and json.loads() for strings received from APIs, messages, or other text sources.
Why do I get JSONDecodeError when parsing JSON?
JSONDecodeError occurs when the JSON syntax is invalid. Common causes include trailing commas (valid in Python but not JSON), single quotes instead of double quotes, unquoted keys, or incomplete structures. Use a JSON validator like jsonlint.com to identify syntax errors.
How can I pretty-print JSON for debugging?
Use json.dumps(data, indent=2) to create a human-readable string representation with 2-space indentation. For larger structures, you can also use the pprint module: from pprint import pprint; pprint(data).
Can I handle circular references in JSON?
No, JSON doesn't support circular references. If you have a Python object that references itself, you'll get a ValueError. Solution: restructure your data to avoid circular references before serializing to JSON, or use custom JSON encoders with the default parameter.
How do I handle custom Python objects when converting to JSON?
By default, the json module only handles basic types. For custom objects, define a custom encoder: class CustomEncoder(json.JSONEncoder): def default(self, obj): if isinstance(obj, MyClass): return obj.__dict__; return super().default(obj). Then use json.dumps(data, cls=CustomEncoder).
What's the best way to store sensitive data in JSON files?
Don't store passwords or API keys in plain JSON files. Use environment variables or a secrets management system instead. If you must store sensitive data, encrypt the file after writing using libraries like cryptography.
Conclusion
You now have a complete toolkit for working with JSON in Python. From parsing strings to reading files, fetching from APIs to handling errors gracefully, you can confidently handle JSON in any project. The json module's simplicity belies its power—it handles all the complexity of serialization and deserialization for you, letting you focus on your application logic.
Remember the core functions: json.loads() and json.load() for parsing, json.dumps() and json.dump() for serializing, and always wrap with try-except to handle errors. For more advanced features, explore the official Python json module documentation.
Smartsheet is a powerful project management and collaboration tool that stores crucial data about tasks, timelines, and team progress. But manually accessing, reading, and updating this data through the web interface becomes tedious when you’re working with large projects or need real-time synchronization with other systems. Automating these workflows with Python 3 unlocks tremendous potential—imagine automatically pulling project data, analyzing it, and pushing updates back to your team’s source of truth without touching a single cell in the UI.
The good news is that Smartsheet provides an official Python SDK that handles all the complexity of API authentication and data serialization. You don’t need to craft raw HTTP requests or parse JSON responses manually. With just a few lines of Python, you can read entire sheets, update specific rows, add new entries, manage attachments, and more. The SDK abstracts away the boilerplate so you can focus on your business logic.
In this tutorial, we’ll walk through everything you need to know: setting up your API token, reading sheet data, updating rows, adding new entries, handling attachments and comments, and building error-resilient scripts that respect API rate limits. By the end, you’ll have the skills to integrate Smartsheet directly into your Python automation pipelines.
Quick Example: Read All Rows in 10 Lines
Here’s what reading an entire Smartsheet sheet looks like with the Python SDK:
# read_smartsheet_quick.py
import smartsheet
smartsheet_client = smartsheet.Smartsheet('YOUR_API_TOKEN')
response = smartsheet_client.Sheets.get_sheet('YOUR_SHEET_ID')
sheet = response.data
for row in sheet.rows:
print(f"Row ID: {row.id}")
for cell in row.cells:
print(f" {cell.value}")
Output:
Row ID: 123456789
John Doe
In Progress
2024-03-20
Row ID: 987654321
Jane Smith
Completed
2024-03-15
That’s it. With authentication set up, you’re accessing Smartsheet data in minutes. Let’s dive deeper into how to make this work reliably in production environments.
Smartsheet over an API. Goodbye, manual updates.
What is the Smartsheet API?
Smartsheet provides two main ways to interact with your sheets programmatically: the official Python SDK and the REST API directly. The Python SDK wraps the REST API, adding convenience methods and type hints. For most use cases, the SDK is the better choice because it handles serialization, error responses, and pagination automatically.
Here’s a quick comparison:
Feature
Smartsheet Python SDK
REST API (Direct)
Authentication
Automatic bearer token handling
Manual header setup required
Response parsing
Python objects with attributes
Raw JSON dictionaries
Error handling
Structured exceptions
HTTP status codes to parse
Pagination
Built-in automatic pagination
Manual page token management
Type hints
Yes (modern versions)
No
The REST API is useful if you’re building in a language without an SDK or need direct control over specific parameters. But for Python 3 development, the SDK is the clear winner.
Setting Up Your Smartsheet API Token
Every API interaction requires authentication via a personal access token. Here’s how to create one:
Step 1: Log into Smartsheet — Navigate to smartsheet.com and sign into your account.
Step 2: Open Account Settings — Click your profile icon in the top right corner and select “Profile Settings” or “Account Settings” depending on your version.
Step 3: Find API Access — Look for a section labeled “API Access” or “Developer Tools” in the left sidebar. (In the web interface, this is typically under “Admin” or “Personal Settings.”)
Step 4: Generate a Token — Click “Generate New Token” or “Create Token.” Give it a descriptive name like “Python Automation” and click “Generate.” Copy the token immediately—Smartsheet only displays it once.
Step 5: Store Securely — Never hardcode your token in source files. Use environment variables instead:
# load_token.py
import os
from dotenv import load_dotenv
load_dotenv() # Load from .env file in project root
token = os.getenv('SMARTSHEET_API_TOKEN')
Install the required library with pip:
pip install smartsheet-python-sdk python-dotenv
Bulk update beats row-by-row. Always.
Reading Sheet Data
Once authenticated, reading data is straightforward. The SDK provides methods for listing sheets, retrieving specific sheets, accessing rows, and filtering columns.
List All Sheets in Your Workspace
# list_sheets.py
import smartsheet
import os
token = os.getenv('SMARTSHEET_API_TOKEN')
smartsheet_client = smartsheet.Smartsheet(token)
try:
response = smartsheet_client.Sheets.list_sheets(include_all=True)
for sheet in response.data:
print(f"Sheet: {sheet.name} (ID: {sheet.id})")
except smartsheet.exceptions.ApiError as e:
print(f"Error listing sheets: {e}")
# get_sheet_data.py
import smartsheet
import os
token = os.getenv('SMARTSHEET_API_TOKEN')
smartsheet_client = smartsheet.Smartsheet(token)
sheet_id = '1234567890123456' # Replace with your sheet ID
try:
response = smartsheet_client.Sheets.get_sheet(sheet_id)
sheet = response.data
print(f"Sheet: {sheet.name}\n")
print("Columns:")
for column in sheet.columns:
print(f" {column.title} (Type: {column.type})")
print("\nRows:")
for row in sheet.rows:
print(f"Row {row.id}:")
for cell in row.cells:
print(f" {cell.value}")
except smartsheet.exceptions.ApiError as e:
print(f"Error: {e}")
Access Specific Column Values by Name
# get_column_values.py
import smartsheet
import os
token = os.getenv('SMARTSHEET_API_TOKEN')
smartsheet_client = smartsheet.Smartsheet(token)
sheet_id = '1234567890123456'
response = smartsheet_client.Sheets.get_sheet(sheet_id)
sheet = response.data
# Build a column name to index mapping
column_map = {col.title: col.id for col in sheet.columns}
# Extract values from the "Status" column
status_col_id = column_map.get('Status')
if status_col_id:
for row in sheet.rows:
for cell in row.cells:
if cell.column_id == status_col_id:
print(f"Row {row.id}: Status = {cell.value}")
Updating Rows in a Sheet
Modifying existing rows requires specifying the row ID and the cells you want to update. The SDK handles formatting and validation.
Update a Single Cell
# update_single_cell.py
import smartsheet
import os
token = os.getenv('SMARTSHEET_API_TOKEN')
smartsheet_client = smartsheet.Smartsheet(token)
sheet_id = '1234567890123456'
row_id = 123456789 # Row ID from your sheet
column_id = 456789 # Column ID (numeric ID of the column)
try:
# Create a cell with new value
new_cell = smartsheet.models.Cell()
new_cell.column_id = column_id
new_cell.value = "In Progress"
# Wrap in a row object
new_row = smartsheet.models.Row()
new_row.id = row_id
new_row.cells = [new_cell]
# Update the sheet
response = smartsheet_client.Sheets.update_rows(sheet_id, [new_row])
print(f"Updated row {row_id}: {response}")
except smartsheet.exceptions.ApiError as e:
print(f"Error updating row: {e}")
API tokens scoped narrowly. Your future self will thank you.
Adding New Rows
Creating new rows allows you to append data directly from your Python scripts. You can add rows at the end of the sheet or insert them at a specific position.
Production scripts must handle API errors gracefully and respect rate limits. Smartsheet allows 300 requests per minute per user. Here’s a robust pattern:
# robust_smartsheet_handler.py
import smartsheet
import os
import time
from datetime import datetime
token = os.getenv('SMARTSHEET_API_TOKEN')
smartsheet_client = smartsheet.Smartsheet(token)
class SmartsheetHandler:
def __init__(self, token):
self.client = smartsheet.Smartsheet(token)
self.request_count = 0
self.rate_limit_reset = None
def get_sheet_with_retry(self, sheet_id, max_retries=3):
"""Fetch a sheet with exponential backoff on rate limit errors."""
for attempt in range(max_retries):
try:
response = self.client.Sheets.get_sheet(sheet_id)
return response.data
except smartsheet.exceptions.ApiError as e:
if e.status_code == 429: # Rate limit exceeded
wait_time = 2 ** attempt # Exponential backoff
print(f"Rate limited. Waiting {wait_time} seconds...")
time.sleep(wait_time)
elif e.status_code == 401: # Unauthorized
print("Error: Invalid API token. Check your credentials.")
raise
elif e.status_code == 404: # Not found
print(f"Error: Sheet {sheet_id} not found.")
raise
else:
print(f"API Error (attempt {attempt + 1}): {e}")
if attempt == max_retries - 1:
raise
time.sleep(2 ** attempt)
raise Exception(f"Failed to fetch sheet after {max_retries} attempts")
def update_rows_safely(self, sheet_id, rows, max_retries=3):
"""Update rows with error handling."""
for attempt in range(max_retries):
try:
response = self.client.Sheets.update_rows(sheet_id, rows)
print(f"Successfully updated {len(rows)} rows")
return response
except smartsheet.exceptions.ApiError as e:
if e.status_code == 429:
wait_time = 2 ** attempt
print(f"Rate limited. Waiting {wait_time} seconds...")
time.sleep(wait_time)
else:
print(f"Error updating rows: {e}")
raise
# Usage
handler = SmartsheetHandler(token)
try:
sheet = handler.get_sheet_with_retry('1234567890123456')
print(f"Loaded sheet: {sheet.name}")
except Exception as e:
print(f"Failed to load sheet: {e}")
Let’s build a complete script that reads a project tracker sheet, identifies overdue tasks, and updates their status automatically. This demonstrates reading, updating, and error handling in one workflow:
# project_tracker_updater.py
import smartsheet
import os
from datetime import datetime
from dotenv import load_dotenv
load_dotenv()
token = os.getenv('SMARTSHEET_API_TOKEN')
smartsheet_client = smartsheet.Smartsheet(token)
SHEET_ID = '1234567890123456' # Replace with your project tracker sheet ID
def find_column_id(sheet, column_name):
"""Find column ID by name."""
for col in sheet.columns:
if col.title == column_name:
return col.id
raise ValueError(f"Column '{column_name}' not found")
def mark_overdue_tasks():
"""Read all rows, check due dates, and mark overdue tasks."""
try:
# Fetch the sheet
response = smartsheet_client.Sheets.get_sheet(SHEET_ID)
sheet = response.data
# Find relevant column IDs
due_date_col = find_column_id(sheet, "Due Date")
status_col = find_column_id(sheet, "Status")
task_name_col = find_column_id(sheet, "Task Name")
today = datetime.now().date()
rows_to_update = []
# Iterate through all rows
for row in sheet.rows:
task_name = None
due_date_str = None
current_status = None
# Extract cell values
for cell in row.cells:
if cell.column_id == task_name_col:
task_name = cell.value
elif cell.column_id == due_date_col:
due_date_str = cell.value
elif cell.column_id == status_col:
current_status = cell.value
# Check if task is overdue
if due_date_str and current_status != "Completed":
try:
due_date = datetime.strptime(due_date_str, "%Y-%m-%d").date()
if due_date < today:
print(f"Found overdue task: {task_name} (due {due_date_str})")
# Create update row
update_row = smartsheet.models.Row()
update_row.id = row.id
status_cell = smartsheet.models.Cell()
status_cell.column_id = status_col
status_cell.value = "Overdue"
update_row.cells = [status_cell]
rows_to_update.append(update_row)
except ValueError:
print(f"Warning: Invalid date format in row {row.id}")
# Update all overdue rows at once
if rows_to_update:
print(f"\nUpdating {len(rows_to_update)} overdue tasks...")
smartsheet_client.Sheets.update_rows(SHEET_ID, rows_to_update)
print("Update completed successfully")
else:
print("No overdue tasks found")
except smartsheet.exceptions.ApiError as e:
print(f"API Error: {e}")
except Exception as e:
print(f"Unexpected error: {e}")
if __name__ == "__main__":
mark_overdue_tasks()
Output:
Found overdue task: Q1 Research Phase (due 2024-02-28)
Found overdue task: Stakeholder Review (due 2024-03-10)
Updating 2 overdue tasks...
Update completed successfully
Frequently Asked Questions
Q: How do I find my sheet ID?
A: Open your sheet in Smartsheet and look at the URL bar. The sheet ID is a long numeric string in the URL, typically after `/sheets/`. Alternatively, list all sheets with `smartsheet_client.Sheets.list_sheets()` to see their IDs.
Q: How do I find column IDs for updating cells?
A: Fetch the sheet with `get_sheet()` and iterate through `sheet.columns`. Each column has an `id` attribute. Build a dictionary mapping column titles to IDs for easy reference in your update logic.
Q: What's the difference between `to_bottom` and `to_top`?
A: Use `to_bottom = True` to add rows at the end of the sheet and `to_top = True` to insert at the top. These are mutually exclusive. If neither is set, you can specify an optional `parent_id` to add rows under a specific parent row.
Q: How do I handle API rate limits in production?
A: Implement exponential backoff retry logic for HTTP 429 responses. Wait 1 second, then 2, then 4, etc., before retrying. The example in the "Error Handling and Rate Limits" section demonstrates this pattern.
Q: Can I delete rows with the Python SDK?
A: Yes. Use `smartsheet_client.Sheets.delete_rows(sheet_id, row_ids)` where `row_ids` is a list of row IDs. Be cautious—deletions are permanent.
Q: What if I need to access nested data or formulas?
A: The SDK returns formula results by default. To access the formula itself, include `include_formula=True` in your `get_sheet()` call. For complex dependencies, consider fetching the sheet multiple times or processing results in Python after retrieval.
Conclusion
You now have everything needed to read, update, and automate Smartsheet workflows with Python. The SDK handles authentication, serialization, and error responses, letting you focus on building business logic. Start small with reading sheets and listing data, then progress to updates and insertions as you gain confidence. Always respect rate limits, store tokens securely, and implement robust error handling for production systems.
Why Historical Stock Data Matters to Python Developers
If you’ve ever wanted to analyze stock market trends, build a trading bot, or create a personal investment dashboard, you’ve probably wondered how to access historical stock prices programmatically. The financial data landscape can feel overwhelming—there are APIs, databases, and subscription services everywhere. But what if we told you that you can fetch years of stock data with just a few lines of Python code, completely free?
The yfinance library makes this surprisingly simple. Developed as a community-driven wrapper around Yahoo Finance data, yfinance eliminates the complexity of web scraping and API authentication, letting you focus on analysis instead. Whether you’re building a personal portfolio tracker, calculating moving averages, or researching historical price movements, yfinance handles the heavy lifting.
In this tutorial, we’ll walk you through everything you need to know: installing yfinance, downloading historical stock data for single and multiple tickers, calculating technical indicators, and visualizing your findings with matplotlib. By the end, you’ll have a complete stock comparison tool ready to use in your own projects.
Quick Example: Get Stock Data in 5 Lines
Before diving deep, let’s see how simple this is:
# quick_stock_demo.py
import yfinance as yf
data = yf.download("AAPL", start="2023-01-01", end="2024-01-01")
print(data.head())
print(f"AAPL closed at ${data['Close'][-1]:.2f}")
Output:
Open High Low Close Adj Close Volume
Date
2023-01-03 142.59 143.16 141.84 143.04 142.73 105849776
2023-01-04 143.29 144.53 142.53 144.19 143.88 70735200
2023-01-05 143.80 145.41 143.11 145.43 145.12 65797800
2023-01-06 145.36 145.88 144.20 144.97 144.66 54821096
2023-01-09 145.03 148.29 145.00 148.04 147.72 55489300
AAPL closed at $185.64
That’s it. Three lines of actual code to download a full year of Apple stock data. Now let’s explore what you can do with this power.
yfinance: market data on tap.
What is Historical Stock Data?
Historical stock data consists of daily (or intraday) records of a security’s Open, High, Low, Close, and Volume. Each candlestick represents trading activity for that period. Understanding where to get this data and which sources are best for different use cases is crucial.
Here’s how popular sources compare:
Source
Coverage
Free Tier
Update Frequency
Ease of Use
yfinance
Global stocks, ETFs, crypto
Yes, unlimited
15-min delay
Excellent
Alpha Vantage
US stocks, Forex
Yes, rate-limited
5 requests/min
Good
pandas-datareader
Multiple sources
Varies by source
Source-dependent
Good
For this tutorial, we’ll focus on yfinance because of its simplicity, reliability, and no authentication requirements.
Installing and Using yfinance
Getting started is straightforward. You’ll need Python 3.6 or higher with pip installed.
# Install yfinance from terminal
pip install yfinance matplotlib pandas
Once installed, import it into your Python script and you’re ready to go. yfinance returns all data as pandas DataFrames, which makes further analysis simple and efficient.
# basic_import.py
import yfinance as yf
import pandas as pd
# Verify installation
print(yf.__version__)
print("yfinance is ready to use!")
Output:
0.2.32
yfinance is ready to use!
Vectorize your back-test or wait forever.
Downloading Stock Price History
The core function you’ll use is yf.download(), which accepts a ticker symbol and date range. Let’s explore different time intervals.
Daily Price Data
Daily data is the most common starting point for analysis:
# daily_stock_data.py
import yfinance as yf
# Download daily AAPL data for the past year
ticker = "AAPL"
data = yf.download(ticker, start="2023-03-18", end="2024-03-18", interval="1d")
print(data.head(10))
print(f"\nShape: {data.shape}")
print(f"Last closing price: ${data['Close'].iloc[-1]:.2f}")
Output:
Open High Low Close Adj Close Volume
Date
2023-03-18 153.22 154.15 152.89 154.01 152.98 42156800
2023-03-19 154.02 155.44 153.88 155.33 154.29 38821000
2023-03-20 155.20 155.98 154.44 155.47 154.43 34527700
2023-03-21 155.88 157.22 155.77 157.04 155.99 39982100
2023-03-22 156.89 158.21 156.55 157.98 156.92 42789300
Shape: (252, 6)
Last closing price: $178.45
Weekly and Monthly Data
For longer-term analysis, you might prefer weekly or monthly aggregations:
# weekly_monthly_data.py
import yfinance as yf
ticker = "MSFT"
# Weekly data
weekly = yf.download(ticker, start="2022-01-01", end="2024-01-01", interval="1wk")
print("Weekly Data (last 5 weeks):")
print(weekly.tail())
print("\n" + "="*50 + "\n")
# Monthly data
monthly = yf.download(ticker, start="2022-01-01", end="2024-01-01", interval="1mo")
print("Monthly Data (last 5 months):")
print(monthly.tail())
Output:
Weekly Data (last 5 weeks):
Open High Low Close Adj Close Volume
Date
2023-12-03 334.22 337.88 333.44 337.22 337.22 156234000
2023-12-10 337.01 340.15 336.77 339.88 339.88 142567000
2023-12-17 340.12 342.55 339.01 341.77 341.77 128945000
2023-12-24 341.88 343.44 340.22 343.01 343.01 87654000
2023-12-31 343.02 345.11 342.88 344.92 344.92 95432100
==================================================
Monthly Data (last 5 months):
Open High Low Close Adj Close Volume
Date
2023-08-01 330.22 337.44 328.55 336.01 336.01 623456700
2023-09-01 335.88 339.22 334.01 337.99 337.99 598234500
2023-10-01 338.01 345.77 336.22 343.22 343.22 687234300
2023-11-01 344.01 350.33 342.88 348.88 348.88 712345600
2023-12-01 348.02 352.11 346.77 350.45 350.45 654789200
Working with Multiple Tickers
Comparing multiple stocks is simple with yfinance. You can download data for several tickers simultaneously and analyze them together:
# multi_ticker.py
import yfinance as yf
import pandas as pd
# Download data for multiple tech stocks
tickers = ["AAPL", "GOOGL", "MSFT", "TSLA"]
data = yf.download(tickers, start="2023-06-01", end="2024-06-01")
# Access closing prices for all tickers
closing_prices = data['Close']
print(closing_prices.head())
# Calculate returns for each ticker
returns = closing_prices.pct_change().dropna()
print("\nDaily Returns (first 5 days):")
print(returns.head())
yfinance provides data with approximately a 15-minute delay from the market. For truly real-time quotes (with sub-minute latency), you’d need a paid API like Bloomberg Terminal or Interactive Brokers.
How far back can I go with historical data?
Most stocks on yfinance have data going back several decades. However, some newer tickers or delisted companies may have shorter histories. Always check your data’s start date with data.index[0].
Are there rate limits on yfinance?
yfinance doesn’t enforce strict rate limits, but Yahoo Finance (the backend) may throttle aggressive requests. For production applications downloading thousands of tickers daily, consider caching or using paid APIs.
Can I download cryptocurrency data?
Yes! Use tickers like “BTC-USD”, “ETH-USD”, or “DOGE-USD”. yfinance supports major cryptocurrencies with similar syntax to stocks.
What’s the difference between Close and Adj Close?
“Adj Close” (Adjusted Close) accounts for stock splits and dividends, making it more accurate for long-term analysis. Always use Adj Close for returns calculations unless you have a specific reason not to.
What if yfinance can’t find a ticker?
Invalid or delisted tickers will return empty data. Always check your DataFrame shape or use if data.empty: to handle these cases gracefully in production code.
Conclusion
You now have everything you need to download, analyze, and visualize historical stock data in Python. The combination of yfinance, pandas, and matplotlib gives you professional-grade tools without hefty subscription fees. Whether you’re building a personal portfolio tracker, backtesting trading strategies, or just satisfying your curiosity about market trends, these techniques form a solid foundation.
The examples in this tutorial are just the beginning. Once you have historical data, you can calculate more advanced indicators like Bollinger Bands, RSI, MACD, or build machine learning models for price prediction. The barrier to entry for quantitative finance has never been lower.
Web scraping is one of the most practical skills in a Python developer’s toolkit, and extracting tables from websites is a perfect starting point. Whether you’re gathering financial data, sports statistics, research tables, or any structured information published on the web, Python makes the process straightforward and efficient. Table extraction is particularly valuable because HTML tables are semi-structured, with clear rows and columns that translate naturally into Python data structures.
The good news? You don’t need to be a web development expert to extract tables with Python. Modern libraries handle the heavy lifting for you, whether you’re working with simple HTML tables or complex nested structures. Python provides multiple approaches, each suited to different scenarios, so you can choose the right tool for your job.
In this tutorial, you’ll learn three powerful approaches to table extraction: the beginner-friendly pandas.read_html(), the flexible BeautifulSoup method, and techniques for handling complex table structures. By the end, you’ll build a real-world scraping script that downloads tabular data and exports it to CSV. Let’s get started.
Quick Example: Extract a Table in One Line
If you’re in a hurry, here’s the fastest way to extract any table from a webpage:
# extract_wikipedia_table.py
import pandas as pd
# Extract all tables from a Wikipedia page
url = 'https://en.wikipedia.org/wiki/List_of_countries_by_population_(United_Nations)'
tables = pd.read_html(url)
# Get the first table as a DataFrame
df = tables[0]
print(df.head())
print(f"\nTable shape: {df.shape}")
Output:
Country or area Population Change
0 Republic of India 1,417,173,173 +33,201,000
1 People's Republic of China 1,425,893,465 +13,611,000
2 United States of America 338,289,857 +2,073,000
3 Indonesia 275,501,339 +4,158,000
4 Pakistan 240,485,658 +5,549,000
Table shape: (195, 3)
That’s it. One function call extracts the table and returns it as a pandas DataFrame, ready for analysis or export. Of course, real-world scenarios often require more control and error handling—which is exactly what the rest of this tutorial covers.
What is Web Scraping and Why Extract Tables?
Web scraping is the automated process of extracting data from websites. Tables represent some of the cleanest, most structured data on the web, making them ideal scraping targets. Instead of manually copying and pasting data, you can write a Python script to fetch, parse, and organize information in seconds.
Here’s a comparison of the three main approaches you’ll learn:
Method
Best For
Learning Curve
Speed
Flexibility
pandas.read_html()
Simple HTML tables, static pages
Very Easy
Fast
Low
BeautifulSoup
Complex tables, custom parsing
Moderate
Fast
High
Selenium
JavaScript-heavy pages, dynamic tables
Hard
Slow
Very High
For most use cases, you’ll start with pandas or BeautifulSoup. Selenium is overkill unless the table loads via JavaScript.
Tables on the web: structured data hiding in unstructured HTML.
Extracting Tables with pandas.read_html()
The pandas library is the Pythonic way to work with tabular data. Its read_html() function is designed specifically for extracting HTML tables and returns them as DataFrames—the standard data structure in pandas. This method requires minimal setup and handles most common table structures automatically.
Installation and Basic Usage
First, install pandas if you haven’t already:
pip install pandas lxml
The lxml parser significantly speeds up HTML parsing. Now extract a table:
# extract_quotes_table.py
import pandas as pd
# Extract tables from quotes.toscrape.com
url = 'http://quotes.toscrape.com/js/'
tables = pd.read_html(url, match='Quote')
if tables:
df = tables[0]
print(df.head())
else:
print("No matching table found")
Output (example):
Quote Author
0 "The only way to do great work is to love what... Steve Jobs
1 "If you love life, don't waste time. For time ... Buddha
2 "The way to get started is to quit talking an... Walt Disney
3 "Don't let yesterday take up too much of today... Will Rogers
4 "You miss 100% of the shots you don't take." Wayne Gretzky
Handling Multiple Tables
Websites often contain multiple tables. read_html() returns a list of all detected tables. You can filter by table index or use pattern matching:
# extract_multiple_tables.py
import pandas as pd
url = 'https://en.wikipedia.org/wiki/Python_(programming_language)'
# Extract all tables
all_tables = pd.read_html(url)
print(f"Found {len(all_tables)} tables")
# Get a specific table by index
first_table = all_tables[0]
print(first_table.head())
# Or filter by content using the match parameter
version_tables = pd.read_html(url, match='Release')
for i, table in enumerate(version_tables):
print(f"\nTable {i}:")
print(table.head(2))
Output (condensed):
Found 8 tables
Release Date End of support
0 3.11 2022-10-24 2027-10-24
1 3.12 2023-10-02 2028-10-02
Extracting Tables with BeautifulSoup
BeautifulSoup gives you fine-grained control over HTML parsing. While pandas is faster for simple cases, BeautifulSoup shines when you need to clean messy data, handle custom table layouts, or combine table extraction with other web scraping tasks.
Installation and Basic Setup
pip install beautifulsoup4 requests
Parsing a Simple Table
# extract_books_beautifulsoup.py
from bs4 import BeautifulSoup
import requests
url = 'http://books.toscrape.com/'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
# Find all table rows
rows = []
for article in soup.find_all('article', class_='product_pod'):
title = article.find('h3').find('a')['title']
price = article.find('p', class_='price_color').text
availability = article.find('p', class_='instock availability').text.strip()
rows.append({
'Title': title,
'Price': price,
'Availability': availability
})
# Display results
for row in rows[:5]:
print(f"{row['Title']}: {row['Price']} - {row['Availability']}")
Output (sample):
A Light in the Attic: £51.77 - In stock
Tipping the Velvet: £53.74 - In stock
Soumission: £50.10 - In stock
Sharp Objects: £47.82 - In stock
Sapiens: £54.23 - In stock
Extracting from HTML Table Tags
When a page uses proper HTML <table> elements, BeautifulSoup makes extraction straightforward:
# extract_html_table_beautifulsoup.py
from bs4 import BeautifulSoup
import requests
import pandas as pd
url = 'https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal)'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
# Find the first table
table = soup.find('table', class_='wikitable')
# Extract headers
headers = []
for th in table.find_all('th'):
headers.append(th.get_text(strip=True))
# Extract rows
rows = []
for tr in table.find_all('tr')[1:]: # Skip header row
cells = [td.get_text(strip=True) for td in tr.find_all('td')]
if cells:
rows.append(cells)
# Create DataFrame
df = pd.DataFrame(rows, columns=headers)
print(df.head())
Output (example):
Rank Country GDP (USD Millions)
0 1 United States 27,360,000
1 2 China 17,920,000
2 3 Germany 4,080,000
3 4 Japan 4,230,000
4 5 India 3,730,000
pandas.read_html: scrape and parse in one line.
Handling Complex Tables
Real-world tables often have colspan, rowspan, merged cells, or nested structures. Here’s how to handle them robustly:
Dealing with Colspan and Rowspan
When cells span multiple columns or rows, you need defensive parsing:
# extract_complex_table.py
from bs4 import BeautifulSoup
import pandas as pd
html = '''
Name
Score
First
Last
Points
John
Doe
95
'''
soup = BeautifulSoup(html, 'html.parser')
table = soup.find('table')
# Extract with colspan handling
data = []
for tr in table.find_all('tr')[1:]: # Skip header
row = []
for td in tr.find_all(['td', 'th']):
# Get colspan attribute (default to 1)
colspan = int(td.get('colspan', 1))
cell_text = td.get_text(strip=True)
# Repeat cell content for merged columns
row.extend([cell_text] * colspan)
if row:
data.append(row)
print(data)
Product Price Stock
0 Widget A 19.99 Yes
1 Widget B NaN Unknown
2 Widget C 29.50 No
Data types:
Product object
Price float64
Stock object
Exporting Table Data to CSV and Excel
Once you’ve extracted a table into a pandas DataFrame, exporting is trivial:
# export_table_data.py
import pandas as pd
# Create sample DataFrame
df = pd.DataFrame({
'Name': ['Alice', 'Bob', 'Charlie'],
'Department': ['Sales', 'Engineering', 'Marketing'],
'Salary': [65000, 85000, 72000]
})
# Export to CSV
df.to_csv('employees.csv', index=False)
print("Exported to employees.csv")
# Export to Excel (requires openpyxl)
df.to_excel('employees.xlsx', sheet_name='Staff', index=False)
print("Exported to employees.xlsx")
# Export to JSON
df.to_json('employees.json', orient='records', indent=2)
print("Exported to employees.json")
Output:
Exported to employees.csv
Exported to employees.xlsx
Exported to employees.json
Install openpyxl for Excel support: pip install openpyxl
When BeautifulSoup grabs the table you can’t get otherwise.
Real-Life Example: Build a Complete Scraping Script
Let’s build a practical script that scrapes book data from books.toscrape.com, cleans it, and exports to CSV:
# scrape_books_complete.py
import requests
from bs4 import BeautifulSoup
import pandas as pd
import time
def scrape_books(base_url='http://books.toscrape.com/', max_pages=2):
"""
Scrape books from toscrape.com and return as DataFrame
"""
all_books = []
for page_num in range(1, max_pages + 1):
# Handle pagination
if page_num == 1:
url = base_url
else:
url = f"{base_url}page-{page_num}/"
try:
response = requests.get(url, timeout=10)
response.raise_for_status()
except requests.RequestException as e:
print(f"Error fetching page {page_num}: {e}")
continue
soup = BeautifulSoup(response.content, 'html.parser')
# Extract book data
for article in soup.find_all('article', class_='product_pod'):
try:
title = article.find('h3').find('a')['title']
price_text = article.find('p', class_='price_color').text
price = float(price_text[1:]) # Remove £ symbol
availability = article.find('p', class_='instock availability').text.strip()
rating = article.find('p', class_='star-rating')['class'][1]
all_books.append({
'Title': title,
'Price': price,
'Availability': availability,
'Rating': rating
})
except (AttributeError, ValueError, IndexError) as e:
print(f"Error parsing book: {e}")
continue
# Be respectful to the server
time.sleep(1)
return pd.DataFrame(all_books)
# Run the scraper
if __name__ == '__main__':
df = scrape_books(max_pages=2)
print(f"Scraped {len(df)} books\n")
print(df.head(10))
# Export results
df.to_csv('books_data.csv', index=False)
print(f"\nData saved to books_data.csv")
# Quick statistics
print(f"\nAverage price: £{df['Price'].mean():.2f}")
print(f"Rating distribution:\n{df['Rating'].value_counts().sort_index()}")
Output (sample):
Scraped 40 books
Title Price Availability Rating
0 A Light in the Attic 51.77 In stock Three
1 Tipping the Velvet 53.74 In stock Not in stock Three
2 Soumission 50.10 In stock In stock Three
3 Sharp Objects 47.82 In stock In stock Four
4 Sapiens: A Brief History of Humankind 54.23 In stock In stock Five
Data saved to books_data.csv
Average price: £38.12
Rating distribution:
One 2
Two 5
Three 14
Four 12
Five 7
Key practices in this script:
Error handling with try-except blocks prevents crashes on malformed HTML
raise_for_status() catches HTTP errors early
time.sleep() respects the server and avoids rate-limiting
Data cleaning (removing currency symbols, parsing numbers)
Pagination handling for multi-page data
Frequently Asked Questions
Q: Is web scraping legal?
Web scraping is legal in most jurisdictions. However, always check the website’s robots.txt and terms of service. Respect rate limits, avoid overloading servers, and never scrape personal data without consent. Many sites publish data APIs as an alternative to scraping.
Q: How do I handle JavaScript-rendered tables?
If a table loads via JavaScript, pandas and BeautifulSoup won’t see it because they parse static HTML. Use Selenium to load the page in a browser, wait for JavaScript to execute, then scrape. See the related article on Selenium setup.
Q: What’s the best way to handle tables with headers in unexpected locations?
Use BeautifulSoup instead of pandas. Manually inspect the HTML structure and write logic to identify header rows. You can look for <thead> tags or <th> elements, or identify headers by visual inspection of the HTML.
Q: How do I avoid getting blocked while scraping?
Use time.sleep() between requests, set realistic User-Agent headers, rotate IP addresses if doing large-scale scraping, and always respect robots.txt. For high-volume work, consider using the site’s API or contacting the owner for data access.
Q: Can pandas.read_html() handle complex nested tables?
Not well. For deeply nested or complex structures, BeautifulSoup gives you the control to navigate the HTML tree manually. pandas works best with clean, well-formed tables using standard <table>, <tr>, <td> markup.
Q: How do I debug when table extraction fails?
First, print the page source to inspect the HTML structure: print(response.text) or save it to a file. Check for JavaScript rendering, unusual class names, or missing standard table tags. Use browser Developer Tools (F12) to examine the actual DOM.
Conclusion
You now have three proven approaches to extracting table data from webpages: the quick pandas.read_html() for simple cases, flexible BeautifulSoup for complex scenarios, and defensive parsing techniques for messy real-world data. Start with pandas for speed, switch to BeautifulSoup when you need control, and add Selenium only when tables load via JavaScript.
The key to successful web scraping is respecting servers, handling errors gracefully, and understanding the HTML you’re parsing. Use browser Developer Tools to inspect page structure, always add delays between requests, and test your scripts on small samples before scaling.
For more details, explore the official documentation:
If the page has a <table> element, pd.read_html turns every table on the page into a DataFrame in one line:
# pip install pandas lxml html5lib
import pandas as pd
# Returns a list of DataFrames — one per
on the page
tables = pd.read_html("https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal)")
# Usually the first match is what you want
gdp = tables[0]
print(gdp.head())
# Filter and sort like any DataFrame
top_10 = gdp.head(10)
print(top_10[["Country", "GDP (nominal, billion USD)"]])
For semi-structured pages (Wikipedia, government data, financial reports), this is often all you need. Five seconds of code beats an hour of BeautifulSoup parsing.
BeautifulSoup for Hand-Rolled Tables
When tables are nested, irregular, or don’t use proper <table> tags, fall back to BeautifulSoup:
import requests
from bs4 import BeautifulSoup
resp = requests.get("https://example.com/data-page")
soup = BeautifulSoup(resp.text, "html.parser")
# Find the right table — by id, class, or position
table = soup.find("table", {"class": "data-grid"})
rows = []
for tr in table.find_all("tr"):
cells = [td.get_text(strip=True) for td in tr.find_all(["td", "th"])]
rows.append(cells)
# Convert to DataFrame
import pandas as pd
df = pd.DataFrame(rows[1:], columns=rows[0])
print(df.head())
Handling Dynamic / JavaScript-Rendered Tables
Single-page apps render tables client-side via JavaScript. requests + pd.read_html sees only an empty shell. Two paths to fix:
Path 1 — Find the underlying API. Open browser DevTools, Network tab, filter to XHR. The table data usually comes from a JSON endpoint. Hit that endpoint directly with requests:
import requests, pandas as pd
resp = requests.get(
"https://api.example.com/v2/countries/gdp",
headers={"Accept": "application/json"},
)
data = resp.json()
df = pd.DataFrame(data["items"])
print(df.head())
This is dramatically faster than rendering the page in a browser and parsing the resulting HTML. Always check for an API first.
Path 2 — Render the page with Playwright. When the API isn’t accessible (auth, anti-bot, generated state), Playwright runs the JS and returns the fully rendered HTML:
# pip install playwright
# python -m playwright install
from playwright.sync_api import sync_playwright
import pandas as pd
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://example.com/dynamic-table")
page.wait_for_selector("table.data-grid") # wait for data to load
html = page.content()
browser.close()
tables = pd.read_html(html)
print(tables[0].head())
Pagination and Multi-Page Tables
Many tables span multiple pages. Loop over pages, accumulate, then concat:
import pandas as pd
all_dfs = []
for page_num in range(1, 11):
url = f"https://example.com/data?page={page_num}"
tables = pd.read_html(url)
if not tables:
break
all_dfs.append(tables[0])
combined = pd.concat(all_dfs, ignore_index=True)
print(combined.shape, combined.head())
Cleaning Extracted Data
Tables on the web are messy. Common cleanups after extraction:
# Trim whitespace and remove footnote markers like '[1]', '[2]'
df = df.applymap(lambda x: str(x).strip().replace("[1]", "").replace("[2]", ""))
# Convert numeric columns (often imported as strings)
df["GDP"] = (
df["GDP"]
.str.replace(",", "")
.str.replace("$", "")
.str.replace("billion", "")
.astype(float)
)
# Drop rows that are all-NaN or just delimiters
df = df.dropna(how="all").reset_index(drop=True)
Common Pitfalls
Forgetting the lxml dependency.pd.read_html raises ImportError without lxml or html5lib installed. pip install lxml fixes it.
Skipping the API check. Scraping the rendered HTML when a JSON API exists is 10x slower and 10x more brittle. Always check DevTools first.
Ignoring robots.txt. Some sites prohibit scraping. Check robots.txt and Terms of Service before automating heavy traffic.
Rate-limit ignorance. Hitting a site with 1000 requests in a minute earns a ban. Add time.sleep(1) between requests, or use the Retry-After header from 429 responses.
Mishandling unicode.read_html defaults to a system encoding. If you see mojibake, pass encoding="utf-8" or fetch with requests first and parse the response.
FAQ
Q: pandas.read_html or BeautifulSoup?
A: read_html first — it’s one line. BeautifulSoup when the table isn’t a proper <table> tag or you need fine control over which cells to extract.
Q: How do I scrape a table behind login?
A: Authenticate via requests.Session() first (POST credentials, persist cookies), then GET the page. Playwright handles complex login flows automatically — point it at the login page, fill the form, wait for redirect.
Q: The page returns 403 / 429 — what do I do?
A: Set a real User-Agent header. Throttle to one request per second. If it’s still blocked, the site is using Cloudflare or similar — see anti-scraping countermeasures.
Q: How do I handle merged cells (rowspan / colspan)?
A: read_html and BeautifulSoup don’t always unfold spans correctly. Manual fix: walk the rows tracking active spans, repeating values across the implied cells.
Q: Polars or pandas for big tables?
A: For scraped data, pandas is fine. If you’ll process millions of rows downstream, switch to Polars after extraction (pl.from_pandas(df)).
Wrapping Up
Most table scraping comes down to three lines of pandas. When the page is static, pd.read_html is the right answer. When JS renders the data, look for the underlying JSON API first; fall back to Playwright if you must. BeautifulSoup is the escape hatch for irregular markup. Combine throttling, real User-Agents, and respect for robots.txt — and you’re a good citizen who also gets clean data.
Have you ever noticed how modern command-line applications display colorful output? Success messages appear in green, warnings shine in yellow, and errors stand out in red. Whether you’re building CLI tools, debugging with colored logging output, or creating interactive terminal applications, adding color to your Python strings can dramatically improve user experience and make information easier to scan and understand.
The good news is that printing colored text in Python is surprisingly simple and doesn’t require complex external dependencies. Whether you prefer using built-in ANSI escape codes, lightweight libraries like colorama and termcolor, or powerful formatting tools like rich, Python offers multiple approaches to suit your needs. Even beginners can master colored text output in just a few minutes.
In this tutorial, we’ll explore four different methods to print colored strings in Python, from the most straightforward ANSI codes to advanced formatting options. By the end, you’ll understand how terminal colors work, how to choose the right approach for your project, and how to implement colored output in real-world applications like logging systems and status reporters.
Quick Example: Print Colored Text
Can’t wait to see colored text? Here’s the simplest way using ANSI escape codes:
# quick_color.py
print("\033[92mSuccess! Operation completed.\033[0m")
print("\033[93mWarning: Check this value.\033[0m")
print("\033[91mError: Something went wrong.\033[0m")
Output:
Success! Operation completed. [displayed in green]
Warning: Check this value. [displayed in yellow]
Error: Something went wrong. [displayed in red]
Color is the difference between unreadable logs and useful ones.
What are ANSI Escape Codes?
ANSI escape codes are special character sequences that tell your terminal to change text color and styling. They follow the pattern \033[CODEm where CODE represents a specific color or style. The sequence always ends with \033[0m to reset formatting back to normal.
Here’s a quick reference table of common ANSI color codes:
Color
Foreground Code
Background Code
Black
30
40
Red
31
41
Green
32
42
Yellow
33
43
Blue
34
44
Magenta
35
45
Cyan
36
46
White
37
47
Bright Green
92
102
Bright Yellow
93
103
Bright Red
91
101
ANSI codes also support text styling. Use 1 for bold, 2 for dim, 4 for underline. Combine multiple codes with semicolons: \033[1;92m creates bold bright green text.
Using ANSI Escape Codes Directly
The most minimal approach is to use ANSI codes directly in your strings. This works on macOS, Linux, and modern Windows 10+ systems. No installation required—just pure Python.
# ansi_colors.py
# Define reusable color constants
GREEN = "\033[92m"
YELLOW = "\033[93m"
RED = "\033[91m"
BLUE = "\033[94m"
RESET = "\033[0m"
# Use in print statements
print(f"{GREEN}Success!{RESET} Operation completed.")
print(f"{YELLOW}Warning:{RESET} Check this configuration.")
print(f"{RED}Error:{RESET} Connection failed.")
print(f"{BLUE}Info:{RESET} Processing 1000 items.")
# Bold and underline
BOLD_GREEN = "\033[1;92m"
UNDERLINE = "\033[4m"
print(f"{BOLD_GREEN}{UNDERLINE}Important Notice{RESET}")
# Background colors
BG_RED = "\033[41m"
WHITE = "\033[37m"
print(f"{BG_RED}{WHITE}CRITICAL{RESET} System failure detected.")
Output:
Success! Operation completed. [green text]
Warning: Check this configuration. [yellow warning]
Error: Connection failed. [red error]
Info: Processing 1000 items. [blue info]
Important Notice [bold green underlined]
CRITICAL System failure detected. [white text on red background]
Red for errors, green for success. Cognitive labor saved.
Using the colorama Library (Cross-Platform)
While ANSI codes work great on most systems, Windows PowerShell can be finicky. The colorama library handles platform differences automatically and provides a cleaner API. Install it with pip install colorama.
# colorama_example.py
from colorama import Fore, Back, Style, init
# Initialize colorama (enables support on Windows)
init(autoreset=True)
print(f"{Fore.GREEN}Success!{Style.RESET_ALL} Your file was saved.")
print(f"{Fore.YELLOW}Warning:{Style.RESET_ALL} Disk space running low.")
print(f"{Fore.RED}Error:{Style.RESET_ALL} Database connection timeout.")
# Background colors
print(f"{Back.CYAN}{Fore.BLACK}Info{Style.RESET_ALL} Processing complete.")
# Text styling
print(f"{Style.BRIGHT}{Fore.BLUE}Important update available!{Style.RESET_ALL}")
print(f"{Style.DIM}{Fore.WHITE}Deprecated function{Style.RESET_ALL}")
# Combine multiple attributes
print(f"{Style.BRIGHT}{Fore.RED}{Back.YELLOW}ALERT{Style.RESET_ALL} Manual intervention required.")
Output:
Success! Your file was saved. [green]
Warning: Disk space running low. [yellow]
Error: Database connection timeout. [red]
Info Processing complete. [black text on cyan]
Important update available! [bright blue]
Deprecated function [dim white]
ALERT Manual intervention required. [bright red on yellow background]
Using the termcolor Library
For a lightweight alternative, termcolor offers a simple function-based API. Install with pip install termcolor. It’s ideal when you need quick colored output without extra features.
Success! Your changes were saved. [green]
Warning: This feature is deprecated. [yellow]
Error: Authentication failed. [red]
Debug: Current value = 42 [cyan]
CRITICAL System overload detected. [white on red]
INFO Backup completed successfully. [black on green]
Important [bold blue]
Deprecated [dark white]
Required [bold underlined red]
ALERT [bold yellow with blinking]
Same data, different signal-to-noise.
Using the rich Library for Advanced Formatting
The rich library is a powerhouse for terminal formatting, supporting colors, tables, panels, progress bars, and syntax highlighting. Install with pip install rich. Use it when you need professional-looking terminal output.
# rich_example.py
from rich.console import Console
from rich import print as rprint
from rich.panel import Panel
from rich.table import Table
console = Console()
# Simple colored text
console.print("[green]Success![/green] Operation completed.")
console.print("[yellow]Warning:[/yellow] Check system resources.")
console.print("[red]Error:[/red] Network timeout occurred.")
# Using rprint function (shorthand)
rprint("[bold blue]Important notice[/bold blue]")
rprint("[dim italic cyan]System info[/dim italic cyan]")
# Panels for highlighted messages
console.print(Panel("[green bold]✓ Deployment Successful[/green bold]", expand=False))
console.print(Panel("[yellow bold]⚠ Review Required[/yellow bold]", expand=False))
console.print(Panel("[red bold]✗ Critical Error[/red bold]", expand=False))
# Tables with colors
table = Table(title="System Status", show_header=True, header_style="bold magenta")
table.add_column("Component", style="cyan")
table.add_column("Status", style="green")
table.add_row("Database", "[green]Online[/green]")
table.add_row("Cache", "[yellow]Warming[/yellow]")
table.add_row("API", "[red]Offline[/red]")
console.print(table)
Output:
Success! Operation completed. [green]
Warning: Check system resources. [yellow]
Error: Network timeout occurred. [red]
Important notice [bold blue]
System info [dim italic cyan]
┌─ Deployment Successful ─┐ [green box]
⚠ Review Required [yellow box]
✗ Critical Error [red box]
[System Status table with colored columns]
Adding Color to Logging Output
Colored output becomes especially valuable in logging systems where different severity levels are immediately recognizable. Here’s how to add colors to Python’s logging module:
# colored_logging.py
import logging
from colorama import Fore, Style, init
init()
class ColoredFormatter(logging.Formatter):
COLORS = {
"DEBUG": Fore.CYAN,
"INFO": Fore.GREEN,
"WARNING": Fore.YELLOW,
"ERROR": Fore.RED,
"CRITICAL": f"{Fore.RED}{Style.BRIGHT}"
}
def format(self, record):
log_color = self.COLORS.get(record.levelname, Fore.WHITE)
record.levelname = f"{log_color}{record.levelname}{Style.RESET_ALL}"
return super().format(record)
# Configure logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler()
handler.setFormatter(ColoredFormatter(
"%(levelname)s - %(name)s - %(message)s"
))
logger.addHandler(handler)
# Use the logger
logger.debug("Debugging information retrieved")
logger.info("Application started successfully")
logger.warning("Low memory condition detected")
logger.error("Failed to connect to database")
logger.critical("System shutdown initiated")
Output:
DEBUG - __main__ - Debugging information retrieved [cyan]
INFO - __main__ - Application started successfully [green]
WARNING - __main__ - Low memory condition detected [yellow]
ERROR - __main__ - Failed to connect to database [red]
CRITICAL - __main__ - System shutdown initiated [bright red]
Real-Life Example: Colored CLI Status Reporter
Let’s build a practical application that monitors system status and displays results with color-coded output. This demonstrates how colors improve information clarity in real scenarios.
# system_status.py
import psutil
from colorama import Fore, Style, init
init(autoreset=True)
def get_status_color(value, thresholds):
"""Determine color based on value thresholds."""
if value < thresholds["warning"]:
return Fore.GREEN, "OK"
elif value < thresholds["critical"]:
return Fore.YELLOW, "WARNING"
else:
return Fore.RED, "CRITICAL"
def report_system_status():
"""Display colored system status report."""
print(f"{Style.BRIGHT}=== System Status Report ==={Style.RESET_ALL}\n")
# CPU Usage
cpu_percent = psutil.cpu_percent(interval=1)
color, status = get_status_color(cpu_percent,
{"warning": 50, "critical": 80})
print(f"CPU Usage: {color}{cpu_percent:>6.1f}%{Style.RESET_ALL} [{status}]")
# Memory Usage
mem = psutil.virtual_memory()
color, status = get_status_color(mem.percent,
{"warning": 70, "critical": 90})
print(f"Memory: {color}{mem.percent:>6.1f}%{Style.RESET_ALL} [{status}]")
# Disk Usage
disk = psutil.disk_usage("/")
color, status = get_status_color(disk.percent,
{"warning": 75, "critical": 90})
print(f"Disk Usage: {color}{disk.percent:>6.1f}%{Style.RESET_ALL} [{status}]")
# Process Count
process_count = len(psutil.pids())
color = Fore.GREEN if process_count < 200 else Fore.YELLOW
print(f"Processes: {color}{process_count:>6}{Style.RESET_ALL}")
print(f"\n{Fore.CYAN}Generated at {Style.BRIGHT}{get_timestamp()}{Style.RESET_ALL}")
def get_timestamp():
"""Return current timestamp."""
from datetime import datetime
return datetime.now().strftime("%Y-%m-%d %H:%M:%S")
if __name__ == "__main__":
report_system_status()
Output:
=== System Status Report ===
CPU Usage: 45.2% [OK] [green]
Memory: 68.3% [OK] [green]
Disk Usage: 82.1% [WARNING] [yellow]
Processes: 187 [green]
Generated at 2026-03-18 14:32:15 [cyan]
Frequently Asked Questions
Q1: Why don’t colors work on my Windows system?
By default, older Windows versions don’t support ANSI codes. Solution: Use the colorama library which automatically enables support on Windows, or enable ANSI in Windows 10+ through system settings. The init() function in colorama handles this automatically.
Q2: Can I use colored text in files instead of just the console?
ANSI codes appear as raw escape sequences in text files. If you want colors in saved files, use HTML or Markdown instead. For console output only, ANSI codes work perfectly. When writing to log files, consider stripping codes with libraries like colorama.deinit() or use separate formatting for file vs. console output.
Q3: Which library should I choose for my project?
Choose based on your needs: Use raw ANSI codes for minimal dependencies; use colorama for reliable cross-platform support; use termcolor for simple, lightweight colored text; use rich for advanced formatting, tables, and panels. For most projects, colorama offers the best balance.
Q4: How do I create custom colors beyond the basic set?
Standard ANSI codes provide 8 basic colors plus 8 bright variants. For 256-color support, use escape codes like \033[38;5;196m (where 196 is a color index). For true color (16 million colors), use RGB format: \033[38;2;255;0;0m for red. The rich library handles 256-color and true color automatically.
Q5: Are ANSI colors supported in Jupyter notebooks?
Most Jupyter implementations don’t render terminal ANSI colors. However, the rich library integrates beautifully with Jupyter and automatically detects the environment to output proper HTML formatting. For Jupyter notebooks, rich is your best choice.
Q6: What about accessibility? Are colored outputs accessible?
Never rely on color alone to convey information. Always combine color with text labels, symbols, or other indicators. For example, use both color and text: “[ERROR]” in red rather than just red text. This ensures visually impaired users can understand your application.
Conclusion
Printing colored strings in Python is simple, powerful, and available through multiple approaches. Whether you choose raw ANSI codes for minimal overhead, colorama for cross-platform reliability, termcolor for lightweight simplicity, or rich for professional formatting, you now have the knowledge to add vibrant output to your applications.
Colored output transforms CLI applications from bland text dumps into clear, scannable interfaces. Your logging becomes easier to understand, your error messages stand out, and your users appreciate the improved clarity. Start with the approach that fits your project, and remember: colors enhance understanding—never use them as the only way to communicate critical information.
Ready to explore more? Check out the official documentation for colorama and rich to discover even more advanced features and capabilities.
Every non-trivial Python application needs a configuration file. Whether you are storing database credentials, feature flags, API endpoints, or user preferences, you need a format that is easy for humans to read and edit, but structured enough for your code to parse reliably. For years, Python developers defaulted to INI files with configparser, but INI has serious limitations — no native data types, no nested sections, and everything is a string. TOML fixes all of that while staying just as readable, and since Python 3.11, you can parse it without installing anything.
Python 3.11 added tomllib to the standard library for reading TOML files. For writing TOML, the community standard is tomli_w, a lightweight package you can install with pip install tomli_w. Between these two tools, you have everything you need to replace INI, JSON, or YAML configuration files with something cleaner and more powerful. If you are on Python 3.10 or earlier, the tomli package provides the same reading API as tomllib.
In this article we will cover everything you need to work with TOML in Python. We will start with a quick example to get you up and running, then explain what TOML is and how it compares to INI, JSON, and YAML. From there we will dive into reading TOML files, understanding TOML data types, working with nested tables and arrays, writing TOML files, and handling common patterns like environment-specific configs. We will finish with a real-life project that builds a complete application configuration system. By the end, you will be ready to use TOML for all your Python configuration needs.
Python TOML Configuration: Quick Example
Here is a complete example that creates a TOML configuration file and reads it back. You can run this immediately to see TOML in action.
# quick_example.py
import tomllib
from pathlib import Path
# First, create a sample TOML config file
config_content = """
[app]
name = "MyWebApp"
version = "2.1.0"
debug = false
[database]
host = "localhost"
port = 5432
name = "myapp_db"
"""
Path("config.toml").write_text(config_content)
# Now read it back
with open("config.toml", "rb") as f:
config = tomllib.load(f)
print(f"App: {config['app']['name']} v{config['app']['version']}")
print(f"Debug mode: {config['app']['debug']}")
print(f"Database: {config['database']['host']}:{config['database']['port']}")
Notice something important: the debug value came back as a Python bool (capital-F False), and the port came back as an int. TOML preserves data types natively — unlike INI files where everything is a string and you have to convert manually. The tomllib.load() function reads the file in binary mode (that is why we open with "rb") and returns a regular Python dictionary.
Want to go deeper? Below we explain the TOML format in detail, compare it to alternatives, and build a production-ready configuration system you can drop into any project.
What Is TOML and Why Use It Instead of INI?
TOML stands for “Tom’s Obvious, Minimal Language.” It was created by Tom Preston-Werner (co-founder of GitHub) as a configuration file format that is easy for humans to read and write while being unambiguous for machines to parse. If you have ever used an INI file or edited a pyproject.toml for a Python package, you have already seen TOML in action — it is the official configuration format for Python packaging tools like pip, setuptools, and poetry.
The key advantage of TOML over INI is that TOML has real data types. In an INI file, the value port = 5432 is the string "5432", and you need to call int() to convert it. In TOML, that same line produces an actual integer. TOML also supports nested sections (tables within tables), arrays, dates, and inline tables — none of which INI can handle without awkward workarounds.
Here is how TOML compares to the other popular configuration formats.
Feature
TOML
INI
JSON
YAML
Human readable
Excellent
Good
Moderate
Good
Comments
Yes (#)
Yes (;/#)
No
Yes (#)
Native data types
String, int, float, bool, datetime, array, table
Strings only
String, number, bool, null, array, object
All of the above plus anchors
Nested sections
Yes (dotted keys and tables)
No
Yes
Yes
Python stdlib support
Read only (tomllib, 3.11+)
Read/write (configparser)
Read/write (json)
No (needs PyYAML)
Trailing commas
Yes (in arrays)
N/A
No
N/A
Security concerns
Minimal
Minimal
Minimal
High (code execution risk)
TOML hits the sweet spot for configuration files: it is more expressive than INI, more readable than JSON (which lacks comments), and safer than YAML (which can execute arbitrary code if loaded unsafely). The fact that Python’s packaging ecosystem chose TOML as its standard format tells you a lot about where the community is headed. Now let us learn the format.
Understanding TOML Syntax
Before writing code, it helps to understand what a TOML file looks like. The format is simple — if you can read an INI file, you can read TOML. The main building blocks are key-value pairs, tables (sections), and arrays.
# sample_config.toml
# This is a comment — TOML uses the hash symbol
# Basic key-value pairs (top level)
title = "My Application"
version = 1
# A table (like an INI section)
[owner]
name = "Alice Johnson"
email = "alice@example.com"
# Nested table using dotted notation
[database.primary]
host = "db.example.com"
port = 5432
enabled = true
[database.replica]
host = "replica.example.com"
port = 5433
enabled = false
# Array of tables (list of objects)
[[servers]]
name = "alpha"
ip = "10.0.0.1"
role = "web"
[[servers]]
name = "beta"
ip = "10.0.0.2"
role = "worker"
Let us read this file and see how Python interprets each section. The tomllib.load() function converts the entire TOML file into a nested Python dictionary, preserving all the structure and data types.
# read_syntax.py
import tomllib
from pathlib import Path
# Create the config file
toml_content = '''
title = "My Application"
version = 1
[owner]
name = "Alice Johnson"
email = "alice@example.com"
[database.primary]
host = "db.example.com"
port = 5432
enabled = true
[database.replica]
host = "replica.example.com"
port = 5433
enabled = false
[[servers]]
name = "alpha"
ip = "10.0.0.1"
role = "web"
[[servers]]
name = "beta"
ip = "10.0.0.2"
role = "worker"
'''
Path("app_config.toml").write_text(toml_content)
with open("app_config.toml", "rb") as f:
config = tomllib.load(f)
# Top-level keys
print(f"Title: {config['title']}")
print(f"Version: {config['version']} (type: {type(config['version']).__name__})")
# Table access
print(f"\nOwner: {config['owner']['name']}")
# Nested tables via dotted keys
print(f"\nPrimary DB: {config['database']['primary']['host']}")
print(f"Replica DB: {config['database']['replica']['host']}")
# Array of tables becomes a list of dicts
print(f"\nServers ({len(config['servers'])}):")
for server in config['servers']:
print(f" {server['name']} ({server['ip']}) - {server['role']}")
Output:
Title: My Application
Version: 1 (type: int)
Owner: Alice Johnson
Primary DB: db.example.com
Replica DB: replica.example.com
Servers (2):
alpha (10.0.0.1) - web
beta (10.0.0.2) - worker
The key things to notice: [database.primary] in TOML becomes config['database']['primary'] in Python — the dot creates nesting. The [[servers]] syntax (double brackets) creates an array of tables — each repeated [[servers]] block adds another dictionary to a list. And version = 1 became a Python int, not a string. These are the three patterns you will use most often in TOML configuration files.
INI files: everything is a string. TOML files: everything is what it should be. Not a hard choice.
Reading TOML Files With tomllib
The tomllib module (Python 3.11+) provides two functions: load() for reading from a file object and loads() for parsing a string. Both return a Python dictionary. The file must be opened in binary mode ("rb") because TOML files are always UTF-8, and tomllib handles the decoding internally to avoid encoding issues.
# reading_toml.py
import tomllib
# Method 1: Load from a file
with open("config.toml", "rb") as f:
config = tomllib.load(f)
print(f"From file: {config['app']['name']}")
# Method 2: Parse from a string
toml_string = '''
[server]
host = "0.0.0.0"
port = 8080
workers = 4
'''
config_from_string = tomllib.loads(toml_string)
print(f"From string: {config_from_string['server']['host']}:{config_from_string['server']['port']}")
# The result is a regular dict — you can use all dict methods
print(f"\nTop-level keys: {list(config_from_string.keys())}")
print(f"Server config: {dict(config_from_string['server'])}")
Output:
From file: MyWebApp
From string: 0.0.0.0:8080
Top-level keys: ['server']
Server config: {'host': '0.0.0.0', 'port': 8080, 'workers': 4}
One gotcha to watch out for: tomllib.load() requires binary mode ("rb"), not text mode ("r"). If you forget, you will get a TypeError. This is by design — it prevents encoding mismatches. The loads() function takes a regular string, which is convenient for testing or when your TOML content comes from an environment variable or API response rather than a file.
TOML Data Types and Python Mapping
One of TOML’s biggest advantages is its rich type system. Every value in a TOML file maps to a specific Python type, and the parser handles the conversion automatically. Let us see each type in action.
# data_types.py
import tomllib
from datetime import datetime, date, time
toml_data = '''
# Strings
name = "Alice"
path = 'C:\Users\alice' # Single quotes = literal (no escapes)
bio = """
This is a
multi-line string."""
# Numbers
integer_val = 42
negative = -17
float_val = 3.14
scientific = 6.022e23
hex_val = 0xff
bin_val = 0b1010
# Booleans
enabled = true
debug = false
# Dates and times
created = 2025-01-15T10:30:00
birthday = 2025-01-15
alarm = 07:30:00
# Arrays (can be mixed types, but best practice is same type)
ports = [8080, 8081, 8082]
features = ["auth", "logging", "caching"]
# Inline table
point = {x = 10, y = 20}
'''
config = tomllib.loads(toml_data)
# Check the Python types
print(f"name: {config['name']!r} ({type(config['name']).__name__})")
print(f"integer_val: {config['integer_val']} ({type(config['integer_val']).__name__})")
print(f"float_val: {config['float_val']} ({type(config['float_val']).__name__})")
print(f"enabled: {config['enabled']} ({type(config['enabled']).__name__})")
print(f"created: {config['created']} ({type(config['created']).__name__})")
print(f"birthday: {config['birthday']} ({type(config['birthday']).__name__})")
print(f"ports: {config['ports']} ({type(config['ports']).__name__})")
print(f"point: {config['point']} ({type(config['point']).__name__})")
print(f"hex_val: {config['hex_val']} ({type(config['hex_val']).__name__})")
print(f"bio: {config['bio']!r}")
Every TOML type has a clean Python equivalent. Strings become str, integers become int (even hex and binary), floats become float, booleans become bool, dates and times become datetime.date, datetime.time, or datetime.datetime, arrays become list, and tables (including inline tables) become dict. Compare this to INI where port = 5432 gives you the string "5432" and you have to write config.getint('section', 'port') to get a number. TOML eliminates that entire category of boilerplate.
Writing TOML Files With tomli_w
The standard library only handles reading TOML. For writing, the community standard is tomli_w, a small package that converts Python dictionaries back to TOML format. Install it with pip install tomli_w.
# writing_toml.py
import tomli_w
import tomllib
# Build a configuration as a Python dictionary
config = {
"app": {
"name": "DataPipeline",
"version": "1.0.0",
"debug": False,
},
"database": {
"host": "localhost",
"port": 5432,
"name": "pipeline_db",
"pool_size": 10,
},
"logging": {
"level": "INFO",
"file": "/var/log/pipeline.log",
"rotate_mb": 50,
},
"features": ["retry", "caching", "metrics"],
}
# Write to file
with open("pipeline_config.toml", "wb") as f:
tomli_w.dump(config, f)
# Write to string (useful for previewing)
toml_string = tomli_w.dumps(config)
print("Generated TOML:\n")
print(toml_string)
# Verify by reading it back
with open("pipeline_config.toml", "rb") as f:
verified = tomllib.load(f)
print(f"Verified: {verified['app']['name']} v{verified['app']['version']}")
print(f"Features: {verified['features']}")
Output:
Generated TOML:
features = [
"retry",
"caching",
"metrics",
]
[app]
name = "DataPipeline"
version = "1.0.0"
debug = false
[database]
host = "localhost"
port = 5432
name = "pipeline_db"
pool_size = 10
[logging]
level = "INFO"
file = "/var/log/pipeline.log"
rotate_mb = 50
Verified: DataPipeline v1.0.0
Features: ['retry', 'caching', 'metrics']
Like tomllib.load(), the tomli_w.dump() function uses binary mode ("wb"). The dumps() function returns a string, which is handy for logging or previewing. Notice that tomli_w formats the output cleanly with proper indentation and section headers. The round-trip (write then read) produces identical data, so you can safely use this for configuration management tools that need to update config files programmatically.
TOML syntax looks friendly until you forget a closing bracket in a nested table. Then it’s personal.
Handling TOML Parsing Errors
When a TOML file has syntax errors, tomllib raises a TOMLDecodeError with a helpful message that includes the line and column number. You should always wrap your config loading in a try-except block so your application fails gracefully with a clear error message instead of a cryptic traceback.
# error_handling.py
import tomllib
import sys
# Example 1: Invalid TOML syntax
bad_toml = '''
[database]
host = "localhost"
port = not_a_number
'''
try:
config = tomllib.loads(bad_toml)
except tomllib.TOMLDecodeError as e:
print(f"TOML syntax error: {e}")
# Example 2: Safe config loading function
def load_config(filepath, required_keys=None):
"""Load and validate a TOML configuration file."""
try:
with open(filepath, "rb") as f:
config = tomllib.load(f)
except FileNotFoundError:
print(f"Error: Config file '{filepath}' not found.")
print("Create one from config.example.toml")
return None
except tomllib.TOMLDecodeError as e:
print(f"Error: Invalid TOML in '{filepath}': {e}")
return None
# Validate required keys
if required_keys:
missing = [k for k in required_keys if k not in config]
if missing:
print(f"Error: Missing required sections: {missing}")
return None
return config
# Test with a valid file
config = load_config("config.toml", required_keys=["app", "database"])
if config:
print(f"\nConfig loaded: {config['app']['name']}")
else:
print("\nFailed to load config")
Output:
TOML syntax error: Invalid value (at line 4, column 8)
Config loaded: MyWebApp
The load_config() function demonstrates the defensive loading pattern you should use in production code. It handles three failure modes: file not found, invalid TOML syntax, and missing required sections. This approach gives users clear, actionable error messages instead of stack traces. You could extend this with schema validation using a library like pydantic if you need to verify value types and ranges as well.
Environment-Specific Configuration
A common pattern in real applications is having different configurations for development, staging, and production. TOML handles this cleanly by using separate tables for each environment, with a shared defaults section.
# env_config.py
import tomllib
from pathlib import Path
import copy
# Create a multi-environment config
config_content = '''
[defaults]
log_level = "INFO"
workers = 2
cache_ttl = 300
[defaults.database]
port = 5432
pool_size = 5
[development]
log_level = "DEBUG"
workers = 1
[development.database]
host = "localhost"
name = "myapp_dev"
pool_size = 2
[production]
log_level = "WARNING"
workers = 8
[production.database]
host = "db.production.internal"
name = "myapp_prod"
pool_size = 20
'''
Path("environments.toml").write_text(config_content)
def get_config(env_name):
"""Load config for a specific environment, merged with defaults."""
with open("environments.toml", "rb") as f:
all_configs = tomllib.load(f)
# Start with defaults
defaults = all_configs.get("defaults", {})
config = copy.deepcopy(defaults)
# Merge environment-specific overrides
env_overrides = all_configs.get(env_name, {})
for key, value in env_overrides.items():
if isinstance(value, dict) and key in config and isinstance(config[key], dict):
config[key].update(value) # Merge nested dicts
else:
config[key] = value # Override scalar values
return config
# Get config for each environment
for env in ["development", "production"]:
config = get_config(env)
print(f"\n--- {env.upper()} ---")
print(f"Log level: {config['log_level']}")
print(f"Workers: {config['workers']}")
print(f"DB: {config['database']['host']}:{config['database']['port']}")
print(f"Pool size: {config['database']['pool_size']}")
Output:
--- DEVELOPMENT ---
Log level: DEBUG
Workers: 1
DB: localhost:5432
Pool size: 2
--- PRODUCTION ---
Log level: WARNING
Workers: 8
DB: db.production.internal:5432
Pool size: 20
The merging function starts with a deep copy of the defaults, then overlays the environment-specific values on top. Nested dictionaries (like database) are merged rather than replaced, so you only need to specify the values that differ from the defaults. Notice that the port value (5432) was inherited from defaults in both environments because neither development nor production overrode it. This pattern keeps your configuration DRY while still allowing full customization per environment.
One config file per environment. tomllib.load() picks the right one. No more if-else chains.
Let us build a practical configuration system that you can drop into any Python project. It reads from a TOML file, supports defaults, validates required fields, and provides convenient dot-notation access to nested values.
# config_manager.py
import tomllib
import tomli_w
import copy
from pathlib import Path
class AppConfig:
"""A configuration manager backed by TOML files."""
def __init__(self, filepath="config.toml"):
self.filepath = Path(filepath)
self._data = {}
self._defaults = {
"app": {"name": "Unnamed", "version": "0.0.0", "debug": False},
"server": {"host": "127.0.0.1", "port": 8000, "workers": 2},
"logging": {"level": "INFO", "file": None},
}
def load(self):
"""Load config from file, merged with defaults."""
self._data = copy.deepcopy(self._defaults)
if self.filepath.exists():
with open(self.filepath, "rb") as f:
file_data = tomllib.load(f)
self._deep_merge(self._data, file_data)
print(f"Loaded config from {self.filepath}")
else:
print(f"No config file found, using defaults")
return self
def _deep_merge(self, base, override):
"""Recursively merge override dict into base dict."""
for key, value in override.items():
if key in base and isinstance(base[key], dict) and isinstance(value, dict):
self._deep_merge(base[key], value)
else:
base[key] = value
def get(self, dotted_key, default=None):
"""Access nested values with dot notation: config.get('server.port')."""
keys = dotted_key.split(".")
current = self._data
for key in keys:
if isinstance(current, dict) and key in current:
current = current[key]
else:
return default
return current
def set(self, dotted_key, value):
"""Set a nested value: config.set('server.port', 9000)."""
keys = dotted_key.split(".")
current = self._data
for key in keys[:-1]:
current = current.setdefault(key, {})
current[keys[-1]] = value
def save(self):
"""Write current config back to the TOML file."""
with open(self.filepath, "wb") as f:
tomli_w.dump(self._data, f)
print(f"Config saved to {self.filepath}")
def show(self):
"""Display the current configuration."""
print(tomli_w.dumps(self._data))
# --- Demo usage ---
if __name__ == "__main__":
# Create a sample config file
sample = """
[app]
name = "TaskTracker"
version = "3.2.1"
debug = true
[server]
host = "0.0.0.0"
port = 5000
[database]
url = "postgresql://localhost/tasks"
pool_size = 10
"""
Path("config.toml").write_text(sample)
# Load and use the config manager
config = AppConfig("config.toml").load()
print(f"\nApp: {config.get('app.name')} v{config.get('app.version')}")
print(f"Server: {config.get('server.host')}:{config.get('server.port')}")
print(f"Workers: {config.get('server.workers')}") # From defaults
print(f"Debug: {config.get('app.debug')}")
print(f"DB: {config.get('database.url')}")
print(f"Missing key: {config.get('cache.redis', 'not configured')}")
# Modify and save
config.set("server.workers", 8)
config.set("cache.redis", "redis://localhost:6379")
config.save()
print("\nUpdated config:")
config.show()
Output:
Loaded config from config.toml
App: TaskTracker v3.2.1
Server: 0.0.0.0:5000
Workers: 2
Debug: True
DB: postgresql://localhost/tasks
Missing key: not configured
Config saved to config.toml
Updated config:
[app]
name = "TaskTracker"
version = "3.2.1"
debug = true
[server]
host = "0.0.0.0"
port = 5000
workers = 8
[logging]
level = "INFO"
[database]
url = "postgresql://localhost/tasks"
pool_size = 10
[cache]
redis = "redis://localhost:6379"
This AppConfig class gives you a clean, reusable configuration layer. The get() method supports dot-notation for nested access (config.get('server.port')), the set() method creates intermediate dictionaries automatically, and the deep merge ensures that defaults fill in any gaps without overwriting values from the config file. You can extend this with environment variable overrides (reading os.environ and merging on top) or schema validation with pydantic for type checking.
Frequently Asked Questions
What if I am using Python 3.10 or earlier?
Install the tomli package with pip install tomli. It has the exact same API as the built-in tomllib. A common pattern is to use a try/except import: try: import tomllib except ModuleNotFoundError: import tomli as tomllib. This way your code works on both old and new Python versions without changes. Many popular packages like pip itself use this exact pattern.
Why does tomllib only read TOML and not write it?
The Python core developers decided to include only the reading side because there is a clear, agreed-upon way to parse TOML, but writing TOML involves style choices (formatting, ordering, comment preservation) that are harder to standardize. The tomli_w package fills this gap perfectly and is maintained by the same developer who wrote tomli (which became tomllib). Installing one extra package for writing is a small price for having a stable reading implementation in the standard library.
How does pyproject.toml relate to this?
The pyproject.toml file is a TOML file that Python packaging tools use to define project metadata, dependencies, and build configuration. It follows the exact same TOML syntax covered in this article. You read it with tomllib.load() just like any other TOML file. Tools like pip, setuptools, poetry, and hatch all read from pyproject.toml — it replaced the older setup.py and setup.cfg approach for modern Python projects.
Can I preserve comments when modifying a TOML file?
Neither tomllib nor tomli_w preserves comments — when you load a TOML file, comments are discarded, and when you write it back, only the data is included. If comment preservation is important (for example, in a user-facing config file), consider the tomlkit package (pip install tomlkit). It parses TOML while preserving formatting, comments, and whitespace, making it ideal for tools that need to edit config files without disturbing the user’s layout.
Should I use TOML or YAML for my project?
For application configuration files, TOML is generally the better choice. It is simpler, safer (no code execution risk), and has standard library support in Python. YAML is better suited for complex data serialization tasks where you need features like anchors, references, and custom types — but those same features are what make YAML a security risk if you use yaml.load() instead of yaml.safe_load(). The Python community’s adoption of TOML for pyproject.toml is a strong signal that TOML is the preferred format for configuration going forward.
How do I migrate from configparser (INI) to TOML?
The structure is similar enough that migration is usually straightforward. INI sections become TOML tables, and key-value pairs stay the same syntactically. The main changes are: remove the type conversion calls (getint(), getboolean(), etc.) because TOML handles types natively, convert comma-separated values to TOML arrays, and add proper quoting to string values. For nested sections, replace [section:subsection] with [section.subsection]. A typical migration takes less than an hour even for large config files.
Conclusion
You now know how to use TOML for Python configuration files using tomllib (reading) and tomli_w (writing). We covered the TOML syntax and its data types, how to read and parse TOML files, how to write TOML from Python dictionaries, error handling and validation, environment-specific configuration patterns, and a complete configuration manager class you can use in your own projects. TOML gives you the readability of INI with the expressiveness of JSON and the safety that YAML lacks — it is the right default choice for Python configuration files in 2025 and beyond.
Try extending the AppConfig class we built — add environment variable overrides, integrate it with pydantic for schema validation, or build a CLI tool that reads and modifies TOML configs. The patterns you learned here apply to any Python project that needs configuration management.
Discord has over 150 million monthly active users, and bots are what keep servers running smoothly. Whether you want to build a moderation bot that auto-kicks spammers, a music bot that plays audio in voice channels, or a utility bot that fetches data from APIs and posts it in a channel, Python and the discord.py library make it surprisingly approachable. If you have ever wished your Discord server could do something automatically, a bot is the answer.
The discord.py library handles all the heavy lifting — connecting to Discord’s WebSocket gateway, managing authentication, parsing events, and sending messages. You just need Python 3.8 or higher and a free Discord account. You will install discord.py with a single pip install command, and within minutes you will have a bot running in your own server responding to commands.
In this article we will walk through the entire process from start to finish. We will begin with setting up the Discord Developer Portal and creating your bot application. Then we will cover bot events, text commands, modern slash commands, and rich embed messages. Along the way we will explain how Discord’s event-driven architecture works and why it matters. Finally, we will build a complete Server Welcome Bot as a real-life project that greets new members, assigns roles, and logs activity.
Discord Bot in Python: Quick Example
Here is the simplest possible Discord bot that connects to a server and responds when someone types !hello. You can copy this code, replace the token placeholder with your actual bot token (we will show you how to get one in the next section), and have a working bot in under a minute.
# quick_example.py
import discord
intents = discord.Intents.default()
intents.message_content = True # Required to read message text
client = discord.Client(intents=intents)
@client.event
async def on_ready():
print(f"Bot is online as {client.user}")
@client.event
async def on_message(message):
if message.author == client.user:
return # Ignore the bot's own messages
if message.content == "!hello":
await message.channel.send(f"Hello, {message.author.display_name}!")
client.run("YOUR_BOT_TOKEN_HERE")
Output (in your terminal):
Bot is online as MyBot#1234
When someone types !hello in any channel the bot can see, it replies with a personalized greeting. The on_ready event fires once when the bot successfully connects to Discord, and the on_message event fires every time a message is sent in a channel the bot has access to. The intents object tells Discord what types of events your bot needs — we enable message_content because reading message text requires explicit permission since 2022.
Want to learn how to set up your bot properly, use modern slash commands, and build something more substantial? Below we cover everything step by step.
Setting Up the Discord Developer Portal
Before you write any code, you need to create a bot application on Discord’s Developer Portal. This gives you a bot token (like a password for your bot) and lets you configure what permissions the bot needs. Here is the step-by-step process.
Go to https://discord.com/developers/applications and log in with your Discord account. Click “New Application” in the top right, give your application a name (this will be your bot’s display name), and click “Create.” On the left sidebar, click “Bot” to open the bot settings. Under the “Privileged Gateway Intents” section, enable Message Content Intent — this is required for your bot to read the text content of messages. You may also want to enable Server Members Intent if your bot needs to track when members join or leave.
To get your bot token, click “Reset Token” on the Bot page. Copy the token immediately — Discord only shows it once. Store it somewhere safe and never share it publicly or commit it to a Git repository. Anyone with your token can control your bot. If you accidentally expose it, go back to the Developer Portal and reset it immediately.
To invite your bot to a server, go to the “OAuth2” section in the left sidebar, then “URL Generator.” Under “Scopes” check bot and applications.commands (for slash commands). Under “Bot Permissions” select the permissions your bot needs — for this tutorial, check Send Messages, Read Message History, Manage Roles, and Embed Links. Copy the generated URL at the bottom and open it in your browser to invite the bot to your test server.
Installing discord.py
Install the library using pip. We recommend installing inside a virtual environment to keep your project dependencies isolated.
# install_discord.py
# Run these commands in your terminal (not in a Python file):
# pip install discord.py
#
# To verify the installation:
import discord
print(f"discord.py version: {discord.__version__}")
Output:
discord.py version: 2.5.2
The discord.py library installs with all its dependencies, including aiohttp for async HTTP requests and websockets for the real-time connection to Discord’s gateway. The version number may differ depending on when you install it, but any version 2.x will work with the code in this tutorial. If you need voice channel support, install discord.py[voice] instead, which includes the PyNaCl library for audio encoding.
Your bot just went online. Time to test it by talking to yourself in an empty Discord server like a normal person.
Understanding Bot Events
Discord bots are event-driven. Instead of running code in a loop, your bot sits idle and waits for Discord to send it events — a message was sent, a member joined, a reaction was added, and so on. You register handler functions for the events you care about using the @client.event decorator. Discord’s gateway sends events over a WebSocket connection, and discord.py automatically parses them into Python objects for you.
Here are the most commonly used events and when they fire:
Event
When It Fires
Common Use
on_ready
Bot connects to Discord
Print status message, initialize data
on_message
Any message is sent
Text commands, auto-moderation
on_member_join
A user joins the server
Welcome messages, auto-role assignment
on_member_remove
A user leaves/is kicked
Goodbye messages, logging
on_reaction_add
Someone reacts to a message
Reaction roles, polls
on_message_delete
A message is deleted
Audit logging, anti-spam
Let us see a bot that responds to multiple events. This example logs when the bot starts, when messages are sent, and when new members join the server.
# event_demo.py
import discord
from datetime import datetime
intents = discord.Intents.default()
intents.message_content = True
intents.members = True # Required for member join/leave events
client = discord.Client(intents=intents)
@client.event
async def on_ready():
print(f"[{datetime.now():%H:%M:%S}] {client.user} is online!")
print(f"Connected to {len(client.guilds)} server(s)")
@client.event
async def on_message(message):
if message.author.bot:
return # Ignore all bot messages
print(f"[{datetime.now():%H:%M:%S}] {message.author}: {message.content}")
@client.event
async def on_member_join(member):
print(f"[{datetime.now():%H:%M:%S}] {member.display_name} joined {member.guild.name}")
client.run("YOUR_BOT_TOKEN_HERE")
Output (in your terminal):
[14:30:00] MyBot#1234 is online!
Connected to 1 server(s)
[14:30:15] Alice: Hello everyone!
[14:30:22] Bob: Hey Alice!
[14:31:05] Charlie joined My Test Server
Notice that we check message.author.bot instead of comparing to client.user. This is a best practice because it prevents your bot from responding to messages from any bot, not just itself. Without this check, two bots could get into an infinite loop responding to each other. The intents.members = True line is required for the on_member_join event to work — Discord requires you to explicitly opt into member-related events for privacy reasons.
Creating Text Commands
Text commands (also called prefix commands) are the classic way to interact with a Discord bot — users type a prefix like ! followed by a command name. While Discord now recommends slash commands for new bots, text commands are still widely used and are simpler to understand for beginners. The discord.py library provides a commands.Bot class that makes text commands easy to build.
# text_commands.py
import discord
from discord.ext import commands
intents = discord.Intents.default()
intents.message_content = True
# Use commands.Bot instead of discord.Client for command support
bot = commands.Bot(command_prefix="!", intents=intents)
@bot.event
async def on_ready():
print(f"{bot.user} is online with prefix '!'")
@bot.command(name="ping")
async def ping_command(ctx):
"""Check if the bot is responsive."""
latency = round(bot.latency * 1000) # Convert to milliseconds
await ctx.send(f"Pong! Latency: {latency}ms")
@bot.command(name="say")
async def say_command(ctx, *, text: str):
"""Make the bot repeat your message."""
await ctx.send(text)
@bot.command(name="userinfo")
async def userinfo_command(ctx, member: discord.Member = None):
"""Show information about a user."""
member = member or ctx.author # Default to the command caller
joined = member.joined_at.strftime("%B %d, %Y") if member.joined_at else "Unknown"
roles = ", ".join(role.name for role in member.roles[1:]) or "No roles"
await ctx.send(
f"**{member.display_name}**\n"
f"Joined: {joined}\n"
f"Roles: {roles}\n"
f"Account created: {member.created_at.strftime('%B %d, %Y')}"
)
bot.run("YOUR_BOT_TOKEN_HERE")
Output (in Discord chat):
User: !ping
Bot: Pong! Latency: 45ms
User: !say Hello from the bot!
Bot: Hello from the bot!
User: !userinfo
Bot: **Alice**
Joined: January 15, 2025
Roles: Moderator, Developer
Account created: March 10, 2020
The commands.Bot class extends discord.Client with a command framework. Instead of manually parsing message content in on_message, you define commands with the @bot.command() decorator. The ctx parameter (short for context) gives you access to the message, the channel, the author, and the server — everything you need to respond. The * in *, text: str tells discord.py to capture all remaining text as a single string instead of splitting on spaces. The member: discord.Member type hint enables automatic user lookup — users can mention someone or type their name and discord.py will find the matching member.
Slash commands: because apparently typing / is cooler than typing ! now.
Creating Slash Commands
Slash commands are Discord’s modern command system. When a user types /, Discord shows a menu of available commands with descriptions and parameter hints — no guessing what commands exist or what arguments they need. Discord strongly recommends slash commands for all new bots because they provide a better user experience and integrate with Discord’s permission system.
# slash_commands.py
import discord
from discord import app_commands
intents = discord.Intents.default()
client = discord.Client(intents=intents)
tree = app_commands.CommandTree(client)
@tree.command(name="ping", description="Check bot latency")
async def ping_slash(interaction: discord.Interaction):
latency = round(client.latency * 1000)
await interaction.response.send_message(f"Pong! Latency: {latency}ms")
@tree.command(name="roll", description="Roll a dice with a given number of sides")
@app_commands.describe(sides="Number of sides on the dice (default: 6)")
async def roll_slash(interaction: discord.Interaction, sides: int = 6):
import random
result = random.randint(1, sides)
await interaction.response.send_message(f"You rolled a **{result}** (d{sides})")
@tree.command(name="poll", description="Create a simple yes/no poll")
@app_commands.describe(question="The question to ask")
async def poll_slash(interaction: discord.Interaction, question: str):
message = await interaction.response.send_message(f"**Poll:** {question}")
# Fetch the message object to add reactions
poll_message = await interaction.original_response()
await poll_message.add_reaction("✅")
await poll_message.add_reaction("❌")
@client.event
async def on_ready():
await tree.sync() # Register commands with Discord
print(f"{client.user} is online with slash commands!")
client.run("YOUR_BOT_TOKEN_HERE")
Output (in Discord chat):
User types: /ping
Bot: Pong! Latency: 38ms
User types: /roll sides:20
Bot: You rolled a **14** (d20)
User types: /poll question:Should we do movie night?
Bot: **Poll:** Should we do movie night?
✅ ❌ (reactions added automatically)
Slash commands use a different API pattern than text commands. Instead of ctx, you receive an interaction object, and you must respond with interaction.response.send_message() within 3 seconds — otherwise Discord shows a “This interaction failed” error to the user. The app_commands.CommandTree manages your slash commands, and tree.sync() in on_ready registers them with Discord. Note that syncing can take up to an hour for global commands, but you can speed this up during development by syncing to a specific server using tree.sync(guild=discord.Object(id=YOUR_SERVER_ID)).
The @app_commands.describe() decorator adds parameter descriptions that Discord shows in the command menu. Type hints like sides: int tell Discord to validate the input — if a user types text instead of a number, Discord will show an error before the command even runs.
Creating Embed Messages
Plain text messages work fine, but embed messages look professional. Embeds support titles, descriptions, fields, colors, thumbnails, and footers — all rendered in a rich card format. They are perfect for displaying structured information like user profiles, search results, or status dashboards.
# embed_demo.py
import discord
from discord import app_commands
from datetime import datetime
intents = discord.Intents.default()
client = discord.Client(intents=intents)
tree = app_commands.CommandTree(client)
@tree.command(name="serverinfo", description="Show information about this server")
async def serverinfo_slash(interaction: discord.Interaction):
guild = interaction.guild
embed = discord.Embed(
title=guild.name,
description=f"Server information for **{guild.name}**",
color=discord.Color.blue(),
timestamp=datetime.now()
)
embed.add_field(name="Owner", value=guild.owner.display_name if guild.owner else "Unknown", inline=True)
embed.add_field(name="Members", value=guild.member_count, inline=True)
embed.add_field(name="Channels", value=len(guild.channels), inline=True)
embed.add_field(name="Roles", value=len(guild.roles), inline=True)
embed.add_field(name="Created", value=guild.created_at.strftime("%B %d, %Y"), inline=True)
embed.add_field(name="Boost Level", value=f"Level {guild.premium_tier}", inline=True)
if guild.icon:
embed.set_thumbnail(url=guild.icon.url)
embed.set_footer(text=f"Server ID: {guild.id}")
await interaction.response.send_message(embed=embed)
@client.event
async def on_ready():
await tree.sync()
print(f"{client.user} is online!")
client.run("YOUR_BOT_TOKEN_HERE")
Output (in Discord chat — rendered as a rich card):
┌─────────────────────────────────┐
│ My Test Server │
│ Server information for │
│ **My Test Server** │
│ │
│ Owner: Alice Members: 42 │
│ Channels: 15 Roles: 8 │
│ Created: Jan 15, 2024 │
│ Boost Level: Level 2 │
│ │
│ Server ID: 123456789012345678 │
└─────────────────────────────────┘
The discord.Embed constructor accepts a title, description, color, and timestamp. The add_field() method adds labeled data — setting inline=True places fields side by side (up to 3 per row), while inline=False gives each field its own row. The set_thumbnail() method adds a small image in the top-right corner, and set_footer() adds gray text at the bottom. Embeds can also include set_image() for a large image and set_author() for a clickable author name with an icon. The color parameter accepts hex values, RGB tuples, or convenience methods like discord.Color.blue(), discord.Color.red(), and discord.Color.green().
Cogs, error handling, logging, and a clean architecture. This bot scales. Yours should too.
Real-Life Example: Server Welcome Bot
Let us build a complete, practical bot that you can actually deploy to your Discord server. This Server Welcome Bot greets new members with a personalized embed message, automatically assigns them a default role, logs all join and leave events to a designated channel, and provides a /stats slash command that shows server statistics.
# welcome_bot.py
import discord
from discord import app_commands
from datetime import datetime
intents = discord.Intents.default()
intents.message_content = True
intents.members = True # Required for join/leave events
client = discord.Client(intents=intents)
tree = app_commands.CommandTree(client)
# Configuration — update these for your server
WELCOME_CHANNEL_NAME = "welcome"
LOG_CHANNEL_NAME = "bot-logs"
DEFAULT_ROLE_NAME = "Member"
join_count = 0 # Track joins during this session
def find_channel(guild, name):
"""Find a text channel by name, returns None if not found."""
return discord.utils.get(guild.text_channels, name=name)
@client.event
async def on_ready():
await tree.sync()
print(f"{client.user} is online!")
print(f"Monitoring {len(client.guilds)} server(s)")
@client.event
async def on_member_join(member):
global join_count
join_count += 1
# Send welcome embed
welcome_ch = find_channel(member.guild, WELCOME_CHANNEL_NAME)
if welcome_ch:
embed = discord.Embed(
title=f"Welcome, {member.display_name}!",
description=f"{member.mention} just joined **{member.guild.name}**! "
f"You are member #{member.guild.member_count}.",
color=discord.Color.green(),
timestamp=datetime.now()
)
if member.avatar:
embed.set_thumbnail(url=member.avatar.url)
embed.set_footer(text="Enjoy your stay!")
await welcome_ch.send(embed=embed)
# Assign default role
role = discord.utils.get(member.guild.roles, name=DEFAULT_ROLE_NAME)
if role:
try:
await member.add_roles(role)
except discord.Forbidden:
print(f"Missing permission to assign {role.name}")
# Log the event
log_ch = find_channel(member.guild, LOG_CHANNEL_NAME)
if log_ch:
await log_ch.send(f"[JOIN] {member.display_name} ({member.id}) joined at {datetime.now():%H:%M:%S}")
@client.event
async def on_member_remove(member):
log_ch = find_channel(member.guild, LOG_CHANNEL_NAME)
if log_ch:
await log_ch.send(f"[LEAVE] {member.display_name} ({member.id}) left at {datetime.now():%H:%M:%S}")
@tree.command(name="stats", description="Show server statistics")
async def stats_command(interaction: discord.Interaction):
guild = interaction.guild
embed = discord.Embed(title="Server Stats", color=discord.Color.blue())
embed.add_field(name="Total Members", value=guild.member_count, inline=True)
embed.add_field(name="Joins This Session", value=join_count, inline=True)
embed.add_field(name="Channels", value=len(guild.channels), inline=True)
embed.add_field(name="Roles", value=len(guild.roles), inline=True)
await interaction.response.send_message(embed=embed)
client.run("YOUR_BOT_TOKEN_HERE")
Output (in Discord — welcome channel):
┌─────────────────────────────────┐
│ Welcome, Charlie! │
│ @Charlie just joined │
│ **My Server**! You are member │
│ #43. │
│ │
│ Enjoy your stay! │
└─────────────────────────────────┘
Output (in Discord — bot-logs channel):
[JOIN] Charlie (987654321098765432) joined at 14:30:00
Output (in Discord — /stats command):
┌─────────────────────────────────┐
│ Server Stats │
│ Total Members: 43 │
│ Joins This Session: 1 │
│ Channels: 15 Roles: 8 │
└─────────────────────────────────┘
This bot uses the discord.utils.get() helper function to find channels and roles by name, which is cleaner than looping through lists manually. The try/except discord.Forbidden block around role assignment handles the case where the bot does not have permission to manage roles — without it, the entire on_member_join handler would crash silently. To deploy this bot to your server, create channels named “welcome” and “bot-logs” and a role named “Member,” then update the configuration constants at the top. You can extend this project by adding a /setwelcome command that lets admins change the welcome channel, a database to persist join counts between restarts, or a reaction-role system where members pick their own roles by clicking emoji reactions.
Frequently Asked Questions
How do I keep my bot token secure?
Never hardcode your token directly in your Python file, especially if you push code to GitHub. Instead, store it in an environment variable and read it with os.environ["DISCORD_TOKEN"], or use a .env file with the python-dotenv library. Add .env to your .gitignore file so it never gets committed. If your token is ever exposed, go to the Discord Developer Portal immediately and click “Reset Token” to invalidate the old one.
Should I use slash commands or text commands?
Use slash commands for new bots. Discord officially recommends them, and they provide a better user experience because Discord shows a command menu with descriptions and type validation. Text commands are still supported in discord.py 2.x, but they require the Message Content Intent, which Discord may restrict further in the future. If you are maintaining an older bot that already uses text commands, you can support both by using commands.Bot and adding slash commands alongside existing prefix commands.
Why are my slash commands not showing up in Discord?
Slash commands need to be synced with Discord using tree.sync(). Global commands can take up to one hour to appear. For instant testing, sync to a specific server with tree.sync(guild=discord.Object(id=SERVER_ID)). Also make sure you invited the bot with the applications.commands scope — without it, the bot cannot register slash commands even if you call sync.
Where can I host my Discord bot?
For small bots, a Raspberry Pi or an old laptop running 24/7 works fine. For production, popular hosting options include Railway, Render, and Oracle Cloud Free Tier (which gives you a free VPS). Avoid Heroku’s free tier for bots because it sleeps after 30 minutes of inactivity, which disconnects your bot. Whatever you choose, make sure the host supports long-running processes — Discord bots need a persistent WebSocket connection, not just HTTP request handling.
How do I handle Discord API rate limits?
The discord.py library handles rate limiting automatically. It tracks the rate limit headers from Discord’s API and pauses requests when you are close to the limit. If you are still hitting rate limits, it usually means your bot is sending too many messages too quickly — add delays between bulk operations using asyncio.sleep(). The typical rate limit is 5 requests per 5 seconds per route, but some endpoints like sending messages have stricter limits.
Can my bot run on multiple servers at the same time?
Yes, a single bot instance handles all servers it is invited to. Discord’s gateway sends events from all servers, and discord.py routes them to your event handlers with the appropriate guild context. The interaction.guild or message.guild object tells you which server the event came from. You do not need separate bot instances per server — one process handles everything.
Conclusion
In this article we walked through building a Discord bot from scratch with Python and discord.py. We covered setting up the Developer Portal, understanding the event-driven architecture, building text commands with commands.Bot, creating modern slash commands with app_commands.CommandTree, and designing rich embed messages. The Server Welcome Bot project ties all of these concepts together into a practical, deployable bot that greets new members, assigns roles, and logs activity.
From here, you can extend the welcome bot with a database backend using sqlite3 or aiosqlite for persistent storage, add a moderation system with kick, ban, and mute commands, or integrate external APIs to build a weather bot, trivia bot, or music bot. The discord.py library supports almost everything the Discord API offers, including voice channels, threads, forums, and scheduled events.
You have a Python script that needs to fetch data from five different APIs, and right now it calls them one after another. Each call takes about two seconds, so the whole thing crawls along for ten seconds total. The frustrating part is that those API calls are completely independent — there is no reason your program should sit idle waiting for one response before sending the next request. This is exactly the problem that Python’s asyncio module solves, and once you understand it, you will never look at I/O-bound code the same way again.
The good news is that asyncio is part of Python’s standard library, so there is nothing extra to install. It has been available since Python 3.4 and has matured significantly — Python 3.11 introduced TaskGroup for structured concurrency, and Python 3.12 refined the event loop internals for better performance. All you need is Python 3.11 or later to use every feature covered in this article, though most examples work on Python 3.7 and above.
In this article we will cover everything you need to know to write concurrent Python code with asyncio. We will start with the fundamentals of async and await, then explore coroutines and the event loop. From there we will dive into running multiple tasks concurrently with asyncio.gather(), handling errors gracefully, and using the modern TaskGroup API for structured concurrency. We will also cover asyncio.wait(), timeouts, semaphores for rate limiting, and finish with a real-life project that fetches data from multiple URLs concurrently. By the end, you will be writing async Python code with confidence.
Python Asyncio: Quick Example
Before diving deep, here is a taste of what asyncio can do. This example runs three simulated tasks concurrently instead of sequentially, cutting the total time from six seconds down to about two.
# quick_example.py
import asyncio
import time
async def fetch_data(name, delay):
"""Simulate an API call that takes 'delay' seconds."""
print(f"Starting {name}...")
await asyncio.sleep(delay) # Non-blocking sleep
print(f"Finished {name}!")
return f"{name}: {delay}s of data"
async def main():
start = time.perf_counter()
# Run all three tasks concurrently
results = await asyncio.gather(
fetch_data("Users API", 2),
fetch_data("Orders API", 2),
fetch_data("Products API", 2),
)
elapsed = time.perf_counter() - start
print(f"\nAll done in {elapsed:.2f} seconds")
for r in results:
print(f" {r}")
asyncio.run(main())
Output:
Starting Users API...
Starting Orders API...
Starting Products API...
Finished Users API!
Finished Orders API!
Finished Products API!
All done in 2.00 seconds
Users API: 2s of data
Orders API: 2s of data
Products API: 2s of data
Notice that all three tasks started immediately and finished at roughly the same time, even though each one waited two seconds. If we had run them sequentially with regular time.sleep(), the total would have been six seconds. The magic here is asyncio.gather() — it schedules all three coroutines to run on the event loop and waits until they all complete. The await asyncio.sleep() call is the key: it tells the event loop “I am done for now, go run something else while I wait.”
Want to go deeper? Below we cover how the event loop works under the hood, explore gather() in detail, learn about TaskGroup for safer error handling, and build a real concurrent URL fetcher you can use in your own projects.
What Is Asyncio and Why Use It?
At its core, asyncio is Python’s framework for writing concurrent code using a single thread. Instead of creating multiple threads or processes, asyncio uses an event loop — a central coordinator that switches between tasks whenever one of them is waiting for something (like a network response or a file read). Think of it like a chef in a kitchen who starts boiling water, then chops vegetables while waiting for the water to boil, then checks the oven — one person doing many things by never standing idle.
This approach is called cooperative multitasking because each task voluntarily gives up control when it hits an await expression. The event loop then picks up another task that is ready to run. This is fundamentally different from threads, where the operating system forcibly switches between them. The cooperative model is simpler to reason about because you know exactly where your code can be interrupted — only at await points.
The question most beginners ask is: when should you use asyncio versus threads versus multiprocessing? Here is a comparison table to help you decide.
Feature
asyncio
threading
multiprocessing
Best for
I/O-bound tasks (network, file, database)
I/O-bound tasks with blocking libraries
CPU-bound tasks (math, image processing)
Concurrency model
Single thread, event loop
Multiple threads, OS-scheduled
Multiple processes, separate memory
GIL impact
Not affected (single thread)
Limited by GIL for CPU work
No GIL limitation
Memory overhead
Very low (coroutines are lightweight)
Moderate (each thread has a stack)
High (each process has its own memory)
Complexity
Moderate (async/await syntax)
High (race conditions, locks)
Moderate (serialization overhead)
Scalability
Thousands of concurrent tasks easily
Hundreds of threads at most
Limited by CPU cores
The takeaway is simple: if your code spends most of its time waiting for external resources — API calls, database queries, file downloads — asyncio is usually the best choice. It can handle thousands of concurrent connections with minimal memory, which is why frameworks like FastAPI and aiohttp are built on top of it. Now let us look at the building blocks.
Understanding async and await
The two keywords that make asyncio work are async and await. When you put async before a function definition, it becomes a coroutine function. Calling it does not run the function immediately — it returns a coroutine object that needs to be scheduled on the event loop. The await keyword is how you actually run a coroutine and get its result, while also telling the event loop it can switch to other tasks.
# async_basics.py
import asyncio
async def greet(name, delay):
"""A coroutine that waits, then returns a greeting."""
await asyncio.sleep(delay)
return f"Hello, {name}!"
async def main():
# Calling greet() returns a coroutine object, not the result
coro = greet("Alice", 1)
print(f"Type of coro: {type(coro)}")
# To actually run it, we await it
result = await greet("Alice", 1)
print(result)
asyncio.run(main())
Output:
Type of coro: <class 'coroutine'>
Hello, Alice!
The important thing to understand here is the difference between calling a coroutine function and awaiting it. When we wrote coro = greet("Alice", 1), nothing happened — the function body did not execute. Only when we used await greet("Alice", 1) did the code inside actually run. This is a common source of bugs for beginners: forgetting to await a coroutine means it silently does nothing, and Python will even warn you about it.
The asyncio.run() function is the entry point that creates an event loop, runs your main() coroutine, and shuts everything down cleanly when it finishes. You should call asyncio.run() exactly once at the top level of your program — never from inside another coroutine.
The event loop is a single-threaded traffic controller. Block it and everything stops.
The Event Loop Explained
The event loop is the engine that drives all of asyncio. It runs in a single thread and continuously cycles through a queue of tasks, executing each one until it hits an await, then moving on to the next ready task. Understanding this cycle helps you write better async code because you know exactly when your code runs and when it yields control.
See how the tasks interleave? Each await asyncio.sleep(0) is a zero-second pause that simply says “let others run.” The event loop picks up Task B after Task A yields, then switches back. This is cooperative multitasking in action — the tasks voluntarily take turns. If Task A had a long CPU-bound computation without any await, it would block the entire event loop and prevent Task B from running at all. That is why asyncio is designed for I/O-bound work, not number crunching.
Running Tasks Concurrently With asyncio.gather()
asyncio.gather() is the workhorse function for running multiple coroutines concurrently. You pass it any number of awaitables (coroutines, tasks, or futures), and it schedules them all to run on the event loop simultaneously. It returns a list of results in the same order you passed the coroutines, regardless of which one finishes first.
# gather_example.py
import asyncio
import time
async def fetch_user(user_id):
"""Simulate fetching a user from a database."""
await asyncio.sleep(1.5) # Simulated DB query
return {"id": user_id, "name": f"User_{user_id}", "active": True}
async def fetch_orders(user_id):
"""Simulate fetching orders for a user."""
await asyncio.sleep(2.0) # Simulated API call
return [{"order_id": 101, "amount": 29.99}, {"order_id": 102, "amount": 59.99}]
async def fetch_preferences(user_id):
"""Simulate fetching user preferences."""
await asyncio.sleep(1.0) # Simulated config lookup
return {"theme": "dark", "language": "en", "notifications": True}
async def main():
user_id = 42
start = time.perf_counter()
# Fetch all three pieces of data concurrently
user, orders, prefs = await asyncio.gather(
fetch_user(user_id),
fetch_orders(user_id),
fetch_preferences(user_id),
)
elapsed = time.perf_counter() - start
print(f"Fetched everything in {elapsed:.2f} seconds\n")
print(f"User: {user}")
print(f"Orders: {orders}")
print(f"Preferences: {prefs}")
asyncio.run(main())
The total time was about two seconds — the duration of the slowest task (fetch_orders) — instead of 4.5 seconds if we had called them sequentially. The results list preserves the order we passed to gather(), so we can unpack them directly into variables. This pattern is incredibly common in web applications where a single page might need data from multiple microservices.
Handling Errors in gather()
By default, if any coroutine passed to gather() raises an exception, the entire gather() call raises that exception and the other tasks may or may not have completed. You can change this behavior with the return_exceptions=True parameter, which makes gather() return exception objects in the results list instead of raising them.
# gather_errors.py
import asyncio
async def safe_task(name, delay):
await asyncio.sleep(delay)
return f"{name} completed"
async def failing_task():
await asyncio.sleep(0.5)
raise ValueError("Something went wrong in the API!")
async def main():
# With return_exceptions=True, errors become results
results = await asyncio.gather(
safe_task("Task A", 1),
failing_task(),
safe_task("Task C", 1.5),
return_exceptions=True, # Don't let one failure crash everything
)
for i, result in enumerate(results):
if isinstance(result, Exception):
print(f"Task {i}: FAILED - {type(result).__name__}: {result}")
else:
print(f"Task {i}: {result}")
asyncio.run(main())
Output:
Task 0: Task A completed
Task 1: FAILED - ValueError: Something went wrong in the API!
Task 2: Task C completed
This is a powerful pattern for building resilient applications. Instead of letting one failed API call crash your entire data-fetching pipeline, you collect all results and handle failures individually. The key line is return_exceptions=True — without it, the ValueError from the failing task would propagate up and you would lose the results from the other two tasks that completed successfully.
Race conditions in async code are subtle. asyncio.Lock() is not subtle. Use it.
Structured Concurrency With TaskGroup
asyncio.TaskGroup was introduced in Python 3.11 as a safer alternative to gather(). The main difference is how it handles errors: when any task in a TaskGroup fails, it automatically cancels all remaining tasks and raises an ExceptionGroup containing all the errors. This “fail fast” behavior prevents orphaned tasks from running in the background after something has already gone wrong.
# taskgroup_example.py
import asyncio
async def download_file(filename, size_mb, delay):
"""Simulate downloading a file."""
print(f"Downloading {filename} ({size_mb}MB)...")
await asyncio.sleep(delay)
print(f"Finished {filename}")
return {"file": filename, "size_mb": size_mb, "status": "complete"}
async def main():
results = []
async with asyncio.TaskGroup() as tg:
# create_task schedules coroutines within the group
task1 = tg.create_task(download_file("report.pdf", 5, 2))
task2 = tg.create_task(download_file("data.csv", 12, 3))
task3 = tg.create_task(download_file("image.png", 2, 1))
# If we get here, ALL tasks succeeded
results = [task1.result(), task2.result(), task3.result()]
print("\nAll downloads complete:")
for r in results:
print(f" {r['file']}: {r['size_mb']}MB - {r['status']}")
asyncio.run(main())
The async with asyncio.TaskGroup() as tg context manager creates a scope for your concurrent tasks. You add tasks using tg.create_task(), and when the async with block exits, it waits for all tasks to complete — similar to gather(). The critical difference shows up when errors occur: TaskGroup cancels sibling tasks immediately instead of letting them run to completion with unknown state. This is what the asyncio community calls “structured concurrency” and it prevents a whole class of subtle bugs.
TaskGroup Error Handling
When a task inside a TaskGroup raises an exception, the group cancels all other running tasks and collects the exceptions into an ExceptionGroup. You catch this with the except* syntax (also new in Python 3.11), which lets you handle different exception types selectively.
# taskgroup_errors.py
import asyncio
async def reliable_task(name, delay):
await asyncio.sleep(delay)
return f"{name} done"
async def flaky_api_call():
await asyncio.sleep(0.5)
raise ConnectionError("API server is down")
async def bad_data_task():
await asyncio.sleep(0.8)
raise ValueError("Invalid response format")
async def main():
try:
async with asyncio.TaskGroup() as tg:
tg.create_task(reliable_task("Backup", 2))
tg.create_task(flaky_api_call())
tg.create_task(bad_data_task())
except* ConnectionError as eg:
for exc in eg.exceptions:
print(f"Connection error: {exc}")
except* ValueError as eg:
for exc in eg.exceptions:
print(f"Value error: {exc}")
print("Program continues after handling errors")
asyncio.run(main())
Output:
Connection error: API server is down
Value error: Invalid response format
Program continues after handling errors
Notice that the “Backup” task was cancelled even though it had not failed — that is TaskGroup‘s strict policy. When the flaky_api_call raised ConnectionError, the group immediately cancelled all remaining tasks and collected the exceptions. The except* syntax handles each exception type separately, which is much cleaner than manually iterating through results looking for errors. If you need tasks to continue even when siblings fail, use gather(return_exceptions=True) instead.
gather() vs TaskGroup: When To Use Which
Now that you have seen both approaches, here is a direct comparison to help you choose the right tool for each situation.
Feature
asyncio.gather()
asyncio.TaskGroup
Python version
3.4+
3.11+
Error behavior (default)
First exception propagates, others may still run
All tasks cancelled on first error
return_exceptions option
Yes — collects errors as results
No — always cancels on error
Error handling syntax
Check isinstance() on results
except* ExceptionGroup
Task cancellation
Manual
Automatic on error
Best for
Independent tasks where partial results are OK
Related tasks that should all succeed or all fail
Use gather() when your tasks are independent and you want best-effort results — for example, fetching data from five APIs where getting four out of five is still useful. Use TaskGroup when your tasks are related and a partial result is meaningless — for example, a multi-step transaction where all steps must succeed. In practice, many developers use gather(return_exceptions=True) for resilient data fetching and TaskGroup for transactional workflows.
Creating and Managing Individual Tasks
Sometimes you need more control than gather() or TaskGroup provide. The asyncio.create_task() function lets you schedule a coroutine to run in the background without immediately waiting for its result. This is useful when you want to start something, do other work, and check on it later.
# create_task_demo.py
import asyncio
async def background_sync(data):
"""Simulate syncing data to a remote server."""
print(f"Syncing {len(data)} records in background...")
await asyncio.sleep(3)
print("Sync complete!")
return len(data)
async def process_request(request_id):
"""Simulate processing an incoming request."""
await asyncio.sleep(0.5)
return f"Request {request_id} processed"
async def main():
# Start background sync — don't wait for it yet
sync_task = asyncio.create_task(background_sync(["user1", "user2", "user3"]))
# Process requests while sync runs in background
for i in range(1, 4):
result = await process_request(i)
print(result)
# Now wait for the sync to finish
synced_count = await sync_task
print(f"\nBackground sync finished: {synced_count} records synced")
asyncio.run(main())
Output:
Syncing 3 records in background...
Request 1 processed
Request 2 processed
Request 3 processed
Sync complete!
Background sync finished: 3 records synced
The key here is that asyncio.create_task() returns a Task object immediately without blocking. The sync coroutine starts running in the background while we process requests in the foreground. When we finally await sync_task, it either returns the result instantly (if it already finished) or waits until it completes. This pattern is perfect for fire-and-forget operations like logging, caching, or background data synchronization.
TaskGroup: spawn ten coroutines, guarantee all ten get cleaned up. Even if one explodes.
Fine-Grained Control With asyncio.wait()
While gather() waits for all tasks to complete, asyncio.wait() gives you more flexibility. You can wait for the first task to finish, wait until any task raises an exception, or set a timeout. It returns two sets: done (completed tasks) and pending (still running tasks).
# wait_example.py
import asyncio
async def fetch_from_mirror(mirror_name, delay):
"""Simulate fetching from different mirror servers."""
await asyncio.sleep(delay)
return f"Data from {mirror_name}"
async def main():
tasks = [
asyncio.create_task(fetch_from_mirror("US-East", 3)),
asyncio.create_task(fetch_from_mirror("EU-West", 1)),
asyncio.create_task(fetch_from_mirror("Asia-Pacific", 2)),
]
# Wait for the FIRST task to complete
done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
# Use the fastest result
for task in done:
print(f"First result: {task.result()}")
# Cancel the rest — we already have what we need
print(f"Cancelling {len(pending)} remaining tasks...")
for task in pending:
task.cancel()
asyncio.run(main())
Output:
First result: Data from EU-West
Cancelling 2 remaining tasks...
This pattern is called “first response wins” and it is incredibly useful for redundant requests. If you have multiple mirror servers or backup APIs, you can query all of them simultaneously and use whichever responds first, then cancel the rest. The return_when parameter accepts three values: FIRST_COMPLETED (return when any task finishes), FIRST_EXCEPTION (return when any task raises), and ALL_COMPLETED (wait for everything, the default).
Setting Timeouts With asyncio.wait_for()
When calling external services, you should always set a timeout so your program does not hang indefinitely. The asyncio.wait_for() function wraps any awaitable with a timeout — if it does not complete in time, it raises asyncio.TimeoutError and cancels the task.
# timeout_example.py
import asyncio
async def slow_database_query():
"""Simulate a database query that takes too long."""
print("Running complex query...")
await asyncio.sleep(10) # This takes way too long
return "query results"
async def main():
try:
# Give it 3 seconds max
result = await asyncio.wait_for(slow_database_query(), timeout=3.0)
print(f"Got result: {result}")
except asyncio.TimeoutError:
print("Query timed out after 3 seconds!")
print("Falling back to cached data...")
result = "cached results"
print(f"Using: {result}")
asyncio.run(main())
Output:
Running complex query...
Query timed out after 3 seconds!
Falling back to cached data...
Using: cached results
The timeout mechanism is essential for production code. Without it, a single unresponsive service can bring down your entire application. The asyncio.wait_for() function cancels the underlying coroutine when the timeout expires, so you do not end up with zombie tasks consuming resources in the background. A good practice is to combine timeouts with a fallback strategy — cached data, default values, or a retry with exponential backoff.
Rate Limiting With Semaphores
When you have hundreds of tasks to run, launching them all at once can overwhelm the target server or hit API rate limits. An asyncio.Semaphore acts as a bouncer — it limits how many coroutines can run a particular section of code at the same time. This is essential for being a good citizen when working with external APIs.
# semaphore_example.py
import asyncio
import time
async def fetch_page(session_semaphore, page_num):
"""Fetch a page, but respect the concurrency limit."""
async with session_semaphore:
# Only N tasks can be inside this block at once
print(f" Fetching page {page_num}...")
await asyncio.sleep(1) # Simulated HTTP request
return f"Page {page_num} content"
async def main():
semaphore = asyncio.Semaphore(3) # Max 3 concurrent requests
start = time.perf_counter()
# Launch 9 tasks, but only 3 run at a time
tasks = [fetch_page(semaphore, i) for i in range(1, 10)]
results = await asyncio.gather(*tasks)
elapsed = time.perf_counter() - start
print(f"\nFetched {len(results)} pages in {elapsed:.2f} seconds")
print(f"(3 at a time, ~1 second each = ~3 batches = ~3 seconds)")
asyncio.run(main())
Output:
Fetching page 1...
Fetching page 2...
Fetching page 3...
Fetching page 4...
Fetching page 5...
Fetching page 6...
Fetching page 7...
Fetching page 8...
Fetching page 9...
Fetched 9 pages in 3.00 seconds
(3 at a time, ~1 second each = ~3 batches = ~3 seconds)
The async with session_semaphore context manager blocks when three tasks are already inside it, making the fourth task wait until one finishes. This creates a natural batching effect: pages 1-3 run first, then 4-6, then 7-9. Without the semaphore, all nine would fire at once, which could trigger rate limiting or connection errors. A good rule of thumb is to set the semaphore value to match the API’s rate limit — if an API allows 5 requests per second, use Semaphore(5) with a one-second delay inside the critical section.
asyncio.gather() turns sequential API calls into concurrent ones. Same result, fraction of the time.
Real-Life Example: Concurrent URL Health Checker
Let us put everything together into a practical project. This health checker takes a list of URLs, pings them all concurrently (with a semaphore to limit concurrency), measures response times, and produces a status report. It uses aiohttp for async HTTP requests and demonstrates error handling, timeouts, and semaphores working together.
# url_health_checker.py
import asyncio
import time
# Using asyncio-compatible HTTP simulation
# In production, replace with aiohttp:
# import aiohttp
async def check_url(semaphore, url, timeout_seconds=5):
"""Check a single URL's health with concurrency limiting."""
async with semaphore:
start = time.perf_counter()
try:
# Simulate HTTP GET with varying response times
# In production: async with aiohttp.ClientSession() as session:
# async with session.get(url, timeout=...) as resp:
simulated_delays = {
"https://httpbin.org/get": 0.3,
"https://jsonplaceholder.typicode.com/posts/1": 0.5,
"https://httpbin.org/delay/2": 2.0,
"https://httpbin.org/status/500": 0.2,
"https://nonexistent.invalid/api": None, # Will "fail"
}
delay = simulated_delays.get(url, 1.0)
if delay is None:
raise ConnectionError(f"Cannot resolve host")
result = await asyncio.wait_for(
asyncio.sleep(delay), # Simulates network I/O
timeout=timeout_seconds,
)
elapsed = time.perf_counter() - start
status = 500 if "status/500" in url else 200
return {
"url": url,
"status": status,
"response_time": round(elapsed, 3),
"healthy": 200 <= status < 400,
}
except asyncio.TimeoutError:
elapsed = time.perf_counter() - start
return {
"url": url,
"status": "TIMEOUT",
"response_time": round(elapsed, 3),
"healthy": False,
}
except Exception as e:
elapsed = time.perf_counter() - start
return {
"url": url,
"status": f"ERROR: {e}",
"response_time": round(elapsed, 3),
"healthy": False,
}
def print_report(results, total_time):
"""Print a formatted health check report."""
print("\n" + "=" * 65)
print(" URL HEALTH CHECK REPORT")
print("=" * 65)
healthy = [r for r in results if r["healthy"]]
unhealthy = [r for r in results if not r["healthy"]]
for r in results:
icon = "[OK]" if r["healthy"] else "[FAIL]"
print(f" {icon} {r['url']}")
print(f" Status: {r['status']} | Time: {r['response_time']}s")
print("-" * 65)
print(f" Total: {len(results)} URLs checked in {total_time:.2f}s")
print(f" Healthy: {len(healthy)} | Unhealthy: {len(unhealthy)}")
print("=" * 65)
async def main():
urls = [
"https://httpbin.org/get",
"https://jsonplaceholder.typicode.com/posts/1",
"https://httpbin.org/delay/2",
"https://httpbin.org/status/500",
"https://nonexistent.invalid/api",
]
semaphore = asyncio.Semaphore(3) # Max 3 concurrent checks
start = time.perf_counter()
# Check all URLs concurrently with rate limiting
tasks = [check_url(semaphore, url, timeout_seconds=5) for url in urls]
results = await asyncio.gather(*tasks)
total_time = time.perf_counter() - start
print_report(results, total_time)
asyncio.run(main())
This project demonstrates the key asyncio patterns we covered: gather() runs all checks concurrently, the semaphore limits us to three concurrent requests so we do not overwhelm any server, and wait_for() ensures no single check hangs forever. The error handling inside check_url catches both timeouts and connection errors, returning a structured result either way. To use this with real HTTP requests, install aiohttp with pip install aiohttp and replace the simulated delays with actual session.get() calls — the async structure stays exactly the same.
Frequently Asked Questions
Can I call a regular (synchronous) function from an async function?
Yes, you can call synchronous functions directly from async code, but be careful. If the synchronous function is fast (like a quick calculation or string manipulation), just call it normally. If it blocks for a long time (like time.sleep() or a synchronous HTTP request), it will freeze the entire event loop. For blocking operations, use await asyncio.to_thread(blocking_function, args) to run them in a separate thread without blocking the loop. This was added in Python 3.9 and is the recommended way to bridge sync and async code.
Can I use the requests library with asyncio?
The requests library is synchronous and will block the event loop if used directly inside an async function. You have two options: use aiohttp as a drop-in async replacement (it has a similar API with session.get() and session.post()), or wrap requests calls with await asyncio.to_thread(requests.get, url) to run them in a thread pool. The aiohttp approach is more efficient because it uses the event loop natively, while the thread approach adds thread-switching overhead.
What is the difference between asyncio.run() and get_event_loop()?
The asyncio.run() function (Python 3.7+) is the modern, recommended way to start your async program. It creates a new event loop, runs your coroutine, and cleans up afterward. The older asyncio.get_event_loop() pattern requires more manual management and is deprecated for most use cases since Python 3.10. Always use asyncio.run(main()) at the top level of your program unless you are integrating with a framework like Jupyter that manages its own event loop.
Why should I not use asyncio for CPU-bound tasks?
The asyncio event loop runs in a single thread, so CPU-intensive work blocks it completely. While one coroutine is crunching numbers, no other coroutine can run — there are no await points to yield control. For CPU-bound work like image processing, scientific computation, or data transformation, use multiprocessing or concurrent.futures.ProcessPoolExecutor. You can even combine them with asyncio using loop.run_in_executor() to run CPU work in a process pool while keeping your I/O code async.
How do I debug asyncio code?
Asyncio has a built-in debug mode you can enable by setting the environment variable PYTHONASYNCIODEBUG=1 or by passing debug=True to asyncio.run(). Debug mode warns you about common mistakes like coroutines that were never awaited, callbacks that take too long, and tasks that are destroyed while still pending. You can also use the asyncio logger with logging.getLogger('asyncio').setLevel(logging.DEBUG) to see detailed event loop activity.
What are async for and async with?
These are async versions of regular for loops and context managers. async for iterates over an asynchronous iterator — useful for streaming data from a database cursor or a websocket connection. async with enters and exits an asynchronous context manager — used heavily in aiohttp for managing HTTP sessions and connections. Both allow the event loop to run other tasks between iterations or during setup and teardown, which keeps your program responsive even when working with slow data sources.
Conclusion
You now have a solid understanding of Python's asyncio module and its key tools for concurrent programming. We covered the async/await syntax for defining coroutines, the event loop that coordinates everything, asyncio.gather() for running multiple tasks concurrently, TaskGroup for structured concurrency with automatic cancellation, asyncio.wait() for fine-grained control, timeouts with wait_for(), and semaphores for rate limiting. Each of these tools solves a specific problem, and knowing when to reach for which one is what separates a beginner from an effective async programmer.
Try extending the URL health checker project we built — add real HTTP requests with aiohttp, save results to a JSON file, or schedule periodic checks with asyncio.sleep() in a loop. You could also build an async web scraper that respects rate limits, or a chat application using websockets. The patterns you learned here apply directly to all of these projects.
How To Use Python Dataclasses For Clean Data Structures
You’re building a model to represent a user, a product, an order—some structured data. In older Python, you’d write boilerplate code: __init__ to initialize fields, __repr__ to show a nice string representation, __eq__ to compare instances. A simple 5-field data class would require 30+ lines of code. Dataclasses, added in Python 3.7, solve this problem entirely. One decorator (@dataclass) gives you automatic __init__, __repr__, __eq__, and more. Your class definition becomes clean, readable, and type-safe. This is one of the best quality-of-life improvements in modern Python.
In this article, we’ll explore dataclasses from basics to advanced patterns. You’ll learn when to use them, how to configure them, and how to build practical systems with them. By the end, you’ll write less boilerplate and more focused business logic.
Dataclasses: Quick Example
Here’s what would normally require 20+ lines of boilerplate code, now expressed as pure declarations:
#simple_dataclass.py
from dataclasses import dataclass
@dataclass
class Person:
name: str
age: int
email: str
person = Person("Alice", 30, "alice@example.com")
print(person)
# Person(name='Alice', age=30, email='alice@example.com')
person2 = Person("Alice", 30, "alice@example.com")
print(person == person2)
# True
The @dataclass decorator gave us an __init__ that accepts all three fields, a __repr__ that shows the class and values, and __eq__ that compares instances by their fields. All with zero boilerplate code. This is why dataclasses are a game-changer for Python development.
@dataclass: __init__, __repr__, __eq__ for free.
Why Dataclasses Beat Dictionaries and Simple Classes
You might think “Can’t I just use dictionaries?” or “Why not write a normal class?” Let’s compare the approaches side by side. Each approach has tradeoffs, and dataclasses hit the sweet spot for most scenarios:
#comparison.py
from dataclasses import dataclass
# Dictionary approach (old way)
person_dict = {
'name': 'Bob',
'age': 28,
'email': 'bob@example.com'
}
print(person_dict['name']) # Must use quotes, typos aren't caught
print(person_dict.get('phone')) # Missing keys return None silently
# Regular class (lots of boilerplate)
class PersonClass:
def __init__(self, name, age, email):
self.name = name
self.age = age
self.email = email
def __repr__(self):
return f'PersonClass(name={self.name}, age={self.age}, email={self.email})'
person_class = PersonClass('Bob', 28, 'bob@example.com')
print(person_class.name) # Attribute access is nicer
# Dataclass approach (best of both worlds)
@dataclass
class Person:
name: str
age: int
email: str
person = Person('Bob', 28, 'bob@example.com')
print(person.name) # Attribute access like classes
print(person) # Nice repr for free
# Person(name='Bob', age=28, email='bob@example.com')
Output:
Bob
None
Bob
Person(name='Bob', age=28, email='bob@example.com')
Dataclasses give you attribute access (cleaner than dict[‘key’]), automatic __repr__ (debug-friendly output), type hints (IDE support, safety), and zero boilerplate. Compared to dictionaries, you get type safety and better tooling. Compared to regular classes, you lose nothing but lines of code. They’re the right choice for modeling data.
Basic Dataclass Syntax With Methods
Dataclasses aren’t just data containers—they can have methods too. This makes them useful for modeling entities with both state and behavior:
#dataclass_with_methods.py
from dataclasses import dataclass
@dataclass
class Point:
x: float
y: float
p = Point(3.5, 4.2)
print(f"Point coordinates: ({p.x}, {p.y})")
@dataclass
class Rectangle:
width: float
height: float
def area(self):
# Calculate area using width and height
return self.width * self.height
def perimeter(self):
# Calculate perimeter
return 2 * (self.width + self.height)
rect = Rectangle(5, 3)
print(f"Area: {rect.area()}")
print(f"Perimeter: {rect.perimeter()}")
Output:
Point coordinates: (3.5, 4.2)
Area: 15
Perimeter: 16
Dataclasses combine data (the fields) with behavior (the methods). This is object-oriented programming the right way—encapsulation of related data and operations.
Default Values and Field Configuration
Often your fields have default values, or they need special handling (like lists that start empty). The field() function from dataclasses gives you fine-grained control:
#dataclass_defaults.py
from dataclasses import dataclass, field
from datetime import datetime
from typing import List
@dataclass
class Article:
title: str
content: str
author: str = "Anonymous" # Simple default
created_at: datetime = field(default_factory=datetime.now) # Factory for mutable defaults
tags: List[str] = field(default_factory=list) # Always use field for lists
views: int = 0
is_published: bool = False
article1 = Article("Python Tips", "Learn about dataclasses...")
print(article1)
article2 = Article(
"Flask Tutorial",
"Build APIs with Flask...",
author="John Doe",
is_published=True
)
print(article2)
Notice that created_at uses field(default_factory=…). This is important: mutable defaults (lists, dicts, datetime.now()) must use default_factory, otherwise all instances share the same list. Without field(), if you created two articles, both would share the same tags list (a common bug). The field() function prevents this.
Frozen Dataclasses: Immutability When You Need It
Sometimes you want data that can’t be changed after creation. Colors, coordinates, configuration objects—these are often immutable for good reason. The frozen=True parameter makes dataclass instances immutable:
#frozen_dataclass.py
from dataclasses import dataclass
@dataclass(frozen=True)
class Color:
red: int
green: int
blue: int
sky_blue = Color(135, 206, 235)
print(f"Sky blue: {sky_blue}")
try:
sky_blue.red = 100 # Try to modify
except Exception as e:
print(f"Error: {e}")
darker_blue = Color(115, 186, 215)
print(f"Darker blue: {darker_blue}")
Output:
Sky blue: Color(red=135, green=206, blue=235)
Error: cannot assign to field 'red'
Darker blue: Color(red=115, green=186, blue=215)
Immutable objects are safer—they can’t be modified accidentally, they’re thread-safe, and they work as dictionary keys. Use frozen=True for configuration objects, constants, and anything that should never change.
Inheritance: Building Class Hierarchies
Dataclasses work beautifully with inheritance. Child classes inherit parent fields and can add their own:
#dataclass_inheritance.py
from dataclasses import dataclass
@dataclass
class Employee:
name: str
employee_id: int
salary: float
@dataclass
class Manager(Employee):
department: str
team_size: int
emp = Employee("Alice", 101, 50000)
mgr = Manager("Bob", 102, 80000, "Engineering", 5)
print(emp)
print(mgr)
print(f"Manager {mgr.name} manages {mgr.team_size} people")
Output:
Employee(name='Alice', employee_id=101, salary=50000)
Manager(name='Bob', employee_id=102, salary=80000, department='Engineering', team_size=5)
Manager Bob manages 5 people
Manager inherits name, employee_id, and salary from Employee, then adds department and team_size. The __init__ signature includes all fields in the right order. This is how you model real-world hierarchies—employee types, vehicle types, etc.
slots=True: smaller, faster, no surprises.
Dataclasses vs NamedTuple: Understanding the Differences
NamedTuple is similar to dataclasses but makes different tradeoffs. Both are useful, but for different scenarios. Let’s compare them so you can choose wisely:
#dataclass_vs_namedtuple.py
from dataclasses import dataclass
from typing import NamedTuple
@dataclass
class PersonDataclass:
name: str
age: int
class PersonNamedTuple(NamedTuple):
name: str
age: int
p1 = PersonDataclass("Charlie", 35)
p2 = PersonNamedTuple("Charlie", 35)
print(p1)
print(p2)
p1.age = 36 # Dataclass is mutable
print(f"Updated dataclass: {p1}")
try:
p2.age = 36 # NamedTuple is immutable
except Exception as e:
print(f"NamedTuple error: {e}")
name, age = p2 # NamedTuple supports unpacking
print(f"Unpacked: {name}, {age}")
Dataclasses: Mutable (can change fields), more flexible, better for modeling evolving objects
NamedTuple: Immutable, lighter weight, can be unpacked like tuples, good for fixed data
Use dataclasses when you might modify objects or want methods. Use NamedTuple when you want immutable, lightweight data containers. For most modern Python code, dataclasses are the default choice.
Real-Life Example: Product Inventory System
Let’s build a practical inventory management system using dataclasses. This demonstrates real-world patterns: enums for categories, computed properties, and business logic:
#inventory_system.py
from dataclasses import dataclass, field
from datetime import datetime
from typing import List
from enum import Enum
class Category(Enum):
ELECTRONICS = "Electronics"
CLOTHING = "Clothing"
BOOKS = "Books"
@dataclass
class Product:
sku: str
name: str
category: Category
price: float
quantity_in_stock: int
reorder_level: int = 10
last_restocked: datetime = field(default_factory=datetime.now)
def needs_restock(self) -> bool:
return self.quantity_in_stock <= self.reorder_level
def total_value(self) -> float:
return self.price * self.quantity_in_stock
def sell(self, quantity: int) -> bool:
if quantity > self.quantity_in_stock:
return False
self.quantity_in_stock -= quantity
return True
def restock(self, quantity: int) -> None:
self.quantity_in_stock += quantity
self.last_restocked = datetime.now()
@dataclass
class Inventory:
products: List[Product] = field(default_factory=list)
def add_product(self, product: Product) -> None:
self.products.append(product)
def get_product(self, sku: str) -> Product:
for product in self.products:
if product.sku == sku:
return product
return None
def low_stock_items(self) -> List[Product]:
return [p for p in self.products if p.needs_restock()]
def total_value(self) -> float:
return sum(p.total_value() for p in self.products)
def inventory_report(self) -> None:
print("=== INVENTORY REPORT ===")
for product in self.products:
status = "LOW STOCK" if product.needs_restock() else "OK"
print(f"{product.sku}: {product.name} - {product.quantity_in_stock} units ({status})")
print(f"Total Inventory Value: ${self.total_value():.2f}")
print(f"Items needing restock: {len(self.low_stock_items())}")
inventory = Inventory()
laptop = Product(sku="LAPTOP001",name="MacBook Pro 14 inch",category=Category.ELECTRONICS,price=1999.99,quantity_in_stock=5,reorder_level=3)
book = Product(sku="BOOK001",name="Python Mastery",category=Category.BOOKS,price=29.99,quantity_in_stock=2,reorder_level=10)
inventory.add_product(laptop)
inventory.add_product(book)
laptop.sell(2)
book.sell(1)
inventory.inventory_report()
book.restock(15)
print(f"\nAfter restocking: {book}")
Output:
=== INVENTORY REPORT ===
LAPTOP001: MacBook Pro 14 inch - 3 units (OK)
BOOK001: Python Mastery - 1 units (LOW STOCK)
Total Inventory Value: $5999.97
Items needing restock: 1
After restocking: Product(sku='BOOK001', name='Python Mastery', category=, price=29.99, quantity_in_stock=16, reorder_level=10, last_restocked=datetime.datetime(2026, 3, 12, 10, 30, 45))
This inventory system uses dataclasses effectively: Product holds item data with methods for business logic (sell, restock, total_value). Inventory aggregates products and provides reports. Enums ensure category values are consistent. Timestamps track when restocking happened. This is maintainable, testable code that scales from a small store to enterprise inventory systems.
@dataclass does in one line what takes three dunder methods and twenty lines of boilerplate.
Frequently Asked Questions
Q: When should I use dataclasses vs regular classes?
Use dataclasses when your class is primarily a data container. Use regular classes when you have complex initialization logic, property decorators, or inheritance patterns that don’t fit the dataclass model. When in doubt, start with a dataclass and upgrade to a regular class if needed.
Q: Can I use dataclasses with type hints?
Type hints are required for dataclasses—they tell the decorator which fields to create. This is a feature, not a limitation. Type hints make your code clearer and enable better IDE support, type checking, and documentation. Python’s type system is optional, but use it with dataclasses.
Q: How do I convert a dataclass to a dictionary?
Use the asdict() function from dataclasses module: from dataclasses import asdict; my_dict = asdict(my_dataclass). This is useful for JSON serialization, logging, or comparing to dictionaries.
Q: Can I add validators to dataclasses?
Use the __post_init__ method to validate after initialization: def __post_init__(self): if self.age < 0: raise ValueError("Age must be positive"). You can also use field with validate_before for pre-validation, or external libraries like Pydantic for more complex validation.
Q: Do dataclasses work with JSON serialization?
Convert to dict first with asdict(), then to JSON: json.dumps(asdict(my_dataclass)). For automatic JSON support with validation, consider Pydantic instead. Pydantic is built on dataclasses but adds JSON schema support and validation.
Conclusion
Dataclasses are a game-changer for Python developers. They eliminate boilerplate code, add type safety, and make your code more readable. Whether you're building simple data models, complex business objects, or inventory systems, dataclasses scale from simple to sophisticated use cases. Start using them in your next project and you'll never want to go back to writing __init__ methods by hand.
Key takeaways:
Dataclasses eliminate __init__, __repr__, __eq__ boilerplate with one decorator
Use field() for mutable defaults (lists, dicts) and special handling
Use frozen=True for immutable objects
Dataclasses support inheritance naturally
Add methods to dataclasses for business logic—they're not just data containers
Dataclasses work with type hints, IDE tooling, and modern Python patterns
Use __post_init__ for validation after initialization
Choose dataclasses over dictionaries for type safety and over regular classes for less boilerplate
If you have ever wanted to build a backend that serves data to a mobile app, a frontend framework like React, or even another Python script, you need a REST API. REST APIs are the backbone of modern software — they let different systems communicate over HTTP using a standardized set of operations. Whether you are building a to-do app, a dashboard, or a microservice, the ability to create an API is one of the most practical skills a Python developer can have.
Flask is one of the best frameworks for building REST APIs in Python. It is lightweight, flexible, and stays out of your way — you can go from zero to a working API in under 20 lines of code. Flask does not force you into any particular project structure or ORM, which makes it perfect for learning and for small-to-medium projects. You will need Python 3.8 or later, and installing Flask is a single pip install flask command.
In this article we will build a REST API from scratch using Flask. We will start with a quick working example, then cover what REST means and how HTTP methods map to CRUD operations. From there we will build routes for creating, reading, updating, and deleting resources, add proper error handling, learn how to test our API with curl and Python’s requests library, and finish with a complete real-life project — a Bookmark Manager API that stores bookmarks in a JSON file with tagging and search. By the end, you will be ready to build any API you need.
Building a REST API With Flask: Quick Example
Here is the shortest working Flask API. It defines two endpoints — one that returns a welcome message and one that returns a list of books as JSON. You can run this and test it in your browser immediately.
Run the script with python quick_api.py, then open http://127.0.0.1:5000/api/books in your browser. You will see the JSON list of books. The @app.route() decorator maps a URL path to a Python function, and jsonify() converts Python dictionaries and lists into proper JSON responses with the correct Content-Type header.
Want to go deeper? Below we cover what REST actually means, how to build full CRUD endpoints, handle errors properly, test your API, and build a complete Bookmark Manager project.
What Is a REST API and Why Use Flask?
REST stands for Representational State Transfer. It is an architectural style for designing networked applications. In practice, a REST API is a web service that uses HTTP methods (GET, POST, PUT, DELETE) to perform operations on resources identified by URLs. When you visit /api/books, you are requesting the “books” resource. When you send a POST request to that same URL with JSON data, you are creating a new book.
The beauty of REST is its simplicity — it maps naturally to the four CRUD operations that every application needs. Here is how HTTP methods correspond to database-style operations:
HTTP Method
CRUD Operation
Example URL
What It Does
GET
Read
/api/books
Retrieve all books
GET
Read
/api/books/1
Retrieve book with ID 1
POST
Create
/api/books
Create a new book
PUT
Update
/api/books/1
Update book with ID 1
DELETE
Delete
/api/books/1
Delete book with ID 1
Flask is ideal for building REST APIs because it gives you just enough structure without imposing decisions. Unlike Django (which includes an ORM, admin panel, and templating engine), Flask lets you pick only the pieces you need. For a REST API, that means routes, request parsing, and JSON responses — Flask handles all three beautifully out of the box.
Installing Flask and Project Setup
Flask installs in seconds and has no mandatory dependencies beyond its own core libraries. Let us install it and verify everything works.
# setup_check.py
# Install from terminal: pip install flask
# Verify the installation
import flask
print(f"Flask version: {flask.__version__}")
Output:
Flask version: 3.1.0
That is all you need. Flask includes its own development server, so you do not need to install a separate web server for development. For production, you would use a WSGI server like Gunicorn, but for learning and testing, the built-in server is perfect.
Your First Flask Route
A route in Flask is a URL pattern mapped to a Python function. When someone visits that URL, Flask calls your function and returns whatever it sends back. Let us build a slightly more detailed example that shows how routes, methods, and response codes work together.
# first_routes.py
from flask import Flask, jsonify, request
app = Flask(__name__)
@app.route("/")
def home():
return jsonify({
"message": "Welcome to my API",
"version": "1.0",
"endpoints": ["/api/hello", "/api/greet?name=YourName"]
})
@app.route("/api/hello")
def hello():
return jsonify({"greeting": "Hello, World!"})
@app.route("/api/greet")
def greet():
# Read a query parameter from the URL
name = request.args.get("name", "stranger")
return jsonify({"greeting": f"Hello, {name}!"})
if __name__ == "__main__":
app.run(debug=True, port=5000)
The request.args.get() method reads query parameters from the URL. The second argument ("stranger") is the default value if the parameter is not provided. The debug=True flag enables auto-reloading — Flask will restart the server automatically whenever you save changes to your code, which makes development much faster. It also shows detailed error pages in the browser when something goes wrong.
Flask routing: because every URL deserves a function that loves it.
Building CRUD Endpoints With JSON
Now let us build a complete set of CRUD endpoints for a books resource. This is the pattern you will use for nearly every REST API — a collection endpoint (/api/books) for listing and creating, and an item endpoint (/api/books/<id>) for reading, updating, and deleting individual records.
# crud_api.py
from flask import Flask, jsonify, request
app = Flask(__name__)
# In-memory data store (replace with a database in production)
books = [
{"id": 1, "title": "Python Crash Course", "author": "Eric Matthes", "year": 2019},
{"id": 2, "title": "Fluent Python", "author": "Luciano Ramalho", "year": 2022},
{"id": 3, "title": "Automate the Boring Stuff", "author": "Al Sweigart", "year": 2019},
]
next_id = 4 # Track the next available ID
@app.route("/api/books", methods=["GET"])
def get_all_books():
return jsonify(books)
@app.route("/api/books/<int:book_id>", methods=["GET"])
def get_book(book_id):
book = next((b for b in books if b["id"] == book_id), None)
if book is None:
return jsonify({"error": "Book not found"}), 404
return jsonify(book)
@app.route("/api/books", methods=["POST"])
def create_book():
global next_id
data = request.get_json()
# Validate required fields
if not data or "title" not in data or "author" not in data:
return jsonify({"error": "Title and author are required"}), 400
new_book = {
"id": next_id,
"title": data["title"],
"author": data["author"],
"year": data.get("year", "Unknown")
}
books.append(new_book)
next_id += 1
return jsonify(new_book), 201 # 201 = Created
@app.route("/api/books/<int:book_id>", methods=["PUT"])
def update_book(book_id):
book = next((b for b in books if b["id"] == book_id), None)
if book is None:
return jsonify({"error": "Book not found"}), 404
data = request.get_json()
book["title"] = data.get("title", book["title"])
book["author"] = data.get("author", book["author"])
book["year"] = data.get("year", book["year"])
return jsonify(book)
@app.route("/api/books/<int:book_id>", methods=["DELETE"])
def delete_book(book_id):
global books
book = next((b for b in books if b["id"] == book_id), None)
if book is None:
return jsonify({"error": "Book not found"}), 404
books = [b for b in books if b["id"] != book_id]
return jsonify({"message": f"Book {book_id} deleted"})
if __name__ == "__main__":
app.run(debug=True)
There are several important patterns in this code. The methods parameter on @app.route() restricts which HTTP methods a route accepts — without it, Flask only allows GET. The request.get_json() method parses the request body as JSON. We return appropriate HTTP status codes: 201 for successful creation, 404 when a resource is not found, and 400 for bad requests. The <int:book_id> URL parameter tells Flask to extract an integer from the URL and pass it to your function — if someone sends a non-integer, Flask automatically returns a 404.
Error Handling in Flask APIs
A well-designed API needs consistent error responses. Flask lets you register custom error handlers that return JSON instead of the default HTML error pages. This is critical for APIs — your clients are programs, not browsers, and they need machine-readable error messages.
# error_handling.py
from flask import Flask, jsonify, request
app = Flask(__name__)
# Custom error handlers for common HTTP errors
@app.errorhandler(404)
def not_found(error):
return jsonify({"error": "Resource not found", "status": 404}), 404
@app.errorhandler(405)
def method_not_allowed(error):
return jsonify({"error": "Method not allowed", "status": 405}), 405
@app.errorhandler(400)
def bad_request(error):
return jsonify({"error": "Bad request", "status": 400}), 400
@app.errorhandler(500)
def internal_error(error):
return jsonify({"error": "Internal server error", "status": 500}), 500
# Example route with input validation
@app.route("/api/calculate", methods=["POST"])
def calculate():
data = request.get_json()
if not data:
return jsonify({"error": "Request body must be JSON"}), 400
a = data.get("a")
b = data.get("b")
operation = data.get("operation", "add")
if a is None or b is None:
return jsonify({"error": "Fields 'a' and 'b' are required"}), 400
try:
a, b = float(a), float(b)
except (ValueError, TypeError):
return jsonify({"error": "Fields 'a' and 'b' must be numbers"}), 400
operations = {
"add": a + b,
"subtract": a - b,
"multiply": a * b,
}
if operation == "divide":
if b == 0:
return jsonify({"error": "Cannot divide by zero"}), 400
result = a / b
elif operation in operations:
result = operations[operation]
else:
return jsonify({"error": f"Unknown operation: {operation}"}), 400
return jsonify({"result": result, "operation": operation})
if __name__ == "__main__":
app.run(debug=True)
Output:
# Request: POST /api/calculate with {"a": 10, "b": 3, "operation": "multiply"}
# Response: {"result": 30.0, "operation": "multiply"}
# Request: POST /api/calculate with {"a": 10, "b": 0, "operation": "divide"}
# Response (400): {"error": "Cannot divide by zero"}
# Request: GET /api/nonexistent
# Response (404): {"error": "Resource not found", "status": 404}
The @app.errorhandler() decorator registers functions that handle specific HTTP error codes globally. Every 404 in your entire application will now return JSON instead of an HTML page. This is important because API clients like mobile apps and JavaScript frontends expect JSON responses for everything, including errors. The calculate endpoint demonstrates thorough input validation — checking for missing fields, wrong types, and edge cases like division by zero before processing the request.
404: Resource not found. 200: Everything is fine. 500: Run.
Testing Your API With curl and requests
Building an API is only half the job — you also need to test it. The two most common tools for testing REST APIs from the command line are curl (built into most operating systems) and Python’s requests library. Let us see both approaches for testing our books API.
Testing With curl
The curl command is the fastest way to test an endpoint from your terminal. Here are the commands for each CRUD operation:
# testing_curl.sh
# GET all books
curl http://127.0.0.1:5000/api/books
# GET a single book
curl http://127.0.0.1:5000/api/books/1
# POST - create a new book
curl -X POST http://127.0.0.1:5000/api/books \
-H "Content-Type: application/json" \
-d '{"title": "Clean Code", "author": "Robert Martin", "year": 2008}'
# PUT - update an existing book
curl -X PUT http://127.0.0.1:5000/api/books/1 \
-H "Content-Type: application/json" \
-d '{"title": "Python Crash Course (3rd Ed)", "year": 2023}'
# DELETE - remove a book
curl -X DELETE http://127.0.0.1:5000/api/books/3
The -X flag specifies the HTTP method, -H sets headers, and -d sends the request body. Always include Content-Type: application/json when sending JSON data — without it, Flask’s request.get_json() returns None.
Testing With Python’s requests Library
For more complex testing or when you want to automate API tests, Python’s requests library is more convenient than curl:
# test_api.py
import requests
BASE_URL = "http://127.0.0.1:5000/api/books"
# GET all books
response = requests.get(BASE_URL)
print(f"GET all: {response.status_code}")
print(f"Books: {response.json()}\n")
# POST a new book
new_book = {"title": "Clean Code", "author": "Robert Martin", "year": 2008}
response = requests.post(BASE_URL, json=new_book)
print(f"POST: {response.status_code}")
print(f"Created: {response.json()}\n")
# GET a single book
response = requests.get(f"{BASE_URL}/1")
print(f"GET one: {response.status_code}")
print(f"Book: {response.json()}\n")
# PUT update
update_data = {"title": "Python Crash Course (3rd Ed)"}
response = requests.put(f"{BASE_URL}/1", json=update_data)
print(f"PUT: {response.status_code}")
print(f"Updated: {response.json()}\n")
# DELETE
response = requests.delete(f"{BASE_URL}/3")
print(f"DELETE: {response.status_code}")
print(f"Result: {response.json()}")
The requests library is much easier to work with programmatically. The json= parameter automatically serializes your dictionary to JSON and sets the correct Content-Type header. The response.json() method parses the response body back into a Python dictionary. This makes it trivial to write automated test scripts that verify your API behaves correctly after every code change.
Real-Life Example: Bookmark Manager API
Your Flask API just launched into production. Time to celebrate!
Let us build something practical — a Bookmark Manager API that stores website bookmarks with titles, URLs, tags, and timestamps. It supports full CRUD operations, searching by tag, and persists data to a JSON file so bookmarks survive server restarts. This project uses every concept from the article.
# bookmark_api.py
import os
import json
from datetime import datetime
from flask import Flask, jsonify, request
app = Flask(__name__)
DATA_FILE = "bookmarks.json"
def load_data():
"""Load bookmarks from JSON file."""
if os.path.exists(DATA_FILE):
with open(DATA_FILE, "r") as f:
return json.load(f)
return {"bookmarks": [], "next_id": 1}
def save_data(data):
"""Save bookmarks to JSON file."""
with open(DATA_FILE, "w") as f:
json.dump(data, f, indent=2)
@app.route("/api/bookmarks", methods=["GET"])
def get_bookmarks():
data = load_data()
tag = request.args.get("tag") # Optional tag filter
bookmarks = data["bookmarks"]
if tag:
bookmarks = [b for b in bookmarks if tag.lower() in [t.lower() for t in b["tags"]]]
return jsonify(bookmarks)
@app.route("/api/bookmarks/<int:bookmark_id>", methods=["GET"])
def get_bookmark(bookmark_id):
data = load_data()
bookmark = next((b for b in data["bookmarks"] if b["id"] == bookmark_id), None)
if not bookmark:
return jsonify({"error": "Bookmark not found"}), 404
return jsonify(bookmark)
@app.route("/api/bookmarks", methods=["POST"])
def create_bookmark():
data = load_data()
body = request.get_json()
if not body or "url" not in body:
return jsonify({"error": "URL is required"}), 400
bookmark = {
"id": data["next_id"],
"title": body.get("title", "Untitled"),
"url": body["url"],
"tags": body.get("tags", []),
"created_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
}
data["bookmarks"].append(bookmark)
data["next_id"] += 1
save_data(data)
return jsonify(bookmark), 201
@app.route("/api/bookmarks/<int:bookmark_id>", methods=["PUT"])
def update_bookmark(bookmark_id):
data = load_data()
bookmark = next((b for b in data["bookmarks"] if b["id"] == bookmark_id), None)
if not bookmark:
return jsonify({"error": "Bookmark not found"}), 404
body = request.get_json()
bookmark["title"] = body.get("title", bookmark["title"])
bookmark["url"] = body.get("url", bookmark["url"])
bookmark["tags"] = body.get("tags", bookmark["tags"])
save_data(data)
return jsonify(bookmark)
@app.route("/api/bookmarks/<int:bookmark_id>", methods=["DELETE"])
def delete_bookmark(bookmark_id):
data = load_data()
original_count = len(data["bookmarks"])
data["bookmarks"] = [b for b in data["bookmarks"] if b["id"] != bookmark_id]
if len(data["bookmarks"]) == original_count:
return jsonify({"error": "Bookmark not found"}), 404
save_data(data)
return jsonify({"message": f"Bookmark {bookmark_id} deleted"})
@app.errorhandler(404)
def not_found(error):
return jsonify({"error": "Resource not found"}), 404
@app.errorhandler(400)
def bad_request(error):
return jsonify({"error": "Bad request"}), 400
if __name__ == "__main__":
app.run(debug=True)
This project demonstrates a complete, production-style API pattern. Every endpoint validates its input, returns appropriate status codes, and handles edge cases like missing resources. The tag filtering on the GET endpoint shows how to support query parameters for searching and filtering. The JSON file persistence means bookmarks survive server restarts without needing a database. You could extend this with pagination (limit and offset parameters), full-text search across titles, import/export from browser bookmark files, or a simple HTML frontend.
Frequently Asked Questions
Should I use Flask or Django for my API?
Flask is better for small-to-medium APIs and microservices where you want full control over your stack. Django REST Framework is better when you need an admin panel, ORM, authentication, and other batteries-included features out of the box. If you are building a simple API or learning, start with Flask. If you are building a large application with user accounts and a database, Django might save you time.
How do I connect Flask to a database?
For SQLite (great for learning and small apps), use Python’s built-in sqlite3 module directly. For production, Flask-SQLAlchemy is the most popular choice — it provides an ORM that works with PostgreSQL, MySQL, and SQLite. Install it with pip install flask-sqlalchemy and define your models as Python classes. For simple projects, a JSON file (like we used in the Bookmark Manager) works fine and avoids the complexity of database setup.
Why do I get CORS errors when calling my Flask API from JavaScript?
Browsers block JavaScript from making requests to a different domain than the page was loaded from — this is called Cross-Origin Resource Sharing (CORS). To fix it, install flask-cors with pip install flask-cors, then add CORS(app) to your Flask app. This enables all origins by default. For production, configure it to only allow your specific frontend domain.
How do I deploy a Flask API to production?
Never use Flask’s built-in development server in production — it is single-threaded and not secure. Instead, use a WSGI server like Gunicorn (pip install gunicorn, then gunicorn app:app). Popular deployment platforms include Railway, Render, Heroku, and AWS. For a simple setup, a DigitalOcean droplet with Gunicorn behind Nginx works well and can handle thousands of requests per second.
How do I add authentication to my Flask API?
The simplest approach is API key authentication — check for a key in the request headers. For token-based auth, use Flask-JWT-Extended (pip install flask-jwt-extended), which handles JWT token creation, validation, and refresh. For OAuth and social login, Flask-Login combined with Flask-Dance works well. Start with API keys for internal tools and move to JWT when you need user accounts.
How do I write automated tests for a Flask API?
Flask includes a built-in test client that simulates HTTP requests without starting a server. Use app.test_client() in your test functions with pytest: call client.get("/api/books"), then assert response.status_code == 200 and check response.get_json() for the expected data. This lets you test every endpoint and edge case automatically as part of your CI pipeline.
Conclusion
You now know how to build a complete REST API with Flask. We covered the fundamentals: routing with @app.route(), handling JSON with request.get_json() and jsonify(), building full CRUD endpoints for creating, reading, updating, and deleting resources, returning proper HTTP status codes, implementing custom error handlers, and testing with both curl and Python’s requests library. We tied it all together with a Bookmark Manager API that includes tag-based filtering and JSON file persistence.
The Bookmark Manager is a solid foundation for your own projects. Try extending it with pagination, user authentication, a SQLite database, or a React frontend that consumes the API. The REST patterns you learned here are universal — they apply whether you are building a personal project or a production microservice.
For more advanced Flask features like blueprints, middleware, and extensions, check out the official Flask documentation. For REST API design best practices, the RESTful API tutorial is an excellent reference.
How To Schedule Python Scripts To Run Automatically With cron and schedule
If you’re building Python applications, sooner or later you’ll want them to run on a schedule without you having to sit at your computer and trigger them manually. Whether it’s a daily backup, sending reports, checking for updates, or cleaning up temporary files, automation is a game-changer. In this tutorial, we’ll explore how to schedule Python scripts to run automatically using both Linux cron and the Python schedule library, plus we’ll touch on Windows Task Scheduler for our Windows users.
Quick Example (TLDR)
Here’s the fastest way to schedule a Python script using the schedule library:
# Install: pip install schedule
import schedule
import time
def my_job():
# This code runs on schedule
print("Job executed at", time.strftime("%Y-%m-%d %H:%M:%S"))
# Schedule the job to run every day at 10:00 AM
schedule.every().day.at("10:00").do(my_job)
# Keep the scheduler running
while True:
schedule.run_pending()
time.sleep(60) # Check every minute if a job should run
Output:
Job executed at 2026-03-12 10:00:15
Job executed at 2026-03-13 10:00:12
Why Automate Your Python Scripts?
Let’s think about some real-world scenarios where automation saves you time and money:
Backups: Automatically backup your database every night without remembering
Reports: Generate and email reports every Monday morning at 9 AM
Data Processing: Process new files as they arrive, every hour
Monitoring: Check server health every 5 minutes and alert if something’s wrong
Cleanup: Delete temporary files older than 30 days, weekly
When you automate these tasks, you free up mental energy to focus on more important things. Plus, computers don’t sleep, so they can work 24/7 without complaining!
Method 1: Linux cron Jobs
If you’re running on Linux or macOS, cron is a powerful and built-in scheduler. Let’s walk through how to set up a cron job.
Step 1: Create Your Python Script
First, let’s create a simple backup script:
# backup_script.py
import shutil
import os
from datetime import datetime
def backup_database():
# Get current date and time
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
# Source and destination paths
source = "/home/user/myapp/data.db"
destination = f"/home/user/backups/data_{timestamp}.db"
try:
# Copy the database file
shutil.copy2(source, destination)
print(f"Backup successful: {destination}")
except Exception as e:
# Log errors for troubleshooting
print(f"Backup failed: {e}")
if __name__ == "__main__":
backup_database()
Add this line to run the backup every day at 2 AM:
# Run backup script every day at 2:00 AM
0 2 * * * /usr/bin/python3 /home/user/backup_script.py >> /var/log/backup.log 2>&1
The cron syntax is: minute hour day month weekday command
0 2 * * * = Every day at 2:00 AM
/usr/bin/python3 = Full path to Python interpreter
/home/user/backup_script.py = Your script path
>> /var/log/backup.log 2>&1 = Log output to a file
Method 2: Python schedule Library (Recommended for Cross-Platform)
If you need more control or want your scheduler to run within your Python application, the schedule library is excellent:
# job_scheduler.py
import schedule
import time
import logging
from datetime import datetime
# Setup logging to track what happens
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(message)s'
)
def backup_job():
# This runs every day
logging.info("Starting daily backup...")
# Your backup code here
logging.info("Backup complete!")
def send_report():
# This runs every Monday at 9 AM
logging.info("Sending weekly report...")
# Your email code here
logging.info("Report sent!")
def cleanup_temp_files():
# This runs every hour
logging.info("Cleaning temporary files...")
# Your cleanup code here
logging.info("Cleanup complete!")
# Schedule the jobs
schedule.every().day.at("02:00").do(backup_job)
schedule.every().monday.at("09:00").do(send_report)
schedule.every().hour.do(cleanup_temp_files)
# Keep the scheduler running
if __name__ == "__main__":
print("Scheduler started. Press Ctrl+C to stop.")
while True:
schedule.run_pending()
time.sleep(60) # Check every minute
Windows users can use Task Scheduler to run Python scripts automatically:
Press Win + R and type taskschd.msc to open Task Scheduler
Click “Create Basic Task”
Give it a name: “My Python Backup”
Set trigger (when to run): Daily at 2:00 AM
Set action: Start a program
Program: C:Python312python.exe
Arguments: C:UsersYourNameackup_script.py
Click Finish
Error Handling in Scheduled Tasks
When your script runs automatically, you won’t be there to see errors. That’s why logging is critical:
# robust_scheduler.py
import schedule
import time
import logging
import traceback
from datetime import datetime
# Configure detailed logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler('/var/log/my_jobs.log'),
logging.StreamHandler()
]
)
def safe_job_wrapper(job_func, job_name):
# Wrapper function that catches and logs errors
def wrapper():
try:
logging.info(f"Starting: {job_name}")
job_func()
logging.info(f"Completed: {job_name}")
except Exception as e:
# Log the full error trace for debugging
logging.error(f"Failed: {job_name}")
logging.error(traceback.format_exc())
return wrapper
def my_risky_job():
# This might fail sometimes
result = 10 / 1 # Would be 10 / 0 to cause error
print(f"Calculation result: {result}")
# Schedule with error handling
safe_wrapper = safe_job_wrapper(my_risky_job, "my_risky_job")
schedule.every().hour.do(safe_wrapper)
if __name__ == "__main__":
while True:
schedule.run_pending()
time.sleep(60)
Log Output (when error occurs):
2026-03-12 15:00:00 - INFO - Starting: my_risky_job
2026-03-12 15:00:00 - ERROR - Failed: my_risky_job
2026-03-12 15:00:00 - ERROR - Traceback (most recent call last):
File "scheduler.py", line 15, in wrapper
job_func()
File "scheduler.py", line 28, in my_risky_job
result = 10 / 0
ZeroDivisionError: division by zero
Real-Life Example: Automated Daily Backup Script
Let’s build a complete backup system that you can use right away:
# daily_backup.py
import schedule
import time
import os
import shutil
import logging
from datetime import datetime, timedelta
import gzip
import json
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
filename='/var/log/backup.log'
)
class BackupManager:
def __init__(self, source_dir, backup_dir, keep_days=7):
# Initialize backup settings
self.source_dir = source_dir
self.backup_dir = backup_dir
self.keep_days = keep_days
# Create backup directory if it doesn't exist
os.makedirs(backup_dir, exist_ok=True)
def backup_files(self):
# Create timestamped backup
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
backup_path = os.path.join(self.backup_dir, f"backup_{timestamp}.tar.gz")
try:
# Compress the entire directory
shutil.make_archive(
backup_path.replace('.tar.gz', ''),
'gztar',
self.source_dir
)
logging.info(f"Backup created: {backup_path}")
# Clean old backups
self.cleanup_old_backups()
except Exception as e:
logging.error(f"Backup failed: {e}")
def cleanup_old_backups(self):
# Remove backups older than keep_days
cutoff_date = datetime.now() - timedelta(days=self.keep_days)
for filename in os.listdir(self.backup_dir):
filepath = os.path.join(self.backup_dir, filename)
file_time = datetime.fromtimestamp(os.path.getmtime(filepath))
if file_time < cutoff_date:
try:
os.remove(filepath)
logging.info(f"Deleted old backup: {filename}")
except Exception as e:
logging.error(f"Failed to delete {filename}: {e}")
# Create backup manager instance
backup = BackupManager("/home/user/myapp", "/home/user/backups")
# Schedule daily backup at 2 AM
schedule.every().day.at("02:00").do(backup.backup_files)
# Keep scheduler running
if __name__ == "__main__":
logging.info("Backup scheduler started")
while True:
schedule.run_pending()
time.sleep(60)
Output:
2026-03-12 02:00:00 - INFO - Backup created: /home/user/backups/backup_20260312_020000.tar.gz
2026-03-13 02:00:00 - INFO - Backup created: /home/user/backups/backup_20260313_020000.tar.gz
2026-03-15 02:00:00 - INFO - Deleted old backup: backup_20260305_020000.tar.gz
FAQ
Q1: How do I know if my scheduled job ran?
Use logging! Redirect output to a log file as shown in our examples. Check the log file to verify execution.
Q2: Can I schedule a job for every 30 minutes?
Yes! With schedule library: schedule.every(30).minutes.do(my_job). With cron: */30 * * * * command
Q3: What's the difference between cron and the schedule library?
Cron is system-level, always running. Schedule library runs inside your Python process. Cron is better for production servers; schedule is better for local testing and development.
Q4: How do I stop a scheduled task?
For cron: crontab -e and delete the line. For schedule: Stop the Python process (Ctrl+C). For Windows: Open Task Scheduler and disable the task.
Q5: What if my Python script takes longer to run than scheduled?
Use a queuing system like Celery for production workloads. For simple scripts, the schedule library won't run overlapping instances by default.
Conclusion
Scheduling Python scripts is one of the most practical skills you can develop. Whether you use cron for server environments, the schedule library for cross-platform applications, or Windows Task Scheduler for local automation, you now have the tools to build reliable automated systems. Start small with a simple daily task, add logging to track what happens, and expand from there. Your future self will thank you for automating those repetitive tasks!
schedule.every().monday.at('09:00') — cron syntax for people who value their sanity.
cron Basics for Python Scripts
On Linux and macOS, cron is the system-level scheduler. Run crontab -e to open your user's crontab; each line is a schedule plus a command. The five fields before the command are minute hour day-of-month month day-of-week:
# Run every day at 2:30 AM
30 2 * * * /usr/bin/python3 /home/me/scripts/cleanup.py
# Every 15 minutes
*/15 * * * * /home/me/.venv/bin/python /home/me/scripts/poll.py
# Every Monday at 9 AM
0 9 * * 1 /home/me/.venv/bin/python /home/me/scripts/weekly-report.py
# Twice an hour on weekdays
0,30 * * * 1-5 /home/me/.venv/bin/python /home/me/scripts/business-hours.py
Three rules that save hours of debugging:
Use absolute paths. Cron runs with a minimal PATH. python3 alone may not resolve. Use /usr/bin/python3 or your venv's full path.
Redirect stdout / stderr. Append >> /var/log/cleanup.log 2>&1 so you can debug. Cron-by-default emails errors to the system mail spool, which you'll never see.
Test with a 'run in one minute' schedule first. Set the job to run in 1-2 minutes from now while you're watching the log — much faster than waiting until 2:30 AM.
The schedule Library: Cron in Pure Python
For schedules that should travel with your code (containerized apps, cross-platform scripts), the schedule library gives you a Pythonic API:
The advantage over cron: the schedule travels with the code, no system-level setup. The disadvantage: your script has to stay running. Pair it with systemd on Linux or supervisord to handle restarts.
APScheduler — Production-Grade Scheduling
For real applications, APScheduler beats both cron and schedule: persistent jobs (survive restart), missed-job handling, multiple triggers per job, async support. Three job stores cover most use cases — memory (default), SQLAlchemy (persistent), Redis (distributed):
# pip install apscheduler
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.triggers.cron import CronTrigger
def daily_etl():
print("Running ETL")
def hourly_sync():
print("Syncing...")
scheduler = BackgroundScheduler()
scheduler.add_job(
daily_etl,
trigger=CronTrigger(hour=2, minute=30),
id="daily-etl",
replace_existing=True,
)
scheduler.add_job(hourly_sync, "interval", hours=1)
scheduler.start()
# Keep the main thread alive
import time
while True:
time.sleep(60)
The replace_existing=True idiom is essential — without it, restarting your app fails with "job already exists" against a persistent job store.
Celery Beat — Distributed Scheduled Tasks
For microservices, Celery Beat schedules tasks across a distributed worker pool. The beat process emits scheduled tasks into the queue; any worker consumes them. Survives any single node restart, scales horizontally:
# celery_app.py
from celery import Celery
from celery.schedules import crontab
app = Celery("tasks", broker="redis://localhost:6379")
app.conf.beat_schedule = {
"daily-report": {
"task": "tasks.send_daily_report",
"schedule": crontab(hour=9, minute=0),
},
"every-15-min": {
"task": "tasks.cleanup",
"schedule": 900.0, # 15 minutes in seconds
},
}
@app.task
def send_daily_report():
print("Daily report sent")
@app.task
def cleanup():
print("Cleanup done")
# Run beat alongside the workers:
# celery -A celery_app beat
# celery -A celery_app worker
Common Pitfalls
Time zone confusion. Cron uses the system's local time. APScheduler defaults to UTC. schedule uses local time. Be explicit about which one you intend, or you'll be off by hours after the next DST change.
Long-running jobs blocking the scheduler. If a 15-minute job kicks off every 15 minutes, the next instance fires while the first is still running. Use max_instances=1 in APScheduler or queue jobs through Celery.
Forgetting the venv. Cron's python3 isn't your venv's python. Use the full venv path: /home/me/.venv/bin/python.
Silent failures. Cron emails errors to /var/mail by default. If you've never seen those emails, you're missing failures. Pipe to a log file and tail it occasionally.
Race conditions on overlap. Two cron jobs that need exclusive file access will collide if their schedules overlap. Use file locks (flock) or a job queue.
FAQ
Q: Cron, schedule, APScheduler, or Celery Beat?
A: Cron for one-off OS-level scripts. schedule for in-process scheduling in small apps. APScheduler when you need persistence or async. Celery Beat for distributed multi-worker setups.
Q: How do I make sure my script doesn't run twice if it overlaps?
A: File lock at the top: fd = open("/tmp/myscript.lock"); fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB). Exits cleanly if another instance holds the lock.
Q: How do I run a Python script on a schedule on Windows?
A: Use Task Scheduler. Create a basic task, set the trigger (daily, weekly, on event), set the action to C:\Python312\python.exe C:\scripts\myscript.py. Or use APScheduler in-process — works identically on Windows.
Q: How do I see what cron jobs are scheduled?
A: crontab -l for the current user, sudo crontab -l -u username for another user. System-level cron lives in /etc/cron.d/* and /etc/cron.{hourly,daily,weekly,monthly}/*.
Q: How do I retry a failed scheduled job?
A: APScheduler doesn't retry out of the box. Wrap your job in a try/except + sleep, or use Celery's built-in retry: @app.task(autoretry_for=(Exception,), retry_backoff=True).
Wrapping Up
Pick the right tool for the scale. A single Python script that should run at 2 AM? Plain cron. A small app with a dozen recurring jobs? schedule in-process. Real production app with persistent jobs and async support? APScheduler. Microservices fleet? Celery Beat. The complexity of each tool tracks the complexity of the use case — don't reach for Celery Beat when cron will do.
You have written a function that works perfectly. Then one day it starts running slowly in production, and you have no idea which function is the bottleneck. Or maybe you need to log every time a critical function gets called, but you do not want to litter every function body with print() statements. These are the exact problems that Python decorators solve — they let you wrap extra behavior around your functions without touching the function code itself.
The good news is that decorators are built into Python’s core syntax. You do not need to install anything — just standard Python 3. The concept uses closures and first-class functions, which sounds intimidating, but the pattern is surprisingly simple once you see it in action. By the end of this article you will be writing your own decorators from scratch with confidence.
In this article we will start with a quick working example so you can see the payoff immediately. Then we will break down what decorators actually are and how they work under the hood. From there we will build a logging decorator, a timing decorator, a retry decorator, and learn how to stack multiple decorators together. We will also cover functools.wraps — a small but critical detail that most tutorials skip. Finally, we will tie everything together with a real-life performance monitoring toolkit you can drop into any project.
Python Decorators for Logging: Quick Example
Let us jump straight into a working example. Here is a decorator that automatically logs every time a function is called, including the arguments it received and the value it returned. You can copy this code, run it, and see the result immediately.
# quick_example.py
import functools
def log_calls(func):
"""Decorator that logs function calls with arguments and return values."""
@functools.wraps(func)
def wrapper(*args, **kwargs):
print(f"CALL: {func.__name__}({args}, {kwargs})")
result = func(*args, **kwargs)
print(f"RETURN: {func.__name__} -> {result}")
return result
return wrapper
@log_calls
def add(a, b):
return a + b
print(add(3, 5))
print(add(10, 20))
With just the @log_calls line above the function definition, every call to add() now prints a log message automatically. The function itself has no idea it is being logged — it just does its job and returns the sum. That separation of concerns is the entire point of decorators. The functools.wraps call inside the decorator preserves the original function’s name and docstring, which we will explain in detail later.
Want to go deeper? Below we cover exactly how this pattern works, how to build timing and retry decorators, and how to combine multiple decorators on a single function.
What Are Python Decorators and Why Use Them?
A decorator is a function that takes another function as input, adds some behavior to it, and returns a new function. Think of it like gift wrapping — the present inside (your original function) stays exactly the same, but the wrapping paper (the decorator) adds something extra on the outside. When someone opens the gift, they get both the wrapping experience and the present itself.
In Python, functions are first-class objects. That means you can pass a function as an argument to another function, return a function from a function, and assign a function to a variable. Decorators take advantage of all three of these capabilities. The @decorator_name syntax above a function definition is just shorthand — writing @log_calls above def add() is exactly the same as writing add = log_calls(add) after the function definition.
Here is when decorators are the right tool for the job versus when you should use something else:
Use Case
Decorator?
Why
Add logging to multiple functions
Yes
Same behavior applied to many functions without repeating code
Measure execution time
Yes
Timing logic stays separate from business logic
Retry on failure
Yes
Retry policy wraps around the function cleanly
Input validation
Yes
Validate arguments before the function runs
Change what a function returns
Maybe
Can work, but modifying return values can be confusing to callers
Complex state management
No
Use a class instead — decorators should be stateless or nearly so
The key principle is that decorators work best when you want to add the same cross-cutting behavior to multiple functions. If you find yourself copying the same three lines of logging code into every function, that is a strong signal that a decorator would clean things up. Let us start building decorators from scratch.
Basic Decorator Syntax in Python
Before we build useful decorators, let us understand the basic pattern. A decorator is a function that accepts a function as its argument, defines an inner function (called a wrapper) that adds behavior, and returns the wrapper. Here is the simplest possible decorator that does nothing except call the original function.
# basic_decorator.py
def my_decorator(func):
"""A minimal decorator that wraps a function without changing behavior."""
def wrapper(*args, **kwargs):
# You could add behavior here (before the call)
result = func(*args, **kwargs)
# You could add behavior here (after the call)
return result
return wrapper
@my_decorator
def greet(name):
return f"Hello, {name}!"
print(greet("Alice"))
print(greet("Bob"))
Output:
Hello, Alice!
Hello, Bob!
The wrapper function uses *args and **kwargs to accept any combination of positional and keyword arguments. This is important because the decorator needs to work with functions that have different signatures — a function with two parameters, a function with five parameters, or a function with keyword-only parameters should all work the same way. The result = func(*args, **kwargs) line calls the original function with whatever arguments were passed in, and return result passes the return value back to the caller.
Now let us replace those placeholder comments with real, useful behavior. We will start with a logging decorator.
Building a Logging Decorator
A logging decorator captures information about function calls as they happen. This is incredibly useful for debugging — instead of sprinkling print() statements throughout your code and then removing them later, you simply add or remove the @log_calls decorator. Here is a robust version that formats the output nicely and handles both positional and keyword arguments.
# logging_decorator.py
import functools
from datetime import datetime
def log_calls(func):
"""Log every call to the decorated function with timestamp, args, and result."""
@functools.wraps(func)
def wrapper(*args, **kwargs):
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
# Format arguments for readable output
arg_parts = [repr(a) for a in args]
kwarg_parts = [f"{k}={v!r}" for k, v in kwargs.items()]
all_args = ", ".join(arg_parts + kwarg_parts)
print(f"[{timestamp}] CALL: {func.__name__}({all_args})")
result = func(*args, **kwargs)
print(f"[{timestamp}] RETURN: {func.__name__} -> {result!r}")
return result
return wrapper
@log_calls
def calculate_discount(price, discount_percent=10):
"""Calculate the discounted price."""
discount_amount = price * (discount_percent / 100)
return round(price - discount_amount, 2)
# Test with positional and keyword arguments
print(calculate_discount(99.99))
print()
print(calculate_discount(49.99, discount_percent=25))
Notice how the decorator captures the function name, the exact arguments passed (including keyword arguments like discount_percent=25), and the return value. The !r format specifier uses repr() so strings show up with quotes, making it easy to distinguish "hello" from hello in your logs. The timestamp tells you exactly when each call happened, which is essential for debugging timing-related issues in production. You can easily swap print() for Python’s built-in logging module to write these messages to a file instead of the console.
Clean code isn’t written — it’s decorated. One @wrapper and your function gains superpowers.
Building a Timing Decorator
Performance matters, and the first step to improving performance is measuring it. A timing decorator wraps a function and records how long it takes to execute. This is far more convenient than manually adding time.time() calls at the start and end of every function you want to profile.
# timing_decorator.py
import functools
import time
def timer(func):
"""Measure and print the execution time of the decorated function."""
@functools.wraps(func)
def wrapper(*args, **kwargs):
start_time = time.perf_counter() # High-precision timer
result = func(*args, **kwargs)
end_time = time.perf_counter()
elapsed = end_time - start_time
print(f"⏱ {func.__name__} took {elapsed:.4f} seconds")
return result
return wrapper
@timer
def slow_operation():
"""Simulate a slow operation."""
total = 0
for i in range(1_000_000):
total += i * i
return total
@timer
def fast_operation():
"""Simulate a fast operation."""
return sum(range(1000))
print(f"Result: {slow_operation()}")
print(f"Result: {fast_operation()}")
Output:
⏱ slow_operation took 0.0892 seconds
Result: 333332833333500000
⏱ fast_operation took 0.0000 seconds
Result: 499500
We use time.perf_counter() instead of time.time() because it provides the highest resolution timer available on your operating system. This matters when measuring fast functions — time.time() might show 0.0 seconds for something that actually takes 0.3 milliseconds. The :.4f format gives us four decimal places, which is enough precision for most profiling needs. If you need sub-millisecond accuracy, change it to :.6f for microsecond resolution.
Using functools.wraps for Metadata Preservation
You may have noticed that every decorator we have written includes @functools.wraps(func) on the wrapper function. This is not optional — without it, the decorated function loses its identity. Let us see exactly what goes wrong when you skip it.
# without_wraps.py
def bad_decorator(func):
"""Decorator WITHOUT functools.wraps — demonstrates the problem."""
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
@bad_decorator
def calculate_tax(amount, rate=0.08):
"""Calculate sales tax on a purchase amount."""
return round(amount * rate, 2)
# Check the function's identity
print(f"Function name: {calculate_tax.__name__}")
print(f"Docstring: {calculate_tax.__doc__}")
print(f"Help output:")
help(calculate_tax)
Output:
Function name: wrapper
Docstring: None
Help output:
Help on function wrapper in module __main__:
wrapper(*args, **kwargs)
The function thinks its name is wrapper and its docstring is None. This breaks debugging tools, documentation generators, and any code that inspects function metadata. Now let us see the same decorator with functools.wraps applied.
# with_wraps.py
import functools
def good_decorator(func):
"""Decorator WITH functools.wraps — preserves the original function's metadata."""
@functools.wraps(func)
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
@good_decorator
def calculate_tax(amount, rate=0.08):
"""Calculate sales tax on a purchase amount."""
return round(amount * rate, 2)
# Check the function's identity
print(f"Function name: {calculate_tax.__name__}")
print(f"Docstring: {calculate_tax.__doc__}")
print(f"Help output:")
help(calculate_tax)
Output:
Function name: calculate_tax
Docstring: Calculate sales tax on a purchase amount.
Help output:
Help on function calculate_tax in module __main__:
calculate_tax(amount, rate=0.08)
Calculate sales tax on a purchase amount.
With @functools.wraps(func), the decorated function retains its original name, docstring, and even its parameter signature in the help output. This is a one-line addition that prevents hours of confusion. Every decorator you write should include it — no exceptions. The rule is simple: if your decorator defines a wrapper function, put @functools.wraps(func) on the line directly above it.
Stack too many decorators and suddenly nobody knows what the function actually does. Including you.
Decorators That Accept Arguments
Sometimes you want your decorator to be configurable. For example, a logging decorator where you can set the log level, or a retry decorator where you can specify the number of retries. This requires an extra layer of nesting — a function that returns a decorator, which returns a wrapper. It sounds complicated, but the pattern is consistent once you see it.
# decorator_with_args.py
import functools
def log_with_level(level="INFO"):
"""Decorator factory that accepts a log level argument."""
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
arg_parts = [repr(a) for a in args]
kwarg_parts = [f"{k}={v!r}" for k, v in kwargs.items()]
all_args = ", ".join(arg_parts + kwarg_parts)
print(f"[{level}] Calling {func.__name__}({all_args})")
result = func(*args, **kwargs)
print(f"[{level}] {func.__name__} returned {result!r}")
return result
return wrapper
return decorator
@log_with_level("DEBUG")
def fetch_user(user_id):
"""Simulate fetching a user from a database."""
return {"id": user_id, "name": "Alice", "email": "alice@example.com"}
@log_with_level("WARNING")
def delete_user(user_id):
"""Simulate deleting a user — dangerous operation."""
return f"User {user_id} deleted"
print(fetch_user(42))
print()
print(delete_user(7))
Output:
[DEBUG] Calling fetch_user(42)
[DEBUG] fetch_user returned {'id': 42, 'name': 'Alice', 'email': 'alice@example.com'}
{'id': 42, 'name': 'Alice', 'email': 'alice@example.com'}
[WARNING] Calling delete_user(7)
[WARNING] delete_user returned 'User 7 deleted'
User 7 deleted
The key difference is that log_with_level("DEBUG") is not the decorator itself — it is a decorator factory that returns the actual decorator. When Python sees @log_with_level("DEBUG"), it first calls log_with_level("DEBUG"), which returns the decorator function. Then Python applies that decorator function to fetch_user. This three-layer structure (factory → decorator → wrapper) is the standard pattern for any decorator that needs configuration.
Building a Retry Decorator
Network calls fail. APIs time out. Databases hiccup. A retry decorator automatically re-runs a function when it raises an exception, with a configurable number of attempts and delay between retries. This is one of the most practical decorators you will ever write.
# retry_decorator.py
import functools
import time
import random
def retry(max_attempts=3, delay=1.0):
"""Retry a function up to max_attempts times with a delay between retries."""
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
last_exception = None
for attempt in range(1, max_attempts + 1):
try:
return func(*args, **kwargs)
except Exception as e:
last_exception = e
if attempt < max_attempts:
print(f"⚠ {func.__name__} failed (attempt {attempt}/{max_attempts}): {e}")
print(f" Retrying in {delay}s...")
time.sleep(delay)
else:
print(f"✗ {func.__name__} failed after {max_attempts} attempts: {e}")
raise last_exception
return wrapper
return decorator
@retry(max_attempts=3, delay=0.5)
def unreliable_api_call():
"""Simulate an API call that fails randomly."""
if random.random() < 0.7: # 70% chance of failure
raise ConnectionError("Server unavailable")
return {"status": "ok", "data": [1, 2, 3]}
# Set seed for reproducible output
random.seed(42)
try:
result = unreliable_api_call()
print(f"Success: {result}")
except ConnectionError as e:
print(f"Final failure: {e}")
Output:
⚠ unreliable_api_call failed (attempt 1/3): Server unavailable
Retrying in 0.5s...
⚠ unreliable_api_call failed (attempt 2/3): Server unavailable
Retrying in 0.5s...
✗ unreliable_api_call failed after 3 attempts: Server unavailable
Final failure: Server unavailable
The retry decorator stores the last exception in last_exception so it can re-raise it if all attempts fail. This way the caller still gets the original exception type (ConnectionError) rather than a generic error. The time.sleep(delay) adds a pause between retries, which is important for network operations — hammering a server that just failed with immediate retries usually makes things worse. In production, you would typically use exponential backoff (doubling the delay each time) instead of a fixed delay.
Stacking Multiple Decorators
One of the most powerful features of decorators is that you can stack them. When you put multiple @decorator lines above a function, they are applied from bottom to top — the decorator closest to the function definition runs first, and the outermost decorator wraps everything. This lets you combine logging, timing, and retry logic on a single function.
# stacking_decorators.py
import functools
import time
def log_calls(func):
"""Log function calls."""
@functools.wraps(func)
def wrapper(*args, **kwargs):
print(f"LOG: Calling {func.__name__}")
result = func(*args, **kwargs)
print(f"LOG: {func.__name__} returned {result!r}")
return result
return wrapper
def timer(func):
"""Time function execution."""
@functools.wraps(func)
def wrapper(*args, **kwargs):
start = time.perf_counter()
result = func(*args, **kwargs)
elapsed = time.perf_counter() - start
print(f"⏱ {func.__name__} took {elapsed:.4f}s")
return result
return wrapper
@log_calls # Applied second (outermost)
@timer # Applied first (innermost)
def process_data(items):
"""Process a list of items with a simulated delay."""
time.sleep(0.1) # Simulate work
return [item.upper() for item in items]
result = process_data(["hello", "world", "python"])
Output:
LOG: Calling process_data
⏱ process_data took 0.1003s
LOG: process_data returned ['HELLO', 'WORLD', 'PYTHON']
The execution order matters. Because @timer is closest to the function, it wraps process_data first. Then @log_calls wraps the already-timed version. When you call process_data(), the log decorator runs first (printing the "Calling" message), then the timer starts, then the actual function runs, then the timer stops (printing the elapsed time), and finally the log decorator prints the return value. If you reversed the order, the timer would measure the time including the logging overhead, which is usually not what you want.
@lru_cache — turning O(2^n) into O(n) with one line. Your CPU sends its regards.
Real-Life Example: Performance Monitoring Toolkit
Decorators are just functions with ambition. Stack them right and your code writes itself.
Let us tie everything together into a practical project. This performance monitoring toolkit gives you three decorators that you can drop into any Python project: @monitor for combined logging and timing, @retry for fault tolerance, and @validate_types for runtime type checking. Together they form a lightweight monitoring layer for any application.
# performance_toolkit.py
import functools
import time
from datetime import datetime
def monitor(func):
"""Combined logging and timing decorator for production monitoring."""
@functools.wraps(func)
def wrapper(*args, **kwargs):
timestamp = datetime.now().strftime("%H:%M:%S")
arg_parts = [repr(a) for a in args]
kwarg_parts = [f"{k}={v!r}" for k, v in kwargs.items()]
signature = ", ".join(arg_parts + kwarg_parts)
print(f"[{timestamp}] → {func.__name__}({signature})")
start = time.perf_counter()
try:
result = func(*args, **kwargs)
elapsed = time.perf_counter() - start
print(f"[{timestamp}] ← {func.__name__} returned {result!r} ({elapsed:.4f}s)")
return result
except Exception as e:
elapsed = time.perf_counter() - start
print(f"[{timestamp}] ✗ {func.__name__} raised {type(e).__name__}: {e} ({elapsed:.4f}s)")
raise
return wrapper
def validate_types(**expected_types):
"""Validate argument types at runtime before the function executes."""
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
# Check keyword arguments against expected types
for param_name, expected_type in expected_types.items():
if param_name in kwargs:
value = kwargs[param_name]
if not isinstance(value, expected_type):
raise TypeError(
f"{param_name} must be {expected_type.__name__}, "
f"got {type(value).__name__}"
)
return func(*args, **kwargs)
return wrapper
return decorator
# --- Use the toolkit ---
@monitor
def fetch_user_profile(user_id):
"""Simulate fetching a user profile from a database."""
time.sleep(0.05) # Simulate database query
users = {1: "Alice", 2: "Bob", 3: "Charlie"}
if user_id not in users:
raise ValueError(f"User {user_id} not found")
return {"id": user_id, "name": users[user_id], "active": True}
@monitor
@validate_types(amount=float, description=str)
def process_payment(amount=0.0, description=""):
"""Process a payment transaction."""
time.sleep(0.02) # Simulate payment processing
return {"status": "completed", "amount": amount, "ref": "TXN-001"}
# Run the toolkit
print("=== Performance Monitoring Toolkit Demo ===\n")
# Successful call
profile = fetch_user_profile(1)
print(f"Got profile: {profile}\n")
# Successful payment
payment = process_payment(amount=29.99, description="Monthly subscription")
print(f"Payment result: {payment}\n")
# Failed call (user not found)
try:
fetch_user_profile(999)
except ValueError:
print("(Error handled gracefully)\n")
# Type validation failure
try:
process_payment(amount="not a number", description="Bad payment")
except TypeError as e:
print(f"Type error caught: {e}")
Output:
=== Performance Monitoring Toolkit Demo ===
[10:30:00] → fetch_user_profile(1)
[10:30:00] ← fetch_user_profile returned {'id': 1, 'name': 'Alice', 'active': True} (0.0503s)
Got profile: {'id': 1, 'name': 'Alice', 'active': True}
[10:30:00] → process_payment(amount=29.99, description='Monthly subscription')
[10:30:00] ← process_payment returned {'status': 'completed', 'amount': 29.99, 'ref': 'TXN-001'} (0.0201s)
Payment result: {'status': 'completed', 'amount': 29.99, 'ref': 'TXN-001'}
[10:30:00] → fetch_user_profile(999)
[10:30:00] ✗ fetch_user_profile raised ValueError: User 999 not found (0.0501s)
(Error handled gracefully)
Type error caught: amount must be float, got str
This toolkit demonstrates how decorators keep your business logic clean. The fetch_user_profile function only cares about looking up users — it knows nothing about logging, timing, or error formatting. All of that cross-cutting behavior lives in the decorators. You could add @monitor to every function in your application with a single line per function, giving you instant visibility into how your code performs. To extend this project, try adding a decorator that caches results (memoization) or one that limits how many times a function can be called per minute (rate limiting).
Frequently Asked Questions
What is the difference between a decorator and a regular function call?
A decorator wraps a function at definition time, not at call time. When you put @timer above a function, the wrapping happens once when Python loads the file. After that, every call to the function automatically goes through the wrapper. A regular function call like timer(my_func)(args) does the same thing but only for that one call. Decorators give you permanent, reusable wrapping with clean syntax.
Can you use a class as a decorator instead of a function?
Yes. Any callable object can be a decorator, and classes are callable (calling a class creates an instance). You define __init__ to accept the function and __call__ to act as the wrapper. Class-based decorators are useful when you need to maintain state between calls, such as counting how many times a function has been called. For stateless decorators like logging and timing, function-based decorators are simpler and more common.
How do you debug a decorated function?
Always use @functools.wraps(func) in your decorators — this is the single most important step. Without it, tracebacks show the wrapper function name instead of your actual function name, making debugging nearly impossible. If you need to temporarily remove a decorator for debugging, just comment out the @decorator line. You can also access the original unwrapped function via decorated_func.__wrapped__, which functools.wraps sets automatically.
Do decorators add performance overhead?
Yes, but it is typically negligible. Each decorated call adds the overhead of one extra function call plus whatever your wrapper does. For a simple logging decorator, this is microseconds. The overhead only matters if you are decorating a function that gets called millions of times in a tight loop. In that case, measure with time.perf_counter() and decide whether the convenience is worth the cost. For most applications — web servers, CLI tools, data processing scripts — the overhead is invisible.
Can decorators work with async functions?
Yes, but you need to make the wrapper function async too. Use async def wrapper(*args, **kwargs) and result = await func(*args, **kwargs) inside the decorator. If you want a single decorator that works with both sync and async functions, use import asyncio and check asyncio.iscoroutinefunction(func) to decide whether to use await. The functools.wraps pattern works the same way for async decorators.
Does the order of stacked decorators matter?
Absolutely. Decorators are applied bottom-to-top (closest to the function first), but they execute top-to-bottom when the function is called. If you stack @log_calls on top of @timer, the logging will record the total time including timer overhead. If you reverse them, the timer will only measure the function itself. The general rule is: put the decorator whose behavior you want to see first (outermost) on top, and put the decorator that should run closest to the actual function on the bottom.
Conclusion
In this article we covered everything you need to start using Python decorators effectively in your own projects. We started with the basic decorator pattern — a function that takes a function and returns a wrapper function. We built a logging decorator that captures function calls with timestamps and arguments, a timing decorator that measures execution time with time.perf_counter(), and a retry decorator that gracefully handles transient failures. We also covered functools.wraps (always use it), decorator factories for configurable decorators, and how stacking multiple decorators affects execution order.
The performance monitoring toolkit in the real-life example gives you a foundation you can build on. Try extending it with a caching decorator using functools.lru_cache, a rate limiter that tracks calls per time window, or an authentication decorator for a web application. Decorators are one of Python's most elegant features — once you get comfortable with the pattern, you will find uses for them everywhere.
Every Python developer starts with print() for debugging. It works fine when you are learning, but the moment your code runs in production — on a server, in a scheduled task, or as a background service — print statements become useless. They disappear when the terminal closes, they have no timestamps, no severity levels, and no way to separate important messages from noise. That is where Python’s built-in logging module comes in, and when you combine it with rotating file handlers, you get a production-grade logging system that manages itself.
The good news is that Python ships with everything you need. The logging module is part of the standard library, so there is nothing to install. It supports multiple log levels (DEBUG through CRITICAL), custom formatting, and a variety of handlers that control where your logs go — console, files, network sockets, email, and more. The RotatingFileHandler and TimedRotatingFileHandler are especially useful because they automatically manage log file sizes and rotation, preventing your disk from filling up.
In this article we will set up a complete logging system from scratch. We will start with a quick working example, then explain why logging beats print(), walk through log levels and formatting, set up size-based rotation with RotatingFileHandler, time-based rotation with TimedRotatingFileHandler, combine multiple handlers for simultaneous console and file logging, add structured JSON logging, and finish with a real-life project — a Production Application Logger class you can drop into any project.
Python Logging With Rotation: Quick Example
Here is a minimal setup that logs messages to both the console and a rotating file. The file automatically rolls over when it hits 1 MB, keeping the last 3 backups.
# quick_logging.py
import logging
from logging.handlers import RotatingFileHandler
# Create logger
logger = logging.getLogger("myapp")
logger.setLevel(logging.DEBUG)
# Console handler
console = logging.StreamHandler()
console.setLevel(logging.INFO)
# Rotating file handler (1 MB max, keep 3 backups)
file_handler = RotatingFileHandler("app.log", maxBytes=1_000_000, backupCount=3)
file_handler.setLevel(logging.DEBUG)
# Formatter
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
console.setFormatter(formatter)
file_handler.setFormatter(formatter)
# Add handlers
logger.addHandler(console)
logger.addHandler(file_handler)
# Test it
logger.debug("This goes to the file only")
logger.info("This goes to both console and file")
logger.warning("Something might be wrong")
logger.error("Something is definitely wrong")
Output (console):
2026-03-13 14:30:00,123 - myapp - INFO - This goes to both console and file
2026-03-13 14:30:00,124 - myapp - WARNING - Something might be wrong
2026-03-13 14:30:00,124 - myapp - ERROR - Something is definitely wrong
Notice that the DEBUG message only appears in the file, not the console — because we set the console handler to INFO level. The file handler captures everything from DEBUG up. When app.log reaches 1 MB, it automatically renames to app.log.1, creates a fresh app.log, and deletes the oldest backup beyond 3 files. Zero maintenance required.
Want to go deeper? Below we cover why logging beats print, all five log levels, custom formatting, both types of rotation handlers, and a production-ready logger class you can reuse in any project.
Why Not Just Use print()?
The print() function is great for quick debugging, but it falls apart in any serious application. Understanding its limitations helps explain why the logging module exists and why every production codebase uses it.
Here is a side-by-side comparison of what you get with each approach:
Feature
print()
logging
Timestamps
Manual (you add them yourself)
Automatic with formatters
Severity levels
None
DEBUG, INFO, WARNING, ERROR, CRITICAL
Output destination
Console only (stdout)
Console, files, email, network, etc.
File rotation
Not possible
Built-in with handlers
Easy to disable
Must delete or comment out
Change one level setting
Thread safety
Not guaranteed
Built-in thread safety
Source tracking
Manual
Automatic (module, line number, function)
Production ready
No
Yes
The most important difference is control. With logging, you can set your production server to WARNING level (ignoring all DEBUG and INFO messages) without changing a single line of code. You can send errors to a file while keeping info messages in the console. You can add email alerts for CRITICAL failures. None of this is possible with print().
Basic Logging Setup
The simplest way to start logging is with logging.basicConfig(), which configures the root logger with a single function call. This is fine for scripts and small programs, though for larger applications you will want the more flexible approach we show later.
# basic_setup.py
import logging
# Configure the root logger
logging.basicConfig(
level=logging.DEBUG,
format="%(asctime)s - %(levelname)s - %(message)s",
datefmt="%Y-%m-%d %H:%M:%S"
)
# These all use the root logger
logging.debug("Detailed information for diagnosing problems")
logging.info("Confirmation that things are working as expected")
logging.warning("Something unexpected happened, but the program still works")
logging.error("A more serious problem - something failed")
logging.critical("The program may not be able to continue")
Output:
2026-03-13 14:30:00 - DEBUG - Detailed information for diagnosing problems
2026-03-13 14:30:00 - INFO - Confirmation that things are working as expected
2026-03-13 14:30:00 - WARNING - Something unexpected happened, but the program still works
2026-03-13 14:30:00 - ERROR - A more serious problem - something failed
2026-03-13 14:30:00 - CRITICAL - The program may not be able to continue
The basicConfig() function is a convenience wrapper. The level parameter sets the minimum severity to capture — anything below this level is silently ignored. The format string controls what each log line looks like, using placeholder variables like %(asctime)s for the timestamp and %(levelname)s for the severity. The datefmt parameter controls the timestamp format.
print() is a debugging crutch. logging.getLogger() is a debugging exoskeleton.
Understanding Log Levels
Log levels are the filtering mechanism that makes logging so powerful. Each level has a numeric value, and the logger only processes messages at or above the configured threshold. Understanding when to use each level is critical for writing logs that are actually useful when you need them.
# log_levels.py
import logging
logging.basicConfig(level=logging.DEBUG, format="%(levelname)-8s %(message)s")
# DEBUG (10) - Detailed diagnostic info, only useful during development
logging.debug(f"Processing user_id=42, payload_size=1024 bytes")
# INFO (20) - Routine operational messages
logging.info("Server started on port 8080")
logging.info("User 'alice' logged in successfully")
# WARNING (30) - Something unexpected but not broken
logging.warning("Disk usage at 85% - consider cleanup")
logging.warning("API response took 4.2s (threshold: 3.0s)")
# ERROR (40) - Something failed, but the app continues
logging.error("Failed to connect to database: Connection refused")
logging.error("Payment processing failed for order #1234")
# CRITICAL (50) - The app may crash or is in an unrecoverable state
logging.critical("Out of memory - shutting down worker process")
logging.critical("Security breach detected: unauthorized admin access")
Output:
DEBUG Processing user_id=42, payload_size=1024 bytes
INFO Server started on port 8080
INFO User 'alice' logged in successfully
WARNING Disk usage at 85% - consider cleanup
WARNING API response took 4.2s (threshold: 3.0s)
ERROR Failed to connect to database: Connection refused
ERROR Payment processing failed for order #1234
CRITICAL Out of memory - shutting down worker process
CRITICAL Security breach detected: unauthorized admin access
A common production strategy is to log DEBUG and INFO to files (for post-mortem analysis) while only showing WARNING and above in the console (to avoid drowning operators in noise). The %-8s in the format string left-aligns the level name in an 8-character field, making the output easier to scan visually.
RotatingFileHandler for Size-Based Rotation
The RotatingFileHandler automatically creates a new log file when the current one reaches a specified size. Old log files are renamed with numeric suffixes (.1, .2, etc.) and the oldest files beyond your backup count are deleted automatically. This prevents log files from growing unbounded and filling up your disk.
# rotating_handler.py
import logging
from logging.handlers import RotatingFileHandler
logger = logging.getLogger("rotating_demo")
logger.setLevel(logging.DEBUG)
# Create rotating handler: 500 KB max, keep 5 backups
handler = RotatingFileHandler(
filename="demo.log",
maxBytes=500_000, # 500 KB per file
backupCount=5, # Keep demo.log.1 through demo.log.5
encoding="utf-8" # Always specify encoding
)
formatter = logging.Formatter(
"%(asctime)s | %(levelname)-8s | %(name)s | %(message)s",
datefmt="%Y-%m-%d %H:%M:%S"
)
handler.setFormatter(formatter)
logger.addHandler(handler)
# Simulate logging activity
for i in range(1000):
logger.info(f"Processing record {i}: status=OK, duration=0.{i % 100:02d}s")
if i % 100 == 0:
logger.warning(f"Batch {i // 100} checkpoint reached")
logger.info("Processing complete")
Output (in demo.log):
2026-03-13 14:30:00 | INFO | rotating_demo | Processing record 0: status=OK, duration=0.00s
2026-03-13 14:30:00 | WARNING | rotating_demo | Batch 0 checkpoint reached
2026-03-13 14:30:00 | INFO | rotating_demo | Processing record 1: status=OK, duration=0.01s
...
After running this, you will see files like demo.log, demo.log.1, demo.log.2, and so on in your directory. The demo.log file is always the current, active log. When it hits 500 KB, the handler renames it to demo.log.1 (pushing the previous .1 to .2, etc.) and starts writing to a fresh demo.log. Files beyond demo.log.5 are automatically deleted. The total maximum disk usage is maxBytes x (backupCount + 1) — in this case, about 3 MB.
TimedRotatingFileHandler for Time-Based Rotation
Sometimes you want logs rotated by time rather than size — a new file every day, every hour, or every week. The TimedRotatingFileHandler handles this automatically. This is especially useful for daily log files that you can easily search by date.
# timed_handler.py
import logging
from logging.handlers import TimedRotatingFileHandler
logger = logging.getLogger("timed_demo")
logger.setLevel(logging.DEBUG)
# Create timed rotating handler: rotate at midnight, keep 30 days
handler = TimedRotatingFileHandler(
filename="daily.log",
when="midnight", # Rotate at midnight
interval=1, # Every 1 day
backupCount=30, # Keep 30 days of logs
encoding="utf-8",
utc=False # Use local time, not UTC
)
# Customize the backup file suffix to include the date
handler.suffix = "%Y-%m-%d"
formatter = logging.Formatter(
"%(asctime)s | %(levelname)-8s | %(funcName)s | %(message)s"
)
handler.setFormatter(formatter)
logger.addHandler(handler)
# Example usage
def process_order(order_id, amount):
logger.info(f"Processing order #{order_id} for ${amount:.2f}")
if amount > 1000:
logger.warning(f"High-value order #{order_id}: ${amount:.2f}")
logger.debug(f"Order #{order_id} details sent to payment gateway")
process_order(1001, 49.99)
process_order(1002, 1500.00)
process_order(1003, 25.50)
Output (in daily.log):
2026-03-13 14:30:00,123 | INFO | process_order | Processing order #1001 for $49.99
2026-03-13 14:30:00,123 | DEBUG | process_order | Order #1001 details sent to payment gateway
2026-03-13 14:30:00,124 | INFO | process_order | Processing order #1002 for $1500.00
2026-03-13 14:30:00,124 | WARNING | process_order | High-value order #1002: $1500.00
2026-03-13 14:30:00,124 | DEBUG | process_order | Order #1002 details sent to payment gateway
2026-03-13 14:30:00,125 | INFO | process_order | Processing order #1003 for $25.50
2026-03-13 14:30:00,125 | DEBUG | process_order | Order #1003 details sent to payment gateway
At midnight, the handler renames daily.log to daily.log.2026-03-13 and creates a fresh daily.log. The when parameter accepts several values: "S" for seconds, "M" for minutes, "H" for hours, "D" for days, "midnight" for midnight rotation, and "W0" through "W6" for specific weekdays (Monday through Sunday). The %(funcName)s formatter variable automatically includes the function name where the log call was made — extremely useful for tracing issues across a large codebase.
print(‘debug here’) is not logging. It never was. It never will be.
Multiple Handlers: Console AND File Logging
In practice, you almost always want both console output (for real-time monitoring) and file output (for historical records). Python’s logging module makes this easy — a single logger can have multiple handlers, each with its own level and format.
# multi_handler.py
import logging
from logging.handlers import RotatingFileHandler
def setup_logger(name, log_file="app.log", console_level=logging.INFO, file_level=logging.DEBUG):
"""Create a logger with both console and rotating file handlers."""
logger = logging.getLogger(name)
logger.setLevel(logging.DEBUG) # Capture everything; handlers filter
# Console handler - concise format, higher threshold
console_handler = logging.StreamHandler()
console_handler.setLevel(console_level)
console_fmt = logging.Formatter("%(levelname)-8s %(message)s")
console_handler.setFormatter(console_fmt)
# File handler - detailed format, captures everything
file_handler = RotatingFileHandler(
log_file, maxBytes=5_000_000, backupCount=5, encoding="utf-8"
)
file_handler.setLevel(file_level)
file_fmt = logging.Formatter(
"%(asctime)s | %(levelname)-8s | %(name)s:%(funcName)s:%(lineno)d | %(message)s"
)
file_handler.setFormatter(file_fmt)
logger.addHandler(console_handler)
logger.addHandler(file_handler)
return logger
# Usage
logger = setup_logger("myapp")
def connect_to_database(host, port):
logger.debug(f"Attempting connection to {host}:{port}")
logger.info(f"Connected to database at {host}:{port}")
return True
def fetch_users():
logger.debug("Executing SELECT * FROM users")
logger.info("Fetched 42 users from database")
logger.warning("Query took 2.3 seconds (threshold: 1.0s)")
connect_to_database("localhost", 5432)
fetch_users()
Output (console — concise):
INFO Connected to database at localhost:5432
INFO Fetched 42 users from database
WARNING Query took 2.3 seconds (threshold: 1.0s)
Output (app.log — detailed):
2026-03-13 14:30:00,123 | DEBUG | myapp:connect_to_database:30 | Attempting connection to localhost:5432
2026-03-13 14:30:00,124 | INFO | myapp:connect_to_database:31 | Connected to database at localhost:5432
2026-03-13 14:30:00,125 | DEBUG | myapp:fetch_users:35 | Executing SELECT * FROM users
2026-03-13 14:30:00,125 | INFO | myapp:fetch_users:36 | Fetched 42 users from database
2026-03-13 14:30:00,126 | WARNING | myapp:fetch_users:37 | Query took 2.3 seconds (threshold: 1.0s)
The key insight is that the logger level must be set to the lowest level you want to capture (DEBUG), and each handler then filters independently. The console handler only shows INFO and above, keeping the terminal clean. The file handler captures everything including DEBUG messages, giving you full diagnostic detail when you need to investigate an issue after the fact. The file format includes the function name and line number (%(funcName)s:%(lineno)d), which makes tracing bugs significantly faster.
Real-Life Example: Production Application Logger
Log rotation: because even your log files deserve a fresh start every now and then.
Let us build a reusable AppLogger class that you can drop into any project. It combines everything we have covered — console and file handlers, rotating files, structured formatting, and exception logging — into a clean, configurable package.
# app_logger.py
import os
import logging
import traceback
from logging.handlers import RotatingFileHandler, TimedRotatingFileHandler
from datetime import datetime
class AppLogger:
"""Production-ready logger with console and rotating file output."""
def __init__(self, name, log_dir="logs", console_level="INFO",
file_level="DEBUG", max_bytes=10_000_000, backup_count=10):
# Create log directory if it doesn't exist
os.makedirs(log_dir, exist_ok=True)
self.logger = logging.getLogger(name)
self.logger.setLevel(logging.DEBUG)
# Prevent duplicate handlers if called multiple times
if self.logger.handlers:
return
# Console handler - human-readable, colored by level
console = logging.StreamHandler()
console.setLevel(getattr(logging, console_level.upper()))
console_fmt = logging.Formatter(
"%(asctime)s | %(levelname)-8s | %(message)s",
datefmt="%H:%M:%S"
)
console.setFormatter(console_fmt)
# Rotating file handler - detailed, size-based rotation
log_path = os.path.join(log_dir, f"{name}.log")
file_handler = RotatingFileHandler(
log_path, maxBytes=max_bytes, backupCount=backup_count, encoding="utf-8"
)
file_handler.setLevel(getattr(logging, file_level.upper()))
file_fmt = logging.Formatter(
"%(asctime)s | %(levelname)-8s | %(name)s:%(funcName)s:%(lineno)d | %(message)s"
)
file_handler.setFormatter(file_fmt)
# Error-only file handler - quick access to errors
error_path = os.path.join(log_dir, f"{name}_errors.log")
error_handler = RotatingFileHandler(
error_path, maxBytes=max_bytes, backupCount=5, encoding="utf-8"
)
error_handler.setLevel(logging.ERROR)
error_handler.setFormatter(file_fmt)
self.logger.addHandler(console)
self.logger.addHandler(file_handler)
self.logger.addHandler(error_handler)
def debug(self, msg): self.logger.debug(msg)
def info(self, msg): self.logger.info(msg)
def warning(self, msg): self.logger.warning(msg)
def error(self, msg): self.logger.error(msg)
def critical(self, msg): self.logger.critical(msg)
def exception(self, msg):
"""Log an error with full traceback."""
self.logger.error(f"{msg}\n{traceback.format_exc()}")
# Demo usage
if __name__ == "__main__":
log = AppLogger("myapp")
log.info("Application started")
log.debug("Loading configuration from config.json")
log.info("Database connection established")
# Simulate processing
for i in range(5):
log.info(f"Processing batch {i + 1} of 5")
if i == 2:
log.warning("Batch 3 had 12 skipped records")
# Simulate an error with traceback
try:
result = 1 / 0
except ZeroDivisionError:
log.exception("Math operation failed")
log.info("Application shutdown complete")
Output (console):
14:30:00 | INFO | Application started
14:30:00 | INFO | Database connection established
14:30:00 | INFO | Processing batch 1 of 5
14:30:00 | INFO | Processing batch 2 of 5
14:30:00 | INFO | Processing batch 3 of 5
14:30:00 | WARNING | Batch 3 had 12 skipped records
14:30:00 | INFO | Processing batch 4 of 5
14:30:00 | INFO | Processing batch 5 of 5
14:30:00 | ERROR | Math operation failed
Traceback (most recent call last):
File "app_logger.py", line 74, in <module>
result = 1 / 0
ZeroDivisionError: division by zero
14:30:00 | INFO | Application shutdown complete
This logger class gives you three outputs: a clean console for real-time monitoring, a detailed log file for everything, and a separate error-only file for quick problem diagnosis. The exception() method automatically captures the full Python traceback, which is invaluable for debugging production errors. The duplicate handler check (if self.logger.handlers) prevents the common bug where creating multiple instances of the same logger adds duplicate handlers, causing each message to appear multiple times.
You could extend this logger with JSON-formatted output for log aggregation tools like ELK Stack, email alerts for CRITICAL messages using SMTPHandler, or Slack notifications via a custom handler.
Frequently Asked Questions
When should I use basicConfig vs manual handler setup?
Use basicConfig() for quick scripts, one-file programs, and learning. It is a single function call that handles the most common case. Switch to manual handler setup (creating Logger, Handler, and Formatter objects explicitly) when you need multiple handlers with different levels, custom formatting per output, or when building a library or larger application. The manual approach gives you complete control.
How do I silence noisy logs from third-party libraries?
Third-party libraries like requests, urllib3, and boto3 often produce verbose DEBUG logs. Set their logger level to WARNING: logging.getLogger("urllib3").setLevel(logging.WARNING). This silences their DEBUG and INFO messages without affecting your own logging. You can also use logging.getLogger("urllib3").propagate = False to completely stop their messages from reaching your root logger.
What format variables are available in log formatters?
The most useful ones are: %(asctime)s for timestamp, %(levelname)s for level, %(name)s for logger name, %(funcName)s for function name, %(lineno)d for line number, %(filename)s for file name, %(message)s for the actual message, and %(process)d for process ID. You can combine them in any order. For production, include at minimum the timestamp, level, and message.
How do I log in JSON format for tools like ELK Stack?
Install the python-json-logger package (pip install python-json-logger) and use its JsonFormatter. Replace your standard formatter with JsonFormatter("%(asctime)s %(levelname)s %(name)s %(message)s"). This outputs each log line as a JSON object with those fields as keys, which tools like Elasticsearch, Splunk, and CloudWatch can parse automatically without custom regex patterns.
Is Python logging thread-safe?
Yes. The logging module uses locks internally to ensure that log messages from different threads do not interleave or corrupt each other. Each handler has its own lock. This means you can safely use the same logger from multiple threads without any additional synchronization. For multi-process applications, however, you need to be more careful — RotatingFileHandler can have issues when multiple processes write to the same file. Use QueueHandler with a separate logging process in that case.
Does logging slow down my application?
Logging has minimal overhead when configured correctly. The biggest performance tip is to use lazy formatting: write logger.debug("Processing %d items", count) instead of logger.debug(f"Processing {count} items"). With the first form, the string formatting only happens if DEBUG level is enabled. With the f-string, the formatting happens every time regardless of level. For most applications, logging overhead is negligible compared to I/O, network calls, or database queries.
Conclusion
You now have a complete understanding of Python’s logging system. We covered why print() falls short in production, how to use basicConfig() for quick setup, the five log levels and when to use each one, custom formatting with Formatter, size-based rotation with RotatingFileHandler, time-based rotation with TimedRotatingFileHandler, combining multiple handlers for simultaneous console and file output, and a reusable AppLogger class for production applications.
The AppLogger class from the real-life example is ready to use in your own projects. Try extending it with JSON output for log aggregation, email alerts for critical errors, or integration with monitoring tools like Sentry. Proper logging is one of those investments that pays for itself the first time you need to debug a production issue at 2 AM.
Telegram bots are everywhere — they moderate group chats, send weather alerts, track crypto prices, manage to-do lists, and even run entire customer support workflows. If you have ever wanted to build your own bot that responds to commands, sends images, presents clickable buttons, or holds multi-step conversations, Python and the python-telegram-bot library make it surprisingly straightforward.
The python-telegram-bot library is the most popular Python wrapper for the Telegram Bot API. It handles all the low-level HTTP communication, provides clean handler classes for different message types, and supports both simple command-response patterns and complex multi-turn conversations. You will need Python 3.9 or later and a free Telegram account to follow along. Installation is a single pip install command, and creating a bot token through Telegram’s BotFather takes about two minutes.
In this article we will walk through every step of building a Telegram bot from scratch. We will start by creating a bot token with BotFather, then install the library and build a basic echo bot. From there we will cover command handlers, sending text and media, inline keyboard buttons, conversation handlers for multi-step flows, error handling, and finally a complete real-life project — a Personal Expense Tracker bot that stores expenses in a JSON file and can summarize your spending by category. By the end, you will have the skills to build and deploy any Telegram bot you can imagine.
Building a Telegram Bot in Python: Quick Example
Before we dive into the details, here is a minimal working bot that responds to the /start command and echoes back any text message the user sends. This gives you a working bot in under 15 lines of code.
# quick_bot.py
from telegram import Update
from telegram.ext import ApplicationBuilder, CommandHandler, MessageHandler, filters, ContextTypes
async def start(update: Update, context: ContextTypes.DEFAULT_TYPE):
await update.message.reply_text("Hello! I am your Python bot. Send me any message and I will echo it back.")
async def echo(update: Update, context: ContextTypes.DEFAULT_TYPE):
await update.message.reply_text(f"You said: {update.message.text}")
app = ApplicationBuilder().token("YOUR_BOT_TOKEN").build()
app.add_handler(CommandHandler("start", start))
app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, echo))
app.run_polling()
Output (in Telegram chat):
User: /start
Bot: Hello! I am your Python bot. Send me any message and I will echo it back.
User: Python is awesome!
Bot: You said: Python is awesome!
This tiny script creates a fully functional Telegram bot. The CommandHandler listens for the /start command and calls our start function. The MessageHandler with filters.TEXT catches every regular text message and echoes it back. The run_polling() method keeps the bot running, continuously checking Telegram’s servers for new messages.
Want to go deeper? Below we cover how to get your bot token, handle multiple commands, send photos and documents, create interactive button menus, build multi-step conversations, and put it all together in a real expense tracker project.
What Is a Telegram Bot and Why Build One?
A Telegram bot is a special account on Telegram that is controlled by software rather than a human. When someone sends a message to your bot, Telegram forwards that message to your server (or your script running locally via polling), your code processes it, and sends a response back through the Telegram Bot API. The user experience feels like chatting with a person, but behind the scenes it is your Python code making the decisions.
Bots are useful for automation, notifications, data collection, and interactive tools. Unlike building a full web application with a frontend, a Telegram bot gives you a polished chat interface for free — Telegram handles the UI, push notifications, media rendering, and cross-platform support. You just write the logic.
The python-telegram-bot library abstracts the raw HTTP API into a clean, Pythonic interface. Here is how its key concepts map to what you will be building:
Concept
What It Does
Example Use
Application
The main bot engine that manages handlers and polling
Starting and running the bot
CommandHandler
Responds to slash commands like /start, /help
Greeting users, showing menus
MessageHandler
Responds to regular messages (text, photos, etc.)
Echoing text, processing uploads
CallbackQueryHandler
Responds to inline button presses
Interactive menus, confirmations
ConversationHandler
Manages multi-step conversation flows
Forms, surveys, step-by-step input
filters
Narrows which messages trigger a handler
Only text, only photos, only from groups
Now that you understand the building blocks, let us set up BotFather and get a token so we can start coding.
Setting Up BotFather and Getting Your Bot Token
Every Telegram bot needs a unique token — a long string that authenticates your code with the Telegram API. You get this token by talking to BotFather, which is itself a Telegram bot that manages bot creation. This is a one-time setup that takes about two minutes.
Open Telegram and search for @BotFather. Start a conversation and send the /newbot command. BotFather will ask you for two things: a display name for your bot (anything you like, such as “My Python Bot”) and a username that must end in bot (such as my_python_tutorial_bot). Once you provide both, BotFather responds with your bot token.
# Your token will look something like this (this is a fake example):
# 7891234567:AAF_example-token-string-here-abc123
# IMPORTANT: Never share your real token publicly.
# Store it in an environment variable:
# In your terminal:
# export TELEGRAM_BOT_TOKEN="7891234567:AAF_your-real-token-here"
# In your Python code:
import os
BOT_TOKEN = os.environ.get("TELEGRAM_BOT_TOKEN")
The token is essentially the password to your bot. Anyone who has it can control your bot, so never commit it to a public repository or paste it in shared documents. The safest approach is to store it in an environment variable and read it with os.environ.get() as shown above. For development, you can also use a .env file with the python-dotenv library.
Treat your bot token like your Netflix password. Actually, treat it better.
Installing python-telegram-bot
With your token ready, the next step is installing the library. The python-telegram-bot library is available on PyPI and supports Python 3.9 and above. A single pip command installs everything you need.
# install_check.py
# Install from terminal: pip install python-telegram-bot
# Verify the installation
import telegram
print(f"python-telegram-bot version: {telegram.__version__}")
Output:
python-telegram-bot version: 21.10
If the import works and prints a version number, you are ready to go. The library includes everything — HTTP handling, handler classes, inline keyboards, and conversation management. No additional packages are required for the tutorials in this article.
Handling Commands with CommandHandler
Commands are messages that start with a forward slash, like /start, /help, or /weather. They are the primary way users interact with bots. The CommandHandler class lets you register a Python function for each command your bot supports.
Let us build a bot that responds to three commands: /start for a welcome message, /help for a list of available commands, and /about for information about the bot.
# command_bot.py
import os
from telegram import Update
from telegram.ext import ApplicationBuilder, CommandHandler, ContextTypes
BOT_TOKEN = os.environ.get("TELEGRAM_BOT_TOKEN")
async def start(update: Update, context: ContextTypes.DEFAULT_TYPE):
user_name = update.effective_user.first_name
await update.message.reply_text(
f"Welcome, {user_name}! I am a demo bot.\n"
f"Use /help to see what I can do."
)
async def help_command(update: Update, context: ContextTypes.DEFAULT_TYPE):
help_text = (
"Here are my commands:\n\n"
"/start - Start the bot\n"
"/help - Show this help message\n"
"/about - Learn about this bot"
)
await update.message.reply_text(help_text)
async def about(update: Update, context: ContextTypes.DEFAULT_TYPE):
await update.message.reply_text(
"I was built with Python and python-telegram-bot v21.\n"
"Tutorial: pythonhowtoprogram.com"
)
app = ApplicationBuilder().token(BOT_TOKEN).build()
app.add_handler(CommandHandler("start", start))
app.add_handler(CommandHandler("help", help_command))
app.add_handler(CommandHandler("about", about))
app.run_polling()
Output (in Telegram chat):
User: /start
Bot: Welcome, Alice! I am a demo bot.
Use /help to see what I can do.
User: /help
Bot: Here are my commands:
/start - Start the bot
/help - Show this help message
/about - Learn about this bot
Each CommandHandler takes two arguments: the command name (without the slash) and the async function to call. The update object contains everything about the incoming message — who sent it, the chat ID, the message text, and more. The context object provides access to bot-level data and the bot instance itself. Notice how we access the user’s first name with update.effective_user.first_name to personalize the greeting.
Sending Messages and Media
Text replies are just the beginning. Telegram bots can send photos, documents, audio files, locations, and formatted messages. The python-telegram-bot library provides dedicated methods for each media type, all accessible through the bot instance or the message reply methods.
Here is a bot that demonstrates sending a photo from a URL, a document, and a message with HTML formatting:
# media_bot.py
import os
from telegram import Update
from telegram.ext import ApplicationBuilder, CommandHandler, ContextTypes
BOT_TOKEN = os.environ.get("TELEGRAM_BOT_TOKEN")
async def send_photo(update: Update, context: ContextTypes.DEFAULT_TYPE):
# Send a photo from a public URL
photo_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/c/c3/Python-logo-notext.svg/800px-Python-logo-notext.svg.png"
await update.message.reply_photo(
photo=photo_url,
caption="The Python logo - beautiful, isn't it?"
)
async def send_formatted(update: Update, context: ContextTypes.DEFAULT_TYPE):
# Send a message with HTML formatting
formatted_text = (
"<b>Bold text</b>\n"
"<i>Italic text</i>\n"
"<code>inline_code()</code>\n\n"
"<pre># Code block\nprint('Hello from a code block!')</pre>"
)
await update.message.reply_text(formatted_text, parse_mode="HTML")
async def send_document(update: Update, context: ContextTypes.DEFAULT_TYPE):
# Create and send a text file on the fly
content = "This file was generated by your Telegram bot!"
file_bytes = content.encode("utf-8")
await update.message.reply_document(
document=file_bytes,
filename="bot_message.txt",
caption="Here is your generated document."
)
app = ApplicationBuilder().token(BOT_TOKEN).build()
app.add_handler(CommandHandler("photo", send_photo))
app.add_handler(CommandHandler("format", send_formatted))
app.add_handler(CommandHandler("doc", send_document))
app.run_polling()
Output (in Telegram chat):
User: /photo
Bot: [Sends Python logo image with caption "The Python logo - beautiful, isn't it?"]
User: /format
Bot: Bold text
Italic text
inline_code()
# Code block
print('Hello from a code block!')
User: /doc
Bot: [Sends bot_message.txt file with caption "Here is your generated document."]
The reply_photo() method accepts a URL or a file path. For formatted text, Telegram supports both HTML and Markdown — we use parse_mode="HTML" because it handles nested formatting more reliably. The reply_document() method can send any file, including ones you generate dynamically from bytes. This is incredibly useful for bots that create reports, export data, or generate files on demand.
Sending photos, docs, and audio in one bot. We call that the multimedia flex.
Using Inline Keyboard Buttons for User Interaction
One of the most powerful features in Telegram bots is inline keyboards — rows of clickable buttons that appear directly below a message. These buttons can trigger actions, navigate menus, or collect user choices without the user typing anything. They make your bot feel like a polished application rather than a text-only chat.
Here is a bot that presents a menu with inline buttons and responds when the user clicks one:
# button_bot.py
import os
from telegram import Update, InlineKeyboardButton, InlineKeyboardMarkup
from telegram.ext import ApplicationBuilder, CommandHandler, CallbackQueryHandler, ContextTypes
BOT_TOKEN = os.environ.get("TELEGRAM_BOT_TOKEN")
async def menu(update: Update, context: ContextTypes.DEFAULT_TYPE):
# Create a 2x2 grid of buttons
keyboard = [
[
InlineKeyboardButton("Python Basics", callback_data="basics"),
InlineKeyboardButton("Web Scraping", callback_data="scraping"),
],
[
InlineKeyboardButton("APIs", callback_data="apis"),
InlineKeyboardButton("Automation", callback_data="automation"),
],
]
reply_markup = InlineKeyboardMarkup(keyboard)
await update.message.reply_text("What topic interests you?", reply_markup=reply_markup)
async def button_callback(update: Update, context: ContextTypes.DEFAULT_TYPE):
query = update.callback_query
await query.answer() # Acknowledge the button press
topics = {
"basics": "Python Basics: Start with variables, loops, and functions!",
"scraping": "Web Scraping: Use BeautifulSoup and Selenium to extract data.",
"apis": "APIs: Learn requests, REST APIs, and authentication.",
"automation": "Automation: Automate files, emails, and workflows.",
}
response = topics.get(query.data, "Unknown topic selected.")
await query.edit_message_text(text=f"Great choice!\n\n{response}")
app = ApplicationBuilder().token(BOT_TOKEN).build()
app.add_handler(CommandHandler("menu", menu))
app.add_handler(CallbackQueryHandler(button_callback))
app.run_polling()
Output (in Telegram chat):
User: /menu
Bot: What topic interests you?
[Python Basics] [Web Scraping]
[APIs] [Automation]
User: [clicks "Web Scraping"]
Bot: Great choice!
Web Scraping: Use BeautifulSoup and Selenium to extract data.
Each InlineKeyboardButton has a visible label and a callback_data string that gets sent to your bot when the user clicks it. The CallbackQueryHandler catches these clicks and routes them to your callback function. Always call query.answer() first to remove the loading spinner from the button — without this, Telegram shows a progress indicator that never goes away. Then use query.edit_message_text() to update the original message in place, which gives a smooth, app-like experience.
Building Multi-Step Conversations
Simple command-response bots are useful, but many real-world bots need to collect information across multiple messages — like a form where you ask for the user’s name, then their email, then their preference. The ConversationHandler manages these multi-step flows by tracking which “state” each user is in and routing their messages to the appropriate handler function.
Here is a bot that collects a user’s name, favorite programming language, and experience level through a guided conversation:
# conversation_bot.py
import os
from telegram import Update, ReplyKeyboardMarkup, ReplyKeyboardRemove
from telegram.ext import (
ApplicationBuilder, CommandHandler, MessageHandler,
ConversationHandler, ContextTypes, filters
)
BOT_TOKEN = os.environ.get("TELEGRAM_BOT_TOKEN")
# Define conversation states
NAME, LANGUAGE, EXPERIENCE = range(3)
async def start_survey(update: Update, context: ContextTypes.DEFAULT_TYPE):
await update.message.reply_text(
"Welcome to the developer survey! What is your name?",
reply_markup=ReplyKeyboardRemove() # Remove any previous keyboard
)
return NAME # Move to the NAME state
async def get_name(update: Update, context: ContextTypes.DEFAULT_TYPE):
context.user_data["name"] = update.message.text
languages = [["Python", "JavaScript"], ["Go", "Rust"]]
await update.message.reply_text(
f"Nice to meet you, {update.message.text}! "
f"What is your favorite programming language?",
reply_markup=ReplyKeyboardMarkup(languages, one_time_keyboard=True)
)
return LANGUAGE # Move to the LANGUAGE state
async def get_language(update: Update, context: ContextTypes.DEFAULT_TYPE):
context.user_data["language"] = update.message.text
levels = [["Beginner", "Intermediate", "Advanced"]]
await update.message.reply_text(
f"Great choice! What is your experience level?",
reply_markup=ReplyKeyboardMarkup(levels, one_time_keyboard=True)
)
return EXPERIENCE # Move to the EXPERIENCE state
async def get_experience(update: Update, context: ContextTypes.DEFAULT_TYPE):
context.user_data["experience"] = update.message.text
# Summarize the collected data
data = context.user_data
summary = (
f"Survey complete! Here is your profile:\n\n"
f"Name: {data['name']}\n"
f"Language: {data['language']}\n"
f"Experience: {data['experience']}\n\n"
f"Thanks for participating!"
)
await update.message.reply_text(summary, reply_markup=ReplyKeyboardRemove())
return ConversationHandler.END # End the conversation
async def cancel(update: Update, context: ContextTypes.DEFAULT_TYPE):
await update.message.reply_text("Survey cancelled.", reply_markup=ReplyKeyboardRemove())
return ConversationHandler.END
# Build the conversation handler
survey_handler = ConversationHandler(
entry_points=[CommandHandler("survey", start_survey)],
states={
NAME: [MessageHandler(filters.TEXT & ~filters.COMMAND, get_name)],
LANGUAGE: [MessageHandler(filters.TEXT & ~filters.COMMAND, get_language)],
EXPERIENCE: [MessageHandler(filters.TEXT & ~filters.COMMAND, get_experience)],
},
fallbacks=[CommandHandler("cancel", cancel)],
)
app = ApplicationBuilder().token(BOT_TOKEN).build()
app.add_handler(survey_handler)
app.run_polling()
Output (in Telegram chat):
User: /survey
Bot: Welcome to the developer survey! What is your name?
User: Alice
Bot: Nice to meet you, Alice! What is your favorite programming language?
[Python] [JavaScript]
[Go] [Rust]
User: Python
Bot: Great choice! What is your experience level?
[Beginner] [Intermediate] [Advanced]
User: Intermediate
Bot: Survey complete! Here is your profile:
Name: Alice
Language: Python
Experience: Intermediate
Thanks for participating!
The ConversationHandler is the most complex handler in the library, but the pattern is straightforward once you understand it. You define numbered states (we used range(3) to create NAME=0, LANGUAGE=1, EXPERIENCE=2). Each handler function returns the next state to transition to. The context.user_data dictionary persists across the conversation, letting you accumulate responses step by step. The ReplyKeyboardMarkup shows a custom keyboard with predefined choices, which reduces typos and makes the experience smoother. Always include a /cancel fallback so users can exit the conversation at any point.
ConversationHandler tracks state so your bot remembers where users left off. Unlike your memory at 3am.
Error Handling and Logging
Production bots need to handle errors gracefully. Network timeouts, invalid user input, and API rate limits can all cause exceptions. The python-telegram-bot library provides a built-in error handler that catches any uncaught exception from your handlers, so your bot keeps running even when something goes wrong.
# error_handling_bot.py
import os
import logging
from telegram import Update
from telegram.ext import ApplicationBuilder, CommandHandler, ContextTypes
# Set up logging to see what is happening
logging.basicConfig(
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
level=logging.INFO
)
logger = logging.getLogger(__name__)
BOT_TOKEN = os.environ.get("TELEGRAM_BOT_TOKEN")
async def risky_command(update: Update, context: ContextTypes.DEFAULT_TYPE):
# This will raise an error if the user doesn't provide a number
user_input = context.args[0] if context.args else None
if user_input is None:
await update.message.reply_text("Please provide a number: /divide 10")
return
result = 100 / int(user_input) # Could raise ValueError or ZeroDivisionError
await update.message.reply_text(f"100 / {user_input} = {result}")
async def error_handler(update: object, context: ContextTypes.DEFAULT_TYPE):
logger.error(f"Exception while handling an update: {context.error}")
if update and hasattr(update, "message") and update.message:
await update.message.reply_text(
"Something went wrong. Please try again with valid input."
)
app = ApplicationBuilder().token(BOT_TOKEN).build()
app.add_handler(CommandHandler("divide", risky_command))
app.add_error_handler(error_handler)
app.run_polling()
Output (in Telegram chat):
User: /divide 5
Bot: 100 / 5 = 20.0
User: /divide 0
Bot: Something went wrong. Please try again with valid input.
User: /divide abc
Bot: Something went wrong. Please try again with valid input.
The add_error_handler() method registers a function that catches any unhandled exception from any handler in your bot. The context.error attribute contains the actual exception. We log it for debugging and send a friendly message to the user. The logging module is configured at the top to output timestamped messages — this is essential for debugging issues in production. In the real world, you would also want to log the full traceback and possibly send error notifications to yourself via a separate Telegram message.
Real-Life Example: Personal Expense Tracker Bot
A whole expense tracker in under 60 lines. Your finance app is sweating right now.
Now let us put everything together into a practical project. This expense tracker bot lets users add expenses with a category and amount, view their spending by category, and clear their data. It stores everything in a JSON file so expenses persist between bot restarts. This project uses command handlers, formatted messages, context storage, and file I/O — all the concepts we covered in this article.
# expense_tracker_bot.py
import os
import json
from datetime import datetime
from telegram import Update
from telegram.ext import ApplicationBuilder, CommandHandler, ContextTypes
BOT_TOKEN = os.environ.get("TELEGRAM_BOT_TOKEN")
DATA_FILE = "expenses.json"
def load_expenses():
"""Load expenses from JSON file, return empty dict if file missing."""
if os.path.exists(DATA_FILE):
with open(DATA_FILE, "r") as f:
return json.load(f)
return {}
def save_expenses(data):
"""Save expenses dictionary to JSON file."""
with open(DATA_FILE, "w") as f:
json.dump(data, f, indent=2)
async def start(update: Update, context: ContextTypes.DEFAULT_TYPE):
await update.message.reply_text(
"Welcome to Expense Tracker!\n\n"
"Commands:\n"
"/add <category> <amount> - Add an expense\n"
"/summary - View spending by category\n"
"/history - View recent expenses\n"
"/clear - Clear all expenses"
)
async def add_expense(update: Update, context: ContextTypes.DEFAULT_TYPE):
if len(context.args) < 2:
await update.message.reply_text("Usage: /add food 12.50")
return
category = context.args[0].lower()
try:
amount = float(context.args[1])
except ValueError:
await update.message.reply_text("Amount must be a number. Example: /add food 12.50")
return
user_id = str(update.effective_user.id)
expenses = load_expenses()
if user_id not in expenses:
expenses[user_id] = []
expenses[user_id].append({
"category": category,
"amount": amount,
"date": datetime.now().strftime("%Y-%m-%d %H:%M")
})
save_expenses(expenses)
total = sum(e["amount"] for e in expenses[user_id] if e["category"] == category)
await update.message.reply_text(
f"Added ${amount:.2f} to {category}.\n"
f"Total in {category}: ${total:.2f}"
)
async def summary(update: Update, context: ContextTypes.DEFAULT_TYPE):
user_id = str(update.effective_user.id)
expenses = load_expenses()
user_expenses = expenses.get(user_id, [])
if not user_expenses:
await update.message.reply_text("No expenses yet. Use /add category amount.")
return
# Group by category
categories = {}
for expense in user_expenses:
cat = expense["category"]
categories[cat] = categories.get(cat, 0) + expense["amount"]
grand_total = sum(categories.values())
lines = [f" {cat}: ${amt:.2f}" for cat, amt in sorted(categories.items())]
text = f"Spending Summary:\n\n" + "\n".join(lines) + f"\n\nTotal: ${grand_total:.2f}"
await update.message.reply_text(text)
async def history(update: Update, context: ContextTypes.DEFAULT_TYPE):
user_id = str(update.effective_user.id)
expenses = load_expenses()
user_expenses = expenses.get(user_id, [])
if not user_expenses:
await update.message.reply_text("No expenses yet.")
return
recent = user_expenses[-10:] # Show last 10 expenses
lines = [f" {e['date']} | {e['category']}: ${e['amount']:.2f}" for e in recent]
text = "Recent Expenses:\n\n" + "\n".join(lines)
await update.message.reply_text(text)
async def clear(update: Update, context: ContextTypes.DEFAULT_TYPE):
user_id = str(update.effective_user.id)
expenses = load_expenses()
expenses[user_id] = []
save_expenses(expenses)
await update.message.reply_text("All expenses cleared.")
app = ApplicationBuilder().token(BOT_TOKEN).build()
app.add_handler(CommandHandler("start", start))
app.add_handler(CommandHandler("add", add_expense))
app.add_handler(CommandHandler("summary", summary))
app.add_handler(CommandHandler("history", history))
app.add_handler(CommandHandler("clear", clear))
app.run_polling()
Output (in Telegram chat):
User: /start
Bot: Welcome to Expense Tracker!
Commands:
/add <category> <amount> - Add an expense
/summary - View spending by category
/history - View recent expenses
/clear - Clear all expenses
User: /add food 12.50
Bot: Added $12.50 to food.
Total in food: $12.50
User: /add transport 35.00
Bot: Added $35.00 to transport.
Total in transport: $35.00
User: /add food 8.75
Bot: Added $8.75 to food.
Total in food: $21.25
User: /summary
Bot: Spending Summary:
food: $21.25
transport: $35.00
Total: $56.25
User: /history
Bot: Recent Expenses:
2026-03-13 14:30 | food: $12.50
2026-03-13 14:31 | transport: $35.00
2026-03-13 14:32 | food: $8.75
This project ties together almost every concept from the article. It uses CommandHandler for five different commands, context.args to parse user input, JSON file I/O for persistence, input validation with helpful error messages, and formatted text output. The expenses are stored per user (keyed by Telegram user ID), so multiple people can use the same bot without their data mixing together. You could extend this by adding inline buttons for quick category selection, a /export command that sends a CSV file, or monthly budget limits with alerts.
Frequently Asked Questions
What is the difference between polling and webhooks?
Polling means your bot continuously asks Telegram "any new messages?" in a loop — this is what run_polling() does and it works great for development and small bots. Webhooks are the opposite: you give Telegram a URL, and Telegram sends new messages directly to your server. Webhooks are more efficient for production bots because they eliminate the constant polling overhead. For most tutorials and personal projects, polling is simpler and works perfectly fine.
How do I keep my bot running 24/7?
During development, running the script on your local machine is fine, but it stops when you close the terminal. For production, deploy your bot to a cloud server. Popular free or low-cost options include Railway, Render, PythonAnywhere, and a small VPS from providers like DigitalOcean. You can also use a Raspberry Pi at home. The key is that the Python process needs to stay running continuously — tools like systemd, screen, or pm2 can manage this for you.
Can I run multiple bots from one Python script?
Yes, but it is generally better to run each bot as a separate script or process. Each Application instance manages its own polling loop and handlers, and mixing them in one script can make debugging harder. If you need bots to share data, use a shared database or file rather than trying to run them in the same process.
Does Telegram have rate limits for bots?
Yes. Telegram limits bots to about 30 messages per second overall, and 1 message per second to the same chat. If your bot sends too many messages too quickly, Telegram returns a 429 error with a retry_after value telling you how long to wait. The python-telegram-bot library handles basic rate limiting automatically, but for high-volume bots you should implement message queuing and respect the limits explicitly.
How do I make my bot work in group chats?
By default, bots in groups only see messages that start with a slash command or explicitly mention the bot. To see all messages, you need to disable "Group Privacy" mode in BotFather using the /setprivacy command. In your code, you can filter group messages using filters.ChatType.GROUP or filters.ChatType.SUPERGROUP to create group-specific behavior.
What is the best way to store data for a Telegram bot?
For simple bots with small amounts of data, JSON files work well (as we used in the expense tracker). For anything more complex, use SQLite (built into Python via the sqlite3 module) for structured data without needing an external database server. For production bots with many users, PostgreSQL or MongoDB are popular choices. The python-telegram-bot library also has built-in persistence classes that can automatically save conversation state and user data.
Conclusion
You now have a solid foundation for building Telegram bots with Python. We covered the entire workflow: setting up a bot through BotFather, installing python-telegram-bot, handling commands with CommandHandler, sending text, photos, and documents, creating interactive inline keyboard buttons with CallbackQueryHandler, building multi-step conversations with ConversationHandler, implementing error handling and logging, and tying it all together with a Personal Expense Tracker project.
The expense tracker is a great starting point for your own projects. Try extending it with inline buttons for quick category selection, a monthly budget limit with warnings, or a /export command that generates a CSV report of your spending. The patterns you learned here — handlers, filters, context data, and conversation states — apply to any bot you want to build, from a simple notification bot to a full customer service assistant.
You are building a script that pulls data from an external API — maybe it fetches weather data, processes payment transactions, or syncs records from a third-party service. It works perfectly in testing, but when you deploy it to production and it starts making hundreds of requests, everything falls apart. The API starts returning 429 Too Many Requests errors, your script crashes, and you lose data. This is the rate limit problem, and every developer who works with APIs will hit it eventually.
The good news is that handling rate limits is a solved problem. Python’s requests library combined with a few simple patterns — exponential backoff, jitter, and retry logic — can make your API calls resilient and well-behaved. For more complex scenarios, the tenacity library (pip install tenacity) provides a powerful decorator-based retry system. You will also want the requests library if you do not have it already: pip install requests.
In this article we will cover everything you need to handle API rate limits like a professional. We will start with a quick example that adds retry logic to a simple API call, then explain what rate limits are and why APIs enforce them. From there we will build retry logic from scratch using exponential backoff, learn the tenacity library for production-grade retries, handle the Retry-After header that many APIs send, implement request throttling to stay under limits proactively, and finish with a real-life project that builds a reusable API client with built-in rate limit handling. By the end, your API calls will never crash from a 429 again.
Handling API Rate Limits: Quick Example
Here is the simplest way to add retry logic to an API call. This example retries failed requests with increasing delays between attempts, which is all you need for basic rate limit handling.
# quick_example.py
import requests
import time
def fetch_with_retry(url, max_retries=3):
"""Fetch a URL with automatic retry on failure."""
for attempt in range(max_retries):
response = requests.get(url)
if response.status_code == 200:
return response.json()
elif response.status_code == 429:
wait = 2 ** attempt # Exponential backoff: 1s, 2s, 4s
print(f"Rate limited! Waiting {wait}s before retry {attempt + 1}...")
time.sleep(wait)
else:
response.raise_for_status()
raise Exception(f"Failed after {max_retries} retries")
# Test with a real API
data = fetch_with_retry("https://jsonplaceholder.typicode.com/posts/1")
print(f"Title: {data['title']}")
print(f"User ID: {data['userId']}")
Output:
Title: sunt aut facere repellat provident occaecati excepturi optio reprehenderit
User ID: 1
This simple function checks the response status code after each request. If it gets a 429, it waits progressively longer before retrying — 1 second, then 2, then 4. This is called exponential backoff and it is the foundation of all rate limit handling. The real API at jsonplaceholder.typicode.com does not rate limit us in this case, so the request succeeds on the first try, but the retry logic is ready for when it matters.
Want to go deeper? Below we explain the different retry strategies, build a production-grade solution with the tenacity library, and create a reusable API client you can drop into any project.
What Are API Rate Limits and Why Do They Exist?
A rate limit is a restriction that an API server places on how many requests a client can make within a specific time window. When you exceed that limit, the server responds with HTTP status code 429 Too Many Requests instead of processing your request. This is not an error in your code — it is the API telling you to slow down.
APIs enforce rate limits for several practical reasons. First, they protect the server from being overwhelmed by a single client making thousands of requests per second. Second, they ensure fair usage — if one user monopolizes the server, other users get slow or no responses. Third, they manage infrastructure costs — every request costs the API provider compute time, bandwidth, and money. Most API documentation clearly states the rate limits, and many include rate limit headers in their responses so you can track your usage.
Here is how different APIs typically communicate their rate limits.
Header
Meaning
Example
X-RateLimit-Limit
Maximum requests allowed per window
100
X-RateLimit-Remaining
Requests left in current window
23
X-RateLimit-Reset
When the window resets (Unix timestamp)
1710360000
Retry-After
Seconds to wait before retrying (on 429)
30
The most important header is Retry-After — when an API sends a 429 response with this header, it is telling you exactly how long to wait. Always respect this value because the API knows when your rate limit window resets. If you keep hammering the server, some APIs will escalate to longer blocks or even ban your API key. Now let us build proper retry logic.
Building Retry Logic With Exponential Backoff
Exponential backoff means waiting longer after each failed attempt. Instead of retrying immediately (which would just get rate-limited again), you wait 1 second, then 2, then 4, then 8, and so on. This gives the API server time to recover and your rate limit window time to reset. Adding random jitter (a small random delay) prevents the “thundering herd” problem where multiple clients all retry at the exact same moment.
# exponential_backoff.py
import requests
import time
import random
def fetch_with_backoff(url, max_retries=5, base_delay=1):
"""Fetch a URL with exponential backoff and jitter."""
for attempt in range(max_retries):
try:
response = requests.get(url, timeout=10)
if response.status_code == 200:
return response.json()
if response.status_code == 429:
# Check for Retry-After header first
retry_after = response.headers.get("Retry-After")
if retry_after:
wait = int(retry_after)
print(f" Server says wait {wait}s (Retry-After header)")
else:
# Exponential backoff with jitter
wait = base_delay * (2 ** attempt) + random.uniform(0, 1)
print(f" Rate limited. Backing off {wait:.1f}s (attempt {attempt + 1})")
time.sleep(wait)
continue
if response.status_code >= 500:
# Server errors are also retryable
wait = base_delay * (2 ** attempt) + random.uniform(0, 1)
print(f" Server error {response.status_code}. Retrying in {wait:.1f}s")
time.sleep(wait)
continue
# Client errors (4xx except 429) should not be retried
response.raise_for_status()
except requests.exceptions.Timeout:
wait = base_delay * (2 ** attempt)
print(f" Timeout. Retrying in {wait}s (attempt {attempt + 1})")
time.sleep(wait)
except requests.exceptions.ConnectionError:
wait = base_delay * (2 ** attempt)
print(f" Connection failed. Retrying in {wait}s (attempt {attempt + 1})")
time.sleep(wait)
raise Exception(f"Failed after {max_retries} attempts")
# Test with a real API
print("Fetching user data...")
user = fetch_with_backoff("https://jsonplaceholder.typicode.com/users/1")
print(f"Name: {user['name']}")
print(f"Email: {user['email']}")
print(f"Company: {user['company']['name']}")
Output:
Fetching user data...
Name: Leanne Graham
Email: Sincere@april.biz
Company: Romaguera-Crona
This function handles three categories of retryable failures: rate limits (429), server errors (500+), and network issues (timeouts and connection errors). The random.uniform(0, 1) adds jitter so that if you have 10 scripts running in parallel, they do not all retry at the exact same second and trigger another rate limit. Notice that we check for the Retry-After header first — if the server tells us how long to wait, we trust that over our own backoff calculation.
Retry-After: the header that tells you exactly when to try again. Most developers ignore it. Don’t.
Production-Grade Retries With tenacity
Writing retry logic from scratch works, but the tenacity library makes it dramatically cleaner. It provides a @retry decorator that handles exponential backoff, jitter, conditional retries, and more — all in a single line. Install it with pip install tenacity.
# tenacity_example.py
import requests
from tenacity import (
retry,
stop_after_attempt,
wait_exponential,
retry_if_exception_type,
before_sleep_log,
)
import logging
# Set up logging to see retry activity
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
@retry(
stop=stop_after_attempt(5), # Max 5 attempts
wait=wait_exponential(multiplier=1, max=30), # 1s, 2s, 4s, 8s... up to 30s
retry=retry_if_exception_type(requests.exceptions.RequestException),
before_sleep=before_sleep_log(logger, logging.WARNING),
)
def fetch_data(url):
"""Fetch data from an API with automatic retries."""
response = requests.get(url, timeout=10)
if response.status_code == 429:
raise requests.exceptions.RequestException(
f"Rate limited (429). Retry-After: {response.headers.get('Retry-After', 'unknown')}"
)
response.raise_for_status()
return response.json()
# Use it just like a normal function
print("Fetching posts...")
posts = fetch_data("https://jsonplaceholder.typicode.com/posts?_limit=3")
for post in posts:
print(f" [{post['id']}] {post['title'][:50]}...")
Output:
Fetching posts...
[1] sunt aut facere repellat provident occaecati exc...
[2] qui est esse...
[3] ea molestias quasi exercitationem repellat qui i...
The @retry decorator adds all the retry behavior without cluttering your function with loops and sleep calls. The stop_after_attempt(5) sets the maximum retries, wait_exponential(multiplier=1, max=30) implements exponential backoff capped at 30 seconds, and retry_if_exception_type ensures we only retry on network-related errors. The before_sleep_log callback logs each retry attempt so you can monitor what is happening in production. Your actual function stays clean — it just makes the request and raises if something goes wrong.
Custom Retry Conditions With tenacity
Sometimes you need more control over when to retry. The tenacity library lets you write custom retry conditions using callback functions. This is useful when you want to retry based on the response content, not just the HTTP status code.
# tenacity_custom.py
import requests
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_result
def is_rate_limited(response):
"""Return True if the response indicates rate limiting."""
if response is None:
return True
return response.status_code == 429 or response.status_code >= 500
@retry(
stop=stop_after_attempt(4),
wait=wait_exponential(multiplier=1, max=16),
retry=retry_if_result(is_rate_limited),
)
def make_api_call(url):
"""Make an API call, returning the response object."""
try:
response = requests.get(url, timeout=10)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
print(f" 429 received. Retry-After: {retry_after}")
return response
except requests.exceptions.RequestException as e:
print(f" Network error: {e}")
return None
# Make the call — tenacity handles retries automatically
print("Making API call...")
response = make_api_call("https://httpbin.org/get")
if response and response.status_code == 200:
data = response.json()
print(f"Success! Origin: {data.get('origin', 'unknown')}")
print(f"Headers received: {len(data.get('headers', {}))}")
else:
print("All retries exhausted")
Output:
Making API call...
Success! Origin: 203.0.113.42
Headers received: 5
The retry_if_result condition is powerful because it lets you inspect the actual response, not just catch exceptions. The is_rate_limited() function returns True for 429 responses and server errors (500+), telling tenacity to retry. For successful responses (200-399) or client errors (400-499 except 429), it returns False and the function returns normally. This gives you fine-grained control over exactly which responses trigger a retry.
Respecting the Retry-After Header
Many well-designed APIs include a Retry-After header in their 429 responses. This header tells you exactly how many seconds to wait before your next request will be accepted. Always use this value when available — it is more accurate than your own backoff calculation because the API server knows when your rate limit window actually resets.
# retry_after.py
import requests
import time
def fetch_with_retry_after(url, max_retries=5):
"""Fetch a URL, respecting the Retry-After header."""
for attempt in range(max_retries):
response = requests.get(url, timeout=10)
if response.status_code == 200:
# Log rate limit headers if present
remaining = response.headers.get("X-RateLimit-Remaining")
limit = response.headers.get("X-RateLimit-Limit")
if remaining and limit:
print(f" Rate limit: {remaining}/{limit} requests remaining")
return response.json()
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
if retry_after:
# Retry-After can be seconds or an HTTP date
try:
wait = int(retry_after)
except ValueError:
# It's a date string — parse and calculate seconds
wait = 60 # Fallback
print(f" Rate limited. Server says wait {wait}s.")
else:
wait = 2 ** attempt # Fallback to exponential backoff
print(f" Rate limited. No Retry-After. Backing off {wait}s.")
time.sleep(wait)
continue
# Non-retryable error
response.raise_for_status()
raise Exception(f"Failed after {max_retries} attempts")
# Test with httpbin (won't actually rate limit, but demonstrates the pattern)
print("Fetching data with rate limit awareness...")
data = fetch_with_retry_after("https://httpbin.org/get")
print(f"Origin: {data.get('origin', 'unknown')}")
Output:
Fetching data with rate limit awareness...
Origin: 203.0.113.42
This function checks for rate limit headers on every successful response too — not just on 429s. The X-RateLimit-Remaining header tells you how many requests you have left in the current window. If you see this number getting low, you can proactively slow down your requests before hitting the limit. This is called proactive throttling and it is much better than waiting for 429 errors to react.
Exponential backoff without jitter is just synchronized DDoS with extra steps. Add the randomness.
Proactive Request Throttling
Instead of waiting for rate limit errors and then reacting, you can throttle your requests proactively to stay under the limit. This approach is cleaner, faster, and more respectful to the API provider. The simplest way is to add a delay between requests, but a more sophisticated approach uses a token bucket or sliding window to allow bursts while maintaining an average rate.
# throttling.py
import requests
import time
class ThrottledClient:
"""An HTTP client that limits requests per second."""
def __init__(self, requests_per_second=5):
self.min_interval = 1.0 / requests_per_second
self.last_request_time = 0
def get(self, url, **kwargs):
"""Make a throttled GET request."""
# Calculate how long to wait
elapsed = time.time() - self.last_request_time
if elapsed < self.min_interval:
wait = self.min_interval - elapsed
time.sleep(wait)
self.last_request_time = time.time()
return requests.get(url, timeout=10, **kwargs)
# Demo: fetch 5 posts at a controlled rate
client = ThrottledClient(requests_per_second=2) # Max 2 requests/sec
print("Fetching posts with throttling (2 req/sec max):\n")
start = time.time()
for post_id in range(1, 6):
response = client.get(f"https://jsonplaceholder.typicode.com/posts/{post_id}")
data = response.json()
elapsed = time.time() - start
print(f" [{elapsed:5.2f}s] Post {post_id}: {data['title'][:40]}...")
total = time.time() - start
print(f"\nFetched 5 posts in {total:.2f}s (throttled to 2/sec)")
Output:
Fetching posts with throttling (2 req/sec max):
[ 0.31s] Post 1: sunt aut facere repellat provident occ...
[ 0.82s] Post 2: qui est esse...
[ 1.33s] Post 3: ea molestias quasi exercitationem repel...
[ 1.82s] Post 4: eum et est occaecati...
[ 2.35s] Post 5: nesciunt quas odio...
Fetched 5 posts in 2.35s (throttled to 2/sec)
The ThrottledClient class tracks when the last request was made and adds a delay if needed to maintain the target rate. With requests_per_second=2, it ensures at least 0.5 seconds between requests. This is much better than adding a flat time.sleep(0.5) after every request because it accounts for the actual time the request takes — if a request takes 0.3 seconds, it only sleeps for 0.2 more seconds. This maximizes throughput while staying within limits.
Using requests.Session With HTTPAdapter for Retries
The requests library has a built-in retry mechanism through the urllib3.Retry class and HTTPAdapter. This is useful when you want retry behavior on all requests made through a session without modifying each individual call.
# session_retry.py
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
def create_retry_session(
retries=3,
backoff_factor=1,
status_forcelist=(429, 500, 502, 503, 504),
):
"""Create a requests session with built-in retry logic."""
session = requests.Session()
retry_strategy = Retry(
total=retries,
backoff_factor=backoff_factor, # 1s, 2s, 4s between retries
status_forcelist=status_forcelist,
allowed_methods=["GET", "POST", "PUT", "DELETE"],
raise_on_status=False,
)
adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount("https://", adapter)
session.mount("http://", adapter)
return session
# Create a session with retry logic baked in
session = create_retry_session(retries=3, backoff_factor=1)
# Now every request through this session has automatic retries
print("Fetching with retry session...")
response = session.get("https://jsonplaceholder.typicode.com/users/1")
user = response.json()
print(f"Name: {user['name']}")
print(f"Email: {user['email']}")
# Fetch multiple resources — all have retry protection
print("\nFetching comments...")
response = session.get("https://jsonplaceholder.typicode.com/posts/1/comments")
comments = response.json()
print(f"Found {len(comments)} comments on post 1")
for comment in comments[:2]:
print(f" - {comment['name'][:40]}...")
Output:
Fetching with retry session...
Name: Leanne Graham
Email: Sincere@april.biz
Fetching comments...
Found 5 comments on post 1
- id labore ex et quam laborum...
- quo vero reiciendis velit similique ear...
The HTTPAdapter approach is elegant because you configure retry behavior once on the session and it applies to every request automatically. The status_forcelist parameter specifies which HTTP status codes should trigger a retry — here we include 429 (rate limited) and the common server error codes (500-504). The backoff_factor=1 means retries happen at 1s, 2s, 4s intervals. This is the approach most production Python applications use because it requires zero changes to individual API calls.
asyncio.Semaphore(5) — five concurrent requests, zero 429s, maximum throughput.
Real-Life Example: Resilient API Data Fetcher
Let us build a complete, production-ready API client that combines everything we have covered: exponential backoff, Retry-After handling, proactive throttling, and comprehensive logging. This is a class you can drop into any project that needs to pull data from rate-limited APIs.
# resilient_api_client.py
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
import time
import random
class ResilientAPIClient:
"""A production-ready API client with rate limit handling."""
def __init__(self, base_url, requests_per_second=5, max_retries=5):
self.base_url = base_url.rstrip("/")
self.min_interval = 1.0 / requests_per_second
self.max_retries = max_retries
self.last_request_time = 0
self.request_count = 0
self.retry_count = 0
# Set up session with built-in retry for server errors
self.session = requests.Session()
retry_strategy = Retry(
total=2,
backoff_factor=0.5,
status_forcelist=(500, 502, 503, 504),
)
adapter = HTTPAdapter(max_retries=retry_strategy)
self.session.mount("https://", adapter)
self.session.mount("http://", adapter)
def _throttle(self):
"""Enforce rate limiting between requests."""
elapsed = time.time() - self.last_request_time
if elapsed < self.min_interval:
time.sleep(self.min_interval - elapsed)
self.last_request_time = time.time()
def get(self, endpoint, params=None):
"""Make a GET request with full retry and throttling."""
url = f"{self.base_url}/{endpoint.lstrip('/')}"
self._throttle()
for attempt in range(self.max_retries):
self.request_count += 1
response = self.session.get(url, params=params, timeout=15)
if response.status_code == 200:
return response.json()
if response.status_code == 429:
self.retry_count += 1
retry_after = response.headers.get("Retry-After")
if retry_after:
wait = int(retry_after) + random.uniform(0, 1)
else:
wait = (2 ** attempt) + random.uniform(0, 1)
print(f" [429] Rate limited on {endpoint}. Waiting {wait:.1f}s...")
time.sleep(wait)
continue
response.raise_for_status()
raise Exception(f"Failed to fetch {endpoint} after {self.max_retries} retries")
def get_stats(self):
"""Return usage statistics."""
return {
"total_requests": self.request_count,
"retries": self.retry_count,
"success_rate": f"{((self.request_count - self.retry_count) / max(self.request_count, 1)) * 100:.1f}%",
}
# --- Demo: Fetch data from JSONPlaceholder API ---
if __name__ == "__main__":
client = ResilientAPIClient(
base_url="https://jsonplaceholder.typicode.com",
requests_per_second=3,
max_retries=5,
)
print("=== Resilient API Client Demo ===\n")
# Fetch multiple users
print("Fetching users...")
users = client.get("/users")
for user in users[:3]:
print(f" {user['name']} ({user['email']})")
# Fetch posts for a user
print(f"\nFetching posts for user 1...")
posts = client.get("/posts", params={"userId": 1})
print(f" Found {len(posts)} posts")
for post in posts[:3]:
print(f" [{post['id']}] {post['title'][:45]}...")
# Fetch comments on first post
print(f"\nFetching comments on post 1...")
comments = client.get("/posts/1/comments")
print(f" Found {len(comments)} comments")
# Fetch todos
print(f"\nFetching todos...")
todos = client.get("/todos", params={"userId": 1, "_limit": 5})
completed = sum(1 for t in todos if t["completed"])
print(f" {completed}/{len(todos)} completed")
# Print stats
stats = client.get_stats()
print(f"\n=== Client Stats ===")
print(f" Total requests: {stats['total_requests']}")
print(f" Retries: {stats['retries']}")
print(f" Success rate: {stats['success_rate']}")
Output:
=== Resilient API Client Demo ===
Fetching users...
Leanne Graham (Sincere@april.biz)
Ervin Howell (Shanna@melissa.tv)
Clementine Bauch (Nathan@yesenia.net)
Fetching posts for user 1...
Found 10 posts
[1] sunt aut facere repellat provident occaec...
[2] qui est esse...
[3] ea molestias quasi exercitationem repella...
Fetching comments on post 1...
Found 5 comments
Fetching todos...
2/5 completed
=== Client Stats ===
Total requests: 4
Retries: 0
Success rate: 100.0%
This ResilientAPIClient class is designed for real-world use. It combines proactive throttling (the _throttle method ensures you never exceed your target rate), reactive retry logic (exponential backoff with jitter on 429 responses), and built-in statistics tracking. The session-level HTTPAdapter handles server errors (500s) automatically, while the manual retry loop in get() handles rate limits specifically. You can extend this class with authentication headers, POST/PUT methods, pagination support, or async capabilities using aiohttp for higher throughput.
Frequently Asked Questions
How many retries should I set?
Three to five retries is the standard range for most APIs. With exponential backoff starting at 1 second, five retries means a maximum total wait of about 31 seconds (1 + 2 + 4 + 8 + 16). If an API is still rate-limiting you after 30 seconds of waiting, either your rate is far too high or the API is experiencing an outage. For batch processing jobs that can tolerate longer waits, you might go up to 7-10 retries with a backoff cap of 60 seconds.
What is jitter and why does it matter?
Jitter adds a small random delay to your backoff interval. Without it, if 100 clients all get rate-limited at the same time, they will all retry at exactly 1 second, get limited again, retry at 2 seconds, and so on — creating synchronized waves of traffic. Adding random.uniform(0, 1) spreads out the retries so not everyone hits the server at the same instant. This is especially important in distributed systems where multiple workers or servers are calling the same API.
What is the difference between 429 and 503 errors?
A 429 Too Many Requests means you specifically have exceeded your rate limit — the server is healthy but refusing your requests because you are making too many. A 503 Service Unavailable means the server itself is overloaded or down for maintenance — it is not specific to you. Both are retryable, but 429 usually comes with a Retry-After header telling you exactly when to try again, while 503 is more unpredictable. Treat 429 as "slow down" and 503 as "try again later."
Can I use retry logic with async/await?
Yes. The tenacity library works with async functions out of the box — just decorate your async def function with @retry and it handles everything. For the requests session approach, switch to aiohttp which is the async equivalent. You can also use asyncio.sleep() instead of time.sleep() in your manual retry loops so the event loop can process other tasks during the backoff period.
How do I handle rate limits for multiple different APIs?
Create separate client instances for each API, each with its own rate limit configuration. For example, if the GitHub API allows 5,000 requests per hour and a weather API allows 60 requests per minute, create two ResilientAPIClient instances with different requests_per_second values. This keeps each API's rate limiting independent and prevents one slow API from blocking requests to another.
How do I test retry logic without hitting a real API?
Use the responses library (pip install responses) or unittest.mock to mock HTTP responses. You can simulate a sequence of 429 → 429 → 200 responses to verify your backoff logic works correctly. For integration testing, httpbin.org/status/429 returns a real 429 response that you can use to test your retry handling against an actual HTTP server.
Conclusion
You now have a complete toolkit for handling API rate limits in Python. We covered the fundamentals of what rate limits are and why APIs use them, built retry logic with exponential backoff and jitter from scratch, used the tenacity library for clean decorator-based retries, learned to respect the Retry-After header, implemented proactive request throttling, configured the requests session with HTTPAdapter for automatic retries, and built a production-ready ResilientAPIClient class that combines all these techniques.
Try extending the ResilientAPIClient with authentication support, pagination handling, or aiohttp integration for async requests. These are the natural next steps when building data pipelines or API integrations that need to be both fast and reliable.
