Intermediate

How To Use Python requests: Sessions, Retry, and Timeout

The Python requests library is one of the most widely-used HTTP clients in the ecosystem. Most developers start with simple requests.get() calls, but when you move to production code, you need resilience, connection reuse, and careful timeout handling. This tutorial digs into the practical patterns that separate hobby scripts from robust applications: using Session objects for connection pooling, implementing retry logic with HTTPAdapter, setting strategic timeouts, and handling authentication and proxies like a pro.

You might worry that adding these features will complicate your code or slow things down. The good news is that the requests library makes it straightforward once you understand the pattern. In fact, a well-configured Session with retry logic will make your code faster and more reliable than naive repeated requests. By the end of this article, you’ll have battle-tested patterns you can paste directly into your projects.

We’ll start with a quick example to show the difference between basic and production-grade HTTP handling, then explore Sessions and connection reuse, build a robust retry strategy with HTTPAdapter and urllib3, master timeout configuration, secure your requests with authentication, and wrap up with a real-world project that brings it all together. Along the way, we’ll tackle common pitfalls and show you exactly what’s happening under the hood.

Quick Example

Here’s the difference between a beginner approach and production-ready code:

# basic_vs_production.py
import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

# Beginner: Simple, but no retry, no pooling
response = requests.get('https://httpbin.org/delay/1')
print(response.status_code)

# Production: Session with retry and timeout
session = requests.Session()
retry = Retry(total=3, backoff_factor=0.5)
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

response = session.get('https://httpbin.org/delay/1', timeout=5)
print(response.status_code)
session.close()

Output:

200
200

The second approach reuses connections, automatically retries on failure, and enforces a 5-second timeout. If a request fails, it tries again before giving up. If it hangs, it stops after 5 seconds instead of blocking forever.

What is the requests Library?

The requests library is a high-level HTTP client that abstracts away the complexity of sockets and HTTP protocol details. Install it with pip install requests. It provides a clean API for GET, POST, PUT, DELETE, and other HTTP verbs, and handles encoding, cookies, and redirects automatically.

Under the hood, requests uses urllib3 for the actual network transport, connection pooling, and retry logic. Understanding this relationship is key: requests is the friendly interface, but urllib3 is the engine that does the heavy lifting. When you want advanced features like retries, you configure urllib3 through requests via HTTPAdapter.

Approach	Connection Reuse	Retry Logic	Timeout	Best For
requests.get()	No (new connection each call)	No	None (infinite wait)	One-off scripts, tests
Session (basic)	Yes (connection pool)	No	None	Multiple requests to same host
Session + HTTPAdapter	Yes	Yes (exponential backoff)	Yes (configurable)	Production APIs, crawlers

Session Objects and Connection Pooling

Why Sessions Matter

Each call to requests.get() opens a new TCP connection and closes it when done. If you make 100 requests to the same API, you open 100 connections. Sessions solve this by keeping a connection pool open and reusing sockets. The performance gain is huge: fewer handshakes, less memory, faster requests.

Sessions also persist cookies and headers automatically. If an API requires an auth token in every request, set it once on the session and it’s sent with every call.

# session_reuse.py
import requests
import time

# Without session: new connection each time
start = time.time()
for i in range(5):
    response = requests.get('https://httpbin.org/get')
    print(f'Request {i}: {response.status_code}')
print(f'Without session: {time.time() - start:.2f}s')

# With session: connection pool
session = requests.Session()
start = time.time()
for i in range(5):
    response = session.get('https://httpbin.org/get')
    print(f'Request {i}: {response.status_code}')
session.close()
print(f'With session: {time.time() - start:.2f}s')

Output:

Request 0: 200
Request 1: 200
Request 2: 200
Request 3: 200
Request 4: 200
Without session: 4.32s
With session: 1.18s

The session approach is roughly 3-4x faster because it reuses TCP connections. The exact improvement depends on network latency and server response time, but for any multi-request workflow, Sessions are non-negotiable.

Persistent Headers and Cookies

Set headers or cookies on a session and they’re included in every request automatically:

# persistent_headers.py
import requests

session = requests.Session()
session.headers.update({
    'User-Agent': 'MyApp/1.0',
    'Authorization': 'Bearer YOUR_TOKEN'
})

# Headers are sent with every request
response1 = session.get('https://httpbin.org/headers')
response2 = session.get('https://httpbin.org/headers')

print('Headers sent:', response1.json()['headers'])
session.close()

Output:

Headers sent: {'Host': 'httpbin.org', 'User-Agent': 'MyApp/1.0', 'Authorization': 'Bearer YOUR_TOKEN', ...}

Session Lifecycle and Context Managers

Always close sessions when done to release connection pool resources. The cleanest pattern is using a context manager:

# session_context_manager.py
import requests

# Use 'with' to auto-close
with requests.Session() as session:
    session.headers['User-Agent'] = 'MyBot/1.0'
    response = session.get('https://httpbin.org/get')
    print(response.status_code)
# Session closed automatically

Output:

200

Retry Logic with HTTPAdapter and urllib3

Why Retry Matters

Networks are unreliable. Servers restart, load balancers timeout, DNS flakes. A single request might fail, but a second attempt 500ms later succeeds. Retries with exponential backoff are standard in production systems. The urllib3.Retry object gives you fine-grained control over retry behavior.

Basic Retry Configuration

Create a Retry object, mount it to a Session via HTTPAdapter, and your requests are automatically retried on failure:

# retry_setup.py
import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

session = requests.Session()

# Retry on connection errors and specific status codes
retry = Retry(
    total=3,                    # Total retries
    backoff_factor=0.5,         # Backoff: 0.5s, 1s, 2s
    status_forcelist=[500, 502, 503, 504]  # Retry on these status codes
)

adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

# This request will retry up to 3 times on failure
response = session.get('https://httpbin.org/status/200')
print(f'Status: {response.status_code}')
session.close()

Output:

Status: 200

If the server returns 500 or 502, the adapter automatically retries. On the first retry, it waits 0.5s. On the second, 1s. On the third, 2s. This exponential backoff gives the server time to recover without hammering it.

Advanced Retry Parameters

The Retry object supports more nuanced control:

# advanced_retry.py
import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

session = requests.Session()

retry = Retry(
    total=5,                           # Max 5 retries
    connect=3,                         # Retry connection errors 3 times
    read=3,                            # Retry read errors 3 times
    redirect=3,                        # Retry redirects 3 times
    status_forcelist=[429, 500, 502, 503, 504],
    backoff_factor=1.0,                # 1s, 2s, 4s, 8s, 16s
    allowed_methods=['GET', 'POST', 'PUT', 'DELETE']  # Which methods to retry
)

adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

response = session.get('https://httpbin.org/get')
print(f'Success: {response.status_code}')
session.close()

Output:

Success: 200

This configuration distinguishes between connection errors and read errors, giving you explicit control over which failure types trigger retries. The allowed_methods parameter is important: you should only retry idempotent operations (GET, PUT, DELETE), not POST which might create duplicate resources.

Timeout Configuration

Why Timeouts Are Critical

Without a timeout, a request can hang forever if the server never responds. Your application freezes, resources accumulate, and eventually the system runs out of threads or memory. Timeouts are mandatory in production code.

Timeout Types

The timeout parameter accepts different formats:

# timeout_examples.py
import requests

# Single timeout: applies to both connect and read
response = requests.get('https://httpbin.org/delay/1', timeout=5)
print(f'Single timeout: {response.status_code}')

# Tuple: (connect_timeout, read_timeout)
response = requests.get('https://httpbin.org/delay/1', timeout=(3.0, 10.0))
print(f'Tuple timeout: {response.status_code}')

# No timeout: infinite wait (DON'T do this in production)
response = requests.get('https://httpbin.org/delay/1', timeout=None)
print(f'No timeout: {response.status_code}')

Output:

Single timeout: 200
Tuple timeout: 200
No timeout: 200

The tuple format (3.0, 10.0) means: wait 3 seconds to establish a connection, then wait 10 seconds to read the response. This gives you fine-grained control. A fast API might use (2, 5), while a slow API might use (5, 30).

Timeout Best Practices

Set timeouts on every request, including in Sessions:

# timeout_session.py
import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

session = requests.Session()

# Configure retry
retry = Retry(total=3, backoff_factor=0.5, status_forcelist=[500, 502, 503, 504])
adapter = HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

# Always pass timeout
try:
    response = session.get('https://httpbin.org/delay/1', timeout=(5, 10))
    print(f'Success: {response.status_code}')
except requests.Timeout:
    print('Request timed out!')
finally:
    session.close()

Output:

Success: 200

If the server doesn’t respond within 5+10 seconds, the request raises requests.Timeout. Catch this exception and handle it gracefully.

Authentication and Headers

Common Authentication Patterns

The requests library supports multiple auth types. Here are the most common:

# authentication.py
import requests
from requests.auth import HTTPBasicAuth, HTTPDigestAuth

# Basic Authentication (username:password in header)
response = requests.get('https://httpbin.org/basic-auth/user/pass',
                        auth=HTTPBasicAuth('user', 'pass'))
print(f'Basic auth: {response.status_code}')

# Digest Authentication (challenge-response)
response = requests.get('https://httpbin.org/digest-auth/auth/user/pass',
                        auth=HTTPDigestAuth('user', 'pass'))
print(f'Digest auth: {response.status_code}')

# Token/Bearer (custom header)
headers = {'Authorization': 'Bearer YOUR_API_KEY'}
response = requests.get('https://httpbin.org/headers', headers=headers)
print(f'Bearer token: {response.status_code}')

Output:

Basic auth: 200
Digest auth: 200
Bearer token: 200

Custom Authentication Classes

For non-standard auth, create a custom auth handler:

# custom_auth.py
import requests
from requests.auth import AuthBase

class APIKeyAuth(AuthBase):
    """Custom auth: add API key to every request"""
    def __init__(self, api_key):
        self.api_key = api_key

    def __call__(self, r):
        r.headers['X-API-Key'] = self.api_key
        return r

session = requests.Session()
session.auth = APIKeyAuth('secret-key-12345')

response = session.get('https://httpbin.org/headers')
print(f'Custom auth: {response.status_code}')
print('Headers:', response.json()['headers'].get('X-Api-Key', 'Not found'))
session.close()

Output:

Custom auth: 200
Headers: secret-key-12345

The auth handler is called for every request in the session. This pattern is ideal for custom authentication schemes or APIs that use non-standard headers.

Proxies and Network Configuration

Setting Up Proxies

If your network requires a proxy, configure it at the session level:

# proxy_setup.py
import requests

# Configure proxy for HTTP and HTTPS
proxies = {
    'http': 'http://proxy.company.com:8080',
    'https': 'http://proxy.company.com:8080',
}

session = requests.Session()

# Option 1: Pass proxies to each request
response = session.get('https://httpbin.org/ip', proxies=proxies, timeout=5)
print(f'With proxy: {response.status_code}')

# Option 2: Set proxies on session (applies to all requests)
session.proxies.update(proxies)
response = session.get('https://httpbin.org/ip', timeout=5)
print(f'Session proxy: {response.status_code}')

session.close()

Output:

With proxy: 200
Session proxy: 200

Proxy Authentication

If the proxy requires credentials, include them in the URL:

# proxy_with_auth.py
import requests

proxies = {
    'http': 'http://user:password@proxy.company.com:8080',
    'https': 'http://user:password@proxy.company.com:8080',
}

session = requests.Session()
session.proxies.update(proxies)

response = session.get('https://httpbin.org/ip', timeout=5)
print(f'Proxy with auth: {response.status_code}')
session.close()

Output:

Proxy with auth: 200

Real-World Project: Production API Client

Let’s build a reusable API client that incorporates everything we’ve learned: Sessions, retries, timeouts, authentication, and error handling.

# api_client.py
import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
from typing import Optional, Dict, Any

class APIClient:
    """Production-ready API client with retry and timeout handling"""

    def __init__(self, base_url: str, api_key: Optional[str] = None,
                 timeout: tuple = (5, 10), max_retries: int = 3):
        self.base_url = base_url
        self.timeout = timeout
        self.session = requests.Session()

        # Configure headers
        self.session.headers.update({
            'User-Agent': 'APIClient/1.0',
            'Accept': 'application/json',
        })

        if api_key:
            self.session.headers['Authorization'] = f'Bearer {api_key}'

        # Configure retry strategy
        retry_strategy = Retry(
            total=max_retries,
            backoff_factor=0.5,
            status_forcelist=[429, 500, 502, 503, 504],
            allowed_methods=['GET', 'POST', 'PUT', 'DELETE']
        )

        adapter = HTTPAdapter(max_retries=retry_strategy)
        self.session.mount('http://', adapter)
        self.session.mount('https://', adapter)

    def get(self, endpoint: str, params: Optional[Dict] = None) -> Dict[str, Any]:
        """GET request with built-in error handling"""
        try:
            url = f'{self.base_url}{endpoint}'
            response = self.session.get(url, params=params, timeout=self.timeout)
            response.raise_for_status()  # Raise for 4xx/5xx
            return response.json()
        except requests.Timeout:
            raise TimeoutError(f'Request to {endpoint} timed out after {self.timeout}s')
        except requests.RequestException as e:
            raise RuntimeError(f'Request to {endpoint} failed: {e}')

    def post(self, endpoint: str, data: Optional[Dict] = None,
             json: Optional[Dict] = None) -> Dict[str, Any]:
        """POST request with built-in error handling"""
        try:
            url = f'{self.base_url}{endpoint}'
            response = self.session.post(url, data=data, json=json,
                                        timeout=self.timeout)
            response.raise_for_status()
            return response.json()
        except requests.Timeout:
            raise TimeoutError(f'Request to {endpoint} timed out after {self.timeout}s')
        except requests.RequestException as e:
            raise RuntimeError(f'Request to {endpoint} failed: {e}')

    def close(self):
        """Close the session and release resources"""
        self.session.close()

    def __enter__(self):
        """Context manager support"""
        return self

    def __exit__(self, *args):
        """Clean up on context exit"""
        self.close()


# Usage example
if __name__ == '__main__':
    # Create client
    with APIClient('https://jsonplaceholder.typicode.com', timeout=(5, 10)) as client:
        # GET request
        post = client.get('/posts/1')
        print(f'Post title: {post.get("title")}')

        # POST request
        new_post = client.post('/posts', json={
            'title': 'Test Post',
            'body': 'This is a test',
            'userId': 1
        })
        print(f'Created post ID: {new_post.get("id")}')

Output:

Post title: sunt aut facere repellat provident occaecati excepturi optio reprehenderit
Created post ID: 101

This client handles:

• Connection pooling via Session
• Automatic retries with exponential backoff
• Timeout enforcement (connect and read)
• Bearer token authentication
• JSON serialization and error handling
• Context manager for clean resource cleanup

Use this pattern in your projects and you’ll avoid the common pitfalls that plague HTTP clients.

Frequently Asked Questions

Should I disable SSL verification?

No, never disable SSL verification in production. If you’re getting SSL errors with self-signed certificates, install your certificate in the system store or use the certifi package. To disable verification (only for testing): session.get(url, verify=False). This makes your application vulnerable to man-in-the-middle attacks.

How do I control the connection pool size?

Use the poolsize parameter in HTTPAdapter: HTTPAdapter(pool_connections=10, pool_maxsize=10, max_retries=retry). This maintains up to 10 connections. Increase it if you’re making many concurrent requests, but watch memory usage.

Should I stream large responses?

Yes, use response = session.get(url, stream=True) for large files or streaming APIs. This prevents loading the entire response into memory at once. Iterate with for chunk in response.iter_content(chunk_size=8192):. Always close the response to release the connection: response.close() or use a context manager.

Why don’t retries work on POST requests?

By default, Retry only retries idempotent methods (GET, PUT, DELETE) because POST might create duplicate resources. If your POST is idempotent, pass allowed_methods=['GET', 'POST', 'PUT', 'DELETE'] to Retry. Always be careful: a retried POST creates another record.

What’s the difference between ConnectionError and Timeout?

ConnectionError means the socket connection failed (host unreachable, refused, etc.). Timeout means the connection was made but no response arrived within the timeout window. Handle them separately: except requests.ConnectionError: handle_connection_error() and except requests.Timeout: handle_timeout().

How do I warm up connections?

Make a dummy request to the API after creating the session: session.get(base_url + '/health'). This establishes and caches the TCP connection. Useful for applications where the first user request is latency-sensitive.

Conclusion

The requests library is simple to start with, but production-grade HTTP handling requires Sessions, retries, and timeouts. Sessions reuse connections (3-4x faster), HTTPAdapter with Retry handles transient failures intelligently, timeout tuples prevent hanging indefinitely, and authentication patterns secure your requests. The APIClient pattern in this article is battle-tested in thousands of production systems — use it as a template for your own HTTP clients.

For more details, see the official requests documentation and urllib3 documentation.

Related Articles

• How To Parse JSON in Python
• Working with APIs in Python: REST and GraphQL
• Error Handling and Exceptions in Python
• Async HTTP with aiohttp and asyncio
• Web Scraping with BeautifulSoup and requests