How To Store Data Separately For Testing and Production in Python 3

Intermediate

How To Store Data Separately For Testing and Production in Python 3

Quick Example (TLDR)

Use environment variables to switch between test and production settings:

# config.py
import os

ENVIRONMENT = os.getenv('ENV', 'development')

if ENVIRONMENT == 'production':
    DATABASE_URL = 'postgresql://prod.example.com:5432/maindb'
    DEBUG = False
else:
    DATABASE_URL = 'sqlite:///test.db'
    DEBUG = True

# main.py
from config import DATABASE_URL, DEBUG
print(f"Database: {DATABASE_URL}, Debug: {DEBUG}")

Output (development):

Database: sqlite:///test.db, Debug: True

Output (production):

Database: postgresql://prod.example.com:5432/maindb, Debug: False

Why Separate Configurations Matter

Your test environment should be completely isolated from your production data. Testing on production is a recipe for disaster: you could delete real data, send test emails to real customers, or charge real credit cards. The solution is environment-specific configuration.

Using Environment Variables

Environment variables are the simplest and most secure way to manage different configs:

import os

# Read environment variable, use default if not set
DATABASE_HOST = os.getenv('DB_HOST', 'localhost')
DATABASE_PORT = int(os.getenv('DB_PORT', '5432'))
API_KEY = os.getenv('API_KEY')  # No default - must be set
DEBUG_MODE = os.getenv('DEBUG', 'False').lower() == 'true'

print(f"Host: {DATABASE_HOST}")
print(f"Port: {DATABASE_PORT}")
print(f"Debug: {DEBUG_MODE}")

Output:

Host: localhost
Port: 5432
Debug: False

Using .env Files with python-dotenv

For development, load environment variables from a .env file:

# .env (development)
DATABASE_URL=sqlite:///dev.db
API_KEY=dev-key-12345
DEBUG=True
MAIL_SERVICE=fake

# .env.production
DATABASE_URL=postgresql://prod.example.com/maindb
API_KEY=prod-key-secret
DEBUG=False
MAIL_SERVICE=sendgrid

Load them in Python:

from dotenv import load_dotenv
import os

# Load from .env file
load_dotenv('.env')

DATABASE_URL = os.getenv('DATABASE_URL')
API_KEY = os.getenv('API_KEY')
print(f"Database: {DATABASE_URL}")

Output:

Database: sqlite:///dev.db

ConfigParser for Multi-Environment Files

For complex configurations, use ConfigParser with separate .ini files:

# config_development.ini
[database]
host = localhost
port = 5432
name = testdb
user = testuser
password = testpass

[email]
service = console
debug = True

[api]
key = dev-key-123
timeout = 10

Read it in Python:

import configparser
import os

# Determine which config to load
env = os.getenv('ENV', 'development')
config_file = f'config_{env}.ini'

# Parse the config file
config = configparser.ConfigParser()
config.read(config_file)

# Access values
db_host = config.get('database', 'host')
db_port = config.getint('database', 'port')
email_debug = config.getboolean('email', 'debug')

print(f"Database: {db_host}:{db_port}")
print(f"Email debug: {email_debug}")

Output:

Database: localhost:5432
Email debug: True

The Settings Pattern: Dev, Staging, and Production

Create a flexible settings module that supports multiple environments:

# settings/base.py - Common settings for all environments
import os

class Settings:
    # Common to all environments
    SECRET_KEY = os.getenv('SECRET_KEY', 'dev-key')
    ALLOWED_HOSTS = ['localhost', '127.0.0.1']
    
    # Default values
    DATABASE_URL = 'sqlite:///db.sqlite3'
    API_TIMEOUT = 30
    DEBUG = False
    LOG_LEVEL = 'INFO'

# settings/development.py
from .base import Settings

class DevelopmentSettings(Settings):
    DEBUG = True
    DATABASE_URL = 'sqlite:///dev.db'
    API_TIMEOUT = 60  # Longer timeout for debugging
    LOG_LEVEL = 'DEBUG'

# settings/production.py
from .base import Settings

class ProductionSettings(Settings):
    DEBUG = False
    DATABASE_URL = os.getenv('DATABASE_URL')  # From env vars
    API_TIMEOUT = 10  # Strict timeout in production
    LOG_LEVEL = 'WARNING'

# settings/staging.py
from .base import Settings

class StagingSettings(Settings):
    DEBUG = False  # But with more logging than prod
    DATABASE_URL = os.getenv('STAGING_DATABASE_URL')
    LOG_LEVEL = 'INFO'

Use it in your app:

# main.py
import os
from settings.development import DevelopmentSettings
from settings.production import ProductionSettings
from settings.staging import StagingSettings

# Load appropriate settings
env = os.getenv('ENV', 'development')

if env == 'production':
    settings = ProductionSettings()
elif env == 'staging':
    settings = StagingSettings()
else:
    settings = DevelopmentSettings()

# Use the settings
print(f"Debug mode: {settings.DEBUG}")
print(f"Database: {settings.DATABASE_URL}")
print(f"Log level: {settings.LOG_LEVEL}")

Output (development):

Debug mode: True
Database: sqlite:///dev.db
Log level: DEBUG

Output (production):

Debug mode: False
Database: postgresql://prod.example.com/maindb
Log level: WARNING

Real-Life Example: Flask App With Environment-Based Config

Here’s a complete Flask application setup with separate environments:

# config.py
import os

class Config:
    # Common
    SECRET_KEY = os.getenv('SECRET_KEY', 'dev-secret-key')
    SQLALCHEMY_TRACK_MODIFICATIONS = False

class DevelopmentConfig(Config):
    DEBUG = True
    SQLALCHEMY_DATABASE_URI = 'sqlite:///dev.db'
    SQLALCHEMY_ECHO = True  # Log all SQL
    MAIL_BACKEND = 'console'  # Print emails to console

class TestingConfig(Config):
    TESTING = True
    SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:'  # In-memory DB
    MAIL_BACKEND = 'testing'  # Don't send emails

class ProductionConfig(Config):
    DEBUG = False
    SQLALCHEMY_DATABASE_URI = os.getenv('DATABASE_URL')
    SQLALCHEMY_ECHO = False  # Don't log SQL
    MAIL_BACKEND = 'sendgrid'  # Send real emails

# app.py
from flask import Flask
import os
from config import DevelopmentConfig, TestingConfig, ProductionConfig

def create_app():
    app = Flask(__name__)
    
    # Load config based on environment
    env = os.getenv('FLASK_ENV', 'development')
    
    if env == 'production':
        app.config.from_object(ProductionConfig)
    elif env == 'testing':
        app.config.from_object(TestingConfig)
    else:
        app.config.from_object(DevelopmentConfig)
    
    @app.route('/status')
    def status():
        return {
            'environment': env,
            'debug': app.config['DEBUG'],
            'database': app.config['SQLALCHEMY_DATABASE_URI']
        }
    
    return app

# Run with: FLASK_ENV=production python app.py
if __name__ == '__main__':
    app = create_app()
    app.run()

Output (development):

{
  "environment": "development",
  "debug": true,
  "database": "sqlite:///dev.db"
}

FAQ

Q: Should I commit .env files to git?

A: Never! Add .env to .gitignore. Commit .env.example with dummy values so others know what variables are needed. This keeps secrets out of version control.

Q: Can I use environment variables for all settings?

A: Yes, but config files are often easier for complex setups. Use environment variables for secrets (API keys, passwords) and config files for regular settings.

Q: How do I test with a test database without affecting production?

A: Use an in-memory database or a separate test database for testing. Set ENV=testing to automatically use test configuration with no risk to production.

Q: What if I forget to set an environment variable?

A: Use defaults wisely. For critical values like DATABASE_URL, don’t provide defaults so the app fails loudly. For optional values, provide sensible defaults.

Q: How do I know which environment my code is running in?

A: Always have a way to check: print(os.getenv(‘ENV’)) or check your settings object. In Flask: app.config[‘DEBUG’] or app.config[‘ENV’]

Conclusion

Separating configuration by environment is a fundamental practice in professional software development. It protects your production data, makes testing safer, and allows different teams to work without stepping on each other’s toes. Use environment variables for secrets and configuration files for complex settings, and your code will be more flexible and secure.

References

• Python ConfigParser documentation
• os.getenv documentation
• python-dotenv package
• The Twelve Factor App – Config

The Pattern: Env Var Says Which Config to Load

The most reliable pattern across frameworks: read a single environment variable (APP_ENV or PYTHON_ENV) at startup, branch on it to load the right config file. Everything else flows from that one decision:

# File: config/__init__.py
import os
from importlib import import_module

env = os.environ.get("APP_ENV", "development")
config = import_module(f"config.{env}").Config

# config/development.py
class Config:
    DEBUG = True
    DATABASE_URL = "sqlite:///dev.db"
    CACHE_TYPE = "SimpleCache"
    LOG_LEVEL = "DEBUG"

# config/testing.py
class Config:
    DEBUG = True
    TESTING = True
    DATABASE_URL = "sqlite:///:memory:"
    CACHE_TYPE = "NullCache"

# config/production.py
class Config:
    DEBUG = False
    DATABASE_URL = os.environ["DATABASE_URL"]  # require in prod
    CACHE_TYPE = "RedisCache"
    LOG_LEVEL = "WARNING"

The trick: development and testing have safe defaults inline, but production REQUIRES the env var. os.environ["DATABASE_URL"] raises KeyError at startup if missing — fail-fast is exactly what you want in production.

Pydantic Settings — The Modern Approach

For real applications, hand-rolled config classes get unwieldy fast. pydantic-settings (the modern replacement for pydantic.BaseSettings) gives you type-checked config with automatic env-var loading, validation, and .env file support:

# pip install pydantic-settings

from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8")

    database_url: str
    secret_key: str = Field(min_length=32)
    debug: bool = False
    log_level: str = "INFO"
    redis_url: str | None = None

settings = Settings()
print(settings.database_url)

It loads from .env in dev, from real env vars in production, and the type annotations get validated at startup. secret_key with min_length=32 catches misconfigured staging where someone copy-pasted a short test value.

Secrets vs Config

Treat secrets (API keys, DB passwords, signing keys) differently from configuration (feature flags, timeouts, hostnames). Three rules:

• Never commit secrets. Put them in .env (gitignored), in a secrets manager (AWS Secrets Manager, Vault, GCP Secret Manager), or in your platform’s encrypted-env-var feature.
• Never log secrets. Add a __repr__ override that returns " for any field marked as a secret. Pydantic v2 supports SecretStr for exactly this.
• Rotate periodically. Secrets that never change are secrets that have been leaked. Build rotation into your config-loading code (multiple valid keys at once, with a deprecation window).

Common Pitfalls

• Default-to-production. os.environ.get("APP_ENV", "production") is the opposite of what you want. Default to development so a forgotten env var in CI doesn’t accidentally point at the prod database.
• Config drift between environments. A new feature flag added in dev but not staging means staging tests pass, prod breaks. Use a single Settings class with the same fields across envs; only the values differ.
• Config loaded at module import time. If your settings = Settings() runs at import, you can’t override env vars in tests without re-importing the module. Wrap config in a function (get_settings()) and cache it lazily.
• Reading env vars throughout the codebase. Scattered os.environ.get(...) calls make config hard to audit. Centralize all env-var access in one Settings class; the rest of the code imports from there.
• No validation. A typo in a numeric setting (e.g., PORT="abc") shouldn’t fail when the first request arrives — it should fail at startup. Pydantic Settings gives you this for free.

FAQ

Q: .env file or real environment variables?
A: Both. .env for local development (never committed), real env vars in production via your platform’s secrets management. pydantic-settings reads from both transparently.

Q: How do I share config between Python and other services in my stack?
A: Use a portable format — env vars, JSON, or YAML — not a Python module. Then each service loads from the shared source. Resist the temptation to import config across language boundaries.

Q: What about feature flags?
A: Same Settings class can hold feature flags as booleans. For dynamic flags that change without redeployment, use a flag service like LaunchDarkly or Unleash — they’re worth it once you have more than 5-10 flags.

Q: How do I test config-dependent code?
A: Inject the config object rather than importing it. Tests pass a custom Settings instance with the values they need. pytest fixtures make this easy with autouse=True overrides.

Q: 12-factor app — do I have to do all twelve?
A: Config-in-environment is factor 3 and the one that matters most. The other eleven are great guidelines, but most teams gain 80% of the benefit just by getting config and secrets out of code.

Wrapping Up

Environment-driven config is one of those infrastructure habits that compounds: small now, life-saving when you’re trying to debug a production outage at 2 AM. Start with the APP_ENV pattern, move to pydantic-settings when the codebase grows, and never let secrets touch your git history. The setup cost is an hour; the lifetime cost of getting it wrong is incalculable.

Related Articles

• How To Manage Python Environment Variables With dotenv
• How To Use Python TOML For Configuration Files Instead of INI
• How To Set Up Python Logging With Rotating File Handlers