Intermediate
How To Store Data Separately For Testing and Production in Python 3
Quick Example (TLDR)
Use environment variables to switch between test and production settings:
# config.py
import os
ENVIRONMENT = os.getenv('ENV', 'development')
if ENVIRONMENT == 'production':
DATABASE_URL = 'postgresql://prod.example.com:5432/maindb'
DEBUG = False
else:
DATABASE_URL = 'sqlite:///test.db'
DEBUG = True
# main.py
from config import DATABASE_URL, DEBUG
print(f"Database: {DATABASE_URL}, Debug: {DEBUG}")
Output (development):
Database: sqlite:///test.db, Debug: True
Output (production):
Database: postgresql://prod.example.com:5432/maindb, Debug: False
Why Separate Configurations Matter
Your test environment should be completely isolated from your production data. Testing on production is a recipe for disaster: you could delete real data, send test emails to real customers, or charge real credit cards. The solution is environment-specific configuration.
Using Environment Variables
Environment variables are the simplest and most secure way to manage different configs:
import os
# Read environment variable, use default if not set
DATABASE_HOST = os.getenv('DB_HOST', 'localhost')
DATABASE_PORT = int(os.getenv('DB_PORT', '5432'))
API_KEY = os.getenv('API_KEY') # No default - must be set
DEBUG_MODE = os.getenv('DEBUG', 'False').lower() == 'true'
print(f"Host: {DATABASE_HOST}")
print(f"Port: {DATABASE_PORT}")
print(f"Debug: {DEBUG_MODE}")
Output:
Host: localhost
Port: 5432
Debug: False
Using .env Files with python-dotenv
For development, load environment variables from a .env file:
# .env (development)
DATABASE_URL=sqlite:///dev.db
API_KEY=dev-key-12345
DEBUG=True
MAIL_SERVICE=fake
# .env.production
DATABASE_URL=postgresql://prod.example.com/maindb
API_KEY=prod-key-secret
DEBUG=False
MAIL_SERVICE=sendgrid
Load them in Python:
from dotenv import load_dotenv
import os
# Load from .env file
load_dotenv('.env')
DATABASE_URL = os.getenv('DATABASE_URL')
API_KEY = os.getenv('API_KEY')
print(f"Database: {DATABASE_URL}")
Output:
Database: sqlite:///dev.db
ConfigParser for Multi-Environment Files
For complex configurations, use ConfigParser with separate .ini files:
# config_development.ini
[database]
host = localhost
port = 5432
name = testdb
user = testuser
password = testpass
[email]
service = console
debug = True
[api]
key = dev-key-123
timeout = 10
Read it in Python:
import configparser
import os
# Determine which config to load
env = os.getenv('ENV', 'development')
config_file = f'config_{env}.ini'
# Parse the config file
config = configparser.ConfigParser()
config.read(config_file)
# Access values
db_host = config.get('database', 'host')
db_port = config.getint('database', 'port')
email_debug = config.getboolean('email', 'debug')
print(f"Database: {db_host}:{db_port}")
print(f"Email debug: {email_debug}")
Output:
Database: localhost:5432
Email debug: True
The Settings Pattern: Dev, Staging, and Production
Create a flexible settings module that supports multiple environments:
# settings/base.py - Common settings for all environments
import os
class Settings:
# Common to all environments
SECRET_KEY = os.getenv('SECRET_KEY', 'dev-key')
ALLOWED_HOSTS = ['localhost', '127.0.0.1']
# Default values
DATABASE_URL = 'sqlite:///db.sqlite3'
API_TIMEOUT = 30
DEBUG = False
LOG_LEVEL = 'INFO'
# settings/development.py
from .base import Settings
class DevelopmentSettings(Settings):
DEBUG = True
DATABASE_URL = 'sqlite:///dev.db'
API_TIMEOUT = 60 # Longer timeout for debugging
LOG_LEVEL = 'DEBUG'
# settings/production.py
from .base import Settings
class ProductionSettings(Settings):
DEBUG = False
DATABASE_URL = os.getenv('DATABASE_URL') # From env vars
API_TIMEOUT = 10 # Strict timeout in production
LOG_LEVEL = 'WARNING'
# settings/staging.py
from .base import Settings
class StagingSettings(Settings):
DEBUG = False # But with more logging than prod
DATABASE_URL = os.getenv('STAGING_DATABASE_URL')
LOG_LEVEL = 'INFO'
Use it in your app:
# main.py
import os
from settings.development import DevelopmentSettings
from settings.production import ProductionSettings
from settings.staging import StagingSettings
# Load appropriate settings
env = os.getenv('ENV', 'development')
if env == 'production':
settings = ProductionSettings()
elif env == 'staging':
settings = StagingSettings()
else:
settings = DevelopmentSettings()
# Use the settings
print(f"Debug mode: {settings.DEBUG}")
print(f"Database: {settings.DATABASE_URL}")
print(f"Log level: {settings.LOG_LEVEL}")
Output (development):
Debug mode: True
Database: sqlite:///dev.db
Log level: DEBUG
Output (production):
Debug mode: False
Database: postgresql://prod.example.com/maindb
Log level: WARNING
Real-Life Example: Flask App With Environment-Based Config
Here’s a complete Flask application setup with separate environments:
# config.py
import os
class Config:
# Common
SECRET_KEY = os.getenv('SECRET_KEY', 'dev-secret-key')
SQLALCHEMY_TRACK_MODIFICATIONS = False
class DevelopmentConfig(Config):
DEBUG = True
SQLALCHEMY_DATABASE_URI = 'sqlite:///dev.db'
SQLALCHEMY_ECHO = True # Log all SQL
MAIL_BACKEND = 'console' # Print emails to console
class TestingConfig(Config):
TESTING = True
SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:' # In-memory DB
MAIL_BACKEND = 'testing' # Don't send emails
class ProductionConfig(Config):
DEBUG = False
SQLALCHEMY_DATABASE_URI = os.getenv('DATABASE_URL')
SQLALCHEMY_ECHO = False # Don't log SQL
MAIL_BACKEND = 'sendgrid' # Send real emails
# app.py
from flask import Flask
import os
from config import DevelopmentConfig, TestingConfig, ProductionConfig
def create_app():
app = Flask(__name__)
# Load config based on environment
env = os.getenv('FLASK_ENV', 'development')
if env == 'production':
app.config.from_object(ProductionConfig)
elif env == 'testing':
app.config.from_object(TestingConfig)
else:
app.config.from_object(DevelopmentConfig)
@app.route('/status')
def status():
return {
'environment': env,
'debug': app.config['DEBUG'],
'database': app.config['SQLALCHEMY_DATABASE_URI']
}
return app
# Run with: FLASK_ENV=production python app.py
if __name__ == '__main__':
app = create_app()
app.run()
Output (development):
{
"environment": "development",
"debug": true,
"database": "sqlite:///dev.db"
}
FAQ
Q: Should I commit .env files to git?
A: Never! Add .env to .gitignore. Commit .env.example with dummy values so others know what variables are needed. This keeps secrets out of version control.
Q: Can I use environment variables for all settings?
A: Yes, but config files are often easier for complex setups. Use environment variables for secrets (API keys, passwords) and config files for regular settings.
Q: How do I test with a test database without affecting production?
A: Use an in-memory database or a separate test database for testing. Set ENV=testing to automatically use test configuration with no risk to production.
Q: What if I forget to set an environment variable?
A: Use defaults wisely. For critical values like DATABASE_URL, don’t provide defaults so the app fails loudly. For optional values, provide sensible defaults.
Q: How do I know which environment my code is running in?
A: Always have a way to check: print(os.getenv(‘ENV’)) or check your settings object. In Flask: app.config[‘DEBUG’] or app.config[‘ENV’]
Conclusion
Separating configuration by environment is a fundamental practice in professional software development. It protects your production data, makes testing safer, and allows different teams to work without stepping on each other’s toes. Use environment variables for secrets and configuration files for complex settings, and your code will be more flexible and secure.
References
- Python ConfigParser documentation
- os.getenv documentation
- python-dotenv package
- The Twelve Factor App – Config
