Python Configuration

Configuration management is a critical aspect of Python application development, especially when moving from development to production environments. In this guide, we'll explore various methods to handle configuration in Python applications, helping you choose the right approach for your projects.

Introduction to Configuration Management

Configuration management refers to the process of separating code from configuration settings like database credentials, API keys, feature toggles, and environment-specific parameters. Good configuration practices help you:

• Keep sensitive data out of your code repository
• Easily deploy to different environments
• Change application behavior without code modifications
• Follow the 12-Factor App methodology

Let's explore the various approaches to configuration in Python, from simple to more sophisticated options.

Simple Config Files

The most straightforward approach is to use a dedicated Python file for configuration.

Using Python Files

Creating a config.py file is the simplest approach:

python

# config.py
DATABASE_HOST = "localhost"
DATABASE_PORT = 5432
DATABASE_NAME = "myapp"
DATABASE_USER = "user"
DATABASE_PASSWORD = "password"  # Not recommended for production

DEBUG = True
LOG_LEVEL = "INFO"

Then in your application code:

python

import config

def connect_to_db():
    print(f"Connecting to {config.DATABASE_NAME} on {config.DATABASE_HOST}...")
    # Connection logic here
    
# Example usage
connect_to_db()

Output:

Connecting to myapp on localhost...

Environment-Specific Configuration

For different environments, you can extend this approach:

python

# config.py
import os

# Default configuration (development)
DATABASE_HOST = "localhost"
DATABASE_PORT = 5432
DATABASE_NAME = "myapp_dev"
DEBUG = True

# Override with production settings
if os.getenv("ENVIRONMENT") == "production":
    DATABASE_HOST = "production-db.example.com"
    DATABASE_NAME = "myapp_prod"
    DEBUG = False

INI Configuration Files

Python's built-in configparser module allows you to use INI-style configuration files.

Basic configparser Example

First, create an INI file:

ini

; config.ini
[database]
host = localhost
port = 5432
name = myapp
user = user
password = password

[app]
debug = true
log_level = INFO

Then read it in your Python code:

python

import configparser

def load_config(config_file="config.ini"):
    config = configparser.ConfigParser()
    config.read(config_file)
    return config

# Example usage
config = load_config()
db_name = config['database']['name']
debug_mode = config['app'].getboolean('debug')

print(f"Database name: {db_name}")
print(f"Debug mode: {debug_mode}")

Output:

Database name: myapp
Debug mode: True

JSON Configuration

JSON provides another common format for configuration files:

python

import json

def load_config(config_file="config.json"):
    with open(config_file, 'r') as f:
        return json.load(f)

# Example usage
try:
    config = load_config()
    print(f"Database name: {config['database']['name']}")
    print(f"Debug mode: {config['app']['debug']}")
except (FileNotFoundError, json.JSONDecodeError) as e:
    print(f"Error loading configuration: {e}")

A sample config.json file would look like:

json

{
  "database": {
    "host": "localhost",
    "port": 5432,
    "name": "myapp",
    "user": "user",
    "password": "password"
  },
  "app": {
    "debug": true,
    "log_level": "INFO"
  }
}

YAML Configuration

YAML offers a more human-readable format for configuration files. You'll need to install the PyYAML package:

bash

pip install pyyaml

python

import yaml

def load_config(config_file="config.yaml"):
    with open(config_file, 'r') as f:
        return yaml.safe_load(f)

# Example usage
try:
    config = load_config()
    print(f"Database name: {config['database']['name']}")
    print(f"Debug mode: {config['app']['debug']}")
except (FileNotFoundError, yaml.YAMLError) as e:
    print(f"Error loading configuration: {e}")

A sample config.yaml file:

yaml

database:
  host: localhost
  port: 5432
  name: myapp
  user: user
  password: password

app:
  debug: true
  log_level: INFO

Environment Variables

Following the 12-Factor App methodology, environment variables are an excellent way to configure applications, especially in containerized environments.

Basic Environment Variables

python

import os

# Load configuration from environment variables
DATABASE_HOST = os.getenv("DATABASE_HOST", "localhost")
DATABASE_PORT = int(os.getenv("DATABASE_PORT", "5432"))
DATABASE_NAME = os.getenv("DATABASE_NAME", "myapp")
DEBUG = os.getenv("DEBUG", "False").lower() in ("true", "yes", "1")

print(f"Database host: {DATABASE_HOST}")
print(f"Debug mode: {DEBUG}")

To run this with custom environment variables:

bash

DATABASE_HOST=custom-host.example.com DEBUG=true python app.py

Output:

Database host: custom-host.example.com
Debug mode: True

Using python-dotenv

For local development, .env files can be convenient. The python-dotenv package loads environment variables from a file:

bash

pip install python-dotenv

Create a .env file:

# .env - Do not commit to source control!
DATABASE_HOST=localhost
DATABASE_PORT=5432
DATABASE_NAME=myapp
DATABASE_USER=user
DATABASE_PASSWORD=secret
DEBUG=true

Then in your code:

python

import os
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

# Access variables
DATABASE_HOST = os.getenv("DATABASE_HOST")
DEBUG = os.getenv("DEBUG", "False").lower() in ("true", "yes", "1")

print(f"Database host: {DATABASE_HOST}")
print(f"Debug mode: {DEBUG}")

Configuration Libraries

For more complex applications, dedicated configuration libraries provide additional features.

Using Python-Decouple

Python-decouple is a library that helps separate configuration from code:

bash

pip install python-decouple

Create a .env file as shown earlier, then:

python

from decouple import config

DATABASE_HOST = config('DATABASE_HOST', default='localhost')
DATABASE_PORT = config('DATABASE_PORT', default=5432, cast=int)
DEBUG = config('DEBUG', default=False, cast=bool)

print(f"Database host: {DATABASE_HOST}")
print(f"Database port: {DATABASE_PORT}")
print(f"Debug mode: {DEBUG}")

Using Dynaconf

Dynaconf is a more feature-rich configuration library:

bash

pip install dynaconf

Create a settings.toml file:

toml

[default]
database_host = "localhost"
database_port = 5432
debug = false

[development]
debug = true

[production]
database_host = "prod-db.example.com"

Then in your code:

python

from dynaconf import Dynaconf

# Load settings
settings = Dynaconf(
    settings_files=["settings.toml", ".secrets.toml"],
    environments=True,
)

print(f"Database host: {settings.database_host}")
print(f"Debug mode: {settings.debug}")

To switch environments:

bash

ENV_FOR_DYNACONF=production python app.py

Real-World Example: Configurable Web Application

Let's build a simple Flask application with flexible configuration:

python

import os
from flask import Flask, jsonify
from dynaconf import Dynaconf

# Load configuration
settings = Dynaconf(
    settings_files=["settings.toml", ".secrets.toml"],
    environments=True,
    env=os.getenv("FLASK_ENV", "development"),
)

app = Flask(__name__)

@app.route('/')
def index():
    return jsonify({
        "app_name": settings.app_name,
        "environment": settings.current_env,
        "debug": settings.debug,
    })

@app.route('/config')
def config():
    # Only show non-sensitive configuration
    safe_config = {
        "app_name": settings.app_name,
        "environment": settings.current_env,
        "debug": settings.debug,
        "database": {
            "host": settings.database.host,
            "port": settings.database.port,
            "name": settings.database.name,
        }
    }
    return jsonify(safe_config)

if __name__ == '__main__':
    app.run(
        debug=settings.debug,
        host=settings.server.host,
        port=settings.server.port
    )

With a corresponding settings.toml:

toml

[default]
app_name = "My Configurable App"
debug = false

[default.server]
host = "0.0.0.0"
port = 8000

[default.database]
host = "localhost"
port = 5432
name = "myapp"

[development]
debug = true

[production]
debug = false

[production.server]
port = 5000

[production.database]
host = "production-db.example.com"

Best Practices for Configuration Management

1. Never hardcode sensitive information in your source code
2. Use environment variables for sensitive values and deployment-specific settings
3. Provide sensible defaults for all configuration options
4. Validate configuration at application startup
5. Document all configuration options clearly
6. Keep configuration simple – avoid overly complex hierarchies
7. Use appropriate security measures for sensitive configuration data
8. Version control configuration templates but not actual configuration files with secrets
9. Consider centralized configuration for microservices and distributed applications

Configuration Security Considerations

When dealing with sensitive configuration:

1. Don't store secrets in code repositories
2. Use environment variables or dedicated secret management tools
3. Encrypt sensitive configuration files if they must be stored
4. Use dedicated services like AWS Secrets Manager, HashiCorp Vault, or Azure Key Vault for production secrets
5. Rotate secrets regularly
6. Limit access to production configuration
7. Audit configuration changes

Summary

Python offers many approaches to configuration management, from simple Python files to sophisticated libraries:

• Config files (Python, INI, JSON, YAML) – Simple and flexible
• Environment variables – Good for containerized applications and 12-Factor methodology
• Dedicated libraries (python-dotenv, python-decouple, dynaconf) – Added features for larger applications
• Secret management services – For secure handling of sensitive configuration

The best approach depends on your application's complexity, deployment environment, and security requirements. For simple scripts, a Python file may be sufficient. For microservices in containerized environments, environment variables or a dedicated configuration service might be more appropriate.

Additional Resources

• The Twelve-Factor App: Config
• Python configparser documentation
• Python-dotenv on GitHub
• Dynaconf documentation
• Python-decouple documentation

Exercises

1. Create a simple Python application that reads configuration from a YAML file.
2. Modify the application to fall back to environment variables if the configuration file is not found.
3. Implement a configuration scheme that has different settings for development, testing, and production environments.
4. Create a Flask application that uses Dynaconf for configuration management.
5. Implement a secure way to handle database credentials in a Python application.

By mastering configuration management in Python, you'll build more flexible, secure, and maintainable applications that can be easily deployed across different environments.