Python Environment Variables

Environment variables provide a powerful way to configure your Python applications across different environments. In this guide, we'll explore how to work with environment variables in Python, why they're essential for DevOps practices, and best practices for managing them in your applications.

Introduction to Environment Variables

Environment variables are dynamic-named values that can affect the behavior of running processes on a computer. They are part of the environment in which a process runs and provide a way to influence program behavior without modifying code.

In DevOps and modern application development, environment variables serve several important purposes:

Security: Keep sensitive data like API keys and passwords out of source code
Configuration: Adjust application behavior based on the environment (development, testing, production)
Flexibility: Change settings without redeploying or modifying code
Portability: Run the same code in different environments with different configurations

Accessing Environment Variables in Python

Python provides easy access to environment variables through the os module. Let's look at how to read and set environment variables in your Python applications.

Reading Environment Variables

The os.environ dictionary contains all current environment variables:

python
import os

# Get a specific environment variable
database_url = os.environ.get('DATABASE_URL')
print(f"Database URL: {database_url}")

# Using a default value if the variable doesn't exist
api_key = os.environ.get('API_KEY', 'default-key')
print(f"API Key: {api_key}")

# Get all environment variables
print("All environment variables:")
for key, value in os.environ.items():
    print(f"{key}={value}")

Output (will vary based on your system):

Database URL: None
API Key: default-key
All environment variables:
USER=johndoe
HOME=/home/johndoe
PATH=/usr/local/bin:/usr/bin:/bin
...

Setting Environment Variables

You can set environment variables in Python for the current process:

python
import os

# Set an environment variable
os.environ['APP_ENV'] = 'development'
print(f"App environment: {os.environ.get('APP_ENV')}")

# Change an existing variable
os.environ['APP_ENV'] = 'production'
print(f"App environment: {os.environ.get('APP_ENV')}")

Output:

App environment: development
App environment: production

Note that environment variables set this way only exist for the duration of the current process. They won't affect other processes or persist after your program exits.

Environment Variables from the Command Line

When running Python scripts, you can set environment variables directly from the command line:

On Linux/macOS:

bash
DATABASE_URL=postgres://localhost:5432/mydb python my_script.py

On Windows (Command Prompt):

cmd
set DATABASE_URL=postgres://localhost:5432/mydb
python my_script.py

On Windows (PowerShell):

powershell
$env:DATABASE_URL="postgres://localhost:5432/mydb"
python my_script.py

Using python-dotenv for Environment Variable Management

The python-dotenv package provides a convenient way to load environment variables from a .env file, which is especially useful during development. Let's see how to use it:

Installation

bash
pip install python-dotenv

Create a .env file

Create a file named .env in your project directory:

# Database Configuration
DATABASE_URL=postgres://localhost:5432/mydb
DATABASE_USER=admin
DATABASE_PASSWORD=secret123

# API Configuration
API_KEY=your_api_key_here
DEBUG=True

Load variables from .env file

python
import os
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

# Access variables
database_url = os.environ.get('DATABASE_URL')
api_key = os.environ.get('API_KEY')
debug = os.environ.get('DEBUG')

print(f"Database URL: {database_url}")
print(f"API Key: {api_key}")
print(f"Debug mode: {debug}")

Output:

Database URL: postgres://localhost:5432/mydb
API Key: your_api_key_here
Debug mode: True

Advanced dotenv usage

You can also specify a custom path for your .env file or override existing environment variables:

python
import os
from dotenv import load_dotenv

# Load from a specific file
load_dotenv('/path/to/custom/.env')

# Override existing environment variables (default is False)
load_dotenv(override=True)

Real-World Examples

Let's explore some practical examples of using environment variables in Python applications.

Example 1: Configuring a Database Connection

python
import os
import psycopg2
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

def get_database_connection():
    """Create a database connection based on environment variables."""
    try:
        connection = psycopg2.connect(
            host=os.environ.get('DB_HOST', 'localhost'),
            database=os.environ.get('DB_NAME'),
            user=os.environ.get('DB_USER'),
            password=os.environ.get('DB_PASSWORD'),
            port=os.environ.get('DB_PORT', '5432')
        )
        print("Database connection successful!")
        return connection
    except Exception as e:
        print(f"Error connecting to database: {e}")
        return None

# Usage
conn = get_database_connection()

Example 2: Environment-Specific Configuration

python
import os
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

class Config:
    """Base configuration class."""
    SECRET_KEY = os.environ.get('SECRET_KEY', 'fallback-dev-key')
    DEBUG = False
    TESTING = False
    DATABASE_URI = os.environ.get('DATABASE_URI')

class DevelopmentConfig(Config):
    """Development environment configuration."""
    DEBUG = True
    DATABASE_URI = os.environ.get('DEV_DATABASE_URI', 'sqlite:///dev.db')

class TestingConfig(Config):
    """Testing environment configuration."""
    TESTING = True
    DATABASE_URI = os.environ.get('TEST_DATABASE_URI', 'sqlite:///test.db')

class ProductionConfig(Config):
    """Production environment configuration."""
    # We should never use a fallback for production secrets
    SECRET_KEY = os.environ.get('SECRET_KEY')
    DATABASE_URI = os.environ.get('PROD_DATABASE_URI')

# Get the current configuration based on FLASK_ENV
env = os.environ.get('FLASK_ENV', 'development')

config_classes = {
    'development': DevelopmentConfig,
    'testing': TestingConfig,
    'production': ProductionConfig
}

# Get the appropriate config
config = config_classes.get(env, DevelopmentConfig)

print(f"Current environment: {env}")
print(f"Debug mode: {config.DEBUG}")
print(f"Database URI: {config.DATABASE_URI}")

Example 3: Working with API Keys

python
import os
import requests
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

def fetch_weather(city):
    """Fetch weather data from a weather API."""
    api_key = os.environ.get('WEATHER_API_KEY')
    
    if not api_key:
        raise ValueError("WEATHER_API_KEY environment variable not set")
        
    base_url = "https://api.weatherapi.com/v1/current.json"
    params = {
        'key': api_key,
        'q': city
    }
    
    try:
        response = requests.get(base_url, params=params)
        response.raise_for_status()
        return response.json()
    except requests.RequestException as e:
        print(f"Error fetching weather data: {e}")
        return None

# Usage
weather_data = fetch_weather("London")
if weather_data:
    print(f"Temperature in London: {weather_data['current']['temp_c']}°C")

Best Practices for Environment Variables

Never commit secrets to version control
- Keep .env files out of your repository by adding them to .gitignore
- Use separate .env files for different environments
Validate required environment variables
- Check for critical variables at startup and fail fast if they're missing
Use descriptive variable names
- Follow conventions like DATABASE_URL, API_KEY, etc.
- Consider prefixing variables specific to your application (MYAPP_DATABASE_URL)
Provide defaults where appropriate
- Use os.environ.get('VARIABLE', 'default') for non-critical variables
Document your environment variables
- Create a template .env.example file listing all required variables without values
- Add comments explaining each variable's purpose

Environment Variables in Containerized Python Applications

When working with Docker or Kubernetes, you'll need to provide environment variables to your containerized Python applications.

Docker Example

docker
# Dockerfile
FROM python:3.9-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

CMD ["python", "app.py"]

You can provide environment variables when running the container:

bash
docker run -e DATABASE_URL=postgres://user:pass@host:5432/db -e API_KEY=secret my-python-app

Or use an environment file:

bash
docker run --env-file .env my-python-app

Summary

Environment variables are a powerful mechanism for configuring Python applications securely and flexibly. We've explored:

Reading and setting environment variables with Python's os module
Using python-dotenv to manage environment variables in development
Real-world examples of environment variables for database connections, configuration systems, and API keys
Best practices for working with environment variables in Python applications
Handling environment variables in containerized environments

By leveraging environment variables effectively, you can create Python applications that are more secure, configurable, and ready for deployment in various environments – a key DevOps practice.

Additional Resources

Exercises

Create a Python script that displays the value of PATH environment variable.
Write a program that reads database credentials from environment variables and connects to a database.
Create a configuration system that uses environment variables to determine whether to run in development, testing, or production mode.
Use python-dotenv to create a script that loads different .env files based on the current environment.
Write a utility function that validates the presence of required environment variables and provides helpful error messages.

If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)

Introduction to Environment Variables​

Accessing Environment Variables in Python​

Reading Environment Variables​

Setting Environment Variables​

Environment Variables from the Command Line​

Using python-dotenv for Environment Variable Management​

Installation​

Create a .env file​

Load variables from .env file​

Advanced dotenv usage​

Real-World Examples​

Example 1: Configuring a Database Connection​

Example 2: Environment-Specific Configuration​

Example 3: Working with API Keys​

Best Practices for Environment Variables​

Environment Variables in Containerized Python Applications​

Docker Example​

Summary​

Additional Resources​

Exercises​