Python CI/CD Integration
Introduction
Continuous Integration and Continuous Deployment (CI/CD) are essential practices in modern software development that help teams deliver code changes more frequently and reliably. For Python developers, integrating CI/CD into projects means automating testing, linting, building, and deployment processes to ensure high-quality code reaches production quickly and safely.
In this tutorial, you'll learn:
- What CI/CD is and why it matters for Python projects
- How to set up basic CI/CD pipelines for Python applications
- Popular CI/CD tools for Python development
- Best practices for Python CI/CD workflows
What is CI/CD?
Continuous Integration (CI) is the practice of frequently merging code changes into a shared repository, with automated tests to catch issues early.
Continuous Deployment (CD) extends this by automatically deploying all code changes to a testing or production environment after the build stage.
For Python projects, CI/CD brings several benefits:
- Automatically running tests on every code change
- Ensuring code style consistency with linters
- Building and packaging Python applications reliably
- Deploying to development, staging, or production environments
Getting Started with Python CI/CD
Prerequisites
Before setting up CI/CD for your Python project, you'll need:
- A Python project with tests (e.g., using pytest, unittest)
- A version control system (typically Git)
- A repository on GitHub, GitLab, Bitbucket, or similar platform
- Basic understanding of YAML for configuration files
Popular CI/CD Tools for Python
1. GitHub Actions
GitHub Actions is a popular choice for Python CI/CD due to its tight integration with GitHub repositories.
Example: Basic Python CI Workflow
Create a file at .github/workflows/python-ci.yml:
name: Python CI
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.8, 3.9, 3.10]
steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flake8 pytest
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Lint with flake8
run: |
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
- name: Test with pytest
run: |
pytest
This workflow:
- Triggers on pushes to the main branch or pull requests
- Tests across multiple Python versions
- Installs project dependencies
- Runs linting with flake8
- Executes tests with pytest
2. GitLab CI/CD
If you're using GitLab, you can create a .gitlab-ci.yml file in your project root:
image: python:3.9
stages:
- test
- deploy
before_script:
- pip install -r requirements.txt
test:
stage: test
script:
- pip install pytest pytest-cov
- pytest --cov=myproject
lint:
stage: test
script:
- pip install flake8
- flake8 myproject
deploy:
stage: deploy
script:
- pip install twine
- python setup.py sdist bdist_wheel
- twine upload dist/*
only:
- tags
This GitLab CI pipeline:
- Uses a Python 3.9 base image
- Defines test and deploy stages
- Runs tests with coverage reports
- Performs linting
- Deploys the package to PyPI when a tag is pushed
3. Jenkins
For teams with existing Jenkins infrastructure, you can create a Jenkinsfile for your Python project:
pipeline {
agent {
docker {
image 'python:3.9'
}
}
stages {
stage('Setup') {
steps {
sh 'pip install -r requirements.txt'
sh 'pip install pytest flake8'
}
}
stage('Lint') {
steps {
sh 'flake8 .'
}
}
stage('Test') {
steps {
sh 'pytest'
}
post {
always {
junit 'test-reports/*.xml'
}
}
}
stage('Deploy') {
when {
expression { env.TAG_NAME != null }
}
steps {
sh 'pip install twine'
sh 'python setup.py sdist bdist_wheel'
sh 'twine upload dist/*'
}
}
}
}
Building a Complete CI/CD Pipeline for Python
Now let's create a more comprehensive GitHub Actions workflow that includes testing, building, and deploying a Python application:
Example: Full Python Web App CI/CD Pipeline
name: Python Web App CI/CD
on:
push:
branches: [ main ]
tags:
- 'v*'
pull_request:
branches: [ main ]
jobs:
test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:13
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: test_db
ports:
- 5432:5432
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.9'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install pytest pytest-cov flake8
- name: Lint with flake8
run: |
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
flake8 . --count --max-complexity=10 --max-line-length=127 --statistics
- name: Test with pytest
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/test_db
run: |
pytest --cov=app --cov-report=xml
- name: Upload coverage report
uses: codecov/codecov-action@v3
with:
file: ./coverage.xml
build:
needs: test
runs-on: ubuntu-latest
if: success() && (github.event_name == 'push' && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v')))
steps:
- uses: actions/checkout@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Extract metadata
id: meta
uses: docker/metadata-action@v4
with:
images: yourusername/python-app
- name: Build and push
uses: docker/build-push-action@v4
with:
context: .
push: true
labels: ${{ steps.meta.outputs.labels }}
deploy:
needs: build
runs-on: ubuntu-latest
if: success() && startsWith(github.ref, 'refs/tags/v')
steps:
- name: Deploy to production
uses: appleboy/ssh-action@master
with:
host: ${{ secrets.SERVER_HOST }}
username: ${{ secrets.SERVER_USER }}
key: ${{ secrets.SSH_PRIVATE_KEY }}
script: |
cd /app
docker pull yourusername/python-app:latest
docker-compose down
docker-compose up -d
This comprehensive workflow:
- Sets up a PostgreSQL service for integration testing
- Runs linting and testing with coverage reports
- Builds and pushes a Docker image when tests pass
- Deploys to a production server when a version tag is pushed
Best Practices for Python CI/CD
1. Organize Your Python Project for CI/CD
A well-structured Python project makes CI/CD implementation easier:
my_project/
├── .github/workflows/ # GitHub Actions workflows
│ └── python-ci.yml
├── src/ # Project source code
│ └── my_project/
│ ├── __init__.py
│ └── app.py
├── tests/ # Test directory
│ ├── __init__.py
│ └── test_app.py
├── .gitignore
├── pyproject.toml # Modern Python project config
├── requirements.txt # Dependencies
└── README.md
2. Manage Dependencies Properly
Use dependency pinning to ensure consistent builds:
# requirements.txt
flask==2.2.3
SQLAlchemy==2.0.5
pytest==7.3.1
For development dependencies, consider using a separate file:
# requirements-dev.txt
-r requirements.txt
pytest==7.3.1
pytest-cov==4.1.0
flake8==6.0.0
3. Test Configuration
Create a pytest.ini file to configure your test environment:
[pytest]
testpaths = tests
python_files = test_*.py
python_functions = test_*
python_classes = Test*
addopts = --cov=src --cov-report=term-missing
4. Environment Variables and Secrets
Store sensitive information as secrets in your CI/CD platform:
# GitHub Actions example
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
API_KEY: ${{ secrets.API_KEY }}
Example: Real-World Python Web Application CI/CD
Let's create a complete CI/CD pipeline for a Flask web application:
Project Setup
flask_app/
├── .github/workflows/
│ └── deploy.yml
├── app/
│ ├── __init__.py
│ ├── routes.py
│ ├── models.py
│ └── templates/
├── tests/
│ ├── test_routes.py
│ └── test_models.py
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── wsgi.py
Sample Flask Application (app/__init__.py)
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
def create_app(config=None):
app = Flask(__name__)
app.config.from_mapping(
SECRET_KEY='dev',
SQLALCHEMY_DATABASE_URI='sqlite:///app.db',
SQLALCHEMY_TRACK_MODIFICATIONS=False,
)
if config:
app.config.update(config)
db.init_app(app)
from . import routes
app.register_blueprint(routes.bp)
return app
Dockerfile
FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 5000
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "wsgi:app"]
CI/CD Workflow (.github/workflows/deploy.yml)
name: Flask App CI/CD
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.9'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install pytest pytest-cov
- name: Run tests
run: |
pytest --cov=app tests/
deploy:
needs: test
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
- name: Deploy to Heroku
uses: akhileshns/heroku-[email protected]
with:
heroku_api_key: ${{ secrets.HEROKU_API_KEY }}
heroku_app_name: ${{ secrets.HEROKU_APP_NAME }}
heroku_email: ${{ secrets.HEROKU_EMAIL }}
usedocker: true
Advanced CI/CD Techniques for Python
1. Matrix Testing
Test across multiple Python versions and operating systems:
jobs:
test:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: [3.8, 3.9, 3.10, 3.11]
steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
# ...rest of test job
2. Caching Dependencies
Speed up CI builds by caching pip packages:
- name: Cache pip packages
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-
3. Testing Database Migrations
Ensure your database migrations work correctly:
- name: Test migrations
run: |
flask db upgrade
flask db downgrade
flask db upgrade
Summary
Integrating CI/CD into your Python projects significantly improves code quality and deployment efficiency. In this tutorial, you've learned:
- The fundamentals of CI/CD for Python applications
- How to set up CI/CD pipelines using GitHub Actions, GitLab CI, and Jenkins
- Best practices for organizing Python projects for CI/CD
- How to create comprehensive workflows for testing, building, and deploying Python applications
By implementing CI/CD in your Python projects, you'll catch bugs earlier, ensure consistent code quality, and streamline your deployment process.
Additional Resources
- GitHub Actions Documentation
- GitLab CI/CD Documentation
- pytest Documentation
- Python Packaging User Guide
Exercises
- Set up a basic CI workflow for an existing Python project using GitHub Actions.
- Modify the workflow to test against multiple Python versions.
- Add a deployment step to your CI/CD pipeline that deploys your application to a cloud provider (e.g., Heroku, AWS, or Google Cloud).
- Implement dependency caching to speed up your CI process.
- Configure a workflow that publishes your Python package to PyPI when you create a new release.
By completing these exercises, you'll gain practical experience with Python CI/CD integration that you can apply to your own projects.
If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)