CICD Security Pipeline

Introduction

A CI/CD Security Pipeline (or DevSecOps Pipeline) integrates security practices into your Continuous Integration and Continuous Deployment processes. Rather than treating security as an afterthought or final check, a security pipeline embeds security testing and verification at every stage of your development lifecycle.

By implementing a security pipeline, you can:

Detect vulnerabilities early in development
Automatically enforce security policies
Create auditable security records
Prevent security issues from reaching production
Build security awareness across your development team

This guide will show you how to create a basic CI/CD security pipeline for your projects, even if you're just getting started with CI/CD concepts.

Understanding the Secure CI/CD Pipeline

A secure CI/CD pipeline includes security checks at every stage of development:

Let's break down each of these security components and learn how to implement them.

1. Static Code Analysis (SAST)

Static Application Security Testing analyzes your source code without executing it, looking for security vulnerabilities like SQL injection, XSS, and more.

Example: Setting up a basic SAST scan with SonarQube in GitHub Actions

yaml
name: SAST Scan

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  sonarqube:
    name: SonarQube Scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: SonarQube Scan
        uses: SonarSource/sonarqube-scan-action@master
        env:
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
          SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}

SonarQube results will show issues like:

CRITICAL: SQL Injection vulnerability found in app/controllers/users_controller.rb:45
MAJOR: Cross-site scripting (XSS) vulnerability in app/views/posts/show.html.erb:12

2. Dependency Scanning (SCA)

Software Composition Analysis checks for vulnerabilities in your project dependencies and third-party libraries.

Example: Setting up dependency scanning with OWASP Dependency-Check

yaml
name: Dependency Scan

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  dependency-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run OWASP Dependency-Check
        uses: dependency-check/Dependency-Check_Action@main
        with:
          project: 'My Project'
          path: '.'
          format: 'HTML'
          out: 'reports' 
          args: >
            --failOnCVSS 7
            --enableRetired
      - name: Upload report
        uses: actions/upload-artifact@v3
        with:
          name: Dependency Check Report
          path: ${{ github.workspace }}/reports

Output example:

CVE-2022-22965: Spring Framework RCE vulnerability (CVSS: 9.8) found in spring-core-5.3.10.jar
CVE-2021-44228: Log4Shell vulnerability (CVSS: 10.0) found in log4j-core-2.14.0.jar

3. Secret Scanning

Prevent sensitive credentials like API keys and passwords from being committed to your repository.

Example: Using pre-commit hooks with git-secrets

First, install git-secrets:

bash
git clone https://github.com/awslabs/git-secrets.git
cd git-secrets
make install

Set up in your repository:

bash
cd your-repository
git secrets --install
git secrets --register-aws # Adds patterns for AWS keys

Add custom patterns:

bash
git secrets --add 'private_key'
git secrets --add 'api_key[[:space:]]*=[[:space:]]*[[:alnum:]]+'
git secrets --add --allowed 'AKIAIOSFODNN7EXAMPLE' # Allow test examples

Now, when you try to commit code with secrets:

bash
$ git commit -m "Add new feature"
git-secrets: (ERROR) Matched prohibited pattern
--
api_key = a8a7sd8a79sd87as98d7
--

4. Container Scanning

If you're using containers, you need to scan them for vulnerabilities before deployment.

Example: Scanning Docker containers with Trivy in GitLab CI

yaml
container_scanning:
  stage: test
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  variables:
    DOCKER_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  script:
    - trivy image --severity HIGH,CRITICAL $DOCKER_IMAGE
  only:
    - main

Example output:

CRITICAL: CVE-2023-12345 - Buffer overflow in OpenSSL 1.1.1n affects nginx:1.21
HIGH: CVE-2023-67890 - Remote code execution in Python 3.9.10

5. Dynamic Application Security Testing (DAST)

DAST tests running applications by simulating attacks from the outside.

Example: Setting up OWASP ZAP scanning in Jenkins

groovy
pipeline {
    agent any
    
    stages {
        stage('Deploy Test Environment') {
            steps {
                sh 'docker-compose up -d app'
                sh 'sleep 30' // Wait for app to start
            }
        }
        
        stage('DAST Scan') {
            steps {
                sh '''
                docker run --rm -v $(pwd):/zap/wrk/:rw -t owasp/zap2docker-stable zap-baseline.py \
                    -t http://host.docker.internal:8080 \
                    -r dast-report.html
                '''
            }
        }
    }
    
    post {
        always {
            archiveArtifacts artifacts: 'dast-report.html', fingerprint: true
            sh 'docker-compose down'
        }
    }
}

ZAP will generate a report with findings like:

ALERT: [Cross-Site Scripting (Reflected)] http://host.docker.internal:8080/search?q=test%3Cscript%3Ealert(1)%3C/script%3E
ALERT: [X-Content-Type-Options Header Missing] http://host.docker.internal:8080

6. Infrastructure as Code Scanning

Scan your infrastructure definitions for security issues and compliance violations.

Example: Scanning Terraform code with tfsec

yaml
name: Infrastructure Scanning

on:
  push:
    paths:
      - '**.tf'

jobs:
  tfsec:
    name: TFSec Infrastructure Scan
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
      
      - name: Run tfsec
        uses: aquasecurity/tfsec-[email protected]
        with:
          soft_fail: true

Sample output:

CRITICAL: AWS S3 Bucket has no encryption defined (terraform/s3.tf:15)
HIGH: Security group rule allows traffic from public internet (terraform/security_groups.tf:22)

Building a Complete Security Pipeline

Now let's create a complete security pipeline example using GitHub Actions that incorporates several security checks:

yaml
name: Secure CI/CD Pipeline

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  static-analysis:
    name: Static Code Analysis
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'
      - name: Install Bandit
        run: pip install bandit
      - name: Run Bandit
        run: bandit -r ./src -f json -o bandit-results.json
      - name: Upload Bandit results
        uses: actions/upload-artifact@v3
        with:
          name: bandit-results
          path: bandit-results.json

  dependency-check:
    name: Dependency Scanning
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'
      - name: Install dependencies
        run: pip install safety
      - name: Run safety check
        run: safety check --json > safety-results.json
      - name: Upload Safety results
        uses: actions/upload-artifact@v3
        with:
          name: safety-results
          path: safety-results.json

  secret-scanning:
    name: Secret Scanning
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Detect Secrets
        uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  build-scan:
    name: Build and Image Scan
    needs: [static-analysis, dependency-check, secret-scanning]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Build Docker image
        run: docker build -t my-app:${{ github.sha }} .
      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'my-app:${{ github.sha }}'
          format: 'sarif'
          output: 'trivy-results.sarif'
      - name: Upload Trivy scan results
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-results.sarif'

  deploy-staging:
    name: Deploy to Staging
    needs: build-scan
    runs-on: ubuntu-latest
    if: ${{ success() && github.event_name == 'push' && github.ref == 'refs/heads/main' }}
    steps:
      - uses: actions/checkout@v3
      - name: Deploy to staging
        run: echo "Deploying to staging environment"
        # Add your deployment script here
  
  dast-scan:
    name: Dynamic Application Security Testing
    needs: deploy-staging
    runs-on: ubuntu-latest
    if: ${{ success() && github.event_name == 'push' && github.ref == 'refs/heads/main' }}
    steps:
      - name: Run ZAP Scan
        uses: zaproxy/action-[email protected]
        with:
          target: 'https://staging-app-url.example.com'

Best Practices for CI/CD Security Pipelines

Shift Left: Move security scanning as early in the development process as possible
Fail Fast: Configure your pipeline to fail on high-severity issues
Scan Dependencies Regularly: Schedule regular scans even without code changes
Maintain Allowlists: Add known false positives to allowlists to reduce noise
Centralize Results: Use a central dashboard to track security findings
Automate Remediation: Where possible, automate security fixes
Scan Infrastructure: Don't forget to scan your infrastructure code
Customize Rules: Tune security tools to match your project's risk profile

CI/CD Security Pipeline Implementation Steps

Let's walk through implementing a basic security pipeline for a Python web application:

1. Create a baseline security configuration file

First, create a .security-baseline.yml file in your project:

yaml
# .security-baseline.yml
security:
  sast:
    enabled: true
    fail_on: critical
  dependency_check:
    enabled: true
    fail_on: high
  secrets:
    enabled: true
    fail_on: any
  container_scan:
    enabled: true
    fail_on: critical
  dast:
    enabled: true
    fail_on: critical

2. Set up pre-commit hooks for local security

Create a .pre-commit-config.yaml file:

yaml
# .pre-commit-config.yaml
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0
    hooks:
    -   id: trailing-whitespace
    -   id: end-of-file-fixer
    -   id: check-yaml
    -   id: check-added-large-files

-   repo: https://github.com/pycqa/bandit
    rev: 1.7.5
    hooks:
    -   id: bandit
        args: ['-ll']

-   repo: https://github.com/gitleaks/gitleaks
    rev: v8.16.3
    hooks:
    -   id: gitleaks

Install pre-commit:

bash
pip install pre-commit
pre-commit install

3. Integrate with CI/CD (using GitHub Actions example)

Create .github/workflows/security-pipeline.yml:

yaml
name: Security Pipeline

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main, develop ]
  schedule:
    - cron: '0 0 * * 0'  # Weekly scan

jobs:
  security-checks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'
      
      - name: Load security baseline config
        id: config
        run: |
          echo "::set-output name=sast_enabled::$(yq e '.security.sast.enabled' .security-baseline.yml)"
          echo "::set-output name=deps_enabled::$(yq e '.security.dependency_check.enabled' .security-baseline.yml)"
      
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install bandit safety
          if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
      
      - name: Run SAST scan
        if: steps.config.outputs.sast_enabled == 'true'
        run: bandit -r . -f json -o bandit-results.json
      
      - name: Run dependency check
        if: steps.config.outputs.deps_enabled == 'true'
        run: safety check --json > safety-results.json
      
      - name: Secret scanning
        uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      
      # Add more security checks as needed
      
      - name: Upload security results
        uses: actions/upload-artifact@v3
        with:
          name: security-results
          path: |
            bandit-results.json
            safety-results.json

4. Add a security reporting dashboard

You can use tools like OWASP DefectDojo, Sonar, or ThreadFix to aggregate security results. For a beginner example, let's create a simple HTML report:

yaml
  generate-report:
    needs: security-checks
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v3
        with:
          name: security-results
      
      - name: Generate HTML report
        run: |
          echo '<html><head><title>Security Report</title>' > report.html
          echo '<style>body{font-family:sans-serif}table{border-collapse:collapse;width:100%}td,th{border:1px solid #ddd;padding:8px}</style>' >> report.html
          echo '</head><body><h1>Security Scan Results</h1>' >> report.html
          
          echo '<h2>SAST Results</h2>' >> report.html
          echo '<table><tr><th>Severity</th><th>File</th><th>Issue</th></tr>' >> report.html
          cat bandit-results.json | jq -r '.results[] | "<tr><td>\(.issue_severity)</td><td>\(.filename)</td><td>\(.issue_text)</td></tr>"' >> report.html
          echo '</table>' >> report.html
          
          echo '<h2>Dependency Results</h2>' >> report.html
          echo '<table><tr><th>Severity</th><th>Package</th><th>Vulnerability</th></tr>' >> report.html
          cat safety-results.json | jq -r '.[] | "<tr><td>\(.[5])</td><td>\(.[0])</td><td>\(.[3])</td></tr>"' >> report.html
          echo '</table>' >> report.html
          
          echo '</body></html>' >> report.html
      
      - name: Upload HTML report
        uses: actions/upload-artifact@v3
        with:
          name: security-report
          path: report.html

Security Gates: Making Decisions Based on Results

Security gates determine whether a build should proceed based on security findings:

yaml
  security-gate:
    needs: [security-checks, generate-report]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v3
        with:
          name: security-results
      
      - name: Check SAST critical findings
        run: |
          CRITICAL_COUNT=$(cat bandit-results.json | jq '.results[] | select(.issue_severity=="HIGH" or .issue_severity=="CRITICAL")' | jq -s 'length')
          if [ $CRITICAL_COUNT -gt 0 ]; then
            echo "::error::$CRITICAL_COUNT critical security issues found! Pipeline failed."
            exit 1
          fi
      
      - name: Check dependency critical findings
        run: |
          CRITICAL_COUNT=$(cat safety-results.json | jq '.[] | select(.[5]=="Critical" or .[5]=="High")' | jq -s 'length')
          if [ $CRITICAL_COUNT -gt 0 ]; then
            echo "::error::$CRITICAL_COUNT critical dependency issues found! Pipeline failed."
            exit 1
          fi

Real-world Implementation Example

Let's implement a security pipeline for a Python Flask application:

Project structure:

my-flask-app/
├── .github/
│   └── workflows/
│       └── security-pipeline.yml
├── app/
│   ├── __init__.py
│   ├── models.py
│   ├── routes.py
│   └── templates/
│       └── index.html
├── tests/
│   └── test_app.py
├── .pre-commit-config.yaml
├── .security-baseline.yml
├── Dockerfile
├── requirements.txt
└── run.py

Install security checks locally:

bash
pip install bandit safety pre-commit
pre-commit install

Add a vulnerable dependency in requirements.txt for demonstration:

flask==2.0.1
sqlalchemy==1.4.23
werkzeug==2.0.1
pyyaml==5.4.1  # Has known vulnerabilities

Add insecure code in app/routes.py:

python
@app.route('/search')
def search():
    query = request.args.get('q')
    # INSECURE: Vulnerable to SQL Injection
    result = db.engine.execute(f"SELECT * FROM products WHERE name LIKE '%{query}%'")
    return render_template('results.html', results=result)

Run security checks manually:

bash
# SAST check
bandit -r ./app

# Dependency check
safety check -r requirements.txt

The pipeline will automatically find these issues when you push to your repository.

Summary

Implementing a CI/CD security pipeline brings security practices into your development workflow, helping you identify and fix security issues early. Key components include:

Static Analysis (SAST) - Finding code vulnerabilities
Dependency Scanning (SCA) - Detecting vulnerable libraries
Secret Scanning - Preventing credential leaks
Container Scanning - Ensuring secure container images
Dynamic Testing (DAST) - Testing running applications
Infrastructure Scanning - Securing your infrastructure code

By implementing these practices in your CI/CD pipeline, you create a robust security process that catches issues early and builds security awareness in your development team.

Additional Resources

Exercises

Basic: Set up a pre-commit hook with Bandit in a Python project.
Intermediate: Implement a full GitHub Actions workflow with SAST and dependency scanning.
Advanced: Create a complete DevSecOps pipeline with security gates and reporting dashboards.

Try implementing these security checks in your own CI/CD pipeline and see how many vulnerabilities you can catch before they reach production!

If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)

Introduction​

Understanding the Secure CI/CD Pipeline​

1. Static Code Analysis (SAST)​

Example: Setting up a basic SAST scan with SonarQube in GitHub Actions​

2. Dependency Scanning (SCA)​

Example: Setting up dependency scanning with OWASP Dependency-Check​

3. Secret Scanning​

Example: Using pre-commit hooks with git-secrets​

4. Container Scanning​

Example: Scanning Docker containers with Trivy in GitLab CI​

5. Dynamic Application Security Testing (DAST)​

Example: Setting up OWASP ZAP scanning in Jenkins​

6. Infrastructure as Code Scanning​

Example: Scanning Terraform code with tfsec​

Building a Complete Security Pipeline​

Best Practices for CI/CD Security Pipelines​

CI/CD Security Pipeline Implementation Steps​

1. Create a baseline security configuration file​

2. Set up pre-commit hooks for local security​

3. Integrate with CI/CD (using GitHub Actions example)​

4. Add a security reporting dashboard​

Security Gates: Making Decisions Based on Results​

Real-world Implementation Example​

Summary​

Additional Resources​

Exercises​