Skip to main content

CI/CD Secret Management

Introduction​

When building modern applications, you'll inevitably need to manage sensitive information like API keys, database credentials, and encryption keys. While developing locally, you might store these in a .env file or even hardcode them (not recommended!). But what happens when you set up CI/CD pipelines to automate your build, test, and deployment processes?

This is where secret management becomes crucial. Mishandling secrets in your CI/CD pipelines can lead to devastating security breaches, data leaks, and compromised systems. In this guide, we'll explore how to properly manage secrets in CI/CD workflows to maintain both security and automation.

Why Secret Management Matters​

Consider this scenario: A developer accidentally commits an AWS access key to a public GitHub repository. Within minutes, automated bots scanning GitHub find this key and use it to spin up expensive cloud resources for cryptocurrency mining. This has happened countless times, costing companies thousands of dollars and compromising their infrastructure.

Secret management in CI/CD aims to prevent such scenarios by providing secure ways to:

  1. Store sensitive information
  2. Make secrets available to CI/CD processes when needed
  3. Limit access to secrets based on the principle of least privilege
  4. Rotate and revoke secrets when necessary

Common Secret Management Anti-patterns​

Before diving into the right approaches, let's understand what not to do:

❌ Hardcoding Secrets in Source Code​

javascript
// Never do this!
const apiKey = "sk_live_51HxSZFA8rjKL6k6K7oKwXlVCZ";
const dbPassword = "super_secure_password123";

❌ Storing Secrets in Configuration Files​

yaml
# Don't commit this file to version control!
database:
  username: admin
  password: p@ssw0rd

❌ Logging Secrets in CI/CD Output​

bash
echo "Using API key: $API_KEY" # This will expose your secret in logs

❌ Embedding Secrets in Docker Images​

dockerfile
# Never embed secrets in your Dockerfile
ENV DATABASE_PASSWORD=mypassword

Secure Approaches to Secret Management​

Now, let's explore secure approaches to managing secrets in CI/CD pipelines:

1. Environment Variables​

Most CI/CD platforms offer a way to set secrets as environment variables that are accessible only during pipeline execution.

Example in GitHub Actions:

yaml
# .github/workflows/deploy.yml
name: Deploy Application

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Deploy to production
        env:
          API_KEY: ${{ secrets.API_KEY }}
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
        run: |
          # The secrets are now available as environment variables
          npm run deploy

Example in GitLab CI:

yaml
# .gitlab-ci.yml
deploy:
  stage: deploy
  script:
    - echo "Deploying application..."
    - npm run deploy
  variables:
    # Reference variables stored in GitLab CI/CD settings
    DATABASE_URL: $DATABASE_URL
    API_KEY: $API_KEY

2. Secret Management Services​

For more advanced needs, dedicated secret management services provide additional security features:

  • HashiCorp Vault
  • AWS Secrets Manager
  • Google Secret Manager
  • Azure Key Vault

Here's how you might use HashiCorp Vault in a CI/CD pipeline:

yaml
# .github/workflows/deploy.yml
name: Deploy with Vault

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Fetch secrets from Vault
        id: vault
        uses: hashicorp/vault-action@v2
        with:
          url: https://vault.company.com
          token: ${{ secrets.VAULT_TOKEN }}
          secrets: |
            secret/data/myapp/prod API_KEY | API_KEY
            secret/data/myapp/prod DB_PASSWORD | DB_PASSWORD
      
      - name: Deploy application
        run: |
          # Secrets are available as environment variables
          npm run deploy

3. Using .env Files (Securely)​

If you must use .env files, ensure they are:

  • Never committed to version control
  • Generated securely during CI/CD execution
  • Deleted after use
yaml
# .github/workflows/deploy.yml
name: Deploy with .env file

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Create .env file
        run: |
          echo "API_KEY=${{ secrets.API_KEY }}" > .env
          echo "DATABASE_URL=${{ secrets.DATABASE_URL }}" >> .env
      
      - name: Build and deploy
        run: npm run deploy
          
      - name: Clean up
        run: rm .env

Secret Rotation and Lifecycle Management​

Managing secrets isn't just about storing them securely—it's also about maintaining them over time.

Implement Regular Secret Rotation​

bash
# Script to rotate database passwords
#!/bin/bash

# Generate new password
NEW_PASSWORD=$(openssl rand -base64 32)

# Update password in database
mysql -u admin -p"$CURRENT_PASSWORD" -e "ALTER USER 'app_user'@'%' IDENTIFIED BY '$NEW_PASSWORD';"

# Update password in secret management system
aws secretsmanager update-secret --secret-id my-db-secret --secret-string "{\"password\":\"$NEW_PASSWORD\"}"

Use Temporary or Just-in-Time Credentials​

Many cloud platforms allow for temporary credentials that automatically expire. For example, AWS STS (Security Token Service) provides short-lived credentials:

javascript
// Example of using temporary AWS credentials
const AWS = require('aws-sdk');
const sts = new AWS.STS();

async function getTemporaryCredentials() {
  const params = {
    RoleArn: 'arn:aws:iam::123456789012:role/DeployRole',
    RoleSessionName: 'DeploySession',
    DurationSeconds: 3600 // 1 hour
  };
  
  const result = await sts.assumeRole(params).promise();
  return {
    accessKeyId: result.Credentials.AccessKeyId,
    secretAccessKey: result.Credentials.SecretAccessKey,
    sessionToken: result.Credentials.SessionToken
  };
}

Setting Up Secret Management for Different CI/CD Systems​

Let's look at how to set up secret management in some popular CI/CD platforms:

GitHub Actions​

Step-by-step setup:

  1. Go to your GitHub repository → Settings → Secrets and variables → Actions
  2. Click "New repository secret"
  3. Enter a name (e.g., API_KEY) and the secret value
  4. Reference in your workflow file as ${{ secrets.API_KEY }}

GitLab CI/CD​

Step-by-step setup:

  1. Go to your GitLab project → Settings → CI/CD → Variables
  2. Click "Add variable"
  3. Enter a key, value, and set "Mask variable" to hide it in logs
  4. Reference in your .gitlab-ci.yml as $VARIABLE_NAME

Jenkins​

Step-by-step setup:

  1. Install the Jenkins Credentials Plugin
  2. Go to Manage Jenkins → Manage Credentials
  3. Add a new credential (username/password, secret text, etc.)
  4. Use in Jenkinsfile:
groovy
// Jenkinsfile
pipeline {
    agent any
    
    environment {
        // Define credentials using the credentials() function
        DATABASE_CREDS = credentials('database-credentials')
    }
    
    stages {
        stage('Deploy') {
            steps {
                // Access credentials as environment variables
                sh 'deploy.sh'
                // DATABASE_CREDS_USR and DATABASE_CREDS_PSW are available
            }
        }
    }
}

Best Practices for CI/CD Secret Management​

1. Implement Strict Access Controls​

Limit who can access and manage secrets:

yaml
# GitHub repository permission example
name: Secret access protection

on:
  pull_request:
    paths:
      - '.github/workflows/**'

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Check for secrets access
        run: |
          if grep -r "secrets\." .github/workflows/; then
            echo "Potential secrets access in PR"
            exit 1
          fi

2. Audit Secret Usage​

Regularly monitor and audit who is accessing secrets and when:

bash
# AWS CloudTrail query to detect secret access
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetSecretValue \
  --start-time "2023-01-01" \
  --end-time "2023-12-31"

3. Use Secrets Scanning Tools​

Implement scanning tools to detect accidentally committed secrets:

yaml
# .github/workflows/secret-scanning.yml
name: Secret Scanning

on: [push, pull_request]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Run Gitleaks
        uses: zricethezav/gitleaks-action@master

4. Separate Secrets by Environment​

Keep production secrets separate from development and staging:

yaml
# Example of environment-specific secrets in GitHub Actions
name: Deploy

on:
  push:
    branches:
      - main
      - staging

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ github.ref == 'refs/heads/main' && 'production' || 'staging' }}
    steps:
      - uses: actions/checkout@v3
      
      - name: Deploy to environment
        env:
          # These will be different values depending on the environment
          API_KEY: ${{ secrets.API_KEY }}
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
        run: npm run deploy

Real-world Example: Complete CI/CD Pipeline with Secret Management​

Let's tie everything together with a comprehensive example of a Node.js application deployment with secure secret management:

yaml
# .github/workflows/deploy.yml
name: Deploy Application

on:
  push:
    branches: [main]

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
          
      - name: Install dependencies
        run: npm ci
        
      - name: Run tests
        run: npm test
        
  security-scan:
    needs: build-and-test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Run dependency vulnerability scan
        run: npm audit
        
      - name: Run secret scanning
        uses: zricethezav/gitleaks-action@master
        
  deploy:
    needs: security-scan
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
          
      - name: Install dependencies
        run: npm ci
        
      - name: Get temporary AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          role-to-assume: arn:aws:iam::123456789012:role/DeployRole
          aws-region: us-east-1
          
      - name: Get database credentials from AWS Secrets Manager
        run: |
          DB_CREDS=$(aws secretsmanager get-secret-value --secret-id prod/database --query SecretString --output text)
          echo "DB_USER=$(echo $DB_CREDS | jq -r .username)" >> $GITHUB_ENV
          echo "DB_PASSWORD=$(echo $DB_CREDS | jq -r .password)" >> $GITHUB_ENV
        
      - name: Create configuration
        run: |
          cat > config.json << EOF
          {
            "database": {
              "host": "${{ secrets.DB_HOST }}",
              "user": "$DB_USER",
              "password": "$DB_PASSWORD"
            },
            "api": {
              "key": "${{ secrets.API_KEY }}"
            }
          }
          EOF
          
      - name: Deploy application
        run: npm run deploy
        
      - name: Clean up
        run: rm config.json

Extending Secret Management Beyond CI/CD​

The principles of secret management extend beyond CI/CD into your application runtime:

Kubernetes Secrets​

yaml
# kubernetes-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  db-password: cGFzc3dvcmQxMjM=  # base64 encoded "password123"
  api-key: c2sxMjM0NTY3ODkwMTIzNDU2Nzg5MA==  # base64 encoded key
yaml
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        image: my-app:latest
        env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: db-password
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: api-key

Using External Secret Operators​

For more advanced Kubernetes deployments, consider using External Secret Operators:

yaml
# external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: "15m"
  secretStoreRef:
    name: aws-secretsmanager
    kind: ClusterSecretStore
  target:
    name: database-credentials
  data:
  - secretKey: username
    remoteRef:
      key: prod/database
      property: username
  - secretKey: password
    remoteRef:
      key: prod/database
      property: password

Summary​

Proper secret management is a critical component of secure CI/CD practices. In this guide, we've covered:

  • Why secret management is crucial for CI/CD security
  • Common anti-patterns to avoid
  • Secure approaches using environment variables and secret management services
  • Secret rotation and lifecycle management
  • Platform-specific setup for GitHub Actions, GitLab CI, and Jenkins
  • Best practices for access control, auditing, and scanning
  • Real-world examples with complete CI/CD pipelines
  • How to extend secret management to your application runtime

By implementing these practices, you'll significantly reduce the risk of secret leakage and strengthen your overall security posture.

Additional Resources​

Exercises​

  1. Set up a GitHub Actions workflow that fetches a secret from HashiCorp Vault and uses it in a deployment.
  2. Implement a secret rotation policy for your database credentials.
  3. Create a script to audit secret access in your cloud provider.
  4. Configure your CI/CD pipeline to run secret scanning tools before deployment.
  5. Design a secure process for managing secrets across development, staging, and production environments.

If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)