Echo Graceful Shutdown

Introduction

When running a web server in production, properly handling the shutdown process is critical. A graceful shutdown ensures that your application terminates safely without interrupting ongoing operations or losing data. This is especially important in containerized environments, Kubernetes deployments, or when you need to perform server updates without impacting users.

In this tutorial, we'll explore how to implement graceful shutdown patterns in Echo applications. You'll learn how to:

Understand why graceful shutdown is necessary
Capture termination signals from the operating system
Allow ongoing requests to complete before shutting down
Close database connections and other resources properly

Why Graceful Shutdown Matters

Imagine your web server is handling hundreds of requests when you need to deploy a new version. Without proper shutdown handling:

In-flight requests might be abruptly terminated
Database transactions could be left incomplete
File operations might be interrupted, causing data corruption
Connected clients might experience unexpected errors

A graceful shutdown solves these issues by:

Stopping the server from accepting new requests
Allowing in-progress requests to complete normally
Closing database connections and other resources orderly
Only then terminating the application

Basic Implementation

Let's start with a simple implementation of graceful shutdown in an Echo application:

go
package main

import (
	"context"
	"net/http"
	"os"
	"os/signal"
	"time"

	"github.com/labstack/echo/v4"
	"github.com/labstack/gommon/log"
)

func main() {
	// Create a new Echo instance
	e := echo.New()
	e.Logger.SetLevel(log.INFO)
	
	// Routes
	e.GET("/", func(c echo.Context) error {
		return c.String(http.StatusOK, "Hello, World!")
	})
	
	e.GET("/slow", func(c echo.Context) error {
		// Simulate a slow request
		time.Sleep(5 * time.Second)
		return c.String(http.StatusOK, "Slow operation completed")
	})

	// Start server in a goroutine
	go func() {
		if err := e.Start(":8080"); err != nil && err != http.ErrServerClosed {
			e.Logger.Fatal("shutting down the server")
		}
	}()

	// Wait for interrupt signal to gracefully shutdown
	quit := make(chan os.Signal, 1)
	signal.Notify(quit, os.Interrupt)
	<-quit
	
	// Create a deadline for shutdown
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()
	
	// Attempt graceful shutdown
	if err := e.Shutdown(ctx); err != nil {
		e.Logger.Fatal(err)
	}
}

Let's break down the key components:

We start the Echo server in a separate goroutine so our main function can continue execution
We create a channel to receive OS interruption signals (like CTRL+C)
The program blocks until it receives an interrupt signal
Once a signal is received, we create a context with a timeout (10 seconds)
We call e.Shutdown(ctx) to gracefully shutdown the server
If the shutdown exceeds the timeout, the application will forcefully terminate

Testing Graceful Shutdown

To test this behavior:

Start your application
Make a request to the /slow endpoint that takes 5 seconds to complete
While that request is in progress, press CTRL+C to initiate the shutdown
Observe that the slow request still completes successfully

Terminal output will look like:

⇨ http server started on [::]:8080
^C2023/08/14 15:04:32 Received interrupt signal, shutting down...
2023/08/14 15:04:32 Server is shutting down...
2023/08/14 15:04:37 Slow request completed
2023/08/14 15:04:37 Server gracefully stopped

Advanced Implementation with Resource Cleanup

In real-world applications, you'll need to clean up resources like database connections and external services before shutting down. Here's a more comprehensive example:

go
package main

import (
	"context"
	"database/sql"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/labstack/echo/v4"
	"github.com/labstack/gommon/log"
	_ "github.com/lib/pq"
)

type App struct {
	echo   *echo.Echo
	db     *sql.DB
	logger echo.Logger
}

func NewApp() *App {
	e := echo.New()
	e.Logger.SetLevel(log.INFO)

	// Initialize DB (example)
	db, err := sql.Open("postgres", "postgresql://user:password@localhost/dbname?sslmode=disable")
	if err != nil {
		e.Logger.Fatal(err)
	}

	return &App{
		echo:   e,
		db:     db,
		logger: e.Logger,
	}
}

func (a *App) setupRoutes() {
	a.echo.GET("/", func(c echo.Context) error {
		return c.String(http.StatusOK, "Hello, World!")
	})

	a.echo.GET("/db", func(c echo.Context) error {
		// Example DB query
		var now string
		err := a.db.QueryRowContext(c.Request().Context(), "SELECT NOW()").Scan(&now)
		if err != nil {
			return err
		}
		return c.String(http.StatusOK, "DB Time: "+now)
	})
}

func (a *App) Start() {
	a.setupRoutes()

	// Start server in goroutine
	go func() {
		if err := a.echo.Start(":8080"); err != nil && err != http.ErrServerClosed {
			a.logger.Fatal("shutting down the server: ", err)
		}
	}()

	// Wait for interrupt signal
	quit := make(chan os.Signal, 1)
	signal.Notify(quit, os.Interrupt, syscall.SIGTERM)
	<-quit
	a.logger.Info("Received shutdown signal")

	// Give server 10 seconds to shutdown gracefully
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()

	// Attempt graceful shutdown steps
	if err := a.echo.Shutdown(ctx); err != nil {
		a.logger.Error("Server shutdown error:", err)
	} else {
		a.logger.Info("Server gracefully stopped")
	}

	// Close DB connection
	if err := a.db.Close(); err != nil {
		a.logger.Error("Database closing error:", err)
	} else {
		a.logger.Info("Database connection closed")
	}
}

func main() {
	app := NewApp()
	app.Start()
}

This implementation:

Captures both SIGINT (CTRL+C) and SIGTERM signals (common in containerized environments)
Properly closes the database connection after the server shuts down
Logs each step of the shutdown process
Uses a structured approach that can be extended for additional cleanup tasks

Best Practices for Graceful Shutdown

1. Set Appropriate Timeouts

Choose a reasonable shutdown timeout for your application:

Too short: Ongoing requests may be interrupted
Too long: Deployment might be delayed unnecessarily

go
// Adjust timeout based on your application's needs
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)

2. Handle Multiple Signals

In production environments, especially in containers, handle multiple termination signals:

go
quit := make(chan os.Signal, 1)
signal.Notify(quit, os.Interrupt, syscall.SIGTERM, syscall.SIGQUIT)

3. Resource Cleanup Order

Close resources in the correct order:

First shut down the server (stop accepting new requests)
Then close resources like database connections, caches, message queues, etc.

4. Logging During Shutdown

Log each step of the shutdown process to make debugging easier:

go
logger.Info("Shutdown initiated")
// Shutdown server
logger.Info("Server shutdown complete")
// Close database
logger.Info("Database connections closed")
// etc.

5. Use Context Propagation

Use context propagation to ensure that long-running operations can be cancelled when needed:

go
func (s *Service) LongRunningOperation(ctx context.Context) error {
    // Check context regularly
    select {
    case <-ctx.Done():
        return ctx.Err()
    default:
        // Continue processing
    }
    // ...
}

Real-World Example: HTTP Server with Background Workers

Many applications have background workers alongside the HTTP server. Here's how to gracefully shut down both:

go
package main

import (
	"context"
	"net/http"
	"os"
	"os/signal"
	"sync"
	"syscall"
	"time"

	"github.com/labstack/echo/v4"
)

func main() {
	// Create Echo instance
	e := echo.New()
	
	// Setup routes
	e.GET("/", func(c echo.Context) error {
		return c.String(http.StatusOK, "Hello, World!")
	})
	
	// Create context that will be used to signal shutdown
	ctx, cancel := context.WithCancel(context.Background())

	// Start worker in background
	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		defer wg.Done()
		backgroundWorker(ctx)
	}()
	
	// Start server
	go func() {
		if err := e.Start(":8080"); err != nil && err != http.ErrServerClosed {
			e.Logger.Fatal("server error:", err)
		}
	}()
	
	// Wait for termination signal
	quit := make(chan os.Signal, 1)
	signal.Notify(quit, os.Interrupt, syscall.SIGTERM)
	<-quit
	
	// Cancel context to notify all workers to stop
	cancel()
	
	// Create timeout context for server shutdown
	shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer shutdownCancel()
	
	// Shutdown server
	if err := e.Shutdown(shutdownCtx); err != nil {
		e.Logger.Error("server shutdown error:", err)
	}
	
	// Wait for worker to finish
	wg.Wait()
	e.Logger.Info("All processes have been stopped. Exiting.")
}

func backgroundWorker(ctx context.Context) {
	ticker := time.NewTicker(1 * time.Second)
	defer ticker.Stop()
	
	for {
		select {
		case <-ctx.Done():
			fmt.Println("Worker received shutdown signal, finishing...")
			// Perform cleanup
			time.Sleep(2 * time.Second) // Simulate cleanup work
			fmt.Println("Worker shutdown complete")
			return
		case <-ticker.C:
			fmt.Println("Worker doing work...")
			// Do some work
		}
	}
}

This example demonstrates:

Using a shared context to coordinate shutdown between components
Using a WaitGroup to ensure all goroutines finish before exiting
Handling cleanup within each component

Summary

Implementing graceful shutdown in your Echo application is crucial for maintaining reliability, especially in production environments. It ensures that:

In-flight requests complete successfully
Resources are properly released
Your application behaves predictably during deployments or server restarts

By following the patterns described in this tutorial, you'll create more robust Echo applications that can handle the realities of production deployments and server management.

Additional Resources

Exercises

Basic Implementation: Modify the basic example to log how many requests were active during shutdown.
Custom Middlewares: Create a middleware that tracks active requests and reports on them during shutdown.
Database Connections: Implement a more detailed database shutdown procedure that waits for all active transactions to complete before closing.
Kubernetes Ready: Extend the example to properly handle pre-stop hooks in Kubernetes by implementing a separate health endpoint that changes status during shutdown.
Challenge: Implement a rate limiter that stops accepting new requests but allows existing connections to continue during shutdown.

If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)

Introduction​

Why Graceful Shutdown Matters​

Basic Implementation​

Testing Graceful Shutdown​

Advanced Implementation with Resource Cleanup​

Best Practices for Graceful Shutdown​

1. Set Appropriate Timeouts​

2. Handle Multiple Signals​

3. Resource Cleanup Order​

4. Logging During Shutdown​

5. Use Context Propagation​

Real-World Example: HTTP Server with Background Workers​

Summary​

Additional Resources​

Exercises​