Skip to main content

Echo Circuit Breaker Pattern

Introduction​

The Circuit Breaker pattern is a crucial design pattern in modern distributed systems. It's named after the electrical circuit breaker that stops the flow of electricity when a fault is detected, protecting the system from damage. Similarly, in software, this pattern prevents an application from repeatedly trying to execute an operation that's likely to fail, giving the failing service time to recover.

In this guide, you'll learn how to implement the Circuit Breaker pattern in Echo applications to enhance system resilience and prevent cascading failures across microservices.

Understanding the Circuit Breaker Pattern​

What is a Circuit Breaker?​

A circuit breaker acts as a proxy for operations that might fail. It monitors for failures and, once a threshold is reached, "trips" the circuit to prevent further calls to the failing service. After a specified timeout, the circuit breaker allows a limited number of test requests to pass through. If these succeed, the circuit is closed and normal operation resumes.

Three States of a Circuit Breaker​

  1. Closed: The initial state. Requests pass through normally. The circuit breaker keeps track of failures.
  2. Open: When failures exceed a threshold, the circuit opens. Requests fail fast without attempting the operation.
  3. Half-Open: After a timeout period, the circuit transitions to half-open, allowing a limited number of test requests to determine if the problem is fixed.

Implementing Circuit Breaker in Echo​

For Echo applications, we can implement the Circuit Breaker pattern using middleware. We'll use the popular sony/gobreaker package for this example.

Setting Up the Dependencies​

First, let's install the required dependencies:

bash
go get -u github.com/labstack/echo/v4
go get -u github.com/sony/gobreaker

Creating a Circuit Breaker Middleware​

Here's how to implement a basic circuit breaker middleware for Echo:

go
package middleware

import (
	"github.com/labstack/echo/v4"
	"github.com/sony/gobreaker"
	"net/http"
	"time"
)

// CircuitBreakerConfig defines the config for CircuitBreaker middleware.
type CircuitBreakerConfig struct {
	// Timeout is how long to wait for command to complete, in milliseconds
	Timeout int
	// MaxConcurrentRequests is how many commands can run at the same time
	MaxConcurrentRequests int
	// RequestVolumeThreshold is the minimum number of requests needed before a circuit can be tripped
	RequestVolumeThreshold int
	// SleepWindow is how long to wait after a circuit opens before testing for recovery, in milliseconds
	SleepWindow int
	// ErrorPercentThreshold is the threshold to trip a circuit as a percentage
	ErrorPercentThreshold int
}

// CircuitBreaker returns a Circuit Breaker middleware for Echo
func CircuitBreaker(name string, config CircuitBreakerConfig) echo.MiddlewareFunc {
	cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
		Name:        name,
		MaxRequests: uint32(config.MaxConcurrentRequests),
		Interval:    time.Duration(config.SleepWindow) * time.Millisecond,
		Timeout:     time.Duration(config.SleepWindow) * time.Millisecond,
		ReadyToTrip: func(counts gobreaker.Counts) bool {
			failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
			return counts.Requests >= uint32(config.RequestVolumeThreshold) && 
			       failureRatio >= float64(config.ErrorPercentThreshold)/100
		},
		OnStateChange: func(name string, from gobreaker.State, to gobreaker.State) {
			// You can add logging here
		},
	})

	return func(next echo.HandlerFunc) echo.HandlerFunc {
		return func(c echo.Context) error {
			result, err := cb.Execute(func() (interface{}, error) {
				return nil, next(c)
			})

			if err != nil {
				if err == gobreaker.ErrOpenState {
					return c.JSON(http.StatusServiceUnavailable, map[string]string{
						"error": "Service unavailable, circuit breaker is open",
					})
				}
				return err
			}

			return result.(error)
		}
	}
}

Using the Circuit Breaker Middleware​

Now, let's see how to use this middleware in an Echo application:

go
package main

import (
	"errors"
	"math/rand"
	"net/http"
	"time"

	"github.com/labstack/echo/v4"
	"github.com/labstack/echo/v4/middleware"
	"yourpackage/middleware" // Import the custom middleware package
)

func main() {
	e := echo.New()

	// Add basic middleware
	e.Use(middleware.Logger())
	e.Use(middleware.Recover())

	// Configure the circuit breaker
	cbConfig := middleware.CircuitBreakerConfig{
		Timeout:               1000,
		MaxConcurrentRequests: 100,
		RequestVolumeThreshold: 10,
		SleepWindow:           5000,
		ErrorPercentThreshold: 50,
	}

	// Apply circuit breaker to a specific group
	api := e.Group("/api")
	api.Use(middleware.CircuitBreaker("api-breaker", cbConfig))

	// Handler that sometimes fails
	api.GET("/flaky", flakyHandler)
	
	// Normal handler for comparison
	e.GET("/healthy", healthyHandler)

	e.Start(":8080")
}

func flakyHandler(c echo.Context) error {
	// Simulate a service that fails 70% of the time
	rand.Seed(time.Now().UnixNano())
	if rand.Intn(10) < 7 {
		return errors.New("service error")
	}
	return c.JSON(http.StatusOK, map[string]string{"status": "success"})
}

func healthyHandler(c echo.Context) error {
	return c.JSON(http.StatusOK, map[string]string{"status": "healthy"})
}

How It Works​

Let's break down what happens in our implementation:

  1. We create a circuit breaker configuration with specific thresholds for failures and recovery.
  2. When requests go through the middleware, they are executed via the circuit breaker.
  3. If too many failures occur (based on the configured thresholds), the circuit breaker opens.
  4. While open, the circuit breaker quickly rejects requests without attempting to call the failing service.
  5. After a configured timeout, the circuit transitions to half-open state and allows test requests.
  6. If these test requests succeed, the circuit closes again; otherwise, it remains open.

Real-World Application: Protecting an External API Call​

Here's a more realistic example of using a circuit breaker to protect calls to an external API:

go
package main

import (
	"encoding/json"
	"net/http"
	"time"

	"github.com/labstack/echo/v4"
	"github.com/labstack/echo/v4/middleware"
	"yourpackage/middleware" // Your custom middleware
)

type ExternalService struct {
	client *http.Client
}

func NewExternalService() *ExternalService {
	return &ExternalService{
		client: &http.Client{
			Timeout: 5 * time.Second,
		},
	}
}

func (s *ExternalService) FetchData() (map[string]interface{}, error) {
	resp, err := s.client.Get("https://api.example.com/data")
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		return nil, echo.NewHTTPError(resp.StatusCode, "External API error")
	}

	var data map[string]interface{}
	if err := json.NewDecoder(resp.Body).Decode(&data); err != nil {
		return nil, err
	}

	return data, nil
}

func main() {
	e := echo.New()
	e.Use(middleware.Logger())
	e.Use(middleware.Recover())

	service := NewExternalService()

	cbConfig := middleware.CircuitBreakerConfig{
		Timeout:               2000,
		MaxConcurrentRequests: 50,
		RequestVolumeThreshold: 5,
		SleepWindow:           10000,
		ErrorPercentThreshold: 50,
	}

	// Apply circuit breaker specifically to the external API endpoint
	api := e.Group("/api")
	api.Use(middleware.CircuitBreaker("external-api", cbConfig))

	api.GET("/external-data", func(c echo.Context) error {
		data, err := service.FetchData()
		if err != nil {
			return err
		}
		return c.JSON(http.StatusOK, data)
	})

	e.Start(":8080")
}

This example protects your application from an unreliable external API. If the external API starts failing, the circuit breaker will open, preventing cascading failures and giving the external service time to recover.

Benefits of the Circuit Breaker Pattern​

  1. Prevents Cascading Failures: Stops failure in one service from bringing down the entire system
  2. Improves User Experience: Fails fast rather than making users wait for timeouts
  3. Reduces Resource Consumption: Avoids wasting resources on calls that are likely to fail
  4. Enables Graceful Degradation: Services can operate in a degraded mode rather than completely failing
  5. Helps Services Recover: Gives failing services time to recover without being overwhelmed

Common Scenarios for Using Circuit Breakers​

  • Database Operations: Protect against database overload or temporary failures
  • External API Calls: Guard against third-party service failures
  • File System Operations: Handle disk I/O failures gracefully
  • Microservice Communications: Prevent failures in one service from affecting others
  • Resource-Intensive Operations: Limit concurrent resource-intensive tasks

Circuit Breaker Customization​

Depending on your specific needs, you might want to customize your circuit breaker:

go
// Example of a more customized circuit breaker
customCB := middleware.CircuitBreakerConfig{
    Timeout:               500,  // Shorter timeout for more sensitive operations
    MaxConcurrentRequests: 10,   // Limit concurrent requests
    RequestVolumeThreshold: 3,   // Trip after fewer failures for critical paths
    SleepWindow:           15000, // Longer recovery time
    ErrorPercentThreshold: 25,   // Lower threshold for more critical services
}

// Apply to a specific critical endpoint
e.GET("/critical-operation", criticalHandler, middleware.CircuitBreaker("critical", customCB))

Summary​

The Circuit Breaker pattern is an essential tool for building resilient distributed systems with Echo. By implementing circuit breakers:

  • You prevent cascading failures across services
  • Your system degrades gracefully under pressure
  • Failed services have time to recover
  • Users experience faster feedback rather than timeouts

Remember to carefully tune your circuit breaker parameters based on the specific characteristics of each service or endpoint. Too sensitive settings might cause unnecessary service disruptions, while overly lenient settings might not provide adequate protection.

Additional Resources​

Exercises​

  1. Implement a circuit breaker for a database connection in an Echo application
  2. Create a dashboard that shows the current state of all circuit breakers in your application
  3. Extend the circuit breaker to include fallback functionality when the circuit is open
  4. Implement different circuit breaker configurations for different types of operations
  5. Add detailed logging to your circuit breaker to track state changes and failure rates

If you spot any mistakes on this website, please let me know at [email protected]. I’d greatly appreciate your feedback! :)