Circuit Breaker Pattern

Difficulty: Intermediate

Question

What is the circuit breaker pattern? How does it prevent cascading failures in distributed systems?

Answer

The circuit breaker pattern prevents a failing service from causing cascading failures across the entire system. It works like an electrical circuit breaker.

Three states: 1. CLOSED (normal): Requests pass through. Failures are counted. 2. OPEN (tripped): Requests fail immediately without calling the downstream service. A timer starts. 3. HALF-OPEN (testing): After the timer expires, a limited number of test requests are allowed through. If they succeed, circuit closes. If they fail, circuit opens again.

Without a circuit breaker: Service A calls Service B (which is down). All threads in A block waiting for B's timeout. A's own requests start timing out. Services calling A also start failing. Entire system cascades.

With a circuit breaker: After N failures, requests to B fail fast (no waiting). A returns a fallback response. System remains partially functional.

Code examples

Circuit Breaker Implementation

class CircuitBreaker {
  constructor(options = {}) {
    this.state = 'CLOSED';
    this.failureCount = 0;
    this.successCount = 0;
    this.failureThreshold = options.failureThreshold || 5;
    this.resetTimeout = options.resetTimeout || 30000; // 30 seconds
    this.halfOpenMaxCalls = options.halfOpenMaxCalls || 3;
    this.halfOpenCalls = 0;
    this.lastFailureTime = null;
  }

  async execute(fn, fallback) {
    if (this.state === 'OPEN') {
      // Check if reset timeout has passed
      if (Date.now() - this.lastFailureTime >= this.resetTimeout) {
        this.state = 'HALF_OPEN';
        this.halfOpenCalls = 0;
        console.log('Circuit HALF_OPEN: testing...');
      } else {
        console.log('Circuit OPEN: returning fallback');
        return fallback ? fallback() : Promise.reject(new Error('Circuit is open'));
      }
    }

    if (this.state === 'HALF_OPEN' && this.halfOpenCalls >= this.halfOpenMaxCalls) {
      return fallback ? fallback() : Promise.reject(new Error('Circuit is half-open, max test calls reached'));
    }

    try {
      if (this.state === 'HALF_OPEN') this.halfOpenCalls++;
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      if (fallback) return fallback();
      throw error;
    }
  }

  onSuccess() {
    if (this.state === 'HALF_OPEN') {
      this.successCount++;
      if (this.successCount >= this.halfOpenMaxCalls) {
        this.state = 'CLOSED';
        this.failureCount = 0;
        this.successCount = 0;
        console.log('Circuit CLOSED: service recovered');
      }
    }
    this.failureCount = 0;
  }

  onFailure() {
    this.failureCount++;
    this.lastFailureTime = Date.now();

    if (this.failureCount >= this.failureThreshold) {
      this.state = 'OPEN';
      console.log('Circuit OPEN: too many failures');
    }
  }
}

The circuit breaker tracks failures and trips open after the threshold is reached. In the OPEN state, requests fail fast without trying. After a timeout, it allows test requests to check if the service has recovered.

Using the Circuit Breaker

// Create circuit breakers for each downstream service
const paymentCircuit = new CircuitBreaker({
  failureThreshold: 5,
  resetTimeout: 30000
});

const emailCircuit = new CircuitBreaker({
  failureThreshold: 3,
  resetTimeout: 60000
});

// API endpoint with circuit breaker and fallback
app.post('/api/orders', auth, async (req, res) => {
  const order = await createOrder(req.body);

  // Payment with circuit breaker
  const paymentResult = await paymentCircuit.execute(
    // Primary: call payment service
    async () => {
      return await fetch('http://payment-service/charge', {
        method: 'POST',
        body: JSON.stringify({ amount: order.total }),
        timeout: 5000
      }).then(r => r.json());
    },
    // Fallback: queue payment for later
    async () => {
      await paymentQueue.addJob('RETRY_PAYMENT', { orderId: order.id });
      return { status: 'QUEUED', message: 'Payment will be processed shortly' };
    }
  );

  // Email notification with circuit breaker (non-critical)
  await emailCircuit.execute(
    () => sendOrderConfirmation(order),
    () => console.log('Email service down, will retry later')
  );

  res.json({ order, payment: paymentResult });
});

Each downstream service gets its own circuit breaker with appropriate thresholds. Critical services (payment) use queued retries as fallback. Non-critical services (email) can silently fail.

Bulkhead Pattern

// Bulkhead: isolate failures by limiting concurrent calls per service
// Like compartments in a ship - one flooding doesn't sink the whole ship

class Bulkhead {
  constructor(maxConcurrent, maxQueue = 10) {
    this.maxConcurrent = maxConcurrent;
    this.maxQueue = maxQueue;
    this.active = 0;
    this.queue = [];
  }

  async execute(fn) {
    if (this.active >= this.maxConcurrent) {
      if (this.queue.length >= this.maxQueue) {
        throw new Error('Bulkhead full: request rejected');
      }

      // Queue the request
      return new Promise((resolve, reject) => {
        this.queue.push({ fn, resolve, reject });
      });
    }

    return this.run(fn);
  }

  async run(fn) {
    this.active++;
    try {
      return await fn();
    } finally {
      this.active--;
      this.processQueue();
    }
  }

  processQueue() {
    if (this.queue.length > 0 && this.active < this.maxConcurrent) {
      const { fn, resolve, reject } = this.queue.shift();
      this.run(fn).then(resolve).catch(reject);
    }
  }
}

// Usage: limit concurrent calls to each service
const paymentBulkhead = new Bulkhead(10, 20);  // max 10 concurrent, 20 queued
const searchBulkhead = new Bulkhead(20, 50);

// Combined: circuit breaker + bulkhead
const result = await paymentBulkhead.execute(() =>
  paymentCircuit.execute(
    () => callPaymentService(data),
    () => queuePaymentForLater(data)
  )
);

The bulkhead pattern limits concurrent requests to a downstream service, preventing one slow service from consuming all available connections. Combined with circuit breakers, it provides comprehensive fault isolation.

Key points

Concepts covered

Circuit Breaker, Fault Tolerance, Retry Pattern, Bulkhead, Fallback