Design a Logging & Monitoring System

Difficulty: Intermediate

Question

Design a logging and monitoring system for a microservices architecture. How would you handle log aggregation, metrics, and alerting?

Answer

Observability has three pillars: Logs, Metrics, and Traces.

1. Logs: Record discrete events (errors, requests, business events). Use structured logging (JSON) for searchability. 2. Metrics: Numeric measurements over time (request rate, error rate, latency, CPU). Use Prometheus + Grafana. 3. Traces: Follow a request across multiple services. Use distributed tracing (Jaeger, OpenTelemetry).

Log pipeline: Application -> Log Shipper (Filebeat/Fluentd) -> Message Queue (Kafka) -> Processing (Logstash) -> Storage (Elasticsearch) -> Visualization (Kibana)

Key alerting principles: Alert on symptoms (high error rate), not causes (CPU usage). Use severity levels. Avoid alert fatigue.

Code examples

Structured Logging

// Structured logging with Winston (Node.js)
const winston = require('winston');

const logger = winston.createLogger({
  level: 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.json()
  ),
  defaultMeta: { service: 'order-service', version: '1.2.0' },
  transports: [
    new winston.transports.Console(),
    new winston.transports.File({ filename: 'logs/app.log' })
  ]
});

// Structured log entries (easy to search and filter)
logger.info('Order created', {
  orderId: 'ord_123',
  userId: 'usr_456',
  amount: 99.99,
  items: 3,
  duration: 245  // ms
});
// Output: {"level":"info","message":"Order created",
//  "orderId":"ord_123","userId":"usr_456","amount":99.99,
//  "service":"order-service","timestamp":"2025-03-17T10:30:00Z"}

logger.error('Payment failed', {
  orderId: 'ord_123',
  errorCode: 'CARD_DECLINED',
  provider: 'stripe',
  traceId: req.headers['x-trace-id']  // Distributed tracing
});

// Request logging middleware
app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    logger.info('HTTP request', {
      method: req.method,
      path: req.path,
      statusCode: res.statusCode,
      duration: Date.now() - start,
      userAgent: req.headers['user-agent'],
      ip: req.ip,
      traceId: req.headers['x-trace-id']
    });
  });
  next();
});

Structured JSON logs are searchable and filterable in Elasticsearch. Including service name, trace ID, and duration in every log enables powerful cross-service debugging.

Metrics with Prometheus

// Prometheus metrics (Node.js with prom-client)
const promClient = require('prom-client');

// Counter: monotonically increasing value
const httpRequestsTotal = new promClient.Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'path', 'status_code']
});

// Histogram: distribution of values (latency)
const httpDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration in seconds',
  labelNames: ['method', 'path'],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5]  // seconds
});

// Gauge: value that goes up and down
const activeConnections = new promClient.Gauge({
  name: 'active_connections',
  help: 'Number of active connections'
});

// Middleware to collect metrics
app.use((req, res, next) => {
  const end = httpDuration.startTimer({ method: req.method, path: req.route?.path });
  activeConnections.inc();

  res.on('finish', () => {
    end();
    activeConnections.dec();
    httpRequestsTotal.inc({
      method: req.method,
      path: req.route?.path || 'unknown',
      status_code: res.statusCode
    });
  });
  next();
});

// Expose /metrics endpoint for Prometheus to scrape
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', promClient.register.contentType);
  res.end(await promClient.register.metrics());
});

// Key metrics to track (RED method):
// Rate:   requests per second
// Errors: error rate percentage
// Duration: request latency (p50, p95, p99)

Prometheus scrapes /metrics endpoints from all services. Use counters for totals, histograms for latency distributions, and gauges for current values. The RED method (Rate, Errors, Duration) covers the most important signals.

Alerting Rules

# Prometheus alerting rules (alert.rules.yml)
groups:
  - name: application
    rules:
      # High error rate
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status_code=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% for 5 minutes"

      # High latency
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "p95 latency above 2 seconds"

      # Service down
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is down"

# Alerting best practices:
# 1. Alert on symptoms (high error rate) not causes (high CPU)
# 2. Set appropriate thresholds (don't alert on noise)
# 3. Use 'for' duration to avoid flapping alerts
# 4. Include runbook links in annotations
# 5. Route by severity: critical->PagerDuty, warning->Slack

Alerts should fire on user-facing symptoms, not internal metrics. The 'for' duration prevents transient spikes from triggering alerts. Critical alerts page on-call engineers; warnings go to Slack.

Key points

Concepts covered

Structured Logging, ELK Stack, Metrics, Alerting, Distributed Tracing