Monitoring & Alerting

Difficulty: Intermediate

Question

How do you monitor applications in production? Explain the three pillars of observability and how to set up effective alerting.

Answer

Monitoring tells you when something is broken. Observability tells you why. The three pillars of observability are logs, metrics, and traces.

Logs record discrete events with context. Structured JSON logs are searchable and parsable. Centralize logs from all services into one platform (ELK Stack, Datadog, CloudWatch) for correlation.

Metrics are numeric measurements over time: request rate, error rate, latency percentiles, CPU/memory usage. They answer 'how much' and 'how fast.' The RED method (Rate, Errors, Duration) covers the essentials for any service.

Traces follow a single request through multiple services. Essential for debugging microservices where a request might touch 5-10 services. Each span records timing and metadata.

Alerting rules turn metrics into notifications. Good alerts are actionable (someone can fix it), relevant (not noise), and tiered (warning vs critical). Alert on symptoms (high error rate) not causes (high CPU) because symptoms directly impact users.

Key metrics to monitor: - The Four Golden Signals: latency, traffic, errors, saturation - Business metrics: signups, orders, payments - Infrastructure: CPU, memory, disk, network

Code examples

Structured Logging

// Winston structured logging setup
import winston from 'winston';

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.errors({ stack: true }),
    winston.format.json()
  ),
  defaultMeta: { service: 'api-server' },
  transports: [
    new winston.transports.Console(),
    new winston.transports.File({ filename: 'error.log', level: 'error' }),
  ],
});

// Structured log output (easy to search and parse)
logger.info('User logged in', {
  userId: 123,
  email: 'john@example.com',
  ip: '192.168.1.1',
  method: 'password',
  duration: 45,
});
// {"level":"info","message":"User logged in","userId":123,...,"timestamp":"2024-01-15T10:00:00Z"}

logger.error('Payment processing failed', {
  orderId: 456,
  amount: 99.99,
  error: err.message,
  stack: err.stack,
  provider: 'razorpay',
});

// Request logging middleware
app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    logger.info('HTTP Request', {
      method: req.method,
      path: req.path,
      status: res.statusCode,
      duration: Date.now() - start,
      userAgent: req.headers['user-agent'],
    });
  });
  next();
});

Structured JSON logs are machine-parseable and searchable. Always include context (userId, orderId) so you can correlate events. Log at appropriate levels: error for failures, info for events, debug for troubleshooting.

Prometheus Metrics and Grafana

// Express metrics with prom-client
import promClient from 'prom-client';

// Collect default metrics (CPU, memory, event loop)
promClient.collectDefaultMetrics();

// Custom metrics
const httpRequestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status'],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5],
});

const httpRequestTotal = new promClient.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status'],
});

// Middleware to record metrics
app.use((req, res, next) => {
  const end = httpRequestDuration.startTimer();
  res.on('finish', () => {
    const route = req.route?.path || req.path;
    end({ method: req.method, route, status: res.statusCode });
    httpRequestTotal.inc({ method: req.method, route, status: res.statusCode });
  });
  next();
});

// Expose metrics endpoint for Prometheus scraping
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', promClient.register.contentType);
  res.end(await promClient.register.metrics());
});

// Prometheus scrapes /metrics every 15s
// Grafana visualizes the data in dashboards
// Alert rules fire when thresholds are exceeded

Prometheus collects time-series metrics. Histograms track latency distributions (p50, p95, p99). Counters track totals. Grafana creates dashboards and Alertmanager sends notifications.

Health Checks and Alerting Rules

// Health check endpoint with dependency checks
app.get('/health', async (req, res) => {
  const checks = {
    uptime: process.uptime(),
    timestamp: Date.now(),
    database: 'unknown',
    cache: 'unknown',
  };

  try {
    await prisma.$queryRaw`SELECT 1`;
    checks.database = 'healthy';
  } catch {
    checks.database = 'unhealthy';
  }

  try {
    await redis.ping();
    checks.cache = 'healthy';
  } catch {
    checks.cache = 'unhealthy';
  }

  const isHealthy = checks.database === 'healthy' && checks.cache === 'healthy';
  res.status(isHealthy ? 200 : 503).json(checks);
});

// Alerting rules (Prometheus/Alertmanager format)
// groups:
//   - name: app-alerts
//     rules:
//       - alert: HighErrorRate
//         expr: rate(http_requests_total{status=~"5.."}[5m])
//               / rate(http_requests_total[5m]) > 0.05
//         for: 2m
//         labels:
//           severity: critical
//         annotations:
//           summary: "Error rate above 5% for 2 minutes"
//
//       - alert: HighLatency
//         expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 2
//         for: 5m
//         labels:
//           severity: warning
//         annotations:
//           summary: "p99 latency above 2 seconds"
//
//       - alert: HealthCheckFailing
//         expr: probe_success{job="health-check"} == 0
//         for: 1m
//         labels:
//           severity: critical

Health checks report status of all dependencies. Alert on error rate and latency (symptoms), not CPU (causes). Use 'for' durations to avoid false alarms from brief spikes.

Key points

Concepts covered

Monitoring, Logging, Metrics, Alerting, Observability, APM