Ops — Alerting & Incident Response
Alerting is the bridge between monitoring and action. A well-designed alerting system notifies the right people at the right time with enough context to take immediate action. Poor alerting leads to alert fatigue, where engineers ignore notifications because most are false positives.
This section covers alerting philosophy, severity levels, PagerDuty and Opsgenie integration, on-call rotations, runbooks, the incident response lifecycle, post-mortems, and SLA/SLO monitoring.
The core principle: alert on symptoms, not causes. Alert when users are affected, not when a single server is high on CPU. Users do not care about CPU — they care about response time and availability.
✓ Alert on symptoms
High error rate, elevated latency, failed health checks, SLA breach — these directly indicate user impact.
✗ Alert on causes
High CPU, high memory, disk usage — these are causes, not user impact. Alert only if they lead to user-facing symptoms.
| 1 | # ✓ Good: Alert on user-facing symptom |
| 2 | alert: HighErrorRate |
| 3 | expr: | |
| 4 | sum(rate(http_requests_total{status=~"5.."}[5m])) |
| 5 | / sum(rate(http_requests_total[5m])) |
| 6 | > 0.05 |
| 7 | for: 5m |
| 8 | labels: |
| 9 | severity: critical |
| 10 | annotations: |
| 11 | summary: "Error rate above 5% - users are seeing failures" |
| 12 | |
| 13 | # ✗ Bad: Alert on infrastructure cause |
| 14 | alert: HighCPU |
| 15 | expr: node_cpu_seconds_total > 0.9 |
| 16 | for: 5m |
| 17 | # This fires constantly and teaches engineers to ignore alerts |
info
Severity levels determine who is notified and how urgently. A clear severity classification prevents every alert from being treated as critical.
| Level | Description | Response |
|---|---|---|
| P0 | Complete outage or data loss | Page immediately, war room, all hands |
| P1 | Major feature degraded for all users | Page on-call, 15-min response SLA |
| P2 | Minor feature issue or partial degradation | Ticket created, next business day response |
| P3 | Non-urgent: warnings, anomalies | Log for review, no immediate action |
| P4 | Informational: low-priority observations | Dashboard only, no notification |
warning
PagerDuty and Opsgenie are incident management platforms that handle alert routing, escalation, on-call scheduling, and notification delivery. They integrate with Prometheus, Grafana, Datadog, and virtually every monitoring tool.
| 1 | # Prometheus Alertmanager: PagerDuty integration |
| 2 | global: |
| 3 | resolve_timeout: 5m |
| 4 | |
| 5 | route: |
| 6 | receiver: pagerduty-critical |
| 7 | group_by: ["alertname", "severity"] |
| 8 | group_wait: 30s |
| 9 | group_interval: 5m |
| 10 | repeat_interval: 4h |
| 11 | routes: |
| 12 | - match: |
| 13 | severity: critical |
| 14 | receiver: pagerduty-critical |
| 15 | - match: |
| 16 | severity: warning |
| 17 | receiver: slack-warning |
| 18 | |
| 19 | receivers: |
| 20 | - name: pagerduty-critical |
| 21 | pagerduty_configs: |
| 22 | - service_key: "<your-pagerduty-key>" |
| 23 | severity: critical |
| 24 | description: '{{ .GroupLabels.alertname }}' |
| 25 | details: |
| 26 | firing: '{{ .GroupLabels }}' |
| 27 | num_firing: '{{ .Alerts.Firing | len }}' |
| 28 | |
| 29 | - name: slack-warning |
| 30 | slack_configs: |
| 31 | - channel: "#alerts" |
| 32 | send_resolved: true |
| 33 | text: '{{ .GroupAnnotations.summary }}' |
best practice
On-call rotations ensure someone is always available to respond to incidents. The two main patterns are primary-secondary (one engineer leads, another backs up) and follow-the-sun (rotation across time zones).
| Pattern | Best For | Drawback |
|---|---|---|
| Primary-Secondary | Small teams, single timezone | Night pages disrupt sleep |
| Follow-the-Sun | Global teams, 24/7 coverage | Handoff complexity, context loss |
| No On-Call | Systems with zero user-facing SLA | Only for non-critical internal tools |
| 1 | // PagerDuty: Schedule configuration (API example) |
| 2 | const schedule = { |
| 3 | name: "API Team On-Call", |
| 4 | type: "schedule", |
| 5 | time_zone: "America/New_York", |
| 6 | schedule: { |
| 7 | shifts: [ |
| 8 | { |
| 9 | type: "rotation", |
| 10 | rotation_starts: "2025-01-01T00:00:00Z", |
| 11 | users: [ |
| 12 | "user_abc123", // Week 1 |
| 13 | "user_def456", // Week 2 |
| 14 | "user_ghi789", // Week 3 |
| 15 | ], |
| 16 | rotation_length: { |
| 17 | duration: 7, |
| 18 | unit: "days", |
| 19 | }, |
| 20 | }, |
| 21 | ], |
| 22 | restrictions: [ |
| 23 | { |
| 24 | type: "daily_restriction", |
| 25 | start_time: "09:00:00", |
| 26 | duration: { |
| 27 | duration: 12, |
| 28 | unit: "hours", |
| 29 | }, |
| 30 | start_day_of_week: 1, // Monday |
| 31 | }, |
| 32 | ], |
| 33 | }, |
| 34 | }; |
A runbook is a step-by-step guide for responding to a specific alert. Every alert that pages someone should have an attached runbook so the responder knows exactly what to check and how to mitigate.
| 1 | // Runbook attached to alert annotations |
| 2 | const alertRule = { |
| 3 | alert: "HighErrorRate", |
| 4 | expr: "...", |
| 5 | annotations: { |
| 6 | summary: "Error rate above 5%", |
| 7 | runbook_url: |
| 8 | "https://wiki.internal/runbooks/high-error-rate", |
| 9 | dashboard: |
| 10 | "https://grafana.internal/d/api-errors", |
| 11 | escalation: "PagerDuty API Team", |
| 12 | }, |
| 13 | }; |
| 14 | |
| 15 | // Runbook content (markdown on wiki): |
| 16 | // ## High Error Rate Runbook |
| 17 | // |
| 18 | // ### 1. Check the dashboard |
| 19 | // Open the Grafana dashboard linked in the alert. |
| 20 | // Identify which endpoints are failing and the error type. |
| 21 | // |
| 22 | // ### 2. Check recent deployments |
| 23 | // Run: kubectl rollout history deployment/api -n production |
| 24 | // If a deploy happened in the last 30 minutes, consider rollback. |
| 25 | // |
| 26 | // ### 3. Check downstream services |
| 27 | // Verify database, cache, and external API health. |
| 28 | // |
| 29 | // ### 4. Mitigate |
| 30 | // If deploy-related: kubectl rollout undo deployment/api -n production |
| 31 | // If dependency: check connection pool, restart affected service |
| 32 | // If external: enable circuit breaker, return cached/default response |
info
The incident response lifecycle provides a structured process for handling incidents from detection through resolution and learning. Consistency in process reduces chaos during high-pressure situations.
1. Detect
Monitoring alerts, user reports, or automated health checks identify an issue. Alert fires and routes to on-call.
2. Respond
On-call acknowledges the alert, opens a war room (Slack channel or bridge call), and begins triage.
3. Mitigate
Restore service first, root cause later. Rollback, failover, enable circuit breaker, or apply a hotfix.
4. Resolve
Apply permanent fix, verify metrics have returned to normal, update status page, notify stakeholders.
5. Learn
Conduct a blameless post-mortem. Document timeline, root cause, action items, and prevention measures.
Post-mortems are blameless reviews of incidents. They focus on systemic improvements, not individual blame. Every significant incident should produce a post-mortem with actionable follow-ups.
| 1 | // Post-mortem template |
| 2 | const postMortem = { |
| 3 | title: "API Error Rate Spike - 2025-01-15", |
| 4 | severity: "P1", |
| 5 | duration: "47 minutes", |
| 6 | impact: "12% of API requests returned 500 errors", |
| 7 | |
| 8 | timeline: [ |
| 9 | "10:00 UTC - Deploy v2.3.0 to production", |
| 10 | "10:02 UTC - Alert fires: HighErrorRate", |
| 11 | "10:05 UTC - On-call acknowledges, opens war room", |
| 12 | "10:08 UTC - Identified: database connection pool exhaustion", |
| 13 | "10:12 UTC - Mitigation: rollback to v2.2.1", |
| 14 | "10:15 UTC - Error rate returning to normal", |
| 15 | "10:47 UTC - Confirmed resolved, metrics stable", |
| 16 | ], |
| 17 | |
| 18 | rootCause: |
| 19 | "v2.3.0 introduced a new query that opened database connections " + |
| 20 | "without releasing them, exhausting the connection pool after ~2 min", |
| 21 | |
| 22 | actionItems: [ |
| 23 | { |
| 24 | action: "Add connection pool monitoring and alerting", |
| 25 | owner: "backend-team", |
| 26 | priority: "high", |
| 27 | dueDate: "2025-01-22", |
| 28 | }, |
| 29 | { |
| 30 | action: "Add integration test for connection pool under load", |
| 31 | owner: "backend-team", |
| 32 | priority: "high", |
| 33 | dueDate: "2025-01-29", |
| 34 | }, |
| 35 | { |
| 36 | action: "Enforce max connection lifetime in ORM config", |
| 37 | owner: "platform-team", |
| 38 | priority: "medium", |
| 39 | dueDate: "2025-02-05", |
| 40 | }, |
| 41 | ], |
| 42 | |
| 43 | lessonsLearned: [ |
| 44 | "Deploy validation should include connection pool stress testing", |
| 45 | "Canary deployments would have limited blast radius", |
| 46 | ], |
| 47 | }; |
best practice
SLIs (Service Level Indicators) are the metrics you measure. SLOs (Service Level Objectives) are the targets you set. SLAs (Service Level Agreements) are the contractual commitments with customers. Error budgets are the gap between your SLO and 100%.
| 1 | // Define SLIs and SLOs |
| 2 | const sloConfig = { |
| 3 | // Availability SLO: 99.9% (allows ~43 min downtime/month) |
| 4 | availability: { |
| 5 | sli: "successful_requests / total_requests", |
| 6 | slo: 0.999, |
| 7 | window: "30d", |
| 8 | }, |
| 9 | |
| 10 | // Latency SLO: 99% of requests under 200ms |
| 11 | latency: { |
| 12 | sli: "requests_under_200ms / total_requests", |
| 13 | slo: 0.99, |
| 14 | window: "30d", |
| 15 | }, |
| 16 | |
| 17 | // Error budget: 0.1% of total requests can fail |
| 18 | // If budget is consumed, freeze non-critical deploys |
| 19 | errorBudget: { |
| 20 | remaining: calculateErrorBudget(sloConfig.availability), |
| 21 | policy: "When error budget is < 10%, freeze feature deploys " + |
| 22 | "and focus on reliability work", |
| 23 | }, |
| 24 | }; |
| 25 | |
| 26 | // Prometheus alert: Error budget burn rate |
| 27 | const errorBudgetBurnAlert = { |
| 28 | alert: "ErrorBudgetBurnRateHigh", |
| 29 | expr: | |
| 30 | ( |
| 31 | 1 - rate(http_requests_total{status=~"2.."}[1h]) |
| 32 | / rate(http_requests_total[1h]) |
| 33 | ) > 14.4 * (1 - 0.999) |
| 34 | or |
| 35 | ( |
| 36 | 1 - rate(http_requests_total{status=~"2.."}[6h]) |
| 37 | / rate(http_requests_total[6h]) |
| 38 | ) > 6 * (1 - 0.999) |
| 39 | , |
| 40 | for: "5m", |
| 41 | labels: { severity: "critical" }, |
| 42 | annotations: { |
| 43 | summary: "Error budget is burning too fast", |
| 44 | }, |
| 45 | }; |
info
Alert fatigue occurs when responders receive too many alerts, especially false positives. Over time, they begin ignoring all alerts, including critical ones. The solution is fewer, better, actionable alerts.
Consolidate related alerts
Group multiple symptoms into a single incident. 5 separate alerts for one outage = 5 interruptions. One incident alert = 1.
Remove no-action alerts
If an alert does not require a human to take action, it should not be an alert. Make it a dashboard metric instead.
Tune thresholds regularly
Review alert firing rates monthly. An alert that fires daily without action needed should be removed or its threshold adjusted.
Use multi-window, multi-burn-rate
Combine short-window (fast burn) and long-window (slow burn) alerts to catch both sudden spikes and gradual degradation.
✓ Every alert needs a runbook
If a human must respond, give them a documented procedure. Runbooks reduce MTTR (Mean Time to Resolution) significantly.
✓ Page only for user-facing impact
Reserve pages for P0/P1. Everything else goes to tickets or dashboards. Night pages are extremely expensive in engineer well-being.
✓ Conduct blameless post-mortems
Every significant incident produces a post-mortem with action items. Track those items to completion — they prevent future incidents.
✓ Measure and reduce MTTR
Track Mean Time to Acknowledge (MTTA) and Mean Time to Resolve (MTTR). Automate responses where possible (auto-scaling, auto-rollback, circuit breakers).