|$ curl https://forge-ai.dev/api/markdown?path=docs/system-design/reliability

$cat docs/reliability.md

updated Recently·35 min read·published

Reliability

◆System Design◆Advanced🎯Free Tools

Introduction

Reliability is the ability of a system to operate correctly and continue operating despite failures. In distributed systems, failure is not an exception — it is the default mode. Networks partition, disks fail, nodes crash, and deployments go wrong. A reliable system anticipates these events and contains their impact.

Reliability is not the same as availability, though they are related. Availability measures whether the system is up; reliability measures whether it behaves correctly and recovers safely. A system that returns corrupt data is available but not reliable.

This guide covers the foundational patterns of reliability engineering: fault tolerance, retries, circuit breakers, bulkheads, timeouts, and the SLI/SLO/SLA framework for measuring and communicating reliability.

Fault Tolerance

Fault tolerance is the ability to continue operating when components fail. It is achieved through redundancy, isolation, and graceful degradation. The goal is not to prevent all failures — that is impossible — but to ensure that no single failure causes a total outage.

fault-tolerance.txt

TEXT

1	Fault-tolerant architecture:
2
3	┌─────────────┐
4	│ Client │
5	└──────┬──────┘
6	│
7	┌──────▼──────┐
8	│ Load Balancer│
9	│ (health checks)│
10	└──────┬──────┘
11	│
12	┌───────────────┼───────────────┐
13	│ │ │
14	┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
15	│ App-1 │ │ App-2 │ │ App-3 │
16	│ (zone A) │ │ (zone B) │ │ (zone C) │
17	└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
18	│ │ │
19	└───────────────┼───────────────┘
20	│
21	┌──────▼──────┐
22	│ Multi-AZ DB │
23	│ (primary + │
24	│ replicas) │
25	└─────────────┘

✓

best practice

Design for redundancy at every layer: multiple application instances across availability zones, replicated databases, and independent upstream providers. A failure in one zone or component should not take down the whole service.

Retries

Transient failures are common in distributed systems. A network packet may be dropped, a server may briefly be overloaded, or a connection may time out. Retries give the operation another chance to succeed without bothering the user.

However, blind retries can amplify problems. If a service is overloaded, retrying every failed request increases its load. Use backoff, jitter, and retry budgets to retry safely.

retries.txt

TEXT

1	Retry strategies:
2
3	Exponential backoff:
4	retry after 100ms, 200ms, 400ms, 800ms, 1600ms...
5
6	Exponential backoff with jitter:
7	delay = random_between(0, min(max_delay, base * 2^attempt))
8
9	Jitter prevents synchronized retries from a fleet
10	of clients, which can cause thundering herds.
11
12	Retry budget:
13	Allow retries only for a small fraction of total
14	requests, e.g., 10% of requests may be retries.
15	This caps the extra load sent to a struggling service.

⚠

warning

Only retry idempotent operations. Retrying a non-idempotent request such as a payment charge can cause double charges. Use idempotency keys for any operation that might be retried.

Circuit Breakers

A circuit breaker stops requests to a failing service, giving it time to recover and preventing cascading failures. It has three states: closed (requests pass through), open (requests fail fast), and half-open (a few test requests are allowed to probe recovery).

circuit-breaker.txt

TEXT

1	Circuit breaker states:
2
3	┌──────────────┐
4	│ CLOSED │ Normal operation; requests flow.
5	│ (healthy) │
6	└──────┬───────┘
7	│ failure threshold exceeded
8	▼
9	┌──────────────┐
10	│ OPEN │ Requests fail fast for a cooldown.
11	│ (unhealthy) │
12	└──────┬───────┘
13	│ after timeout
14	▼
15	┌──────────────┐
16	│ HALF-OPEN │ Allow limited probes.
17	│ (probing) │
18	└──────┬───────┘
19	│ probes succeed/fail
20	▼
21	CLOSED / OPEN

ℹ

info

Combine circuit breakers with fallbacks. When the breaker is open, return cached data, a degraded response, or a queued acknowledgment instead of failing the entire request.

Bulkheads and Timeouts

Bulkheads isolate resources so that a failure in one area does not exhaust resources needed by others. The name comes from ship design: watertight compartments prevent a single breach from sinking the vessel. In software, bulkheads separate thread pools, connection pools, and queues per dependency.

bulkhead.txt

TEXT

1	Bulkhead pattern:
2
3	┌─────────────────────────────────────────┐
4	│ Application Thread Pool │
5	│ ┌────────┐ ┌────────┐ ┌────────┐ │
6	│ │ Pool A │ │ Pool B │ │ Pool C │ │
7	│ │Payment │ │Inventory│ │Shipping│ │
8	│ └───┬────┘ └───┬────┘ └───┬────┘ │
9	│ │ │ │ │
10	│ ▼ ▼ ▼ │
11	│ Payment Inventory Shipping │
12	│ Service Service Service │
13	└─────────────────────────────────────────┘
14
15	If Inventory Service is slow, Pool B is exhausted,
16	but Payment and Shipping continue to function.

Timeouts

Timeouts prevent a slow dependency from holding resources indefinitely. Every outbound call should have a timeout, and timeouts should be shorter at the edges than in the core. If an upstream call times out, fail fast and free the thread.

✓

best practice

Set timeouts based on the latency distribution of the dependency, not arbitrary values. If the p99 is 200 ms, a 5-second timeout wastes resources. Use 2x-3x the p99 with a hard ceiling.

Graceful Degradation

Graceful degradation is the art of providing reduced functionality when dependencies fail. Instead of returning a 500 error, the system returns a useful but limited response. A product page might show cached prices if the pricing service is down. A social feed might omit recommendations if the ML service is slow.

degradation.txt

TEXT

1	Graceful degradation example:
2
3	Request: product page
4
5	Path when all services healthy:
6	Product DB ──▶ Inventory ──▶ Pricing ──▶ Recommendations
7
8	Path when Pricing is down:
9	Product DB ──▶ Inventory ──▶ Cached price ──▶ Recommendations
10
11	Path when Recommendations is down:
12	Product DB ──▶ Inventory ──▶ Pricing ──▶ (no recommendations)

ℹ

info

Identify non-critical features that can be disabled during incidents. Document fallback behavior and test it regularly. Degradation is only graceful if it is planned and rehearsed.

SLIs, SLOs, and SLAs

Reliability must be measured. The industry uses three related concepts: Service Level Indicators (SLIs), Service Level Objectives (SLOs), and Service Level Agreements (SLAs). They form a hierarchy from measurement to target to contract.

Term	Definition	Example
SLI	A quantitative measure of service behavior	Request latency, error rate, throughput
SLO	A target value for an SLI over time	p99 latency < 200 ms over 30 days
SLA	A contract with consequences for missing SLOs	99.99% uptime or service credits

📝

note

SLOs should be user-centric. Measure what users actually experience — latency of successful requests, availability of critical flows — rather than internal metrics that do not correlate with satisfaction.

Chaos Engineering

Chaos engineering is the practice of intentionally injecting failures into a system to validate that it behaves as expected. By breaking things in a controlled way, teams build confidence in their reliability mechanisms before production incidents occur.

chaos.txt

TEXT

1	Chaos experiments:
2
3	- Terminate a random application instance
4	- Introduce latency on a downstream dependency
5	- Drop network packets between services
6	- Fill a disk on a database replica
7	- Simulate a full availability-zone failure
8
9	Start small in staging, then move to production
10	with blast-radius controls and abort conditions.

🔥

pro tip

Do not start chaos engineering until you have basic observability and on-call processes in place. Chaos experiments are only useful if you can detect, measure, and learn from the failures you introduce.

Reliability Testing

Testing for reliability goes beyond unit tests. It validates how the system behaves under failure, load, and unexpected conditions. Without these tests, reliability patterns exist only in theory.

Test Type	Purpose	Example
Load testing	Verify behavior at expected peak load	Simulate 10K concurrent users
Stress testing	Find the breaking point	Increase load until errors spike
Fault injection	Validate failure handling	Kill a database replica mid-request
Soak testing	Detect degradation over time	Run sustained load for 48 hours
Game days	Practice incident response	Simulate a region failover with the on-call team

✓

best practice

Automate reliability tests in CI/CD. A circuit breaker that has never been exercised will fail when you need it most. Run fault-injection tests regularly and treat regressions as high-priority bugs.

$Blueprint — Engineering Documentation·Section ID: SD-REL-01·Revision: 1.0

X GitHub YouTube Instagram Facebook Discord Slack Telegram Buy me a coffee Ko-fi Sponsor