|$ curl https://forge-ai.dev/api/markdown?path=docs/deploy/zero-downtime

$cat docs/deploy-—-zero-downtime-deployments.md

updated Recently·14 min read·published

Deploy — Zero-Downtime Deployments

◆CI/CD◆Intermediate

Introduction

Zero-downtime deployments ensure your application remains available to users while new versions are being deployed. Users never see a maintenance page or a connection error during a release.

This section covers the major deployment strategies — blue-green, rolling updates, canary releases — along with health checks, database migration strategies, rollback approaches, and platform-specific implementations.

Why Zero-Downtime?

Downtime during deployments directly impacts revenue, user trust, and SLA compliance. Even brief outages — a few seconds during a container restart — can fail health checks, drop active connections, and trigger alert fatigue.

Impact	Cost
Lost revenue during downtime	$5,600/minute (average for Fortune 1000)
SLA breach penalties	Service credits, contract renegotiation
User churn	Users leave after repeated poor experiences

ℹ

info

Zero-downtime is not just a deployment strategy — it is an application architecture requirement. Your code must handle graceful shutdown, connection draining, and stateless requests to make zero-downtime deployments possible.

Blue-Green Deployments

Blue-green deployments maintain two identical production environments. The "blue" environment serves live traffic while the "green" environment receives the new deployment. Once the green environment is validated, traffic is switched instantly.

blue-green.yaml

YAML

1	# AWS ECS: Blue-Green with CodeDeploy
2	apiVersion: apps/v1
3	kind: Deployment
4	metadata:
5	name: app-green
6	spec:
7	replicas: 3
8	selector:
9	matchLabels:
10	app: web
11	slot: green
12	template:
13	metadata:
14	labels:
15	app: web
16	slot: green
17	spec:
18	containers:
19	- name: app
20	image: ghcr.io/myorg/web-app:v2.0.0
21	ports:
22	- containerPort: 3000
23	---
24	# Service switches between blue/green via target groups
25	# CodeDeploy handles the traffic shift:
26	# 1. Deploys to green target group
27	# 2. Runs validation tests
28	# 3. Shifts traffic from blue to green
29	# 4. Keeps blue as rollback target

Pros	Cons
Instant rollback (switch traffic back)	Double infrastructure cost during deployment
Full testing of green before traffic switch	Database schema changes must be backward-compatible

Rolling Updates

Rolling updates replace pods (or instances) incrementally. At any point, a mix of old and new versions are running. This is the default strategy for Kubernetes Deployments.

rolling-update.yaml

YAML

1	apiVersion: apps/v1
2	kind: Deployment
3	metadata:
4	name: web-app
5	spec:
6	replicas: 6
7	strategy:
8	type: RollingUpdate
9	rollingUpdate:
10	maxSurge: 2 # Allow 2 extra pods during update
11	maxUnavailable: 0 # Never reduce below 6 available pods
12	selector:
13	matchLabels:
14	app: web
15	template:
16	metadata:
17	labels:
18	app: web
19	spec:
20	containers:
21	- name: app
22	image: ghcr.io/myorg/web-app:v2.0.0
23	readinessProbe:
24	httpGet:
25	path: /ready
26	port: 3000
27	periodSeconds: 5
28	terminationGracePeriodSeconds: 60

✓

best practice

Set maxUnavailable: 0 to ensure you never lose capacity during a rollout. SetmaxSurge to control how many extra pods are created during the transition. For stateless apps, maxSurge: 1 is usually sufficient.

Canary Releases

Canary releases send a small percentage of traffic to the new version while most users remain on the old version. Monitor error rates and metrics before gradually increasing traffic.

canary-ingress.yaml

YAML

1	# NGINX Ingress: Canary with header routing
2	apiVersion: networking.k8s.io/v1
3	kind: Ingress
4	metadata:
5	name: web-app-canary
6	annotations:
7	nginx.ingress.kubernetes.io/canary: "true"
8	nginx.ingress.kubernetes.io/canary-weight: "10"
9	spec:
10	ingressClassName: nginx
11	rules:
12	- host: app.example.com
13	http:
14	paths:
15	- path: /
16	pathType: Prefix
17	backend:
18	service:
19	name: web-app-canary
20	port:
21	number: 80

istio-canary.yaml

YAML

1	# Istio: Traffic splitting with VirtualService
2	apiVersion: networking.istio.io/v1beta1
3	kind: VirtualService
4	metadata:
5	name: web-app
6	spec:
7	hosts:
8	- web-app
9	http:
10	- route:
11	- destination:
12	host: web-app-stable
13	weight: 90
14	- destination:
15	host: web-app-canary
16	weight: 10

ℹ

info

Use traffic-mirroring (shadow traffic) to test new versions with real production traffic without affecting users. Mirrored responses are discarded, but errors and latency are captured for analysis.

Health Checks & Readiness Probes

Health checks are critical for zero-downtime deployments. A readiness probe tells the load balancer when a pod is ready to receive traffic. A liveness probe tells Kubernetes when to restart a crashed container.

health-checks.ts

TypeScript

1	// Express.js: Health check endpoints
2	app.get("/healthz", (req, res) => {
3	// Liveness: is the process alive?
4	res.status(200).json({ status: "ok" });
5	});
6
7	app.get("/ready", async (req, res) => {
8	try {
9	// Readiness: can the app serve traffic?
10	await db.query("SELECT 1");
11	await redis.ping();
12	res.status(200).json({ status: "ready" });
13	} catch (error) {
14	res.status(503).json({ status: "not ready", error: error.message });
15	}
16	});
17
18	// Graceful shutdown: drain connections before terminating
19	process.on("SIGTERM", async () => {
20	console.log("SIGTERM received, starting graceful shutdown");
21
22	// Stop accepting new connections
23	server.close(() => {
24	console.log("HTTP server closed");
25	});
26
27	// Close database connections
28	await db.end();
29
30	// Wait for in-flight requests to complete (max 30s)
31	setTimeout(() => process.exit(0), 30000);
32	});

⚠

warning

Without graceful shutdown, a SIGTERM kills in-flight requests immediately. Always handle SIGTERM, stop accepting new connections, drain existing ones, and exit cleanly. This is essential for zero-downtime rolling updates.

Database Migrations During Deploy

Database schema changes are the hardest part of zero-downtime deployments. Both the old and new application versions run simultaneously, so the schema must be compatible with both.

Expand-and-Contract Pattern

1. Add new column (expand) → 2. Deploy code that reads/writes both columns → 3. Backfill data → 4. Remove old column (contract). Never do step 1 and 4 in the same deploy.

Never drop columns in the same deploy as code removal

The old code version still reads the column until all old pods are terminated. Deploy code changes first, then clean up the schema in a separate migration.

Use backward-compatible migrations

Add columns as nullable, add indexes CONCURRENTLY, avoid transactions that lock tables for extended periods.

safe-migration.sql

SQL

1	-- Safe migration: expand step
2	-- Add new column (nullable, no lock)
3	ALTER TABLE users ADD COLUMN display_name TEXT;
4
5	-- Backfill existing rows (batched)
6	UPDATE users SET display_name = full_name
7	WHERE display_name IS NULL
8	LIMIT 10000;
9
10	-- Add index concurrently (no table lock)
11	CREATE INDEX CONCURRENTLY idx_users_display_name
12	ON users (display_name);
13
14	-- Later deploy (contract step, after code fully migrated):
15	-- ALTER TABLE users DROP COLUMN full_name;

Rollback Strategies

When a deployment goes wrong, you need a fast, reliable rollback. The strategy depends on what went wrong — code issue, database migration, or infrastructure problem.

Scenario	Rollback Approach
Code bug only	Revert to previous container image version
Code + DB migration	Roll forward: deploy fix, then reverse migration
Bad config	Fix ConfigMap/Secret and restart pods

rollback.sh

Bash

1	# Kubernetes: Rollback a deployment
2	kubectl rollout undo deployment/web-app -n production
3
4	# Rollback to a specific revision
5	kubectl rollout history deployment/web-app -n production
6	kubectl rollout undo deployment/web-app --to-revision=3 -n production
7
8	# Check rollout status
9	kubectl rollout status deployment/web-app -n production
10
11	# ECS: Update service to use previous task definition
12	aws ecs update-service \
13	--cluster production \
14	--service web-app \
15	--task-definition web-app:3 \
16	--force-new-deployment

ℹ

info

Prefer "roll forward" over rollback for database migrations. Rolling back code is fast, but rolling back a migration can lose data. Instead, deploy a fix and migrate forward.

Load Balancer Drain

Connection draining ensures that in-flight requests complete before a pod or instance is removed from the load balancer. Without it, active connections are severed abruptly.

drain.sh

Bash

1	# AWS ALB: Connection draining via target group attribute
2	aws elbv2 modify-target-group-attributes \
3	--target-group-arn arn:aws:elasticloadbalancing:... \
4	--attributes \
5	Key=deregistration_delay.timeout_seconds,Value=60 \
6	Key=target_group_health.unhealthy_state_routing.minimum_healthy_targets.percentage,Value=50
7
8	# NGINX Ingress: Drain annotations
9	# metadata:
10	# annotations:
11	# nginx.ingress.kubernetes.io/upstream-keepalive-connections: "32"
12	# nginx.ingress.kubernetes.io/server-snippet: \|
13	# keepalive_timeout 60s;

Implementation on Major Platforms

Each platform has its own primitives for zero-downtime deployments. Choose the approach that matches your infrastructure.

Platform	Strategy
Kubernetes	Rolling update (default), blue-green, canary via Argo Rollouts or Istio
AWS ECS	Rolling update, blue-green via CodeDeploy
Vercel	Automatic: serves old version while new one builds, instant switch
Cloud Run	Revision-based: new revision receives traffic after readiness, instant rollback

Best Practices

✓ Implement graceful shutdown

Handle SIGTERM, stop accepting new connections, drain in-flight requests, and close database connections cleanly.

✓ Use readiness probes

Never route traffic to a pod that is not fully initialized. Readiness probes ensure the load balancer only sends requests to healthy instances.

✓ Make database migrations backward-compatible

Use expand-and-contract. Never assume the schema is clean — both old and new code versions run against it simultaneously.

✓ Automate rollback on health check failure

Configure your orchestrator to automatically revert if readiness probes fail for a configured number of consecutive checks.

$Blueprint — Engineering Documentation·Section ID: CICD-ZDT·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor