|$ curl https://forge-ai.dev/api/markdown?path=docs/ops/logging

$cat docs/ops-—-logging-&-structured-logs.md

updated Recently·14 min read·published

Ops — Logging & Structured Logs

◆Ops◆Intermediate

Introduction

Logging is the foundation of observability. Structured logs — machine-parseable, JSON-formatted log entries — enable efficient search, filtering, and alerting across your entire system. Unstructured text logs are nearly impossible to query at scale.

This section covers structured vs unstructured logging, Pino and Winston configuration, log levels, centralized aggregation (Loki, Datadog, Elasticsearch), correlation IDs, log-based alerting, and cost management.

Structured vs Unstructured Logging

Structured logs output JSON objects instead of free-form text. Each field is named and typed, making logs searchable, filterable, and analyzable by log aggregation tools.

structured-vs-unstructured.ts

TypeScript

1	// ✗ Unstructured (hard to search and parse)
2	console.log("User " + userId + " placed order " + orderId + " for $" + total);
3
4	// ✓ Structured (machine-readable, searchable)
5	logger.info("Order placed", {
6	event: "order.placed",
7	userId,
8	orderId,
9	total,
10	currency: "USD",
11	items: orderItems.length,
12	timestamp: new Date().toISOString(),
13	});
14
15	// Output (JSON):
16	// {"level":"info","event":"order.placed","userId":"u_123",
17	// "orderId":"ord_456","total":99.99,"currency":"USD",
18	// "items":3,"timestamp":"2025-01-15T10:30:00.000Z"}

ℹ

info

Structured logs enable powerful queries like "show me all orders over $100 in the last hour for users in the EU" — something impossible with unstructured text logs.

Pino & Winston

Pino is the fastest Node.js logger — it offloads JSON serialization to a worker thread. Winston is the most feature-rich logger with extensive transport support. Pino is preferred for performance-critical applications.

pino-setup.ts

TypeScript

1	// Pino: Fast structured logging
2	import pino from "pino";
3
4	const logger = pino({
5	level: process.env.LOG_LEVEL \|\| "info",
6	formatters: {
7	level(label) {
8	return { level: label };
9	},
10	},
11	timestamp: pino.stdTimeFunctions.isoTime,
12	serializers: {
13	err: pino.stdSerializers.err,
14	req: pino.stdSerializers.req,
15	res: pino.stdSerializers.res,
16	},
17	redact: ["req.headers.authorization", "password", "token"],
18	});
19
20	// Create child loggers with context
21	const apiLogger = logger.child({ service: "api" });
22	const dbLogger = logger.child({ service: "database" });
23
24	// Log with context
25	apiLogger.info({ userId: "u_123", method: "POST", path: "/api/orders" },
26	"Request received");
27
28	dbLogger.error({ err, query: "SELECT * FROM users" },
29	"Database query failed");

winston-setup.ts

TypeScript

1	// Winston: Feature-rich logging
2	import winston from "winston";
3
4	const logger = winston.createLogger({
5	level: process.env.LOG_LEVEL \|\| "info",
6	format: winston.format.combine(
7	winston.format.timestamp(),
8	winston.format.errors({ stack: true }),
9	winston.format.json()
10	),
11	defaultMeta: { service: "api", environment: process.env.NODE_ENV },
12	transports: [
13	new winston.transports.Console({
14	format: winston.format.combine(
15	winston.format.colorize(),
16	winston.format.printf(({ timestamp, level, message, ...meta }) => {
17	const metaStr = Object.keys(meta).length ? JSON.stringify(meta) : "";
18	return `${timestamp} [${level}]: ${message} ${metaStr}`;
19	})
20	),
21	}),
22	new winston.transports.File({ filename: "logs/error.log", level: "error" }),
23	new winston.transports.File({ filename: "logs/combined.log" }),
24	],
25	});

✓

best practice

Use Pino for new projects — it is 5-10x faster than Winston and outputs structured JSON by default. Use pino-pretty in development for human-readable output.

Log Levels

Log levels indicate the severity and purpose of a log entry. Use them consistently across your codebase so operators can filter and alert on the right signals.

Level	Purpose	Production?
debug	Detailed diagnostic info for developers	No
info	Normal operations: request served, job completed	Yes
warn	Unexpected but handled: deprecated API used, retry triggered	Yes
error	Failures requiring attention: unhandled exception, DB failure	Yes
fatal	Process will crash: out of memory, un recoverable error	Yes (triggers alert)

⚠

warning

Do not log sensitive data at any level. Pino'sredact option automatically removes fields like passwords, tokens, and authorization headers from all log output.

Logging in Node.js Applications

Production Node.js applications need structured logging middleware, uncaught exception handlers, and request context propagation.

express-logging.ts

TypeScript

1	// Express middleware: Log every request with context
2	import pino from "pino";
3	import { randomUUID } from "crypto";
4
5	const logger = pino({ level: "info" });
6
7	app.use((req, res, next) => {
8	const requestId = randomUUID();
9	req.log = logger.child({
10	requestId,
11	method: req.method,
12	url: req.url,
13	});
14	req.requestId = requestId;
15
16	const start = Date.now();
17	res.on("finish", () => {
18	req.log.info({
19	statusCode: res.statusCode,
20	duration: Date.now() - start,
21	}, "Request completed");
22	});
23
24	next();
25	});
26
27	// Catch unhandled errors
28	process.on("uncaughtException", (err) => {
29	logger.fatal({ err }, "Uncaught exception");
30	process.exit(1);
31	});
32
33	process.on("unhandledRejection", (reason) => {
34	logger.error({ err: reason }, "Unhandled rejection");
35	});

Centralized Log Aggregation

In production, logs from multiple services and instances must be aggregated into a central system for search, analysis, and alerting. The three main approaches are Loki, Datadog, and Elasticsearch.

Platform	Type	Best For
Grafana Loki	Open-source, index-free	Cost-effective, Grafana ecosystem
Datadog	Managed SaaS	Unified observability (logs, metrics, traces)
Elasticsearch	Full-text search engine	Complex search queries, ELK stack

docker-compose.yml

YAML

1	# Docker Compose: Loki + Promtail for log aggregation
2	version: "3.8"
3	services:
4	loki:
5	image: grafana/loki:2.9.0
6	ports:
7	- "3100:3100"
8	volumes:
9	- ./loki-config.yml:/etc/loki/local-config.yaml
10
11	promtail:
12	image: grafana/promtail:2.9.0
13	volumes:
14	- /var/log:/var/log:ro
15	- ./promtail-config.yml:/etc/promtail/config.yml
16	command: -config.file=/etc/promtail/config.yml
17
18	grafana:
19	image: grafana/grafana:10.2.0
20	ports:
21	- "3000:3000"
22	environment:
23	GF_SECURITY_ADMIN_PASSWORD: admin

Log Search & Filtering

Effective log search relies on structured fields and consistent naming conventions. Use LogQL (Loki), KQL (Datadog), or Kibana queries (Elasticsearch) to find relevant logs quickly.

log-search.ts

TypeScript

1	// LogQL (Grafana Loki): Query examples
2	// Find all error logs for the API service
3	{service="api"} \|= "error"
4
5	// Filter by structured field
6	{service="api"} \| json \| level="error"
7
8	// Rate of errors per minute
9	rate({service="api"} \| json \| level="error" [1m])
10
11	// Top 10 slowest requests
12	{service="api"} \| json \| line_format "{{.url}} {{.duration}}ms"
13	\| sort by duration desc \| limit 10
14
15	// Datadog KQL: Query examples
16	// All 5xx errors in the last hour
17	service:api status:5xx @now-1h
18
19	// Errors with specific user
20	service:api status:error @user_id:u_12345

ℹ

info

Standardize your log field names across all services:service,requestId,userId,level,duration. This makes cross-service queries trivial.

Log-Based Alerting

Log-based alerts trigger when specific patterns appear in logs — error spikes, missing heartbeats, or critical events. They complement metric-based alerts for comprehensive monitoring.

loki-alerts.yml

TypeScript

1	// Loki alerting rules (in Prometheus format)
2	groups:
3	- name: app-alerts
4	rules:
5	- alert: HighErrorRate
6	expr: \|
7	sum(rate({service="api"} \| json \| level="error" [5m]))
8	/ sum(rate({service="api"} \| json [5m]))
9	> 0.05
10	for: 5m
11	labels:
12	severity: critical
13	annotations:
14	summary: "Error rate exceeds 5% for API service"
15
16	- alert: MissingHeartbeat
17	expr: \|
18	count_over_time({service="api"} \| json \| event="heartbeat" [5m]) == 0
19	for: 2m
20	labels:
21	severity: warning
22	annotations:
23	summary: "No heartbeat received from API for 5 minutes"

Log Retention & Costs

Logs are expensive to store at scale. A busy service can generate gigabytes of logs per day. Set retention policies based on compliance requirements and debug needs.

Tier	Retention	Storage
Hot	7 days	Fast SSD, full-text search
Warm	30 days	Cheaper storage, indexed
Cold	90 days - 1 year	S3/GCS archive, slow query

ℹ

info

Reduce log volume before storing: drop debug logs in production, sample high-frequency events (e.g., health checks), and use log levels correctly. A 50% reduction in log volume saves 50% in storage and query costs.

Correlation IDs for Request Tracing

Correlation IDs (also called trace IDs or request IDs) link log entries across services. When a request flows through API → worker → database, the same ID appears in every log entry, making end-to-end debugging possible.

correlation-ids.ts

TypeScript

1	// Generate and propagate correlation IDs
2	import { randomUUID } from "crypto";
3
4	// Middleware: Accept or generate request ID
5	app.use((req, res, next) => {
6	req.requestId = req.headers["x-request-id"] \|\| randomUUID();
7	res.setHeader("x-request-id", req.requestId);
8	req.log = logger.child({ requestId: req.requestId });
9	next();
10	});
11
12	// Pass correlation ID to downstream services
13	async function callDownstream(path: string, requestId: string) {
14	const response = await fetch(`https://downstream-service${path}`, {
15	headers: {
16	"x-request-id": requestId,
17	},
18	});
19	return response.json();
20	}
21
22	// Every downstream service logs with the same requestId
23	// enabling full trace reconstruction in your log aggregator

Best Practices

✓ Always use structured logging

JSON logs enable search, filtering, and analytics. Text logs are unparseable at scale.

✓ Use consistent field names

Standardize fields across services: service, requestId, userId, level, event, duration. Create a shared logging schema.

✓ Redact sensitive data

Use log redaction to strip passwords, tokens, and PII. Configure redaction at the logger level, not in individual log calls.

✓ Propagate correlation IDs

Pass request IDs across service boundaries via HTTP headers. This enables full request tracing across microservices.

$Blueprint — Engineering Documentation·Section ID: OPS-LOG·Revision: 1.0

X GitHub YouTube Instagram Facebook Buy me a coffee Sponsor