Incidents

Someone was tidying up old Secrets this morning. Shortly after, the database Pod restarted, and now every page of the shop fails. The web and API Pods look fine.

A teammate was preparing a v2 of the API and "just tweaked the Service". The API Pods are all Running and Ready, but the website fails on every request.

The API team shipped a small config change a minute ago: "point at the new database host". The rollout finished, every Pod is Ready, and yet every request now fails.

Security asked for the database to be locked down, and someone applied a NetworkPolicy before lunch. Postgres is Running and Ready. The API says it can't reach it.

Someone added a liveness probe to the Redis cache "so Kubernetes restarts it if it hangs". Since then the shop fails in waves: fine for a bit, then errors, then fine again.

One of the worker machines just lost power. Kubernetes should reschedule everything, right? The web and API Pods moved, but the shop is still down.

Someone wanted to pause the fake customers (shop-users) while they read some logs, so they ran a scale command from their shell history. The customers kept coming. The shop stopped answering them.

A teammate tidied up the Services so that "everything uses port 80". The API Pods are Running and Ready, and its Service still lists them as endpoints. Every request fails anyway.

Someone switched the Redis cache to the smaller alpine image. The cache Deployment uses the Recreate strategy, so the old Pod stopped first. The new one never started.

During a cleanup, a teammate deleted a Service with no cluster IP: "it said None, it looked broken". The database Pod is Running and Ready, but the API can't find it.

Someone made the Secret keys match the team's style: in Secret shop-db, POSTGRES_PASSWORD is now just `password`. The database Pod was restarted soon after, and it keeps crashing.

Ops patched the kernel on both worker Nodes: drain, patch, reboot. The Nodes are back and Ready. Every Pod of the shop is Pending.

Someone was trimming kube-system to save memory and scaled down a Deployment they didn't recognise. Every shop Pod is Running and Ready, yet no customer gets through.

The database felt slow, and the DBA asked for 8Gi of memory for Postgres. Someone set it as the memory request on the StatefulSet. shop-db-0 went away and never came back.

Security asked to restrict what the API can talk to. The new policy allows exactly the database and the cache, each on its own port. Since it went in, the API can't reach either of them.

The platform team added a ResourceQuota to every namespace at lunch, and nothing happened. An hour later the database Pod was restarted during Node maintenance. It never came back, and there isn't even a Pending Pod.