Then production breaks.
Sixteen incidents from our own work in 2026, building with coding agents every day. Names of products, clients and servers are removed; everything else is as it happened. Five come with a small repo that reproduces the failure and a check that won't pass until you fix it.
Handling an incident means writing its postmortem on the incident page: symptom, cause, fix, the rule. That counts towards promotions from On-call upwards.
| # | Incident | Severity | When | Practice repo | You |
|---|---|---|---|---|---|
| INC-01 | The bot that graded its own homeworkWhy 40 green tests can mean nothing, and how to write the 41st that matters. | sev 1 | 8 Sep 2026 | yes | |
| INC-02 | We rolled back a safety fix because a sentence changedHow an automatic safety net can be the thing that hurts you. | sev 2 | 8 Sep 2026 | ||
| INC-03 | The deploy job was green and the site was downWhy "it works on my machine" survives into 2026, and the two commands that catch it. | sev 1 | 6 Oct 2026 | yes | |
| INC-04 | Connection error that wasn't the networkA one-line config bug that only exists in production. | sev 2 | 5 Sep 2026 | yes | |
| INC-05 | 437 rows processed, zero results savedWhy a success counter is not evidence. | sev 2 | 5 Sep 2026 | ||
| INC-06 | The bill didn't come from the volumeWhere AI money actually goes. | sev 3 | 6 Sep 2026 | ||
| INC-07 | The most expensive model gave the worst resultsHow to compare models without fooling yourself. | sev 3 | 5 Sep 2026 | ||
| INC-08 | Login returns 200 and the session never startsThe one command that would have saved an evening. | sev 2 | 3 Sep 2026 | yes | |
| INC-09 | The page loads and no button does anythingWhen the bug isn't in your code at all. | sev 3 | 3 Sep 2026 | ||
| INC-10 | Healthcheck red, app perfectly fineWhy localhost isn't always local. | sev 3 | 2 Aug 2026 | ||
| INC-11 | Freeing port 8000 shut down DockerWhy permissions and hooks exist. | sev 2 | 3 Sep 2026 | ||
| INC-12 | Seven agents at once, quota gone by mid-morningHow to size parallel work. | sev 3 | 17 Nov 2026 | ||
| INC-13 | Our setup script replaced the CI that ran the testsYour tooling is also a contributor that can be wrong. | sev 2 | 16 Nov 2026 | ||
| INC-14 | Secrets in git, across dozens of reposWhere secrets leak when an agent is doing the typing. | sev 1 | 16 Nov 2026 | ||
| INC-15 | The cause we found by elimination was wrongHow agents (and people) talk themselves into the wrong root cause. | sev 3 | 18 Aug 2026 | ||
| INC-16 | The backup rotation deleted the only good backupWhy "keep the last 7" is a dangerous rule. | sev 1 | 19 Nov 2026 | yes |