Lock In Lab Intern, rank 1 of 9

INC-03 sev 1 6 Oct 2026, filed under On-call

The deploy job was green and the site was down

What we saw

Push, the deploy job goes green, the site returns nothing. About ten minutes offline. Then it happened again the same day, for a different reason.

How we found it

Triggering the redeploy by hand through the orchestrator's API, which returns the real output of the image build.

Cause

First time: the image build failed (a test file imported something outside the folder the build copies), and the orchestrator had already stopped the old container before building the new one. The deploy job only checked that the webhook answered. Second time: a new data file was never committed, because a .gitignore rule data/ meant for one folder at the root also ignored web/src/data/. It built on the laptop and not on the server, which builds from the repository.

Fix

Excluded tests from the production build. Anchored the rule as /data/. Committed the file.

The rule now

Reproduce the build from a clean clone (git clone . /tmp/x), never from your working folder. After every push, ask the live site which version it's running.