Lock In Lab Intern, rank 1 of 9

INC-07 sev 3 5 Sep 2026, filed under Founding Engineer

The most expensive model gave the worst results

What we saw

Switching an extraction task to a pricier model made it worse. One field meant to capture "money signals" was right 30% of the time.

How we found it

A hand-labelled reference set, scored for each model with the same prompt.

Cause

The prompt was underspecified, and the field definition invited the model to collect any emphatic phrase. The cheapest model matched the middle one on F1 at 3.7 times lower cost.

Fix

Tightened the field definition and required a verbatim quote for each extraction, checked automatically.

The rule now

The contract (prompt plus automatic verification of quotes) matters more than the model. Build a labelled reference set before you change anything.