The most expensive model gave the worst results
What we saw
Switching an extraction task to a pricier model made it worse. One field meant to capture "money signals" was right 30% of the time.
How we found it
A hand-labelled reference set, scored for each model with the same prompt.
Cause
The prompt was underspecified, and the field definition invited the model to collect any emphatic phrase. The cheapest model matched the middle one on F1 at 3.7 times lower cost.
Fix
Tightened the field definition and required a verbatim quote for each extraction, checked automatically.
The rule now
The contract (prompt plus automatic verification of quotes) matters more than the model. Build a labelled reference set before you change anything.