Small models deserve a second look
For a while the default answer to almost any language task was to reach for the largest model available and hope the bill stayed reasonable. That habit made sense when smaller models could barely follow instructions. It makes less sense now.
A model with a few billion parameters will not write a convincing essay on a niche topic. It does not have to. A surprising share of real work is narrow: pull a date out of an email, decide which of five categories a support ticket belongs to, rewrite a sentence in plain language, check whether two addresses describe the same place. For tasks like these a small model, given clear instructions and a handful of examples, often lands within a few points of a much larger one.
What you get in return
- Latency. Answers arrive fast enough to sit inside an interactive tool instead of behind a spinner.
- Cost. Running thousands of classifications stops being a budgeting decision.
- Control. A model that fits on one machine can run where the data already lives, which simplifies questions about who sees what.
Where it breaks down
Small models are brittle in specific ways. They lose track of long instructions, they are more sensitive to how a prompt is phrased, and they fail quietly: a wrong answer looks exactly as confident as a right one. That last point matters most. If you move a task to a smaller model, you need a way to measure whether it still works, not just a feeling that the outputs look fine.
My rule of thumb is simple. Start with a large model to find out what good output looks like, keep a small set of examples where you know the right answer, and then try the smallest model that still passes them. More often than I expected, the small one is enough.