Evaluating outputs without fooling yourself
The most common way to evaluate a language model feature is also the least reliable: try a few inputs, read the answers, and decide they look good. I have done this many times. It is useful for the first hour and misleading after that.
The problem is not laziness. Fluent text is persuasive. A well-formed paragraph reads as correct unless you already know the answer, and when you test your own feature you tend to pick inputs you expect to work.
A small, honest test set
What helps most is a fixed set of examples with known good answers, written down before you start tuning. It does not need to be large. Thirty to fifty cases that cover the ordinary inputs, the awkward ones, and a few that should be refused will tell you more than a long afternoon of manual testing.
- Include inputs that real users will send, typos and all.
- Include cases where the correct behavior is to say "I don't know".
- Keep the set stable, so that a change in the score means a change in the system.
Checking the answers
Some outputs can be checked exactly: a category, a number, a date. For free text, a short checklist per example works better than a single overall grade. Does the answer mention the key fact? Does it invent anything? Is it the right length? Using another model as a grader can speed this up, but it inherits similar blind spots, so spot-check its judgments by hand.
Watch the failures, not the average
An average score hides the cases that matter. A change that improves the average by two points and breaks three previously correct answers is not obviously an improvement. Reading the failures after every change is slower than looking at a number, and it is where most of the useful information is.