Retrieval first, fine-tuning later
A question that comes up often: should we fine-tune a model on our documents so it knows about them? Usually the better first step is retrieval. Store the documents, find the passages relevant to each question, and include them in the prompt. The model then answers from text it can actually see.
Why retrieval is a good default
- Updates are immediate. Change a document and the next answer reflects it. A fine-tuned model has to be trained again.
- Answers can cite sources. Because the passages are in the prompt, the system can show where an answer came from, and a reader can check it.
- Access control stays simple. You can retrieve only from documents the current user is allowed to see. Knowledge baked into model weights cannot be filtered that way.
The part that takes the effort
Most retrieval problems are search problems. If the right passage is not found, no model will produce the right answer. Splitting documents into sensible chunks, keeping titles and section headings attached to the text, and combining keyword search with embedding search tend to matter more than the choice of model.
It also pays to test retrieval on its own. For a set of questions, check whether the correct passage appears in the top results before looking at the generated answers. When an answer is wrong, this tells you whether search or generation is to blame.
When fine-tuning makes sense
Fine-tuning is better at teaching a model how to behave than what to know: a consistent format, a particular tone, a narrow classification task with many examples. Those are real needs. They are just different from "answer questions about our wiki", which is what most teams are asking for when the idea first comes up.