RAG, fine-tuning or prompting: how to choose
Choose prompting first, because it is the fastest way to test an AI feature. Choose RAG when answers must come from documents that change. Choose fine-tuning when you need a consistent style or format that prompting cannot reach. Many products use prompting and RAG together.
Published , updated 8 min read
What are the three options?
Prompting gives a model instructions and examples at the start of each request. It changes the model's behavior for one request, and it needs no training.
Retrieval-augmented generation (RAG) searches your documents for the text that answers a question, then gives that text to the model with the question. The model answers from the text it receives. The approach was described in a 2020 research paper by Lewis and colleagues (source: Lewis et al., 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks).
Fine-tuning trains a model further on your examples. It changes how the model writes and responds, and it is not a reliable way to store facts that change.
How do the options compare?
The table below states what each option is for and where it falls short. Use it to find the option that fits your first version, then test it on your own examples.
| Option | What it changes | Use when | Main limit |
|---|---|---|---|
| Prompting | Instructions and examples in each request | The task is clear and the needed knowledge fits in the request | Large document sets do not fit in one request, and cost rises with prompt length. Small sets can often go in one request, so test that before you build retrieval. |
| RAG | The source text the model receives for each question | Answers must come from documents that change often, and sources must be shown | Answers depend on search quality, so the document set needs care |
| Fine-tuning | The model's own behavior, style and format | The output needs a fixed style, format or behavior that prompts cannot hold | Facts go stale, and each change needs new training and tests |
When should I start with prompting?
Start with prompting when you are testing whether an AI feature works at all. It is quick to change, and it shows the failure modes early. Write the instructions, add a few examples and run your evaluation set.
Move to another option only when the tests show a clear gap. A gap in facts points to RAG. A gap in style or format points to fine-tuning. A gap in both points to a combination.
When does RAG fit?
RAG fits products that answer questions from manuals, policies, product documents, tickets or research. It lets you update the answer by changing the documents, without retraining the model. It also lets the product show the source of each answer.
RAG adds parts to build and maintain. You need a process to import documents, split them into sections, search them and check that the right section was found. Many RAG failures start in search, not in the model, so test the search results before the answers.
- The answers depend on documents that change
- Users need to see the source of an answer
- The document set is large and cannot fit in one request
- You need access rules, so users see only the documents they may read
When does fine-tuning fit?
Fine-tuning fits a stable task where the output must follow a fixed pattern. Examples include a fixed report format, a classification with set labels or a tone that prompts cannot hold across many requests.
Do not use fine-tuning to add facts that change. The model will not know about a new policy until you train it again, and the old facts may remain in its answers. Use RAG for facts and fine-tuning for behavior.
How do I choose in practice?
Follow these steps. Each step uses your own examples, not a public benchmark.
- Write the task and five to ten example inputs with a good output for each.
- Test prompting alone. Record the failures by type: wrong facts, wrong format or wrong tone.
- If the failures are wrong facts, add RAG over the documents that hold those facts. Test the search results first.
- If the failures are format or tone after RAG is in place, consider fine-tuning on a set of approved examples.
- Repeat the tests after each change, and keep the evaluation set for later releases.
Can I use more than one option?
Yes. Many products use prompting for instructions, RAG for facts and a fixed output format for structure. Add each part only when the tests show it is needed. Each added part makes the product harder to change, so keep the design as small as the results allow.
How do I test RAG before launch?
Test the search step first. For each example question, check that the passages the search returns contain the answer. If they do not, fix the document import, the splitting or the search settings before you test the model.
Then test the answers. Check that each answer matches the passages it cites, that it states when the documents do not answer the question, and that it does not add facts that the passages lack. Record each failure and its cause, so the next change targets the real problem.
Questions about this guide
Is RAG cheaper than fine-tuning?
It depends on the task and the volume. RAG has costs for document storage, search and the longer prompts that carry source text. Fine-tuning has costs for training data, training runs and tests after each change. Compare both on your own volume before you decide.
Does RAG stop a model from making things up?
It reduces the risk, but it does not remove it. The model can still misread the source or add facts the source does not state. Check answers against the sources, and let the product say when no source supports an answer.
Should I fine-tune to teach the model our product?
Use RAG for product facts, because they change. Fine-tune only for a fixed style or format. Keep the product guide in documents that RAG can search, so the answers stay current when the guide changes.