RAG vs. Fine-Tuning: A 2026 Decision Framework for Practitioners

The most common reason teams fine-tune a model is to "teach it our documents."

It is also the one thing fine-tuning is worst at.

That isn't an opinion. It's the finding of a peer-reviewed study presented at EMNLP 2024, and it should change how you spend your next quarter's compute budget.

What the research actually found

Ovadia et al. set out to answer the question directly: if you want a model to know something it doesn't currently know, is it better to fine-tune it on that material, or to retrieve the material at inference time?

They tested both across multiple models and tasks, including a "current events" set built specifically from information the models could not have seen in training, genuinely new knowledge.

Task: new factual knowledge	Fine-tuning	RAG
Mistral	0.504	0.875
Orca	0.511	0.876

RAG didn't win narrowly. It roughly doubled the score. And on the broader MMLU benchmarks, RAG with the base model as generator beat fine-tuning in almost every case.

The authors' conclusion is blunt: LLMs struggle to learn new factual information through unsupervised fine-tuning.

Why, and the mental model that makes it obvious

Here's the framing that makes this stop being surprising.

Fine-tuning adjusts weights. It shapes the model's behaviour, its tone, its output format, its sense of what a good answer to this kind of question looks like. You are teaching a habit, not a fact.

RAG puts the document in the context window. The model reads it. You are handing it the answer sheet.

Think about how you would learn a colleague's phone number. Fine-tuning is reciting it a thousand times and hoping it sticks, and it might, imperfectly, mixed up with the other numbers you've recited. RAG is writing it down and looking at the note when someone asks. One of those is dramatically more reliable, and it isn't the memorisation.

The decision framework

Almost every real decision collapses to one question: do you need the model to know something, or to do something?

You need...	Use	Because
Answers grounded in changing data (policies, prices, tickets, docs)	RAG	Update the source, not the model
Citations and traceability	RAG	The retrieved chunk is the citation
A large or sensitive corpus	RAG	Nothing gets baked into weights
A strict output format or schema prompting can't enforce	Fine-tune	Format is behaviour
A consistent voice or persona	Fine-tune	Style is behaviour
Sub-200ms latency	Fine-tune	Removes the retrieval hop entirely
High volume, where a small tuned model beats a frontier API	Fine-tune	Unit economics
Domain reasoning patterns (how a radiologist reasons, not what they know)	Fine-tune	Reasoning shape is behaviour

Read the "because" column top to bottom and the rule writes itself: RAG for knowledge, fine-tuning for behaviour.

Where the benchmark and production disagree

This is the part most write-ups skip, and it's where the honesty lives. The paper is strong evidence, but it is not the last word, and quoting it as "fine-tuning can't learn facts" over-reads it in three specific ways.

1. It tested unsupervised fine-tuning. Most teams don't do that.

Ovadia et al. fine-tuned on raw text, dumping documents in and continuing pre-training. In production, teams overwhelmingly use supervised fine-tuning on question–answer pairs, or LoRA. That's a materially different procedure, and the paper doesn't test it. The finding is real; the generalization to "all fine-tuning" is not.

2. The paper's own footnote is the workaround.

Buried in the conclusion is the fix: exposing the model to many paraphrased variations of the same fact improves knowledge retention. Teams that do this, generating a dozen rephrasings per fact before training, report far better results than the raw-text baseline. The paper didn't find that fine-tuning can't inject knowledge. It found that the naive way of doing it doesn't work well.

3. Benchmarks assume retrieval works. Production is where it doesn't.

This one cuts the other way, and it's the biggest gap of all. In a clean benchmark, the retriever finds the right passage and RAG's ceiling is the model's reading comprehension. In production, your corpus is a decade of inconsistent PDFs, and RAG's real accuracy is capped by retrieval recall, not by the LLM at all. The benchmark measures RAG at its best. Your users meet it at its worst.

So the honest reading is narrower than the headline, and more useful:

What I'd do differently

Having watched teams burn quarters on this: the failure is almost never picking the wrong one. It's picking either one before you've defined what "working" looks like.

If I started again, in order: write twenty real questions with the answers you'd accept, before touching a model. Ship RAG first, because it's the fastest path to a system you can inspect. Then, when it disappoints, look at the retrieved chunks before the model: nine times out of ten the answer wasn't in the context, and no amount of fine-tuning fixes that. Reach for fine-tuning only when you have a named complaint RAG can't answer: the format won't hold, latency is over budget, or unit cost doesn't clear.

The teams that shipped weren't the ones who chose correctly. They were the ones who could tell they'd chosen wrong, quickly.

The costs nobody budgets for

Both options have a bill that arrives later.

RAG's hidden cost is latency and plumbing. Retrieval adds roughly 50–300ms per query before generation even starts, and that's the happy path, add a reranker and it grows. You also inherit a whole second system to maintain: chunking strategy, embedding model, index freshness, retrieval quality. When RAG fails in production, it's almost never the LLM. It's retrieval returning the wrong chunk.

Fine-tuning's hidden cost is that it freezes you. Every meaningful change to the underlying knowledge means retraining. And you've now coupled yourself to a base model, when a better one ships next quarter, your competitors switch in an afternoon and you re-run your pipeline.

So most production systems use both

This isn't a fence-sit; it's what the architecture actually converges to. Fine-tune a smaller model for the behaviour you need, the format, the voice, the domain reasoning, then feed it retrieved context for the facts. You get consistent output and current knowledge, often on a cheaper model than the frontier API you started with.

Key takeaways

1. Raw-text fine-tuning is a bad way to teach facts: RAG more than doubled it on new knowledge. But that's a claim about unsupervised fine-tuning, not all of it.
2. The rule is knowledge vs. behaviour. Facts, citations, changing data → RAG. Format, voice, latency, unit cost → fine-tuning.
3. Never put changing information in weights. You'll be retraining forever, and the model won't cite its source.
4. The benchmark's blind spot is your corpus. RAG's real ceiling is retrieval recall, and no paper tested your decade of messy PDFs.
5. Write the twenty questions before you pick an architecture. The teams that shipped weren't the ones who chose right, they were the ones who could tell they'd chosen wrong, fast.

If you fine-tuned a model to teach it your documents, did it actually hold up? And if you augmented with paraphrases, did that change things? I'd genuinely like to hear from anyone who's tested both. 👇

---

Source: Ovadia, O. et al. "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs", arXiv

.05934, published at EMNLP 2024. Latency figures are typical production ranges, not results from that paper.