RAG vs fine-tuning: when to use each (and when to use both).

Most teams reach for fine-tuning first. Then they end up with a more expensive system that hallucinates on edge cases, can't keep up with new information, and quietly costs them 10× more per month than the alternative would have.

The actual decision is not "RAG or fine-tuning." It's about picking the right tool for the failure mode you're trying to fix. This article is the framework we use at eventech when we're scoping a custom AI system — and the cases where the answer is "both."

What each technique actually does.

RAG (retrieval-augmented generation)

RAG is a pattern where, at query time, you search a knowledge base for the most relevant chunks of context and prepend them to the prompt you send to the model. The model then answers using that retrieved context, not just what it learned during training.

You don't change the model. You change what the model sees when it answers.

What this fixes: the model doesn't know your private data, your latest docs, or your specific terminology. RAG injects that information at runtime.

Fine-tuning

Fine-tuning takes a base model and runs additional training on a dataset of (input, output) pairs you provide. The model's weights change. It learns the patterns, style, and reasoning present in your data.

You don't change what the model sees. You change how the model thinks.

What this fixes: the model doesn't follow your format, doesn't speak in your domain's voice, doesn't handle your edge cases, or is too slow/expensive to use off-the-shelf for your volume.

The three axes you actually care about.

Stop thinking about "intelligence" in the abstract. Every production decision comes down to three measurable properties:

• Freshness — how quickly the system can reflect new information. Hours? Days? Months?
• Behavior — how reliably the system produces the output shape, style, and reasoning you need.
• Cost & latency — what you pay per query and how fast you get a response.

RAG and fine-tuning trade off differently on these axes. Here's the cheat sheet:

• RAG wins on freshness — change the index, the model picks it up. No retraining.
• RAG wins on cost & latency for "give me context" tasks — prepending retrieved text is cheap and fast; the model is the same one you already use.
• Fine-tuning wins on behavior — when you need the model to consistently follow a format, match a voice, or apply a specific reasoning pattern.
• Fine-tuning wins on cost & latency for "do this 1M times a day" tasks — a smaller fine-tuned model is often 10× cheaper and faster than a giant general model doing the same task.

The decision tree we use at eventech.

When a client asks "should we fine-tune?" we ask four questions. The answers route us to RAG, fine-tuning, both, or neither.

1. Is the system failing because it doesn't know the answer, or because it doesn't behave the right way?

If it doesn't know the answer (it hallucinates facts, doesn't know your product, doesn't have your latest docs) — start with RAG. The information exists somewhere; you just need to find and inject it.

If it doesn't behave the right way (it writes the wrong format, talks too verbosely, doesn't follow your reasoning chain) — start with fine-tuning. The model knows things, it just doesn't act the way you need.

2. How often does the underlying information change?

If the answer is "daily or more often" — RAG. You cannot retrain a model every day (or shouldn't). RAG is the only practical way to keep a system current with high-velocity information.

If the answer is "rarely, or the patterns are stable" — fine-tuning is fine. The behavior you're encoding isn't going out of date next week.

3. Are you processing 100 queries a day, or 1,000,000?

Low volume: just use the best general model with RAG. Don't over-engineer it.

High volume: fine-tune a smaller model for the specific task. You will save real money and get sub-second latency. The fine-tuned 7B model is often as good as the general 70B for a narrow task, and it runs 5-10× faster and cheaper.

4. Can you explain every decision the model makes?

If yes is the requirement — RAG. Every answer comes with citations, and you can show the user exactly which document the model used.

If you need deep behavior control — fine-tuning. The model's reasoning is encoded in its weights; you can sample from it but you cannot easily point to the "evidence."

The hybrid pattern that handles most production cases.

For most production systems we ship at eventech, the answer is "RAG first, fine-tune the boundaries." Specifically:

• Retrieval handles the easy 80%. Your knowledge base is the source of truth. The model answers with citations. Latency is fine. Cost is fine. Freshness is fine.
• Fine-tuning handles the boundaries. Tone of voice, output format, edge-case handling, refusal patterns, terminology — all of this is encoded in the model. The retrieval gives the model the right facts; the fine-tuning gives it the right behavior.

Concretely, this means you ship a RAG system first, watch where it fails, and fine-tune to fix the failure modes. The fine-tune gets you from 85% accuracy to 95%. Pure RAG would have plateaued at 85%. Pure fine-tuning without RAG would have hallucinated 20% of the time.

When neither is the right answer.

There are problems where neither RAG nor fine-tuning is the right tool:

• Deterministic, rule-based workflows. If the answer is always the same given the same input, write the rule. Don't use an LLM. We see teams spend months trying to "fine-tune" a routing decision that a 50-line Python function would handle.
• Real-time control systems. If you need a model to make a millisecond decision in a control loop, an LLM is the wrong architecture. Look at classical ML, control theory, or simpler models.
• Compliance-locked decisions that cannot have any variance. If a regulator is going to ask "why did the model say that?" and the answer has to be a rule, not a probability distribution, don't use generative AI. Use deterministic software.

Most of our engagements start with the audit call confirming AI is the right tool. Sometimes the answer is "actually, you don't need us — you need a workflow tool, not an LLM." We will say that out loud.

What this looks like in production.

Here's the architecture we ship most often at eventech for a "give me accurate answers from our internal data" system:

1. Ingestion. Pull source documents (PDFs, Notion, Confluence, Google Drive, SQL) into a chunked, embedded index. Refresh on a schedule or webhook.
2. Retrieval. At query time, embed the user's question, find the top 5-20 most relevant chunks, optionally rerank with a cross-encoder.
3. Generation. Send the chunks plus the question to the model, with a fine-tuned system prompt that defines the output format, citation style, refusal behavior, and tone.
4. Validation. A second pass (often a smaller fine-tuned model) checks the answer against the retrieved chunks, flags unsupported claims, and forces a citation.
5. Observability. Every step is logged: query, retrieved chunks, prompt, model response, validation result, latency, cost. You can replay any session and explain every decision.

The retrieval is RAG. The behavior is fine-tuned. The validation is fine-tuned. The observability layer is what makes the whole thing auditable. This is what "custom AI system" actually means in production.

The cost comparison (rough numbers).

For a mid-size enterprise use case (say, 100K queries/month over a private knowledge base of 50K documents), here's a rough order-of-magnitude cost comparison:

• General model + RAG (off-the-shelf): $0.01–0.05 per query. $1K–5K/month. No fine-tuning cost.
• Fine-tuned smaller model + RAG (hybrid): $0.001–0.005 per query. $100–500/month. Plus $5K–20K one-time fine-tuning.
• Fully custom (private model + custom infra): $50K–200K one-time + $500–3K/month inference. Required for compliance or data residency.

The hybrid wins on most production workloads. Pure custom wins when you have compliance requirements that make public APIs non-starters.

How we help.

If you're at the "should we fine-tune?" stage and want a second opinion before you spend 6 weeks on a fine-tuning run that may not be the right answer, we do 30-minute audit calls. We'll tell you which path is right and what the realistic cost and timeline look like. We will also tell you if the answer is "you don't actually need us."

Our custom AI systems engagements are designed around the hybrid pattern above — RAG, fine-tuning, evaluation, observability, deploy. Fixed scope, shipped in 4–8 weeks.

Or send a brief if you have a specific use case in mind.