Prompt engineering vs fine-tuning: climb the ladder one rung at a time

The question people are actually asking

"Should we fine-tune?" is rarely the real question. The real question is one of two things: "how do I make the model behave the way I want", or "how do I make the model know what we know". Those are different problems with different tools, and confusing them is the most common strategic mistake we see in early LLM projects.

There are three tools. Prompt engineering changes the instructions and examples the model sees on each request. Retrieval-augmented generation changes what information is placed in front of the model per request. Fine-tuning changes the model's weights, and therefore its default behaviour.

They form a ladder. Each rung costs more, takes longer and is harder to undo than the one below. The discipline is to climb only when measurement shows the current rung cannot get you there.

The ladder

Rung one: prompt engineering

Prompting is underrated by people who have only tried "write me a summary". A well-built prompt sets the role and constraints, gives the model a handful of worked examples, defines the output format precisely, and breaks a complex task into steps. Structured output and function calling let you get back JSON your code can rely on rather than prose it must parse.

What prompting cannot do is add knowledge the model does not have, beyond what fits in the context window. It also costs tokens on every call: a long prompt with twenty examples is paid for each time, which matters at volume. And prompts are brittle across model versions; what worked on one release may drift on the next, so keep them under version control and test them like code.

Long-context models blur the line between rung one and rung two. If your entire reference material is a few hundred pages, you can place it all in the prompt and skip retrieval. That is simple and often correct for small, stable corpora. It stops being correct when the material grows, changes often, must be filtered per user, or when paying for all of it on every call becomes the largest line in the bill.

Push this rung hard before leaving it. In our experience most "we need to fine-tune" requests dissolve after a day of serious prompt work and a proper evaluation set.

Rung two: retrieval

When the gap is knowledge rather than behaviour, climb to retrieval. RAG fetches the relevant passages from your own documents and places them in the prompt per request. The model's weights do not change; its reading material does.

This is the right tool for facts that change, for content that must be permission-filtered per user, and for anything that needs a citation. It is the wrong tool for teaching a model to write in your house style or to follow a complex format. It also brings real engineering: ingestion, chunking, embeddings, a vector store and a retrieval evaluation. We walk through all of it in RAG explained.

Rung three: fine-tuning

Fine-tuning continues training a model on your own examples so that its default behaviour shifts. It is excellent for consistency: a specific output format, a tone of voice, a classification scheme, a narrow task done the same way every time. It can also replace a long prompt with a short one, reducing tokens and latency on every call.

There are several methods. Supervised fine-tuning trains on input and desired output pairs. Preference methods, descended from RLHF, train on comparisons between better and worse answers. Parameter-efficient techniques update a small set of weights rather than the whole model, which makes training cheaper and the result easier to manage. Hosted providers offer fine-tuning as a service for some models; open-weight models can be tuned anywhere.

What fine-tuning does badly is store knowledge. A model trained on your 2025 policy manual does not reliably recall the manual, and it certainly does not know the 2026 edition. It also carries risks: the model can lose general ability it had before, it can learn errors present in the training set, and a tune on a hosted model is tied to that provider and that model version. Every base model upgrade means retraining and re-evaluating. Our fine-tuning and evals service exists because the evaluation half is where most tuning projects fail.

Side by side

How to decide: the signals that say climb

Decisions on this ladder should be driven by an evaluation set, not by instinct. Build fifty to two hundred representative cases with expected outputs, score each rung against them, and climb only when a specific signal appears.

Signals to move from prompting to retrieval

• Failures are factual: the model is wrong or out of date about your products, policies or customers.
• Answers must cite a source or respect who is allowed to see what.
• The information you are stuffing into the prompt no longer fits or changes weekly.

Signals to move from prompting to fine-tuning

• Failures are behavioural: wrong format, inconsistent tone, missed edge cases, despite good examples.
• The prompt is long, repetitive and paid for on millions of calls, and latency or cost matters.
• You need a smaller model to do one narrow task as well as a large one does with a huge prompt.
• You have, or can generate and review, enough high-quality examples to teach the behaviour.

If the signal is unclear, stay on the current rung and improve the evaluation instead. Evaluating LLM outputs covers how.

Pitfalls we see

A worked example

Consider an insurer building an assistant that drafts replies to policyholder emails. Rung one: a prompt with the house style, five example replies and a strict output format gets to a usable draft in a week. Evaluation shows two failures: the drafts quote outdated policy wording, and the tone drifts on complaint emails.

The first failure is knowledge, so the team adds retrieval over the current policy documents. Wording errors drop sharply. The second failure is behaviour, and the prompt is now long enough to matter at volume. With a few thousand reviewed example replies from senior handlers, a fine-tune fixes the tone and allows a much shorter prompt. The final system uses all three rungs, in the order the evidence demanded, as part of a typical LLM application build.

Key takeaways

• Prompting changes instructions, retrieval changes knowledge, fine-tuning changes behaviour.
• Climb the ladder one rung at a time, and only when an evaluation set shows a gap.
• Most "we need to fine-tune" requests are solved by better prompts or retrieval.
• Fine-tuning is the right tool for format, tone and narrow task skill, not for facts.
• Fine-tuned models need curated data, continuous evaluation and retraining on upgrades.
• Mature systems combine all three; the order of adoption is what keeps the cost sane.

Related services