Small language models: when smaller wins, and when it does not

What counts as small, and why the line moves

A large language model at the frontier runs on clusters of accelerators and is accessed through a provider's API. A small language model is one you can run yourself on a single GPU, a CPU server or a mobile device. Open-weight families such as Llama, Mistral, Gemma, Phi and Qwen all offer small variants, and they improve every year.

The definition is deliberately about where it runs rather than parameter counts, because the counts keep shifting. What was a capable large model a few years ago now fits on a laptop. The practical question is unchanged: for this task, at this volume, with these privacy constraints, what is the smallest model that meets the quality bar?

Our view is that most organisations will end up running a mix. Frontier models for the hard, open-ended work; small models for the narrow, repetitive, sensitive or latency-critical work; and a router in between.

The trade-off, dimension by dimension

Two of those rows deserve emphasis. Latency is not just "faster is nicer": a small model on local hardware can respond quickly enough for interactive use in ways a remote API cannot guarantee. And "data stays in your perimeter" is often the whole reason a project exists; see on-prem vs cloud LLMs for the wider decision.

Where small models win

Small models shine on tasks that are narrow, frequent and well specified. Some patterns from our work:

• Classification and routing. Tagging support tickets, detecting intent, deciding which queue or which larger model should handle a request.
• Extraction. Pulling fields from invoices, forms and emails into a schema, the core of document processing. A fine-tuned small model is often as good as a frontier model here, at a fraction of the cost.
• High-volume transformation. Summarising short texts, normalising addresses, rewriting product descriptions, millions of times a month.
• On-device features. Suggestions, summaries and assistants in mobile apps where data must not leave the phone and the network may not be there.
• Regulated data. Clinical notes, HR records and financial documents that cannot be sent to a third party, served from your own infrastructure.
• Latency-critical paths. Autocomplete, live agent assist and voice interactions where a round trip to a remote API is too slow.

Where small models lose

Small models carry less knowledge and reason less reliably over long, multi-step problems. They are weaker at following complex instructions with many constraints, at writing nuanced long-form text, and at handling inputs far from anything they were trained on. They also degrade more sharply when the prompt is poorly structured, because they have less capacity to recover from ambiguity.

The practical consequence: do not use a small model as a general assistant, and do not expect it to plan an agent's work across many tools. Use it for the step inside the workflow that it can do well, and let a larger model do the orchestration. The cost of LLM applications shows why that split pays.

Techniques that make small models work

Fine-tune for the narrow task

Small models benefit from fine-tuning far more than frontier models do. A few thousand reviewed examples of your exact task often lift a small model from "usable" to "as good as the large one for this job". Follow the ladder in prompt engineering vs fine-tuning: prompt first, then tune.

Quantise for deployment

Quantisation stores the model's weights at lower precision, shrinking memory and speeding inference with a small quality cost. It is what makes phones and CPU servers viable targets. Measure the quality loss on your evaluation set; it is usually acceptable for narrow tasks and occasionally not.

Constrain the output

Ask for a schema and enforce it. Structured output, limited choices and short answers play to a small model's strengths and remove the failure modes that come from free text. Add deterministic checks so a malformed answer is caught before it reaches anyone.

Supply the facts

Small models know less, so give them what they need. Retrieval places the relevant passage in the prompt; the model's job shrinks to reading and extracting, which it does well. Do not rely on a small model's memory for anything factual.

Route, and escalate

Model routing sends each request to the cheapest model likely to handle it, and escalates when confidence is low, when the output fails a check, or when the task is tagged as complex. Most traffic in a mature system is routine, so most of it can go to the small model while the hard minority still gets frontier quality.

Side by side

Deployment options

Small models can run in more places than large ones, and the choice shapes the economics.

• On device. Inside a mobile or desktop app, quantised, for privacy and offline use. Tight limits on size and battery.
• On CPU servers. Adequate for lower-volume or latency-tolerant tasks; no accelerator procurement.
• On a single GPU. The workhorse option for high-volume internal services; one card can serve a surprising amount of traffic.
• As a hosted small model. Several providers serve small open-weight models behind an API, which gives the cost and latency benefits without the operations, at the price of data leaving your perimeter.

Whichever you choose, treat serving as a production system: health checks, capacity planning, version pinning and the same evaluation gates as any other model, as covered in LLMOps and monitoring.

Pitfalls we see

How to decide

Write the task down precisely. Build a golden set of a few hundred cases. Run a frontier model with a good prompt to establish the ceiling. Then run two or three small candidates, with and without fine-tuning, and compare. If a small model is within your tolerance on quality and clearly better on cost, latency or privacy, use it, behind a router that can escalate. If not, use the large model and revisit in six months; the small ones keep improving.

That is the process we follow in LLM app development, and it ends more often than people expect with a small model doing most of the work.

Key takeaways

• "Small" means it runs on hardware you control; the parameter threshold moves every year.
• Small models win on cost, latency, privacy and offline use; frontier models win on open-ended reasoning.
• Use small models for narrow, high-volume, structured tasks, not as general assistants.
• Fine-tuning, quantisation, structured output and retrieval are what make them reliable.
• Route most traffic to the small model and escalate the hard minority to a large one.
• Self-hosting is a production system; budget the operations and the evaluation.

Related services