Inference
Running a trained model to produce an output from new input, as opposed to training; in LLM systems it is where latency, throughput and most running costs appear.
Definition
Machine learning has two phases. Training adjusts a model's weights using large datasets and enormous compute; it happens rarely. Inference uses the finished weights to answer a request; it happens on every call. For an LLM, inference means reading the prompt (the "prefill" step) and then generating the response one token at a time (the "decode" step). Generation is sequential, which is why long answers take longer and cost more than long prompts.
Inference is measured in latency (time to first token and total time), throughput (tokens or requests per second) and cost per request. It can run on a provider's API, on cloud GPUs you rent, on your own servers or on a device. Techniques such as batching, caching repeated prompt prefixes, quantization and speculative decoding all target this phase.
Example: a document-extraction service processes 40,000 invoices overnight. Because nobody is waiting, the team batches requests and uses a smaller model, cutting cost per invoice sharply. The customer-facing chat on the same platform uses a faster, larger model and streams tokens to keep the interface responsive.
Why it matters for your business
- It is the recurring bill. Training costs are the provider's problem; inference costs are yours.
- Latency shapes product design: streaming, background processing and caching are inference decisions.
- Where inference runs determines where data goes, which drives privacy and residency choices.
- Optimisation is measurable: token counts, time to first token and cost per request can all be tracked.
How Teksmobile uses it
Our LLMOps and monitoring service instruments inference end to end: latency, token usage, cost per feature and error rates. We model the economics up front with the LLM cost calculator and explain the levers in our guide to the cost of LLM applications.
Related terms
Part of the Teksmobile AI Glossary. For longer reads, see our guides.
Frequently asked questions
What is time to first token?
The delay between sending a request and receiving the first token of the answer. For streamed chat interfaces it is the latency users feel most; tokens per second then determines how fast the rest appears.
Is self-hosted inference cheaper?
At high, steady volume it can be. At low or spiky volume, hosted APIs are usually cheaper because you pay per token rather than for idle hardware. See on-prem vs cloud LLM.
How do we reduce inference cost?
Shorter prompts, prompt caching, smaller or quantized models for simple tasks, model routing, batching for offline workloads and limiting output length. Measure first; the biggest saving is usually a prompt nobody has trimmed.