Inference is the act of running a trained model forward to produce an output. Training is what happens once, over vast data, to build a model; inference is what happens every single time you use it - every question answered, every document summarized, every sentence generated.
Why it matters
The distinction matters because inference is where the real-world constraints live:
- Latency. Generating a response token by token takes time. Longer outputs, larger models, or heavier prompts all cost more inference time.
- Cost. Providers charge per token processed during inference - both input (the prompt and retrieved passages) and output (the generated response). Every call has a price.
- Consistency. A trained model's weights don't change during inference; only the input changes. So the same question, asked twice with the same context and a deterministic temperature, should produce the same answer.
For document AI, inference is the last step of every question: retrieve the relevant passages from the document, build a prompt containing them, then run the model to generate an answer grounded in that text. The large language model never changes between questions; what changes is the evidence placed in front of it. That retrieve-then-infer loop is how Sidenote produces a fresh, cited answer for each question you ask.
This is also why inference cost scales with document complexity, not model size alone - a long retrieved context costs more tokens in and pushes latency up. Retrieval-augmented workflows earn their keep by keeping that context focused rather than sprawling.
FAQ
What is the difference between training and inference?
Training builds the model: weights are adjusted over vast data, once, at great cost. Inference uses it: the frozen model processes your input and generates output. Every interaction you ever have with an AI product is inference.
Why does inference cost money?
Each token in and out consumes compute, so providers meter usage per token. That is why focused prompts matter: retrieving three relevant passages costs a fraction of stuffing a whole document into every question.