Inference
The act of running a trained model to produce an output from inputs, as opposed to training it.
Also called
- Model serving
- Serving
Kinds
- Batch inference A type of inference where many requests are sent together and answered later, usually at a lower price.
- Reasoning effort A setting which controls how long or how many tokens are spent by a model reasoning before providing an answer.
The other half
- Training The process of supplying data (often in large quantities) to a model so that it can identify patterns and make predictions.
Related
- Cost per token The posted price of input and output tokens consumed or generated.
How it connects
- Context input to Inference
- Inference runs the Model
- Inference measured in Token
- LLM gateway sits in front of Inference