Batch inference
A type of inference where many requests are sent together and answered later, usually at a lower price.
Also called
- Offline inference
- Batch API
- Batch processing
Kind of
- Inference The act of running a trained model to produce an output from inputs, as opposed to training it.
Related
- Cost per token The posted price of input and output tokens consumed or generated.