LLM gateway
An intermediary that your company's calls to models route through; when it holds the only API keys, it is both a control point and a log of every call.
Keep the two benefits separate: control (key custody, nothing routes around it) and observability (logs of what goes into and comes out of each call). Most gateways also give one API across model providers, with failover and cost tracking. Routing is a capability of the gateway: deciding in real time which model a request goes to. The decision can be a deterministic rule (size, cost, who is asking) or a model that classifies the request. It usually trades off cost, latency and capability: send easy work to a cheap model, escalate hard work to a frontier model. The catch is that the router has to guess how hard a request is before the work is done. Router and gateway can be separate, but splitting them is extra work most companies shouldn't take on.
Also called
- LLM gateway
- AI gateway
- model gateway
- LLM router
- model routing
Kinds
- Model routing A step that picks, for each request, which model handles it, using a fixed rule or a model that judges the request.
Often confused with
- Agent gateway A control point in front of everything an agent can act on (tools, APIs and MCP servers) that governs what it can reach, enforces least privilege and logs its actions.
Related
- Model-agnostic A design that lets your company, product or service swap one model or vendor for another with little rework, so it is not locked in to one provider.
How it connects
- Stack includes LLM gateway
- Observability logs from LLM gateway
- LLM gateway sits in front of Inference