Without a gateway, each product service tends to carry its own provider key, retry logic, model selection, and logging. That is manageable for one prototype. It becomes a reliability and governance problem when several services call several providers with different limits and prices.
A gateway centralizes the rules around model calls. It does not make an unsafe application safe by itself.
What does an AI gateway control?
The gateway authenticates callers, selects a provider or model, applies LLM rate limiting, records latency and cost, and can apply quotas, retries, or fallback rules. Its value is one enforceable place for controls shared by many callers. It can also normalize provider APIs so applications do not need a separate integration for every model.
This is also called an LLM gateway when the boundary focuses specifically on language model traffic. It is not the same as model routing. Routing is one gateway capability that chooses a model based on a policy. The gateway is the broader boundary that owns credentials, metering, and request handling before and after that decision.
It can also supply an audit record for each request. That record should identify the calling service, selected model, policy result, token use, and provider outcome without storing secrets or sensitive prompt content by default.
What should stay outside the gateway?
Business authorization still belongs with the application that knows the user, record, and intended action. A gateway can reject a caller that exceeds a quota, but it cannot decide whether an agent may send a particular message or modify a specific invoice. Keep decision-specific permission checks close to the action they protect. A shared provider boundary cannot replace an action-specific permission decision. MCP controls how an agent reaches tools and data, while the gateway manages the model-provider path.
How should a team introduce one?
Start with the shared controls that already repeat, often credentials, LLM observability, and rate limits. Preserve provider error codes and trace identifiers so failures remain diagnosable. Add model fallback only after an evaluation shows that the substitute meets the task’s quality and safety requirements. Failover is a product behavior that needs the same evaluation as a planned route. A gateway that quietly replaces a model after an error can convert an outage into a wrong answer unless the fallback is tested.
Keep the initial change narrow. Moving one service through the gateway can prove that traces, quotas, and errors preserve their behavior before centralizing every caller. A gateway should reduce duplicated controls, not hide them behind a new opaque layer.