Multi-provider LLM Gateway

An integration pattern composed of a common inference interface, provider adapters, and model capability validation.

A client sends an inference request to an LLM gateway, which routes it to OpenAI, Anthropic, or Gemini.

Intent

Provide a uniform interface for invoking language models from different providers, while isolating provider-specific protocols and preserving explicit differences in model capabilities.

Applicability

Use this pattern when:

An agent runtime must support several LLM providers without embedding their SDKs and message formats in agent logic.
Model selection must vary between inference requests.
Context, tool definitions, and inference results require a common representation.
Unsupported combinations of model capabilities and request options must be rejected before invocation.

The central design force is the tension between a stable application interface and heterogeneous provider semantics. The abstraction should unify common operations without implying that all models are behaviorally interchangeable.

Structure

The following diagram shows the principal roles using Mozaik's implementation names. Provider SDKs and concrete mapper classes are omitted.

Class diagram of the multi-provider LLM gateway. DefaultInferenceRunner implements InferenceRunner, selects GenerativeModel instances, and validates requests with InferenceRequestValidator. Each GenerativeModel has one InferenceEndpoint. OpenAIResponses and AnthropicMessages implement InferenceEndpoint, which delegates translation to InferenceEndpointMapper.

Participants

Participant	Responsibility
Client	Submits an inference request through the common interface.
GatewayDefaultInferenceRunner	Resolves the requested model, validates the request, and delegates inference to its endpoint.
Model DescriptorGenerativeModel	Associates a model's capability specification with an endpoint implementation.
Request ValidatorInferenceRequestValidator	Checks request options against the selected model's declared capabilities.
Provider AdapterInferenceEndpoint	Encapsulates provider SDK invocation behind common inference operations.
MapperInferenceEndpointMapper	Translates common requests into provider formats and provider responses into common results.

InferenceRequest and InferenceResult form the shared data contract between clients and the gateway.

Implementation

Mozaik implements the gateway as DefaultInferenceRunner. Its run operation selects a GenerativeModel by the request's model name, validates the request against its specification, and invokes the associated endpoint.

The essential collaboration is shown below. This is a condensed excerpt; constructor dependencies and streaming are omitted.

TypeScript excerpt of DefaultInferenceRunner.run. It finds a GenerativeModel by name, throws if the model is unsupported, validates the request, and returns model.endpoint.infer(input).

Endpoints such as OpenAIResponses and AnthropicMessages delegate translation to a mapper, invoke the provider SDK, and map the completed response into InferenceResult. Adding a provider requires an endpoint, suitable mappings, and model descriptors; the gateway's dispatch algorithm remains unchanged.

Validation covers declared support for features such as tool calling, structured output, reasoning effort, streaming, and context item types. Capability declarations must be maintained as models evolve.

The same selection and validation process serves streaming. In the inspected OpenAI and Anthropic adapters, intermediate events retain provider-specific forms, followed by a common inference.output event. Thus, completed-result portability does not imply uniform streaming semantics.

Automatic fallback, retries, and cost-based routing are separate policies; they are not supplied by this dispatch mechanism.

Consequences

Benefits

Reduced coupling. Agent logic depends on an inference contract rather than individual provider SDKs.
Localized variation. Provider protocol changes are concentrated in endpoints and mappers.
Explicit compatibility. Capability validation identifies unsupported requests before provider invocation.
Extensibility and testability. Additional endpoints and test doubles can satisfy the same interface.

Liabilities

Incomplete substitutability. A common interface cannot guarantee equivalent model behavior, tool semantics, or output quality.
Translation complexity. Context items and provider features may lack exact equivalents; mappings require deliberate handling of these differences.
Maintenance burden. Adapters and capability specifications must track provider changes.
Abstraction leakage. Native errors, raw response data, and streaming events can reintroduce provider dependencies into clients.
Additional indirection. Model resolution, validation, and mapping add components that require testing and diagnosis.

Adapter. Each endpoint adapts a provider API to the common inference contract; mappers isolate data translation.
Facade. The gateway presents a single entry point to model lookup, validation, and provider invocation.
Strategy. Endpoint implementations provide interchangeable inference mechanisms selected through model descriptors.
Bridge. Separating model descriptors from endpoints allows multiple models to share a protocol implementation; this is a related separation of concerns rather than a strict GoF Bridge structure.
Decorator. Logging, metering, retries, or caching may wrap the inference interface without changing provider adapters.

Sources

Implementation reference: Mozaik v4.1.0

Back to patterns