llm-gateway: the path of one chat completion

Every hop below is a real package in internal/

llm-gateway: the path of one chat completion Every hop below is a real package in internal/ Any OpenAI client · SDK unchanged · Architecture component Any OpenAI client SDK unchanged Middleware chain · recover, id, log · gateway process Middleware chain recover, id, log requireAuth · gateway keys · gateway process › authenticated surface requireAuth gateway keys rateLimit · token bucket · gateway process › authenticated surface rateLimit token bucket chat completions · /v1/chat/completions · gateway process › authenticated surface chat completions /v1/chat/completions Response cache · keyed by digest · gateway process Response cache keyed by digest Router · retry and failover · gateway process Router retry and failover OpenAI · provider · Architecture component OpenAI provider Anthropic · provider · Architecture component Anthropic provider Redis · shared state · Architecture component Redis shared state GET /metrics · Prometheus · gateway process GET /metrics Prometheus HTTPS a caller, not an address hit returns here on a miss failover no credential gateway process authenticated surface Legend Backend Database Cloud Security External

One dialect

  • • Callers speak OpenAI's wire format.
  • • Switching model is a string change.
  • • The client never learns which vendor answered.

Order matters

  • • Authenticate first, then meter.
  • • A limiter placed before auth can only meter an IP address.
  • • /healthz and /metrics stay open: a probe that needs a credential fails when it is most needed.

When a provider misbehaves

  • • Retry with backoff, then fall back to the next provider.
  • • A circuit breaker stops hammering a dead endpoint.
  • • Breaker state is exposed on /healthz.