Portkey AI Gateway
Connect, manage, and secure AI interactions across 1,600+ LLMs with centralised control and real-time monitoring

Universal API Access

Connect to 1,600+ LLMs across providers and modalities via a single unified API. No separate integration is required for each provider – switching models or adding a new provider is a configuration change, not an engineering project.

• OpenAI, Anthropic, Google, Mistral, Cohere, Azure, AWS Bedrock, and 1,600+ more.
• Single API key for all providers via the gateway.
• Multimodal support including text, vision, audio, and embeddings.
• OpenAI-compatible drop-in – change the base URL and API key, nothing else.

Virtual Key Management

Store raw LLM provider API keys securely in Portkey's encrypted vault. Issue scoped virtual keys to individual teams, applications, and services with defined model and budget permissions. Rotate or revoke access at any time without touching application code.

• Encrypted key vault for all provider credentials.
• Scoped virtual keys per team, service, or environment.
• Instant revocation and key rotation.
• Full usage audit log per virtual key.

Semantic Caching

Reduce redundant LLM calls by up to 70% with both exact-match and vector similarity caching. Semantically equivalent requests return cached responses instantly, cutting costs and latency without any loss in output quality.

• Simple (exact-match) and semantic (vector similarity) caching modes.
• Cache hit rate analytics in the dashboard.
• Configurable TTL per cache rule.
• Cost reduction of up to 70% on repetitive workloads.

Automatic Retries

Configure retry logic with exponential backoff across all providers. Transient 5xx errors, rate-limit responses, and timeout failures are retried automatically before surfacing to your application, eliminating most provider-side disruptions invisibly.

• Configurable retry count per provider.
• Exponential backoff with jitter.
• Per-provider retry rule configuration.
• Retry with alternative models or providers on persistent failure.

Request Batching

Scale large-volume LLM workloads using provider batch APIs or Portkey's custom batching layer without impacting real-time application performance. Batch jobs run asynchronously and results are delivered via webhook or polling.

• Provider batch API support (OpenAI Batch, Anthropic Batch).
• Custom batching logic for unsupported providers.
• Asynchronous execution with no real-time performance impact.
• Cost-efficient processing at high volume.

Intelligent Routing and Fallbacks

Dynamically route every LLM request to the optimal provider based on cost, latency, quality, or custom rules. Define multi-step fallback chains so that if a primary provider returns an error or exceeds latency thresholds, traffic automatically fails over to the next configured option without interrupting the user experience.

• Automatic fallback chains with configurable priority order.
• Latency, cost, and quality-based routing rules.
• Zero-downtime provider switching.
• Fallback alerts and routing decision logs.

Load Balancing Across Provider Keys

Distribute request volume across multiple API keys for the same provider to avoid rate-limit bottlenecks at high throughput. Load balancing is configured entirely at the gateway layer and requires no changes to application code.

Canary and A/B Deployments

Roll out new models gradually using canary routing. Send a configurable percentage of traffic to a new model or provider while the remainder continues on the existing configuration. Compare latency, cost, and eval scores side-by-side before fully cutting over.

Rate Limiting and Budget Controls

Enforce per-virtual-key and per-team rate limits (requests per minute and tokens per minute) at the gateway layer. Set soft alert thresholds and hard spending caps to prevent runaway costs across teams and applications.

High Availability and Uptime SLA

Portkey's managed cloud gateway operates across multiple regions with active/active failover and a 99.99% uptime SLA. Self-hosted deployments support active/active Kubernetes cluster configurations for equivalent availability on-premises or in a private cloud.

Portkey AI Gateway Specifications:

Documentation:

View the Portkey AI Gateway Documentation (External Link).

View the Portkey Gateway Routing and Config Reference (External Link).

View the Portkey Virtual Key Management Guide (External Link).