Open Source Model Gateway Built to Rival OpenRouter with Sub-2ms Latency
Summary
A new open source AI model gateway written in Rust has been released as an alternative to OpenRouter, unifying self-hosted, frontier, and open source models under one roof. The gateway adds less than 1ms latency for BYOK requests and supports 1000+ models across all major inference providers.
A development team has taken to Hacker News to announce an ambitious open source project: a Rust-native AI model gateway designed to offer a self-hostable, high-performance alternative to services like OpenRouter. The project consolidates self-hosted models, frontier models, and open source models into a single, unified interface, removing the need for developers to maintain separate integrations for each provider.
Built from the ground up in Rust, the gateway is engineered for concurrency and low-latency performance. It handles the complex configuration differences between models and providers, including streaming formats, tool call implementations, model parameters, rate limiting, and varying error behaviors. This abstraction layer means teams can swap or combine providers without rewriting their application logic.
The performance benchmarks are noteworthy: the gateway introduces less than 1 millisecond of overhead for Bring Your Own Key (BYOK) requests, and under 2 milliseconds when Experiential supplies the provider key. With support for every major inference provider and over 1000 models, it positions itself as a comprehensive solution for teams managing diverse AI infrastructure at scale.
The release has sparked discussion in the developer community around the value of open, auditable infrastructure for AI workloads. Unlike proprietary gateways, this open source approach gives organizations full visibility into routing logic, cost management, and data flow — a compelling proposition for enterprises with strict compliance or privacy requirements.
Need IT help in Stockholm?
Book Sovin IT from 499 SEK
