Blog · Engineering

Engineering

How the gateway is built on Cloudflare Workers — routing, failover, streaming, and the edge trade-offs behind it.

2posts

LatestJuly 12, 2026

How automatic failover works in an LLM gateway

A single provider call has a single point of failure: one rate limit, one outage, one bad deploy on their end, and your request fails. Here's the actual mechanism AnyRouter uses to route around that — circuit breakers, escalating cooldowns, and a fallback chain — and why the extra hop barely registers next to model generation time.

Read the post

Building an LLM gateway on Cloudflare Workers

AnyRouter runs as a set of Cloudflare Workers, not a long-running server. That constrains the design in specific ways — a 128MB memory ceiling that rules out buffering responses, a 3MiB script cap that forced a multi-Worker split, and a hard rule that every upstream call goes through Cloudflare's AI Gateway. Here's what that architecture actually looks like.

July 12, 2026