This is an automated email from the ASF dual-hosted git repository.
hsluoyz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/casbin-gateway.git
The following commit(s) were added to refs/heads/master by this push:
new ad2d65e docs: describe fallbacks and model routing in the README
ad2d65e is described below
commit ad2d65ea87dbfcc0028e1aaf5b437d945ef1b90c
Author: Yang Luo <[email protected]>
AuthorDate: Sun Oct 11 18:06:51 2026 +0800
docs: describe fallbacks and model routing in the README
---
README.md | 19 ++++++++++++++++++-
README_zh.md | 19 ++++++++++++++++++-
2 files changed, 36 insertions(+), 2 deletions(-)
diff --git a/README.md b/README.md
index 4f814da..26dacbf 100644
--- a/README.md
+++ b/README.md
@@ -102,7 +102,7 @@
- **Is that API key what it was sold as?** A reseller can serve a cheaper
model, fake a cache hit, or quietly drop a parameter it claims to support.
[Authenticity](#the-killer-feature-is-the-api-behind-that-key-what-it-was-sold-as)
probes every provider on its own and grades it A–F, and [grades a single
endpoint](#check-one-endpoint-without-installing-anything) without installing
anything.
- **What every agent did, spent and sent.** Every prompt and tool call, the
spend read from the agents' own transcripts, and, on Windows, where each
agent's own processes upload data, including what never passes through any
proxy.
-- **One place for all of them.** Point Claude Code, Codex, Cursor, Gemini CLI
and the rest at any of 44 model vendors, say what each may do, and install,
upgrade or roll back the agents themselves.
+- **One place for all of them.** Point Claude Code, Codex, Cursor, Gemini CLI
and the rest at any of 44 model vendors, [fall over to another
vendor](#when-a-provider-cannot-answer-fallbacks-and-model-routing) when one
runs out or goes down, say what each may do, and install, upgrade or roll back
the agents themselves.
## Run it
@@ -197,6 +197,7 @@ The password behind that account is `admin` / `123`, and it
only matters if you
- **[Is the API behind that key what it was sold
as?](#the-killer-feature-is-the-api-behind-that-key-what-it-was-sold-as)** — a
reseller can quietly swap in a cheaper model or fake a cache hit, and none of
it shows up in the traffic. Authenticity asks the upstream directly and grades
it A–F.
- **[Switch every agent's provider from one
place](#send-an-agents-traffic-through-gateway)** — change an API key or base
URL once, and every agent pointed at Gateway picks it up.
+- **[Keep going when a provider runs
out](#when-a-provider-cannot-answer-fallbacks-and-model-routing)** — a key out
of credit, a rate limit or a relay that is down moves the request on to the
next provider, across vendors and APIs, and model routing rules decide which
model each request is actually sent and what it steps down to.
- **[Add a provider, an MCP server, a prompt or a skill from a
link](#import-from-a-link)** — click a vendor's "add this" button on the web
and Gateway opens with what the link carries, before any of it is written.
- **[Run several instances of one agent side by side](#what-to-do-next)** —
e.g. multiple Claude Desktop instances, each signed in to a different account.
- **[Install, upgrade and roll back the agents themselves](#what-to-do-next)**
— which build each agent is on against what its package manager publishes, and
back to an older release when an update broke something. One click whichever
way it was installed, with the command shown first, its progress on the row and
its console output a click away.
@@ -288,6 +289,22 @@ The environment snippet for such a provider sets the base
URL and nothing else
Codex is the exception: its ChatGPT sign-in talks to an endpoint no provider
stands in for, so a Codex CLI still needs a provider with an API key.
+### When a provider cannot answer: fallbacks and model routing
+
+A key runs out of credit, a relay goes down, a vendor rate-limits you halfway
through a session. Gateway moves the request on to the next provider instead of
handing the agent the error.
+
+**Fallbacks.** On an agent's page, under the provider it is bound to, pick the
other providers to fall back to, in the order they should be tried. They need
not be from the same vendor or speak the same API: Claude Code bound to an
Anthropic relay can fall back to DeepSeek, then Kimi, then Qwen, and Gateway
translates each request for whichever one answers. A fallback that does not
serve the model the agent asked for is sent the closest one it does by size, so
a request for a small model [...]
+
+**What counts as not answering.** A connection that fails, a timeout, and
`401`, `402`, `403`, `404`, `408`, `429` or any `5xx`: a rejected key, an
exhausted quota, a model that provider does not serve, a rate limit or an
outage, all of them specific to that provider. A `400` the request itself
caused would fail the same way everywhere, so it goes straight back to the
agent. Gateway fails over only while nothing has been written back to the agent
yet, so a streamed answer is never stitch [...]
+
+**Cooldown.** A provider that fails twice in a row is taken out of rotation
for 15 seconds, doubling with every further failure up to 5 minutes, so a dead
upstream stops costing every request a timeout. It is tried last rather than
dropped, since it may be the only one left, and one answer, or an edit to the
provider, puts it back. The agent's page says when its bound provider is
cooling down and what its last error was.
+
+**Model routing.** Agents choose model names themselves — Claude Code asks for
a small model for its background work whatever it is configured with — so
**Model routing** in the sidebar decides what each name is actually sent as. A
rule matches a model name, with `*` for any run of characters (`*haiku*` covers
every version, next year's included), for every agent or just one, and lists up
to eight steps: a model, and optionally the one provider that must answer it.
The first step is wher [...]
+
+**Seeing what happened.** The route preview on the Model routing page shows
where a model name would go, step by step and with the providers in cooldown
marked, without sending a request. In **LLM Records** each request names the
provider and model it was **Routed to**, the rule it was **Routed by**, and
every provider it failed over from with the status it got.
+
+Requests that name no agent, on `/v1/chat/completions` or `/v1/messages`, go
by model name to every enabled provider that serves it, lowest priority value
first, and fail over the same way.
+
### Import from a link
Vendors put an "add this to my agent manager" button on their own pages, which
opens a `ccswitch://` link rather than a page. Gateway reads that format
instead of inventing a second one nobody would be given a button for, and one
link carries any of four things: a provider with its base URL, key and model
list; an MCP server, as the JSON block a server is written in; a set of
instructions for an agent to read before every session; or a repository to
install skills from.
diff --git a/README_zh.md b/README_zh.md
index 7e300bd..4205681 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -102,7 +102,7 @@
- **这个API Key，真是卖给你的那个吗？**
中转商可以拿便宜模型冒充、伪造缓存命中，或者悄悄丢掉它声称支持的参数。[真伪检测](#杀手锏那个-key-背后的-api真是卖给你的那个吗)会主动探测每个Provider，给出A–F评级；[不安装也能单测一个接口](#不安装单测一个接口)。
- **每个Agent做了什么、花了多少、发到了哪里。**
每条提示词和工具调用，从Agent自己的会话记录里读出的花费；在Windows上还能看到每个Agent自己的进程把数据上传到了哪里，包括任何代理都看不到的那部分。
-- **所有Agent，一个地方管。** 把Claude Code、Codex、Cursor、Gemini
CLI等接到44个模型厂商中的任意一个，规定每个Agent能做什么，并直接安装、升级或回滚这些Agent本身。
+- **所有Agent，一个地方管。** 把Claude Code、Codex、Cursor、Gemini
CLI等接到44个模型厂商中的任意一个，一家额度用完或挂了就[自动切到另一家](#provider-答不上来时备用-provider-和模型路由)，规定每个Agent能做什么，并直接安装、升级或回滚这些Agent本身。
## 运行
@@ -195,6 +195,7 @@ Gateway 会在自己的窗口里打开 —— 不用登录：它只服务本机
- **[查一查那个 Key 背后的 API 是不是真的](#杀手锏那个-key-背后的-api真是卖给你的那个吗)** ——
中转商可以偷偷换成便宜模型、伪造缓存命中，流量里根本看不出来。Authenticity 直接问上游，测出 A 到 F 的等级。
- **[一个地方切换所有 Agent 的 API 供应商](#让-agent-的流量走-gateway)** —— 改一次 Key 或 base
URL，接进 Gateway 的每个 Agent 都跟着换。
+- **[一家额度用完，自动换下一家](#provider-答不上来时备用-provider-和模型路由)** —— Key
没余额、被限流、中转站挂了，请求自动转给下一个 Provider，可以跨厂商、跨 API；模型路由规则决定每个请求实际发给哪个模型，以及答不上来时降到哪个模型。
- **[一条链接就能导入 Provider、MCP、提示词或技能](#从一条链接导入)** —— 在网页上点厂商的「添加到」按钮，Gateway
就带着链接里的东西打开，写进去之前先给你看。
- **[同一个 Agent 开多个实例](#接下来做什么)** —— 比如同时跑好几个 Claude Desktop，各自登录不同账号。
- **[安装、升级和回退 Agent 本身](#接下来做什么)** ——
本机装的是哪个版本，包管理器又发布到了哪个版本；升级坏了还能退回旧版。不管当初是怎么装的，都是一键完成：命令先摆出来，进度就在那一行上，控制台输出点一下就能看。
@@ -285,6 +286,22 @@ export ANTHROPIC_AUTH_TOKEN="cg-..."
Codex 是例外：它的 ChatGPT 登录走的是没有任何 Provider 能替代的端点，所以 Codex CLI 仍然需要一个带 API Key 的
Provider。
+### Provider 答不上来时：备用 Provider 和模型路由
+
+Key 没余额了、中转站挂了、厂商在会话中途开始限流 —— Gateway 会把请求转给下一个 Provider，而不是把错误直接丢给 Agent。
+
+**备用 Provider。** 在 Agent 的页面上，绑定的 Provider 下面，按想要的尝试顺序选上其他 Provider
作为备用。它们不必是同一家厂商，也不必说同一套 API：Claude Code 绑的是一个 Anthropic 中转站，可以依次降到
DeepSeek、Kimi、Qwen，Gateway 会按实际作答的那家转换请求。备用 Provider 没有 Agent
点名的那个模型时，会按大小发给它最接近的模型：点名小模型（比如
`claude-haiku-4-5`）就用它的小模型，点名大模型就用它的大模型，后台的零碎请求不会按旗舰模型的价格计费。
+
+**什么算答不上来。** 连接失败、超时，以及 `401`、`402`、`403`、`404`、`408`、`429` 和所有 `5xx`：Key
被拒、额度用完、这家没有这个模型、被限流或宕机，都是这一家的问题。请求本身有问题导致的 `400`，换哪家都一样会失败，所以直接返回给
Agent。只有在还没向 Agent 写回任何内容时才会切换，所以一个流式回复不会由两个模型拼起来；所有 Provider 都失败时，Agent
收到的是最后一家的原始错误，而不是 Gateway 编出来的。
+
+**冷却。** 一个 Provider 连续失败两次，就移出轮转 15 秒，之后每多失败一次时间翻倍，最长 5
分钟，挂掉的上游不再让每个请求都白等一次超时。它会被排到最后而不是被剔除，因为它可能是唯一剩下的一家；只要成功应答一次，或者改过这个
Provider，它就恢复正常。绑定的 Provider 在冷却时，Agent 页面上会提示，并写出它最后一次的错误。
+
+**模型路由。** 模型名是 Agent 自己选的 —— 不管怎么配置，Claude Code 的后台工作都会点名一个小模型 —— 所以侧边栏的
**Model routing**（模型路由）决定每个模型名实际发给什么。一条规则匹配一个模型名，`*` 代表任意字符（`*haiku*`
覆盖它的每个版本，包括明年的），可以对所有 Agent 生效，也可以只对一个；规则最多列八级：每级一个模型，可以指定必须由哪个 Provider
作答。第一级是请求实际去的地方，下面每一级是上一级被限流、额度用完或不可用时降到的目标 —— 比如 `claude-opus-*` 先走 Anthropic
Key 上的 Opus，再降到 DeepSeek 上的 `deepseek-reasoner`，再降到任何提供 `kimi-k2` 的
Provider。精确的模型名优先于通配符，只对一个 Agent 的规则优先于通用规则，不用手工排序。规则自己的几级之后，还会接上没有这条规则时
Gateway 本来的做法，所以规�
�的 Provider 全删了，请求也照样能走。规则只管路由，不管权限：它的 [...]
+
+**看实际走了哪条路。** 模型路由页面上的路由预览不发真实请求，就能看到一个模型名会一级一级发到哪里，冷却中的 Provider 会标出来。在 **LLM
Records** 里，每个请求都写着**实际发往**哪个 Provider 和模型、**命中规则**是哪一条，以及中途切换掉的每个 Provider
和它返回的状态码。
+
+不带 Agent 的请求（`/v1/chat/completions`、`/v1/messages`）按模型名发给所有提供该模型的已启用
Provider，priority 值小的优先，切换规则相同。
+
### 从一条链接导入
厂商会在自己的页面上放一个「添加到我的 Agent 管理器」按钮，点开的是一条 `ccswitch://` 链接，而不是一个网页。Gateway
直接读这个格式，而不是再造一个没人会给你按钮的格式。一条链接可以带四样东西之一：一个 Provider，连同它的 base URL、Key 和模型列表；一个
MCP 服务器，就是它平时写成的那段 JSON；一份 Agent 每次会话前要读的提示词；或者一个安装技能用的仓库。
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
