wenjin272 commented on code in PR #1124: URL: https://github.com/apache/flink-agents/pull/1124#discussion_r4033777131
########## docs/content/docs/development/model_routing/_index.md: ########## @@ -0,0 +1,318 @@ +--- +title: Model Routing +weight: 5 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Model Routing + +## Overview + +Model routing lets one chat request choose between several registered chat models at runtime. Instead of naming a chat model in a `ChatRequestEvent`, an agent names a **model router**. The router carries a list of **candidate** chat models and a **routing strategy**. For each request the framework runs the strategy, which either **selects** one candidate or **abstains**, meaning it makes no choice and the router's default model is used. The framework then runs the ordinary chat path against the chosen model. + +The router only selects a model. The chosen model is invoked by the same `ChatModelAction` that serves a plain chat request, so its prompt, tools, skills, retries, token metrics, and event logging apply the same way. A strategy never calls a model itself. When the decision should come from an LLM, the framework runs that **judge** call through its own chat path, see [LLM Judge](#llm-judge). + +Typical uses are sending short requests to a small, cheap model and code, SQL, or multi-step reasoning to a large one; keeping a default model for everything the strategy cannot classify; and falling through to the next candidate when the selected model fails. + +What works with routing, and what does not: + +- Any registered `CHAT_MODEL` from any provider can be a candidate or a judge. Tool calls work: the model that answered the initial request is kept for every tool-call round of that request. +- Routers are declared with `addResource` on the execution environment or on the agent. There is no annotation for declaring a router inside an agent class, and the YAML API has no section for routers. +- `ReActAgent` cannot use a router; it registers its own chat model under a fixed name. Routing is for agents that send `ChatRequestEvent` themselves. +- A router cannot be a judge (rejected when the plan is built), a candidate of another router, or a chat model's `connection` (both fail at request time). + +{{< hint info >}} +Model routing is only supported in Java currently. Python agents cannot register a `MODEL_ROUTER` resource yet and the Python API rejects the attempt with an error. A router declared by a Java agent is still understood when the plan is shared with Python. Python support is planned for a future release. +{{< /hint >}} + +## Declaring a Router + +A router is a resource of type `ResourceType.MODEL_ROUTER`, built with `ModelRouter.of(...)` and registered like any other resource. Its candidates are the names of chat models registered in the same environment, listed in the order [fallback](#default-model-and-fallback) tries them. A name cannot be both a chat model and a router. The routing types live in `org.apache.flink.agents.api.chat.model.routing`. + +{{< tabs "Declaring a Router" >}} + +{{< tab "Java" >}} +```java +import org.apache.flink.agents.api.chat.model.routing.ModelRouter; +import org.apache.flink.agents.api.chat.model.routing.Strategies; +import org.apache.flink.agents.api.resource.ResourceDescriptor; +import org.apache.flink.agents.api.resource.ResourceName; +import org.apache.flink.agents.api.resource.ResourceType; + +import java.util.LinkedHashMap; +import java.util.Map; + +// One Ollama connection shared by the candidates. +agentsEnv.addResource( + "ollamaConnection", + ResourceType.CHAT_MODEL_CONNECTION, + ResourceDescriptor.Builder.newBuilder(ResourceName.ChatModel.OLLAMA_CONNECTION) + .addInitialArgument("endpoint", "http://localhost:11434";) + .build()); + +// Two candidate chat models registered under the names "small" and "big". +agentsEnv + .addResource( + "small", + ResourceType.CHAT_MODEL, + ResourceDescriptor.Builder.newBuilder(ResourceName.ChatModel.OLLAMA_SETUP) + .addInitialArgument("connection", "ollamaConnection") + .addInitialArgument("model", "qwen3:1.7b") + .build()) + .addResource( + "big", + ResourceType.CHAT_MODEL, + ResourceDescriptor.Builder.newBuilder(ResourceName.ChatModel.OLLAMA_SETUP) + .addInitialArgument("connection", "ollamaConnection") + .addInitialArgument("model", "qwen3:8b") + .build()); + +// Requests whose latest user message mentions code or SQL go to "big"; +// everything else abstains and lands on the default, "small". +Map<String, String> rules = new LinkedHashMap<>(); +rules.put("big", "\\b(code|sql|program|analyze|prove)\\b"); + +agentsEnv.addResource( + "router", + ResourceType.MODEL_ROUTER, + ModelRouter.of("small", "big") + .strategy(Strategies.rules(rules)) + .defaultModel("small") + .fallback(true) + .build()); +``` +{{< /tab >}} + +{{< /tabs >}} + +The agent then names the router in its `ChatRequestEvent`. Nothing else in the agent changes. + +{{< tabs "Using a Router in an Agent" >}} + +{{< tab "Java" >}} +```java +public class ModelRoutingAgent extends Agent { + + /** Send each input to the router, which selects the concrete model. */ + @Action(EventType.InputEvent) + public static void processInput(InputEvent event, RunnerContext ctx) { + ctx.sendEvent( + new ChatRequestEvent( + "router", + Collections.singletonList( + new ChatMessage(MessageRole.USER, (String) event.getInput())))); + } + + /** Emit the model's answer as output. */ + @Action(EventType.ChatResponseEvent) + public static void processChatResponse(ChatResponseEvent event, RunnerContext ctx) { + ctx.sendEvent(new OutputEvent(event.getResponse().getContent())); + } +} +``` +{{< /tab >}} + +{{< /tabs >}} + +| Method | Description | +|--------|-------------| +| `of(String... candidates)` | Start a router over the given chat model names. | +| `strategy(RoutingStrategy)` | Required. One of the `Strategies` factories below. | +| `describe(candidate, description)` | Describe a candidate. Descriptions are the criteria the [LLM judge](#llm-judge) reads. Fails immediately if the name is not a candidate. | +| `defaultModel(String)` | Where the router lands when the strategy abstains. Optional; without it the first candidate is the default. | +| `fallback(boolean)` | Try the remaining candidates, in declaration order, after the selected model fails. Off by default. | +| `build()` | Produces the `ResourceDescriptor` to register. | + +To turn routing off, address a candidate directly in the `ChatRequestEvent`, or replace the router registration with a plain chat model under the same name. + +## Routing Strategies + +A `RoutingStrategy` is a serializable declaration produced by a `Strategies` factory. It travels in the agent plan as a type tag plus arguments; the executor for that type runs on the TaskManager. Every strategy either selects a candidate or abstains. Review Comment: Overall, this page feels too implementation-heavy for user documentation. Details such as Plan type tags, TaskManager executors, durable call IDs, provider-specific finish-reason behavior, validation stages, and descriptor wire keys obscure the main usage flow. Could we shorten it around the overview, quick start, strategies, fallback, and observability, and move lower-level details to Javadocs or a short advanced subsection? ########## docs/content/docs/development/model_routing/reference.md: ########## @@ -0,0 +1,51 @@ +--- +title: Reference +weight: 1 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Model Routing Reference + +Reference material for [Model Routing]({{< ref "docs/development/model_routing" >}}): event and response fields, metrics, validation stages, and descriptor keys. Review Comment: I’m not sure a separate Reference page is warranted here. It adds another sidebar level for a relatively small amount of content, which is inconsistent with most other development topics. Could we keep Model Routing as a single page, merge the event and metric fields into Observability, and place validation notes near the relevant configuration sections? The descriptor keys could be omitted or moved to Javadocs. ########## docs/content/docs/development/model_routing/_index.md: ########## @@ -0,0 +1,318 @@ +--- +title: Model Routing +weight: 5 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Model Routing + +## Overview + +Model routing lets one chat request choose between several registered chat models at runtime. Instead of naming a chat model in a `ChatRequestEvent`, an agent names a **model router**. The router carries a list of **candidate** chat models and a **routing strategy**. For each request the framework runs the strategy, which either **selects** one candidate or **abstains**, meaning it makes no choice and the router's default model is used. The framework then runs the ordinary chat path against the chosen model. + +The router only selects a model. The chosen model is invoked by the same `ChatModelAction` that serves a plain chat request, so its prompt, tools, skills, retries, token metrics, and event logging apply the same way. A strategy never calls a model itself. When the decision should come from an LLM, the framework runs that **judge** call through its own chat path, see [LLM Judge](#llm-judge). + +Typical uses are sending short requests to a small, cheap model and code, SQL, or multi-step reasoning to a large one; keeping a default model for everything the strategy cannot classify; and falling through to the next candidate when the selected model fails. + +What works with routing, and what does not: + +- Any registered `CHAT_MODEL` from any provider can be a candidate or a judge. Tool calls work: the model that answered the initial request is kept for every tool-call round of that request. +- Routers are declared with `addResource` on the execution environment or on the agent. There is no annotation for declaring a router inside an agent class, and the YAML API has no section for routers. Review Comment: The examples currently register all resources through `addResource`, while our recommended Agent-facing style is annotation-based. Connections, candidate models, and the judge can already use `@ChatModelConnection` and `@ChatModelSetup`, but `MODEL_ROUTER` has no corresponding annotation. Could we add or track a dedicated annotation such as `@ModelRouterSetup` (`@ModelRouter` would conflict with the existing builder class), and then make the documentation examples fully annotation-first? This can be implemented in a separate PR if we want to keep this one documentation-only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
