GitHub user purushah added a comment to the discussion: [Discussion] Pluggable
model routing — API & design (PR #852 follow-up)
Thanks @wenjin272, I agree with your understanding and suggestions.
1. `ModelRoutingEvent` should be observability-only in v1. It should be used
for logging, tracing, evaluation, and metrics context. It should not have a
built-in consumer, and it should not trigger a second `ChatRequestEvent`. The
flow would be: `ChatModelAction` detects that the requested model is a router,
invokes `ModelRouter.route()`, emits `ModelRoutingEvent`, and then directly
continues the existing chat/tool-call loop with the selected concrete model.
2. I agree that `cache` / `cacheSize` can be removed from the public v1 API.
With `ModelRouter` separated from `ChatModelSetup`, `ChatModelAction` can
associate the selected model / `RoutingDecision` with `initialRequestId` and
store it in the existing tool-call context. Subsequent `ToolResponseEvent`s
should reuse that model until the final `ChatResponseEvent`. That gives us
per-ReAct-loop stickiness without a separate LRU cache keyed by first user
message.
3. I agree that returning only a `String` is too limited. `route()` should
return a structured `RoutingDecision`, so `ModelRoutingEvent` can record
selected model, reason, score/confidence, strategy metadata, abstain/default
behavior, and optional LLM judge output. We can still keep the common case
simple with helpers like `RoutingDecision.of("big")` and
`RoutingDecision.abstain()`.
I also agree with keeping `RoutingStrategy` as the single extension point,
built-ins via factories such as `Strategies.rules(...)` and
`Strategies.llm(...)`, fallback as a simple config flag for now, and the
current `RoutingContext` scope for v1.
GitHub link:
https://github.com/apache/flink-agents/discussions/897#discussioncomment-17744848
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
