GitHub user purushah added a comment to the discussion: [Discussion] Pluggable
model routing — API & design (PR #852 follow-up)
Thanks @weiqingy for the detailed questions. I agree these are the right
contracts to clarify before changing the implementation. My updated thinking is:
**1. Event/control-flow contract**
The user-facing API should stay simple: users send `ChatRequestEvent("router",
...)`. `ChatModelAction` should detect that the name refers to a `ModelRouter`,
resolve a `RoutingDecision`, emit `ModelRoutingEvent`, and then directly call
the selected concrete chat model through the existing chat/tool-call loop.
For v1, I would avoid creating a second user-visible `ChatRequestEvent`. That
keeps user-defined actions from observing an internal dispatch and avoids
adding parent/original request ID semantics in the first version.
**2. Role of `ModelRoutingEvent`**
I propose `ModelRoutingEvent` as an observability event only. It should record
the router, selected model, candidates, reason/metadata, default/fallback
status, and timing if available. It should not be the event that drives
dispatch to the selected model.
**3. Structured `RoutingDecision`**
I agree returning only `String` is too limited. `route()` should return
`RoutingDecision`. The simple case can remain easy with helpers like
`RoutingDecision.of("big")` and `RoutingDecision.abstain()`, but the structured
result can preserve reason, score/confidence, metadata, invalid/abstain status,
and optional LLM judge output.
**4. Router-level caching**
I would remove public `cache`/`cacheSize` from v1. In the old PR, caching was
needed because the router was itself a `ChatModelSetup` and could be invoked
again on each tool-call round. If routing is owned by `ChatModelAction`,
routing should run once per `initialRequestId` / ReAct loop, and later
`ToolResponseEvent` rounds should reuse the stored selected model.
**5. LLM-as-router**
I agree LLM routing is not just a pure function. A judge model call has
latency, retries, token usage, metrics, durability, and EventLog implications.
The best design is for the framework to manage this path so the judge decision
is durable and observable. If that is too much for the first patch, I would
keep v1 focused on the basic routing contract and document fully observable LLM
routing as a follow-up.
**6. Recovery/replay**
`RoutingDecision` should be replayable data, not re-derived data.
`ChatModelAction` should store the decision with the `initialRequestId` / ReAct
loop state before invoking the selected backend. With `actionStateStoreBackend`
enabled, this should be persisted through the existing `ActionStateStore` and
coordinated by Flink checkpoints through recovery markers and sequence numbers.
On recovery, if the decision exists, reuse it and do not call `route()` again.
If the stored selected model is no longer valid, fail clearly or report a
recovery mismatch.
**7. Fallback vs retry**
My proposed v1 behavior is: first route to the selected model, then apply the
existing retry policy for that model. If retries are exhausted and router
fallback is enabled, try the remaining candidates in declaration order. The
metadata should record both the initially selected model and the final model
that produced the response.
**8. Deployable custom strategies**
The lambda examples are useful for API readability and local examples, but the
deployable shape should be a named `RoutingStrategy` class or descriptor that
can be serialized/shipped to TaskManagers. That fits better with the existing
resource/action deployment model.
**9. ResourceType**
I lean toward a distinct `MODEL_ROUTER` resource type, while keeping the
user-facing API unchanged. Users can still put the router name in
`ChatRequestEvent.model`. `ChatModelAction` can resolve it as a router before
resolving a normal `CHAT_MODEL`, or we can reject duplicate names to avoid
ambiguity.
**10. Abstain vs invalid candidate**
I agree these should be different. `null`/abstain should use `defaultModel`.
But returning a non-candidate model name likely means a typo or buggy strategy,
so it should be recorded as an invalid routing decision and fail clearly by
default.
**11. Python/YAML scope**
For v1, I would record this explicitly as Java-first framework routing.
Python/YAML can reference the Java router/strategy descriptors if supported by
the existing resource path, but native Python router SPI and YAML aliases can
be follow-ups unless maintainers want full parity in the first patch.
---
Overall, the revised target is: framework-owned routing in `ChatModelAction`,
structured `RoutingDecision`, observability-only `ModelRoutingEvent`,
route-once-per-ReAct-loop state keyed by `initialRequestId`, deterministic
recovery, no public router cache in v1, and explicit fallback/retry/deployment
scope.
GitHub link:
https://github.com/apache/flink-agents/discussions/897#discussioncomment-17745682
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
