joeyutong opened a new pull request, #955: URL: https://github.com/apache/flink-agents/pull/955
## What changed This PR implements the built-in operational metrics proposed in [Discussion #901](https://github.com/apache/flink-agents/discussions/901). It depends on [#924](https://github.com/apache/flink-agents/pull/924), which introduces the execution lifecycle and trace context consumed by these metrics. The PR is opened as a draft and will be rebased onto `main` after #924 is merged. Until then, the Metrics-only diff is available in [this branch comparison](https://github.com/joeyutong/flink-agents/compare/codex/agent-trace-event-log-pr...codex/agent-operational-metrics-pr). ### Runtime lifecycle integration - `ActionExecutionOperator` records input queue, input-run, Action task, and Action execution boundaries at the points where they actually occur. - `OperatorStateManager` exposes pending Action state for restoring current-count gauges after task recovery. - `BuiltInMetrics` is the central dispatcher. Action lifecycle events feed Action metrics, while LLM and Tool lifecycle events feed execution-entity metrics. Event Log writing and metric aggregation consume the same in-process execution event independently. ### Metric implementations - `BuiltInInputRunMetrics` records run outcomes, end-to-end, queue, and processing latency, pending input Events, and active input runs. - `BuiltInActionMetrics` records scheduling and logical execution latency, pending Action tasks, and active Action executions. - `BuiltInExecutionMetrics` pairs execution start and terminal events by execution id, then dispatches by entity type. - `LlmExecutionMetricRecorder` records model-resource success, failure, and latency. - `ToolExecutionMetricRecorder` records Tool metrics and projects explicit Skill and MCP Server metadata into their own scopes. - Current-count gauges are rebuilt from Flink state after restore. Historical latency and outcome samples that began in a previous task attempt are intentionally not reconstructed. ### Java and Python ChatModel paths - Each framework-observed ChatModel invocation emits its own LLM execution outcome and latency sample, including retry attempts. - Retry count and wait time are recorded under the `model_resource` scope in both Java and Python, including final-failure and `IGNORE` paths. - Existing token metrics remain under the `model` scope. ### Metric scope and documentation - The agent name is used as the Flink operator name, with the existing operator name retained as a fallback. - The monitoring documentation lists the new scopes, metric types, lifecycle boundaries, retry semantics, and recovery limitations. ## Validation - `mvn -T4 -B --no-transfer-progress spotless:check` - `mvn -B --no-transfer-progress -pl plan,runtime -am -DskipITs -Dtest=ChatModelActionRetryTest,CompileUtilsTest,BuiltInActionMetricsTest,BuiltInExecutionMetricsTest,BuiltInInputRunMetricsTest,ActionExecutionOperatorTest -Dsurefire.failIfNoSpecifiedTests=false test` - Plan: 11 tests passed - Runtime: 57 tests passed - `uv run ruff check flink_agents/plan/actions/chat_model_action.py flink_agents/plan/tests/actions/test_chat_model_action_retry.py` - `python -m pytest -q flink_agents/plan/tests/actions/test_chat_model_action_retry.py` - Python: 9 tests passed ## Related work - Agent Trace recording: [#924](https://github.com/apache/flink-agents/pull/924) - Agent Trace proposal: [Discussion #900](https://github.com/apache/flink-agents/discussions/900) - Operational Metrics proposal: [Discussion #901](https://github.com/apache/flink-agents/discussions/901) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
