[
https://issues.apache.org/jira/browse/FLINK-39617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18106431#comment-18106431
]
Herbert Wang commented on FLINK-39617:
--------------------------------------
Thank you for digging into this [~jackeyzhe], glad to hear the additive
approach fits cleanly with the existing aggregated subtask metrics
implementation.
One correction on the original proposal: these should be GET endpoints, not
POST, no request body semantics needed here, so GET is the right fit. I updated
the ticket description above.
On the regex question: we're leaning toward full match (so
{{Pattern.matches()}} style) for the /names field, for consistency with Flink's
existing reporter-side metric filter
(metrics.reporter.<name>.filter.includes/excludes), which already uses
full-string match for this kind of hierarchical metric name filtering. Full
match also avoids accidentally over-matching on metric name prefixes/suffixes,
and gives cleaner REST API semantics.
Separately, agreed on keeping /names and /values as separate endpoints rather
than combining them, /values needs per-vertex metric lists plus aggregation
options, while /names only needs vertex IDs and optional filters, so combining
them would just muddy both contracts.
If this direction looks good, happy to have you pick up the implementation, and
would be glad to collaborate, review, or contribute wherever's useful as well.
Thank you.
> Add batch REST endpoints for aggregated subtask metrics across multiple job
> vertices
> --------------------------------------------------------------------------------------
>
> Key: FLINK-39617
> URL: https://issues.apache.org/jira/browse/FLINK-39617
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics, Runtime / REST
> Reporter: Herbert Wang
> Priority: Major
> Labels: Metrics, metrics, rest_api
>
> The JobManager REST API currently exposes aggregated subtask metrics per job
> vertex via:
> {code}
> GET /jobs/:jobid/vertices/:vertexid/subtasks/metrics
> {code}
> Clients that need the same metric set for many vertices, such as autoscalers
> or monitoring integrations, must issue one request per vertex for metric-name
> discovery and another request per vertex for metric values. For jobs with
> many vertices this creates avoidable REST fan-out, repeated MetricFetcher
> updates, and large repeated payloads.
> h2. Proposal
> Add two batch JobManager REST endpoints for aggregated subtask metrics across
> multiple vertices:
> {code}
> GET /jobs/:jobid/vertices/subtasks/metrics/names
> GET /jobs/:jobid/vertices/subtasks/metrics/values
> {code}
> The existing single-vertex endpoint should remain unchanged for compatibility.
> The endpoints are intentionally split rather than using one POST endpoint
> with mode-switching behavior, so OpenAPI schemas, code generation, and
> capability detection remain straightforward.
> h3. Name discovery endpoint
> Request:
> {code:json}
> {
> "vertexIds": ["<jobVertexId>", "<jobVertexId>"],
> "regex": [".*busyTime.*", ".*numRecords.*"]
> }
> {code}
> Response:
> {code:json}
> [
> {
> "vertexId": "<jobVertexId>",
> "metrics": [{ "id": "busyTimeMsPerSecond" }]
> }
> ]
> {code}
> h3. Value aggregation endpoint
> Request:
> {code:json}
> {
> "vertices": [
> { "vertexId": "<jobVertexId>", "metrics": ["busyTimeMsPerSecond"] },
> { "vertexId": "<jobVertexId>", "metrics": ["numRecordsInPerSecond"] }
> ],
> "agg": ["min", "max", "avg"]
> }
> {code}
> Response:
> {code:json}
> [
> {
> "vertexId": "<jobVertexId>",
> "metrics": [{ "id": "busyTimeMsPerSecond", "min": 0.0, "max": 1.0, "avg":
> 0.5 }]
> }
> ]
> {code}
> h2. Compatibility
> This is additive. The existing endpoint remains unchanged:
> {code}
> GET /jobs/:jobid/vertices/:vertexid/subtasks/metrics
> {code}
> Clients can feature-detect the new endpoints and fall back to the existing
> per-vertex endpoint when unavailable, or we can cherry-pick to earlier 2.x
> versions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
