[
https://issues.apache.org/jira/browse/FLINK-40969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-40969:
-----------------------------------
Fix Version/s: 2.3.1
2.4.0
> Rescale details endpoint returns 404 for latest events evicted from
> AdaptiveScheduler history
> ---------------------------------------------------------------------------------------------
>
> Key: FLINK-40969
> URL: https://issues.apache.org/jira/browse/FLINK-40969
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST, Runtime / Web Frontend
> Affects Versions: 2.3.0
> Environment: Affected code paths were confirmed in:
> * Apache Flink 2.3.0
> Configuration:
> * Streaming job
> * AdaptiveScheduler enabled
> * web.adaptive-scheduler.rescale-history.size set to a positive bounded value
> The issue is deployment-mode independent because it is caused by the
> in-memory rescale snapshot and REST lookup logic.
> Reporter: Hao Yu
> Assignee: Hao Yu
> Priority: Minor
> Labels: pull-request-available
> Fix For: 2.3.1, 2.4.0
>
> Attachments: PixPin_2026-10-12_17-39-13.png
>
>
> h2. Problem
> The AdaptiveScheduler Rescales Overview may expose a rescale UUID whose
> details can no longer be retrieved.
> The Overview keeps the latest rescale event for each terminal state
> (COMPLETED, FAILED, and IGNORED). These entries are stored separately from
> the bounded rescale history.
> When a latest event for one terminal state is older than the configured
> history limit, newer events of other terminal states can evict it from the
> history. The event remains visible in the Overview, but requesting its
> details returns HTTP 404:
> {code:java}
> Could not find rescale details of specified rescaleUuid <rescale-uuid>.
> {code}
> The Web UI automatically requests details for the latest events returned by
> the Overview. Consequently, opening the Rescales page can display a server
> error even though the job is running normally and the UUID was returned by
> the Overview endpoint.
> h2. Steps to reproduce
> # Configure a streaming job to use the AdaptiveScheduler and enable a small
> rescale history:
> {code:java}
> jobmanager.scheduler: adaptive
> web.adaptive-scheduler.rescale-history.size: 1
> {code}
> # Trigger a rescale that terminates as IGNORED, for example by updating the
> resource requirements again while a previous rescale is still in progress.
> # Trigger a later rescale that terminates as COMPLETED.
> # Request:
> {code:java}
> GET /jobs/:jobid/rescales/overview
> {code}
> # Record the UUID returned in {{{}latest.ignored{}}}.
> # Request:
> {code:java}
> GET /jobs/:jobid/rescales/history
> {code}
> The ignored event is no longer present because the bounded history only
> retains the later completed event.
> # Request:
> {code:java}
> GET /jobs/:jobid/rescales/details/:ignored-rescale-uuid
> {code}
> h2. Actual result
> The details endpoint returns HTTP 404:
> {code:java}
> Could not find rescale details of specified rescaleUuid
> <ignored-rescale-uuid>.
> {code}
> The Rescales Web UI surfaces this response as a server error.
> h2. Expected result
> Every rescale event returned by {{/jobs/:jobid/rescales/overview}} should
> remain queryable through the details endpoint while it is exposed by the
> Overview.
> The configured history size should continue to limit the number of entries
> returned by the History endpoint.
> h2. Root cause
> {{DefaultRescaleTimeline}} stores rescale information in two independent
> structures:
> * {{rescaleHistory}} is a {{BoundedFIFOQueue}} limited by
> {{{}web.adaptive-scheduler.rescale-history.size{}}}.
> * {{latestRescales}} retains the latest event for each terminal state
> independently of that limit.
> {{RescalesStatsSnapshot}} copies both structures, but {{idToRescaleMap}} is
> built only from {{{}rescaleHistory{}}}:
> {code:java}
> this.idToRescaleMap =
> rescaleHistory.stream()
> .collect(
> Collectors.toMap(
> r -> r.getRescaleIdInfo().getRescaleUuid(),
> Function.identity()));
> {code}
> {{JobRescaleDetailsHandler}} uses this map to resolve the UUID. Therefore, an
> event can still be returned by the Overview through {{latestRescales}} while
> being absent from the details lookup.
> The archived details have the same inconsistency because
> {{JobRescaleDetailsHandler#archiveJsonWithPath}} only archives entries from
> {{{}rescaleHistory{}}}.
> h2. fix
> * Make {{RescalesStatsSnapshot#getRescale}} fall back to {{latestRescales}}
> when the UUID is not found in the history index.
> * Archive the union of {{rescaleHistory}} and {{{}latestRescales{}}},
> deduplicated by rescale UUID.
> * Optionally make the Web UI handle a transient details 404 gracefully. This
> should be an additional safeguard rather than the primary fix.
> This preserves the configured History limit while ensuring that entries
> exposed by the Overview remain queryable.
> h2. Tests
> Add regression tests covering the following cases:
> * An IGNORED event is evicted from the bounded history but remains
> {{{}latest.ignored{}}}; its details can still be retrieved.
> * Archived JSON contains details for the evicted latest event.
> * An event present in both History and latest is archived only once.
> * An unknown UUID still returns HTTP 404.
> * The History endpoint continues to respect the configured size limit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
