Hao Yu created FLINK-40969:
------------------------------
Summary: Rescale details endpoint returns 404 for latest events
evicted from AdaptiveScheduler history
Key: FLINK-40969
URL: https://issues.apache.org/jira/browse/FLINK-40969
Project: Flink
Issue Type: Bug
Components: Runtime / REST, Runtime / Web Frontend
Affects Versions: 2.3.0
Environment: Affected code paths were confirmed in:
* Apache Flink 2.3.0
Configuration:
* Streaming job
* AdaptiveScheduler enabled
* web.adaptive-scheduler.rescale-history.size set to a positive bounded value
The issue is deployment-mode independent because it is caused by the in-memory
rescale snapshot and REST lookup logic.
Reporter: Hao Yu
Attachments: PixPin_2026-10-12_17-39-13.png
h2. Problem
The AdaptiveScheduler Rescales Overview may expose a rescale UUID whose details
can no longer be retrieved.
The Overview keeps the latest rescale event for each terminal state (COMPLETED,
FAILED, and IGNORED). These entries are stored separately from the bounded
rescale history.
When a latest event for one terminal state is older than the configured history
limit, newer events of other terminal states can evict it from the history. The
event remains visible in the Overview, but requesting its details returns HTTP
404:
{code:java}
Could not find rescale details of specified rescaleUuid <rescale-uuid>.
{code}
The Web UI automatically requests details for the latest events returned by the
Overview. Consequently, opening the Rescales page can display a server error
even though the job is running normally and the UUID was returned by the
Overview endpoint.
h2. Steps to reproduce
# Configure a streaming job to use the AdaptiveScheduler and enable a small
rescale history:
{code:java}
jobmanager.scheduler: adaptive
web.adaptive-scheduler.rescale-history.size: 1
{code}
# Trigger a rescale that terminates as IGNORED, for example by updating the
resource requirements again while a previous rescale is still in progress.
# Trigger a later rescale that terminates as COMPLETED.
# Request:
{code:java}
GET /jobs/:jobid/rescales/overview
{code}
# Record the UUID returned in {{{}latest.ignored{}}}.
# Request:
{code:java}
GET /jobs/:jobid/rescales/history
{code}
The ignored event is no longer present because the bounded history only retains
the later completed event.
# Request:
{code:java}
GET /jobs/:jobid/rescales/details/:ignored-rescale-uuid
{code}
h2. Actual result
The details endpoint returns HTTP 404:
{code:java}
Could not find rescale details of specified rescaleUuid <ignored-rescale-uuid>.
{code}
The Rescales Web UI surfaces this response as a server error.
h2. Expected result
Every rescale event returned by {{/jobs/:jobid/rescales/overview}} should
remain queryable through the details endpoint while it is exposed by the
Overview.
The configured history size should continue to limit the number of entries
returned by the History endpoint.
h2. Root cause
{{DefaultRescaleTimeline}} stores rescale information in two independent
structures:
* {{rescaleHistory}} is a {{BoundedFIFOQueue}} limited by
{{{}web.adaptive-scheduler.rescale-history.size{}}}.
* {{latestRescales}} retains the latest event for each terminal state
independently of that limit.
{{RescalesStatsSnapshot}} copies both structures, but {{idToRescaleMap}} is
built only from {{{}rescaleHistory{}}}:
{code:java}
this.idToRescaleMap =
rescaleHistory.stream()
.collect(
Collectors.toMap(
r -> r.getRescaleIdInfo().getRescaleUuid(),
Function.identity()));
{code}
{{JobRescaleDetailsHandler}} uses this map to resolve the UUID. Therefore, an
event can still be returned by the Overview through {{latestRescales}} while
being absent from the details lookup.
The archived details have the same inconsistency because
{{JobRescaleDetailsHandler#archiveJsonWithPath}} only archives entries from
{{{}rescaleHistory{}}}.
h2. fix
* Make {{RescalesStatsSnapshot#getRescale}} fall back to {{latestRescales}}
when the UUID is not found in the history index.
* Archive the union of {{rescaleHistory}} and {{{}latestRescales{}}},
deduplicated by rescale UUID.
* Optionally make the Web UI handle a transient details 404 gracefully. This
should be an additional safeguard rather than the primary fix.
This preserves the configured History limit while ensuring that entries exposed
by the Overview remain queryable.
h2. Tests
Add regression tests covering the following cases:
* An IGNORED event is evicted from the bounded history but remains
{{{}latest.ignored{}}}; its details can still be retrieved.
* Archived JSON contains details for the evicted latest event.
* An event present in both History and latest is archived only once.
* An unknown UUID still returns HTTP 404.
* The History endpoint continues to respect the configured size limit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
