Andrés Beck-Ruiz created CASSANDRA-21504:
--------------------------------------------
Summary: Reconcile stale persisted operational jobs across Sidecar
restarts and storage failures
Key: CASSANDRA-21504
URL: https://issues.apache.org/jira/browse/CASSANDRA-21504
Project: Apache Cassandra
Issue Type: Bug
Reporter: Andrés Beck-Ruiz
Currently, the {{DurableOperationalJobTracker}} persists only two states per
job: the initial CREATED record on submission and the terminal status on
completion. Intermediate RUNNING and per-node progress are never written, and
there is no reconciliation of persisted records against live state. As a
result, a record can be left stuck at CREATED if the sidecar restarts
mid-operation or if the terminal status update from the
{{DurableOperationalJobTracker}} exhausts its retries. Persisting intermediate
RUNNING and per-node progress is handled separately in CASSSIDECAR-378, which
will narrow but not eliminate this window.
To close this gap, we should add a startup/periodic reconciliation sweep over
{{findAllJobs}} that corrects or flags non-terminal records which no longer
match live state. As part of this,
we need a way to distinguish a record that is genuinely still CREATED from one
whose true state was simply never recorded. This can either be done by:
1. An internal durability-degraded marker on {{OperationalJobRecord}}
2. A new {{OperationalJobStatus}} (e.g. UNKNOWN) surfaced through the existing
status field.
This is a follow-up to CASSSIDECAR-374 and not a regression: the prior
in-memory tracker persisted nothing and returned null once a job left memory,
so stuck at CREATED is strictly more information than before.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
