Siyao Meng created HDDS-17193:
---------------------------------
Summary: Recon OM sync fails on a task status write error
Key: HDDS-17193
URL: https://issues.apache.org/jira/browse/HDDS-17193
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Recon
Reporter: Siyao Meng
Assignee: Siyao Meng
{{ReconTaskStatusUpdater.recordRunStart()}} and {{recordRunCompletion()}} catch
and log {{DataAccessException}}, since a failed {{RECON_TASK_STATUS}} write
only means a stale status row. Two places in
{{OzoneManagerServiceProviderImpl}} call {{updateDetails()}} directly, so the
same failure escapes into the OM sync logic:
1. Delta sync loop, after the OM updates are applied to Recon's OM DB
([code|https://github.com/apache/ozone/blob/df42f7f60a2db3c06ed85e781ffdf89e02e0ccf0/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/OzoneManagerServiceProviderImpl.java#L769]):
{code:java}
deltaReconTaskStatusUpdater.recordRunCompletion();
fullSnapshotReconTaskUpdater.updateDetails();
{code}
The exception lands in the generic {{catch (Exception e)}}, which marks the
delta task failed and sets {{fullSnapshot = true}}. A one row status write
failure costs a full OM DB snapshot download and a reprocess of all tasks.
2. {{executeFullSnapshot()}}, after the new snapshot is installed
([code|https://github.com/apache/ozone/blob/df42f7f60a2db3c06ed85e781ffdf89e02e0ccf0/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/OzoneManagerServiceProviderImpl.java#L913]):
{code:java}
fullSnapshotReconTaskUpdater.recordRunCompletion();
deltaReconTaskStatusUpdater.updateDetails();
{code}
The exception skips {{updateOMMetadataManager()}} and the task reinitialization
event, and the snapshot is reported as failed. Since Recon's OM DB already
holds the new snapshot, the next sync likely continues with delta updates from
it, so the tasks may never reprocess the changes between the old and new
snapshot.
This was seen in HDDS-3561 when Recon used SQLite: a {{RECON_TASK_STATUS}}
update failed with {{SQLITE_BUSY}}, delta sync failed, and Recon fell back to a
full snapshot. With Derby it needs a lock wait timeout (60s by default) or
another DB error, so it is rare, but it matters for any move off Derby
(HDDS-16668).
*Proposed fix:* move the {{DataAccessException}} handling from
{{recordRunStart()}} and {{recordRunCompletion()}} into {{updateDetails()}}, so
every caller treats a task status write failure as non fatal. Add a test that
makes the DAO throw during delta sync and asserts no full snapshot is triggered.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
