[
https://issues.apache.org/jira/browse/HDDS-17193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17193:
------------------------------
Status: Patch Available (was: Open)
> Recon OM sync fails on a task status write error
> ------------------------------------------------
>
> Key: HDDS-17193
> URL: https://issues.apache.org/jira/browse/HDDS-17193
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Recon
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> {{ReconTaskStatusUpdater.recordRunStart()}} and {{recordRunCompletion()}}
> catch and log {{DataAccessException}}, since a failed {{RECON_TASK_STATUS}}
> write only means a stale status row. Two places in
> {{OzoneManagerServiceProviderImpl}} call {{updateDetails()}} directly, so the
> same failure escapes into the OM sync logic:
> 1. Delta sync loop, after the OM updates are applied to Recon's OM DB
> ([code|https://github.com/apache/ozone/blob/df42f7f60a2db3c06ed85e781ffdf89e02e0ccf0/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/OzoneManagerServiceProviderImpl.java#L769]):
> {code:java}
> deltaReconTaskStatusUpdater.recordRunCompletion();
> fullSnapshotReconTaskUpdater.updateDetails();
> {code}
> The exception lands in the generic {{catch (Exception e)}}, which marks the
> delta task failed and sets {{fullSnapshot = true}}. A one row status write
> failure costs a full OM DB snapshot download and a reprocess of all tasks.
> 2. {{executeFullSnapshot()}}, after the new snapshot is installed
> ([code|https://github.com/apache/ozone/blob/df42f7f60a2db3c06ed85e781ffdf89e02e0ccf0/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/OzoneManagerServiceProviderImpl.java#L913]):
> {code:java}
> fullSnapshotReconTaskUpdater.recordRunCompletion();
> deltaReconTaskStatusUpdater.updateDetails();
> {code}
> The exception skips {{updateOMMetadataManager()}} and the task
> reinitialization event, and the snapshot is reported as failed. Since Recon's
> OM DB already holds the new snapshot, the next sync likely continues with
> delta updates from it, so the tasks may never reprocess the changes between
> the old and new snapshot.
> This was seen in HDDS-3561 when Recon used SQLite: a {{RECON_TASK_STATUS}}
> update failed with {{SQLITE_BUSY}}, delta sync failed, and Recon fell back to
> a full snapshot. With Derby it needs a lock wait timeout (60s by default) or
> another DB error, so it is rare, but it matters for any move off Derby
> (HDDS-16668).
> *Proposed fix:* move the {{DataAccessException}} handling from
> {{recordRunStart()}} and {{recordRunCompletion()}} into {{updateDetails()}},
> so every caller treats a task status write failure as non fatal. Add a test
> that makes the DAO throw during delta sync and asserts no full snapshot is
> triggered.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
