[
https://issues.apache.org/jira/browse/HDDS-17618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Rose updated HDDS-17618:
------------------------------
Attachment: junit-ozone-integration-test-recon.zip
> Recon ContainerKeyMapper FSO/OBS tasks can truncate each other's data during
> reprocess
> --------------------------------------------------------------------------------------
>
> Key: HDDS-17618
> URL: https://issues.apache.org/jira/browse/HDDS-17618
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Ethan Rose
> Priority: Major
> Attachments: junit-ozone-integration-test-recon.zip
>
>
> This RCA was determined by AI while examining an [intermittent test
> timeout|https://github.com/sodonnel/hadoop-ozone/actions/runs/41122784678/job/124310237831]
> in TestReconContainerEndpoint.
> ----
>
> TestReconContainerEndpoint.testContainerEndpointForOBSBucket fails
> intermittently in CI with a TimeoutException in
> ReconOmMetaManagerTestUtils.waitUntilReconKeyCounts. The test waits 90s for
> Recon's container-key index to show the key it wrote, and the index never
> does. It passes when run on its own.
> Observed in the CI logs
> - Recon's OM DB is bootstrapped from a full OM snapshot, which triggers
> reInitializeTasks (reason MANUAL_TRIGGER).
> - ContainerKeyMapperTaskFSO and ContainerKeyMapperTaskOBS both log
> "Reprocess completed" and the reinitialization succeeds.
> - Recon's OM DB then stays at the snapshot's sequence number. Every later OM
> sync returns 0 updates, so there is no delta event that could re-add the key.
> - In the failing run, OBS finished its reprocess first, hit the "last task"
> branch, and logged "Cleared shared map and reset initialization flag" at
> 21:19:19,159. FSO's initializeKeyContainerTable log line followed at
> 21:19:19,160, after that reset.
> Proposed root cause (hypothesis, not directly confirmed)
> ContainerKeyMapperHelper guards a one-time init (truncating the container-key
> tables and clearing the shared count map) with the static flag
> ReconConstants.CONTAINER_KEY_MAPPER_INITIALIZED. Two tasks (FSO and OBS)
> share that flag and an ACTIVE_TASK_COUNT.
> The task that brings ACTIVE_TASK_COUNT to 0 resets the flag at the end of
> reprocess(). Whether that is safe depends on both tasks having incremented
> the count before either finishes. If one task finishes before the other has
> started its init, then:
> 1. The finishing task sees the count reach 0 and resets the flag.
> 2. The slower task starts, sees the flag as false, and runs the init again.
> 3. That init truncates the container-key tables and wipes the data the first
> task already wrote.
> I did not find a log line that shows the second truncation directly. The
> timing a Which task ends up with its data wiped depends on scheduling, which
> fits an
> intermittent failure.
> Proposed fix
> In ContainerKeyMapperHelper.reprocess, stop resetting
> CONTAINER_KEY_MAPPER_INITIALIZED from the last-finishing task. The flag is
> already reset once per reprocess cycle by
> ReconConstants.resetTableTruncatedFlags(), which
> ReconTaskControllerImpl.reInitiamitting tasks. The test-only
> clearSharedContainerCountMap() still resets it as
> well. The shared count map is still cleared when the last task finishes.
>
> Risk to review
> Code that calls the FSO or OBS task's reprocess() repeatedly, outside
> reInitializing resetTableTruncatedFlags(), would now skip truncation on later
> runs. I found no
> such caller in main code. Existing unit tests that do this still pass.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
