[
https://issues.apache.org/jira/browse/HDDS-16440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16440:
----------------------------------
Labels: pull-request-available (was: )
> DirectoryDeletingService can NPE on a transient Ratis submit failure (masked
> by BackgroundService catch)
> --------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16440
> URL: https://issues.apache.org/jira/browse/HDDS-16440
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Huang Kuan Hao
> Priority: Minor
> Labels: pull-request-available
> Attachments: HDDS-16440.001.patch
>
>
> h3. Finding
> A missing null check turns a tolerated transient Ratis submit failure
> (ServiceException producing a null response) into an uncaught NPE
> dereferencing resp.getSuccess() in the DirectoryDeletingService background
> task. Masked by BackgroundService.run()'s unconditional catch(Throwable),
> which logs and reschedules next interval, and nothing durable is written;
> absent that guard the NPE would abort the run and risk stalling the periodic
> deletion loop (liveness, no data corruption).
> h3. Classification
> * Verdict: MASKED
> * Severity: Medium
> * Source: Specula TLA+ model checking and confirmation debate, finding CR-5
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: om-fso-recursive-deletion
> Guidance:
> campaigns/ozone-9fbf9ee/targets/005-om-fso-recursive-deletion/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "om-fso-recursive-deletion|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16429 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> # OM is leader and the DirectoryDeletingService background task runs with
> pending deleted directories, so enumeration builds a non-empty purge-path
> list.
> # At batch flush, the OM Ratis submit fails transiently (leader step-down /
> not-leader / timeout / interrupt during OM shutdown).
> OzoneManagerRatisServer.submitRequest throws ServiceException.
> # submitPurgeRequest catches the ServiceException, logs "PurgePaths request
> failed. Will retry at next run.", and returns null.
> # submitPurgePathsWithBatching then evaluates resp.getSuccess() on that null
> response, throwing NullPointerException.
> # The NPE is a RuntimeException that none of the enclosing catch blocks
> (which catch only IOException / ExecutionException / InterruptedException)
> handle, so it propagates out of DirDeletingTask.call() and aborts the whole
> directory-deleting run instead of the intended "skip, retry next run".
> Reproduced in a unit test that spies the service, stubs submitRequest to
> throw ServiceException at its real seam, and drives the package-visible
> optimizeDirDeletesAndSubmitRequest with a non-empty purge list. Without the
> fix the call throws NullPointerException.
> h3. Root cause
> submitPurgeRequest (DirectoryDeletingService) returns null on
> ServiceException (a normal, expected transient submit failure it explicitly
> catches and logs). Its caller submitPurgePathsWithBatching dereferences the
> response unconditionally with resp.getSuccess() at both the mid-loop batch
> flush and the final batch flush, so a null response causes a
> NullPointerException. The intended behavior (return an empty list, retry next
> run) is defeated: the run dies instead of retrying.
> h3. Suggested fix
> Null-check the response at both submitPurgePathsWithBatching flush sites:
> treat a null response the same as a failed submit (stop batching and return
> the empty list), so a transient submit failure ends the run gracefully and
> the pending directories are retried on the next run. See
> [^HDDS-16440.001.patch]. The patch adds
> testPurgeDirectoriesSubmitFailureIsTolerated, which reproduces the NPE
> without the fix and passes with it.
> This finding is masked in production: the aborted run's un-purged directories
> remain in the deleted-directory table and are retried on the next scheduled
> run, so the effect is a delayed reclamation of one run rather than permanent
> loss. The fix removes the abort so the retry path is taken directly.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
