Huang Kuan Hao created HDDS-17257:
-------------------------------------
Summary: Intermittent failure in
TestBlockDeletingService#testBlockDeletion
Key: HDDS-17257
URL: https://issues.apache.org/jira/browse/HDDS-17257
Project: Apache Ozone
Issue Type: Sub-task
Components: test
Reporter: Huang Kuan Hao
Assignee: Huang Kuan Hao
{code}
[ERROR] Tests run: 73, Failures: 1, Errors: 0, Skipped: 4, Time elapsed: 27.20
s <<< FAILURE! -- in
org.apache.hadoop.ozone.container.common.TestBlockDeletingService
[ERROR]
org.apache.hadoop.ozone.container.common.TestBlockDeletingService.testBlockDeletion(ContainerTestVersionInfo)[4]
-- Time elapsed: 0.362 s <<< FAILURE!
org.opentest4j.AssertionFailedError: Failed to read container checksum tree
file:
/tmp/junit-11171629283323802431/hdds/eca15dd6-87ff-4680-9d35-46d58ca44679/current/containerDir1/117454920117519356/metadata/117454920117519356.tree
at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:138)
at
org.apache.hadoop.ozone.container.common.TestBlockDeletingService.assertDeletionsInChecksumFile(TestBlockDeletingService.java:1311)
at
org.apache.hadoop.ozone.container.common.TestBlockDeletingService.lambda$testBlockDeletion$0(TestBlockDeletingService.java:628)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
at
org.apache.hadoop.ozone.container.common.TestBlockDeletingService.testBlockDeletion(TestBlockDeletingService.java:628)
{code}
h3. Root cause
{{deleteAndWait(service, n)}} waits until
{{BlockDeletingServiceTestImpl#getTimesOfProcessed()}} reaches {{n}}. The
counter is incremented when {{BackgroundService.PeriodicalTask#run()}} returns,
but {{run()}} only submits each {{BlockDeletingTask}} with
{{CompletableFuture.runAsync}} and does not wait for it, so the task may still
be running when {{deleteAndWait}} returns.
{{assertDeletionsInChecksumFile}} reads the container checksum file right after
{{deleteAndWait}}, without waiting. If
{{ContainerChecksumTreeManager#addDeletedBlocks}} has not written the file yet,
the read fails as above. The other checks in {{testBlockDeletion}} already wait
for the state they read, e.g. {{waitFor(() -> getBytesUsed() == containerSpace
/ 3)}}. {{testChecksumFileUpdatedWhenDeleteRetried}} calls the same assertion
right after {{deleteAndWait}}.
A 500 ms sleep at the start of
{{ContainerChecksumTreeManager#addDeletedBlocks}} reproduces it:
{{testBlockDeletion}} and {{testChecksumFileUpdatedWhenDeleteRetried}} fail in
all 8 parameterizations with the same error, and pass with the fix.
h3. Fix
Make {{assertDeletionsInChecksumFile}} wait until the checksum file records the
expected number of deleted blocks.
{{deleteAndWait}} could instead wait for the submitted tasks to finish (via
{{BackgroundService#getFuture}}). That would also cover the success count check
in {{testBlockDeletion}}, which is incremented a few statements after
{{bytesUsed}}, but it changes the timing of every {{deleteAndWait}} caller, so
it is left out here.
Seen in:
- https://github.com/apache/ozone/actions/runs/39010100374/job/117333028016
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
