[
https://issues.apache.org/jira/browse/HDDS-16781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16781:
------------------------------
Description:
{code}
[ERROR]
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(ContainerLayoutVersion)[2]
-- Time elapsed: 22.70 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <2> but was: <4>
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:632)
at
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testSkip(TestKeyInputStream.java:368)
at
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(TestKeyInputStream.java:136)
{code}
h3. Root cause
{{testSkip}} writes a RATIS/THREE key and immediately reads it, asserting
exactly 2 ReadChunk ops via {{XceiverClientMetrics}}. The write returns once
the data is committed, but a follower may not have applied the PutBlock yet. In
the failing run both ReadChunk requests first went to such a follower:
{code}
WARN keyvalue.KeyValueHandler - Operation: ReadChunk , Message: Unable to find
the block with bcsID 485. Container 9 bcsId is 483. , Result: UNKNOWN_BCSID
{code}
The client correctly retried on another datanode, but the metric counts every
attempt, so it saw 4 ReadChunk ops (2 failed and 2 successful).
h3. Fix
In {{testSkip}} and {{testSeek}}, assert at least 2 ReadChunk ops after the
read instead of exactly 2, so retried attempts are tolerated. The assertions
that seek and skip issue no ReadChunk stay exact.
- https://github.com/apache/ozone/actions/runs/37727947048/job/113153515921
was:
{code}
[ERROR]
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(ContainerLayoutVersion)[2]
-- Time elapsed: 22.70 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <2> but was: <4>
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:632)
at
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testSkip(TestKeyInputStream.java:368)
at
org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(TestKeyInputStream.java:136)
{code}
h3. Root cause
{{testSkip}} writes a RATIS/THREE key and immediately reads it, asserting
exactly 2 ReadChunk ops via {{XceiverClientMetrics}}. The write returns once
the data is committed, but a follower may not have applied the PutBlock yet. In
the failing run both ReadChunk requests first went to such a follower:
{code}
WARN keyvalue.KeyValueHandler - Operation: ReadChunk , Message: Unable to find
the block with bcsID 485. Container 9 bcsId is 483. , Result: UNKNOWN_BCSID
{code}
The client correctly retried on another datanode, but the metric counts every
attempt, so it saw 4 ReadChunk ops (2 failed and 2 successful).
h3. Fix
Close the key's container and wait until it is CLOSED on all replicas before
reading. Followers apply the close only after the PutBlock (Ratis log order),
so no replica can lag behind the block's bcsId. {{testSeek}} has the same exact
ReadChunk count assertion after a write, so the wait is applied there too.
- https://github.com/apache/ozone/actions/runs/37727947048/job/113153515921
> Intermittent ReadChunk count mismatch in TestKeyInputStream
> -----------------------------------------------------------
>
> Key: HDDS-16781
> URL: https://issues.apache.org/jira/browse/HDDS-16781
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Client, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> [ERROR]
> org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(ContainerLayoutVersion)[2]
> -- Time elapsed: 22.70 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: expected: <2> but was: <4>
> at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:632)
> at
> org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testSkip(TestKeyInputStream.java:368)
> at
> org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(TestKeyInputStream.java:136)
> {code}
> h3. Root cause
> {{testSkip}} writes a RATIS/THREE key and immediately reads it, asserting
> exactly 2 ReadChunk ops via {{XceiverClientMetrics}}. The write returns once
> the data is committed, but a follower may not have applied the PutBlock yet.
> In the failing run both ReadChunk requests first went to such a follower:
> {code}
> WARN keyvalue.KeyValueHandler - Operation: ReadChunk , Message: Unable to
> find the block with bcsID 485. Container 9 bcsId is 483. , Result:
> UNKNOWN_BCSID
> {code}
> The client correctly retried on another datanode, but the metric counts every
> attempt, so it saw 4 ReadChunk ops (2 failed and 2 successful).
> h3. Fix
> In {{testSkip}} and {{testSeek}}, assert at least 2 ReadChunk ops after the
> read instead of exactly 2, so retried attempts are tolerated. The assertions
> that seek and skip issue no ReadChunk stay exact.
> - https://github.com/apache/ozone/actions/runs/37727947048/job/113153515921
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
