[
https://issues.apache.org/jira/browse/HDDS-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125934#comment-18125934
]
Siyao Meng commented on HDDS-16712:
-----------------------------------
Thanks [~adwetaojha] . This issue is unassigned right now. Pls feel free to
pick it up.
> Key creation fails intermittently in k8s after datanodes restart
> ----------------------------------------------------------------
>
> Key: HDDS-16712
> URL: https://issues.apache.org/jira/browse/HDDS-16712
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Client, Ozone Manager, SCM
> Reporter: Attila Doroszlai
> Priority: Major
>
> Intermittent failure of key generator after datanodes are restarted in
> {{kubernetes}}:
> {code}
> Ozone Client Key Generator | FAIL |
> 255 != 0
> {code}
> {code}
> 2026-10-04 10:57:23 INFO BaseFreonGenerator:317 - Executing test with prefix
> ockgpost and number-of-tests 1
> 2026-10-04 10:57:23 INFO ProgressBar:167 - Progress: 0.00 % (0 out of 1)
> 2026-10-04 10:57:24 ERROR BaseFreonGenerator:231 - Error on executing task 0
> INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Unable to
> allocate a container to the block of size: 268435456, replicationConfig:
> RATIS/THREE. Waiting for one of pipelines to be OPEN failed. The input
> pipeline IDs 521df264-010e-43d8-a0e2-dbbe8ac0f757 cannot be found
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:805)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleSubmitRequestAndSCMSafeModeRetry(OzoneManagerProtocolClientSideTranslatorPB.java:2531)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:792)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.openOutputStream(RpcClient.java:1520)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:1435)
> at
> org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:1420)
> at
> org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:500)
> at
> org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:480)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.lambda$createKeyWithData$0(OzoneClientKeyGenerator.java:136)
> at com.codahale.metrics.Timer.time(Timer.java:101)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.createKeyWithData(OzoneClientKeyGenerator.java:135)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.createKey(OzoneClientKeyGenerator.java:129)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:225)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.runTaskLoop(BaseFreonGenerator.java:206)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:180)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:171)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
> at java.base/java.lang.Thread.run(Thread.java:1474)
> 2026-10-04 10:57:24 INFO ProgressBar:167 - Progress: 100.00 % (1 out of 1)
> One or more freon test is failed.
> 2026-10-04 10:57:24 INFO metrics:373 - type=TIMER, name=key-create, count=1,
> min=137.43811, max=137.43811, mean=137.43811, stddev=0.0, median=137.43811,
> p75=137.43811, p95=137.43811, p98=137.43811, p99=137.43811, p999=137.43811,
> mean_rate=4.5610858427462215, m1=0.0, m5=0.0, m15=0.0,
> rate_unit=events/second, duration_unit=milliseconds
> 2026-10-04 10:57:24 INFO BaseFreonGenerator:75 - Total execution time (sec):
> 1
> 2026-10-04 10:57:24 INFO BaseFreonGenerator:75 - Failures: 1
> {code}
> - Datanodes register after the restart one by one, triggering pipeline
> creation and closure. State becomes stable after the third DN is updated.
> - If there are no candidate containers, WritableRatisContainerProvider checks
> if any pipelines are in ALLOCATED state, and waits for them to become OPEN.
> If they are then closed, new ones will be allocated. If they are closed in
> the meantime, SCM gives up.
> - Client only retries write, but not {{openKey}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
