aryangupta1998 commented on code in PR #8757:
URL: https://github.com/apache/ozone/pull/8757#discussion_r2195209117
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java:
##########
@@ -750,12 +752,23 @@ public PendingKeysDeletion getPendingDeletionKeys(
.map(b -> new BlockID(b.getContainerID(),
b.getLocalID()))).collect(Collectors.toList());
BlockGroup keyBlocks =
BlockGroup.newBuilder().setKeyName(kv.getKey())
.addAllBlockIDs(blockIDS).build();
+ serializedSize += keyBlocks.getProto().getSerializedSize();
+ if (serializedSize > ratisByteLimit) {
+ flag = true;
+ LOG.info(
+ "Total size of cumulative keys in a cycle crossed 90%
ratis limit, serialized size: {}",
Review Comment:
Based on my testing, this scenario doesn't appear to be very common. In a
typical production case, a key might have around 10 blocks. From test results,
20,000 such keys occupy roughly 2 MB (1.8 MB to be exact). Given that 90% of
the default Ratis limit is about 29 MB, it would take approximately 300,000
keys in a single iteration to reach that limit — which seems unlikely in most
practical scenarios.
Moreover, to successfully delete 300,000 keys in one iteration, the
following configurations would need to be tuned:
```
hdds.scm.block.deletion.per-interval.max (on SCM)
hdds.datanode.block.deleting.limit.per.interval (on Datanodes)
```
Given the above, it makes sense to downgrade this log message to DEBUG level.
Let me know your thoughts — open to suggestions.
cc @ashishkumar50
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
