sumitagrawl commented on code in PR #8827:
URL: https://github.com/apache/ozone/pull/8827#discussion_r2288100698
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerSet.java:
##########
@@ -70,6 +73,8 @@ public class ContainerSet implements Iterable<Container<?>> {
private final Table<ContainerID, String> containerIdsTable;
// Handler that will be invoked when a scan of a container in this set is
requested.
private OnDemandContainerScanner containerScanner;
+ private CachedPendingDeletion cachedPendingDeletion;
Review Comment:
already BlockDeletingService.java have cached information calculated, why
need another cache here ?
In that class, its calculated every iteration.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerMetadataInspector.java:
##########
@@ -239,6 +239,8 @@ static ObjectNode getDBMetadataJson(Table<String, Long>
metadataTable,
metadataTable.get(containerData.getBytesUsedKey()));
dBMetadata.put(OzoneConsts.PENDING_DELETE_BLOCK_COUNT,
metadataTable.get(containerData.getPendingDeleteBlockCountKey()));
+ dBMetadata.put(OzoneConsts.PENDING_DELETE_BLOCK_BYTES,
+ metadataTable.get(containerData.getPendingDeleteBlockBytesKey()));
Review Comment:
checkAndRepair() method update for BlockCount, do similar is required for
BlockBytes ?
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java:
##########
@@ -313,20 +318,28 @@ private static void populateContainerMetadata(
// Set pending deleted block count.
final long blockPendingDeletion;
+ long blockPendingDeletionBytes = 0L;
+ Long pendingDeletionBlockBytes = metadataTable.get(kvContainerData
+ .getPendingDeleteBlockBytesKey());
Long pendingDeleteBlockCount =
metadataTable.get(kvContainerData
.getPendingDeleteBlockCountKey());
if (pendingDeleteBlockCount != null) {
blockPendingDeletion = pendingDeleteBlockCount;
+ if (pendingDeletionBlockBytes != null) {
+ blockPendingDeletionBytes = pendingDeletionBlockBytes;
+ } else if
(VersionedDatanodeFeatures.isFinalized(HDDSLayoutFeature.DATA_DISTRIBUTION)) {
+ LOG.warn("Missing pendingDeletionSize from {}: recalculate them from
delete txn tables",
+ metadataTable.getName());
+ ObjectNode pendingDeletions = getAggregateValues(store,
kvContainerData, kvContainerData.getSchemaVersion());
+ blockPendingDeletionBytes =
pendingDeletions.get(PENDING_DELETION_BYTES).asLong();
+ }
} else {
- // Set pending deleted block count.
- LOG.warn("Missing pendingDeleteBlockCount from {}: recalculate them from
block table", metadataTable.getName());
- MetadataKeyFilters.KeyPrefixFilter filter =
- kvContainerData.getDeletingBlockKeyFilter();
- blockPendingDeletion = store.getBlockDataTable().getRangeKVs(
- kvContainerData.startKeyEmpty(), Integer.MAX_VALUE,
kvContainerData.containerPrefix(), filter, true)
- // TODO: add a count() method to avoid creating a list
- .size();
+ LOG.warn("Missing pendingDeleteBlockCount/size from {}: recalculate them
from delete txn tables",
+ metadataTable.getName());
+ ObjectNode pendingDeletions = getAggregateValues(store, kvContainerData,
kvContainerData.getSchemaVersion());
+ blockPendingDeletionBytes =
pendingDeletions.get(PENDING_DELETION_BYTES).asLong();
+ blockPendingDeletion =
pendingDeletions.get(PENDING_DELETION_BLOCKS).asLong();
Review Comment:
finalized check required here also as above. And what happens if its null ?
earlier for blocks, there is nothing null.
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/DeleteBlocksCommandHandler.java:
##########
@@ -639,8 +639,17 @@ private void updateMetaData(KeyValueContainerData
containerData,
// update pending deletion blocks count and delete transaction ID in
// in-memory container status
- containerData.updateDeleteTransactionId(delTX.getTxID());
- containerData.incrPendingDeletionBlocks(newDeletionBlocks);
+ long pendingBytes = delTX.getTotalBlockSize();
+ metadataTable
+ .putWithBatch(batchOperation,
+ containerData.getPendingDeleteBlockBytesKey(),
+ pendingBytes);
+ containerData.incrPendingDeletionBlocks(newDeletionBlocks, pendingBytes);
+
+ //update latest delete transactionid
+ if (delTX.getTxID() > containerData.getDeleteTransactionId()) {
Review Comment:
this is redundant check, already its updated inside the method as
`deleteTransactionId = max(transactionId, deleteTransactionId);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
