https://github.com/apache/pulsar/issues/13526
---- ## Motivation Under the current ledger-trimming design in `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#internalTrimLedgers`, we need to collect those ledgers that need to be deleted first, and then perform the asynchronous deletion of the ledger concurrently, but we do not continue to pay attention to whether the deletion operation is completed. If the meta-information update has been successfully completed but an error occurs during the asynchronous deletion, the ledger may not be deleted, but at the logical level we think that the deletion has been completed, which will make this part of the data remain in the storage layer forever (such as bk). As the usage time of the cluster becomes longer, the residual data that cannot be deleted will gradually increase. In order to achieve this goal, we can separate the logic of meta-information update and ledger deletion. In the trimming process, we can first mark which ledgers are deletable, and update the results to the metadatastore. We can perform the deletion of marked ledgers asynchronously in the callback of updating the meta information, so that the original logic can be retained seamlessly. Therefore, when we are rolling upgrade or rollback, the only difference is whether the deleted ledger is marked for deletion. To be more specific: 1. for upgrade, only the marker information of ledger has been added, and the logical sequence of deletion has not changed. 2. for rollback, some ledgers that have been marked for deletion may not be deleted due to the restart of the broker. This behavior is consistent with the original version. In addition, if the ledger that has been marked is not deleted successfully, the marker will not be removed. So for this part of ledgers, every time trimming is triggered, it will be deleted again, which is equivalent to a check and retry mechanism. ## Goal We need to modify some logic in `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#internalTrimLedgers` so that the ledger deletion logic in ledger-trimming is split into two stages, marking and deleting. Once the marker information is updated to the metadatastore, every trimming will try to trigger the ledger deletion until all the deleteable ledgers are successfully deleted. ## Implementation This proposal aims to separate the deletion logic in ledger-trimming, so that `ManagedLedgerImpl#internalTrimLedgers` is responsible for marking the deletable ledgers and then perform actual ledger deletion according to the metadatastore. Therefore, the entire trimming process is broken down into the following steps: 1. mark deletable ledgers and update ledger metadata. 2. do acutual ledger deletion after metadata is updated. For step 1, we can store the marker of deletable information in `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#propertiesMap`. When retrieving the deleted ledger information, we can directly query by iterating `propertiesMap`. If this solution is not accepted, maybe we can create a new znode to store these information, but this approach will not be able to reuse the current design. For step 2, we can perform the deletion of marked ledgers asynchronously in the callback of updating the meta information. And every trimming will trigger the check and delete for those deleteable ledgers. Related PR: https://github.com/apache/pulsar/pull/13575
