+1 (binding) Best, Hang
Zhanpeng Wu <wuzhanpeng.w...@gmail.com> 于2022年1月14日周五 10:37写道： > > This is the voting thread for PIP-129. It will stay open for at least 48 > hours. Pasted below for quoting convenience. > > ---- > > https://github.com/apache/pulsar/issues/13526 > > ---- > > ## Motivation > > Under the current ledger-trimming design in > `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#internalTrimLedgers`, > we need to collect those ledgers that need to be deleted first, and then > perform the asynchronous deletion of the ledger concurrently, but we do not > continue to pay attention to whether the deletion operation is completed. > If the meta-information update has been successfully completed but an error > occurs during the asynchronous deletion, the ledger may not be deleted, but > at the logical level we think that the deletion has been completed, which > will make this part of the data remain in the storage layer forever (such > as bk). As the usage time of the cluster becomes longer, the residual data > that cannot be deleted will gradually increase. > > In order to achieve this goal, we can separate the logic of > meta-information update and ledger deletion. In the trimming process, we > can first mark which ledgers are deletable, and update the results to the > metadatastore. We can perform the deletion of marked ledgers asynchronously > in the callback of updating the meta information, so that the original > logic can be retained seamlessly. Therefore, when we are rolling upgrade or > rollback, the only difference is whether the deleted ledger is marked for > deletion. > > To be more specific: > 1. for upgrade, only the marker information of ledger has been added, and > the logical sequence of deletion has not changed. > 2. for rollback, some ledgers that have been marked for deletion may not be > deleted due to the restart of the broker. This behavior is consistent with > the original version. > > In addition, if the ledger that has been marked is not deleted > successfully, the marker will not be removed. So for this part of ledgers, > every time trimming is triggered, it will be deleted again, which is > equivalent to a check and retry mechanism. > > ## Goal > > We need to modify some logic in > `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#internalTrimLedgers` > so that the ledger deletion logic in ledger-trimming is split into two > stages, marking and deleting. Once the marker information is updated to the > metadatastore, every trimming will try to trigger the ledger deletion until > all the deleteable ledgers are successfully deleted. > > ## Implementation > > This proposal aims to separate the deletion logic in ledger-trimming, so > that `ManagedLedgerImpl#internalTrimLedgers` is responsible for marking the > deletable ledgers and then perform actual ledger deletion according to the > metadatastore. > > Therefore, the entire trimming process is broken down into the following > steps: > > 1. mark deletable ledgers and update ledger metadata. > 2. do acutual ledger deletion after metadata is updated. > > For step 1, we can store the marker of deletable information in > `org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl#propertiesMap`. When > retrieving the deleted ledger information, we can directly query by > iterating `propertiesMap`. If this solution is not accepted, maybe we can > create a new znode to store these information, but this approach will not > be able to reuse the current design. > > For step 2, we can perform the deletion of marked ledgers asynchronously in > the callback of updating the meta information. And every trimming will > trigger the check and delete for those deleteable ledgers. > > Related PR: https://github.com/apache/pulsar/pull/13575
