BewareMyPower commented on code in PR #24771: URL: https://github.com/apache/pulsar/pull/24771#discussion_r2384871191
########## pip/pip-444.md: ########## @@ -0,0 +1,62 @@ + +# PIP-444: Rate limit for deleting ledger to alleviate the zk pressure. + +# Background knowledge + +Apache Pulsar uses Apache BookKeeper as its storage layer. +Each topic consists of multiple ledgers, which are created periodically based on the configuration. +Each ledger corresponds to a ZooKeeper (ZK) node for metadata storage. When a ledger is deleted, the corresponding ZK node will also be deleted. +Retention policies determine how long messages are retained in a topic. When the retention period is reached, the corresponding ledgers will be deleted. + +# Motivation + +When the retention of a large topic is reduced, a significant number of ledgers need to be deleted. +Since this deletion operation is not rate-limited, it could results in ZooKeeper (ZK) latency of several minutes, +which threatens the stability of the entire Pulsar cluster. + +# Goals + +Add rate limit feature for deleting ledgers. + +# Detailed Design + +Introduce two new configuration parameters: +``` + @FieldContext( + category = CATEGORY_STORAGE_ML, + doc = "Max number of concurrent requests for deleting ledgers at broker level" + ) + private int managedLedgerDeleteMaxConcurrentRequests = 1000; + @FieldContext( + category = CATEGORY_STORAGE_ML, + doc = "Number of threads to be used for deleting ledgers at broker level" + ) + private int managedLedgerDeleteThreadPoolSize = 5; Review Comment: Using 5 threads only to call an asynchronous operation is a waste, it's better to change the default value to 1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
