J-HowHuang opened a new pull request, #16422: URL: https://github.com/apache/pinot/pull/16422
## Description In PR https://github.com/apache/pinot/pull/15891, we introduced a new algorithm for scheduling tenant rebalance. This new algo removed the design of parallel queue and sequential queue duality--it made every table to run in the same degree of parallelism. This is backward incompatible. We were discussing whether to deprecate the old usage of this duality and thus deprecate the parameters `parallelWhitelist` and `parallelBlacklist` see issue: https://github.com/apache/pinot/issues/16057 We've got a consensus that we should maintain the support of this duality and incorporate the design with the new algorithm. The previous PR broke the API into two code path, where the old algo was executed if the old parameters were specified, vice versa. This PR is to amend that breaking up, to incorporate the old design with the new. ## Change Now the tenant rebalance API has only one code path. The execution of the tenant rebalance is: 1. Resolve the tables to be included in the rebalance job, based on `includeTables` and `excludeTables` parameters 2. Schedule the tables into two queues, parallel queue and sequential queue 3. Run the parallel threads to consume tables and run table rebalance jobs from the parallel queue 4. After the parallel queue becomes empty, run one single thread to consume tables and run table rebalance jobs from the sequential queue In addition, in favor of making tenant rebalance job fault tolerant in terms of controller crash, where the tenant rebalance job progress and context should be track in Zookeeper and have retry mechanism like table rebalance, a new class `TenantTableRebalanceJobContext` is added to assist this future change, though no related mechanism will be added in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
