Hello everyone, A a true cloud native platform, which supports scale up and scale down, I feel like there is a need to be able to reduce partition count in pulsar to truly achieve a scale down after events like sales (akin to black friday, etc) or huge temporary publish burst due to backfill.
I looked through the archives (upto 2021) and did not find any prior discussion on the same topic. I have given this an initial thought to figure out what would it need to support such a feature in the lowest footprint possible. I am attaching the document explaining the need, requirements and initial high level details [0]. What I would like is to understand if the community also finds this feature helpful and does the approach described in the document have some fatal flaw? Summarizing the approach here as well: - Introduce an ability to convert a normal topic object into a read-only topic via admin api and an additional partitioned-topic metadata property (just like shadow source, etc) - Add logic to block produce but allow new consumers and dispatch call based on this flag - Add logic in GC to clean out read only topics when all of their ledgers expire (TTL/retention) Goal is that there is no data movement involved and no impact on existing partitions during this scale down. Looking forward to the discussion. [0] https://docs.google.com/document/d/1sbGQSwDihQftIRsxAXg5Zm4uxKQ0kRk9HadKYRFTswI/edit?usp=sharing Regards -- Girish Sharma
