Hi, How's the proposal going?
On 2024/01/19 06:28:14 Girish Sharma wrote: > Hello everyone, > > A a true cloud native platform, which supports scale up and scale down, I > feel like there is a need to be able to reduce partition count in pulsar to > truly achieve a scale down after events like sales (akin to black friday, > etc) or huge temporary publish burst due to backfill. > > I looked through the archives (upto 2021) and did not find any prior > discussion on the same topic. > > I have given this an initial thought to figure out what would it need to > support such a feature in the lowest footprint possible. I am attaching the > document explaining the need, requirements and initial high level details > [0]. What I would like is to understand if the community also finds this > feature helpful and does the approach described in the document have some > fatal flaw? Summarizing the approach here as well: > > - Introduce an ability to convert a normal topic object into a read-only > topic via admin api and an additional partitioned-topic metadata property > (just like shadow source, etc) > - Add logic to block produce but allow new consumers and dispatch call > based on this flag > - Add logic in GC to clean out read only topics when all of their > ledgers expire (TTL/retention) > > Goal is that there is no data movement involved and no impact on existing > partitions during this scale down. > > Looking forward to the discussion. > > [0] > https://docs.google.com/document/d/1sbGQSwDihQftIRsxAXg5Zm4uxKQ0kRk9HadKYRFTswI/edit?usp=sharing > > Regards > -- > Girish Sharma >
