I have a simpler question. Just storing the message-ids raw will fit ~300K entries in one ledger entry. With the bitmap changes, we can store a couple of million within one 5MB ledger entry. So can you tell us what numbers of unacked messages are creating a problem? What exactly are the issues you face, and at what numbers of unacked messages/memory use etc?
I have my own concerns about this proposal, but I would like to understand the problem first Joe On Sun, Jan 17, 2021 at 10:16 PM Sijie Guo <guosi...@gmail.com> wrote: > Hi Lin, > > Thanks you and Penghui for drafting this! We have seen a lot of pain points > of `managedLedgerMaxUnackedRangesToPersist` when enabling delayed messages. > Glad that you and Penghui are spending time on resolving this! > > Overall the proposal looks good. But I have a couple of questions about the > proposal. > > 1. What happens if the broker fails to write the entry marker? For example, > at t0, the broker flushes dirty pages and successfully writes an entry > marker. At t1, the broker tries to flushes dirty pages but failed to write > the new entry marker. How can you recover the entry marker? > > 2. When a broker crashes and recovers the managed ledger, the cursor > ledger is not writable anymore. Are you going to create a new cursor ledger > and copy all the entries from the old cursor ledger to the new one? > > It would be good if you can clarify these two questions. > > - Sijie > > On Sun, Jan 17, 2021 at 9:48 PM linlin <lin...@apache.org> wrote: > > > Hi, community: > > Recently we encountered some problems when using individual > > acknowledgments, such as: > > when the amount of acknowledgment is large, entry writing fails; a large > > amount of cache causes OOM, etc. > > So I drafted a PIP in ` > > > > > https://docs.google.com/document/d/1uQtyb8t6X04v2vrSrdGWLFkuCkBcGYZbqK8XsVJ4qkU/edit?usp=sharing` > <https://docs.google.com/document/d/1uQtyb8t6X04v2vrSrdGWLFkuCkBcGYZbqK8XsVJ4qkU/edit?usp=sharing> > > < > https://docs.google.com/document/d/1uQtyb8t6X04v2vrSrdGWLFkuCkBcGYZbqK8XsVJ4qkU/edit?usp=sharing > > > > , > > any voice is welcomed. > > >
