We can look at ManagedCursorImpl.buildIndividualDeletedMessageRanges What is saved in the entry is not a bitSet, but a messageRange one by one, which contains information such as ledgerId and entryId. BitSet only exists in the memory and is used to quickly determine whether it already exists. In addition, the position of each ack will be stored in the individualDeletedMessages queue. When persisted to the entry, the queue will be traversed, and the position information of each ack will generate a messageRange. A messageRange contains lowerEndpoint (ledgerId+entryId), upperEndpoint (ledgerId+entryId), 4 longs, about 256 bits.
We assume a more extreme scenario, 300K messages, every other ack has an unacknowledged, that is, 150K location information will be stored in individualDeletedMessages. 150K * 256/8/1024 /1024 ≈ 4.6MB Of course, there are also scenarios where the customer's ack spans several ledgers. On 2021/01/20 00:38:47, Joe F <j...@gmail.com> wrote: > I have a simpler question. Just storing the message-ids raw will fit ~300K> > entries in one ledger entry. With the bitmap changes, we can store a> > couple of million within one 5MB ledger entry. So can you tell us what> > numbers of unacked messages are creating a problem? What exactly are the> > issues you face, and at what numbers of unacked messages/memory use etc?> > > I have my own concerns about this proposal, but I would like to understand> > the problem first> > > Joe> > > On Sun, Jan 17, 2021 at 10:16 PM Sijie Guo <gu...@gmail.com> wrote:> > > > Hi Lin,> > >> > > Thanks you and Penghui for drafting this! We have seen a lot of pain points> > > of `managedLedgerMaxUnackedRangesToPersist` when enabling delayed messages.> > > Glad that you and Penghui are spending time on resolving this!> > >> > > Overall the proposal looks good. But I have a couple of questions about the> > > proposal.> > >> > > 1. What happens if the broker fails to write the entry marker? For example,> > > at t0, the broker flushes dirty pages and successfully writes an entry> > > marker. At t1, the broker tries to flushes dirty pages but failed to write> > > the new entry marker. How can you recover the entry marker?> > >> > > 2. When a broker crashes and recovers the managed ledger, the cursor> > > ledger is not writable anymore. Are you going to create a new cursor ledger> > > and copy all the entries from the old cursor ledger to the new one?> > >> > > It would be good if you can clarify these two questions.> > >> > > - Sijie> > >> > > On Sun, Jan 17, 2021 at 9:48 PM linlin <li...@apache.org> wrote:> > >> > > > Hi, community:> > > > Recently we encountered some problems when using individual> > > > acknowledgments, such as:> > > > when the amount of acknowledgment is large, entry writing fails; a large> > > > amount of cache causes OOM, etc.> > > > So I drafted a PIP in `> > > >> > > >> > > https://docs.google.com/document/d/1uQtyb8t6X04v2vrSrdGWLFkuCkBcGYZbqK8XsVJ4qkU/edit?usp=sharing`> > > < https://docs.google.com/document/d/1uQtyb8t6X04v2vrSrdGWLFkuCkBcGYZbqK8XsVJ4qkU/edit?usp=sharing>> > > > <> > > https://docs.google.com/document/d/1uQtyb8t6X04v2vrSrdGWLFkuCkBcGYZbqK8XsVJ4qkU/edit?usp=sharing> > > >> > > > ,> > > > any voice is welcomed.> > > >> > >> >
