Hello Jack, This idea sounds really interesting. As you are citing in the doc, one tricky part from my point of view is about deleting fragments in order to release space (and be compliant with legal stuff...) We will also have to add the fragment as a top level concept of BooKeeper in order to allow maintenance of the ledgers.
I wonder if it is better to keep this here in BookKeeper, or is it better to keep this complexity at a higher level, like in Pulsar and Pravega, that, to make it simple are basically abstractions over BK to implement such unbounded streams of data. As in Pulsar and in Pravega, when you deal with an unbounded amount of data you will have to deal with Offloading to some external (cheaper) storage, so that's another thing that we should move into BooKeeper. Thanks for sharing Enrico Il giorno sab 11 dic 2021 alle ore 10:53 Jack Vanlightly < jack.vanligh...@gmail.com> ha scritto: > Hi all, > > As a mostly thought exercise, I have put together the protocol changes > required to make ledgers unbounded. I have written up the changes here: > > https://jack-vanlightly.com/blog/2021/12/9/tweaking-the-bookkeeper-protocol-unbounded-ledgers > > The benefits of such an approach is a simplified stream protocol that > doesn't require chaining ledgers together. An individual ledger is already > segmented and we don't lose the benefits of a segmented log if we make one > stream a single ledger. > > I open it up for discussion here so people can comment, find issues or > indicate they like the idea. It is not a pressing need so I haven't > contemplated making a BP out of it yet. > > Jack >
