Hi Jack, Thanks for your comments. I have added a new section on Log Retention which describes the behaviour of the SPSO as the LSO advances. That makes total sense and was an omission from the KIP.
I have added the other ideas as potential future work. I do like the idea of having the SPSO influence the advancements of the LSO for topics which are primarily being using with share groups. I have published an updated version of the KIP. Thanks, Andrew > On 4 Oct 2023, at 10:09, Jack Vanlightly <vanligh...@apache.org> wrote: > > I would like to see more explicit discussion of topic retention and share > groups. There are a few options here from simple to more sophisticated. There > are also topic-level and share-group level options. > > The simple thing would be to ensure that the SPSO of each share group is > bounded by the Log Start Offset (LSO) of each partition which itself is > managed by the retention policy. This is a topic-level control which applies > to all share-groups. I would say that this shared retention is the largest > drawback of modeling queues on shared logs and this is worth noting. > > More sophisticated approaches can be to allow the LSO to advance not (only) > by retention policy but by the advancement of the lowest SPSO. This can keep > the amount of data lower by garbage collecting messages that have been > acknowledged by all share groups. Some people may like that behaviour on > those topics where share groups are the only consumption model and no replay > is needed. > > There are per-share-group possibilities such as share-group TTLs where > messages can be archived on a per share group basis. > > Thanks > Jack
