Hi Chris, Since you are using read_committed mode, the txn marker from the `endOffsets()` should indeed be skipped, no matter for committed of aborted txns. For example if the log looks like this:
offsets: 0, 1, 2, 3, t_c(4) // or t_a(4) which means "abort marker" then the endOffset should return "5". That's why I'm unclear why you're seeing this issue. Guozhang On Tue, Mar 22, 2022 at 6:19 AM Chris Jansen <chris.jan...@permutive.com> wrote: > Thanks Luke, > > If the transaction marker should be hidden, does it follow that aborted > transaction at the end of the log should also be hidden for clients that > are in read committed mode? > > Happy to do a KIP/PR. > > Thanks again, > > Chris > > On Tue, Mar 22, 2022 at 10:21 AM Luke Chen <show...@gmail.com> wrote: > > > Hi Chris, > > > > Yes, the transaction marker should be hidden to clients. > > There is similar issues reported: > > https://issues.apache.org/jira/browse/KAFKA-10683 > > Welcome to submit KIP/PR to improve it. > > > > Thank you. > > Luke > > > > > > On Tue, Mar 22, 2022 at 5:16 PM Chris Jansen <chris.jan...@permutive.com > > > > wrote: > > > > > Hi Guozhang, > > > > > > Sorry, I should have been more clear. By "an unreadable end of the > log", > > I > > > mean the `endOffsets` method returns an offset for a record that is > never > > > surfaced to the caller of `poll`. > > > > > > I've done some more digging and I think I understand why that is now. > The > > > API `endOffsets` calls tells the client, at a low level, what offset it > > can > > > read up to in the log. This may include a transaction marker or an > > aborted > > > transaction that gets delivered, but the client skips over when > returning > > > records via the `poll` method. I understand why the client must be told > > > where to read up to, as the decision to skip chunks of the log must be > > made > > > by the client. > > > > > > What I was looking for when calling `endOffsets` was the last offset > that > > > would be surfaced via the `poll` method rather than where the low level > > > client should read up to. If this is at all possible, it seems like > this > > > might require a broker change. > > > > > > Thanks > > > > > > Chris > > > > > > On Mon, Mar 21, 2022 at 5:38 PM Guozhang Wang <wangg...@gmail.com> > > wrote: > > > > > > > Hi Chris, > > > > > > > > What do you mean by "an unreadable end of the log"? Could you > > elaborate? > > > > > > > > The version you used is recent enough, and the configs seems okay. > So I > > > > think there are some issues elsewhere. > > > > > > > > On Mon, Mar 21, 2022 at 5:51 AM Chris Jansen < > > chris.jan...@permutive.com > > > > > > > > wrote: > > > > > > > > > So an update on this, it seems that the consumer reports an > > unreadable > > > > end > > > > > of the log if the transaction has been aborted. Is this the > intended > > > > > behaviour? > > > > > > > > > > Thanks again, > > > > > > > > > > Chris > > > > > > > > > > On Mon, Mar 21, 2022 at 12:25 PM Chris Jansen < > > > > chris.jan...@permutive.com> > > > > > wrote: > > > > > > > > > > > Hi Guozhang, > > > > > > > > > > > > Thanks for getting back to me. I'm using Confluent's distribution > > > > version > > > > > > 6.2.2, so Kafka 2.8.1. Could there be some consumer or broker > > > > > configuration > > > > > > I'm missing that would change this behaviour? > > > > > > > > > > > > Just to confirm, here are my consumer settings: > > > > > > > > > > > > auto.offset.reset -> earliest, isolation.level -> read_committed, > > > > > > group.instance.id -> <redacted>, group.id -> > > > > > > ephemeral-6c5f8338-3d94-4296-b64a-d3a5a331117e, bootstrap.servers > > -> > > > > > > PLAINTEXT://localhost:58300, enable.auto.commit -> false > > > > > > > > > > > > Thanks again, > > > > > > > > > > > > > > > > > > Chris > > > > > > > > > > > > > > > > > > On Sun, Mar 20, 2022 at 12:46 AM Guozhang Wang < > wangg...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > >> Hi Chris, > > > > > >> > > > > > >> The broker does take the isolation level in the ListOffset API > > > (which > > > > is > > > > > >> used for the endoffset call) in to consideration: > > > > > >> > > > > > >> val lastFetchableOffset = isolationLevel match { > > > > > >> case Some(IsolationLevel.READ_COMMITTED) => > > > > localLog.lastStableOffset > > > > > >> case Some(IsolationLevel.READ_UNCOMMITTED) => > > > localLog.highWatermark > > > > > >> case None => localLog.logEndOffset > > > > > >> } > > > > > >> > > > > > >> > > > > > >> If it is READ_COMMITTED, it would only return the last stable > > > offset. > > > > > >> > > > > > >> But this requires the broker version to be newer enough to > > actually > > > > > >> know the new format of the request, is your broker on the newer > > > > > >> version? Other than that, I cannot think of a reason explaining > > what > > > > > >> you saw. > > > > > >> > > > > > >> > > > > > >> > > > > > >> On Wed, Mar 16, 2022 at 5:12 AM Chris Jansen < > > > > > chris.jan...@permutive.com> > > > > > >> wrote: > > > > > >> > > > > > >> > Hello all, > > > > > >> > > > > > > >> > I have the need to query end offsets for a particular topic > > using > > > a > > > > > >> > consumer that has been configured with the "READ_COMMITTED" > > > > isolation > > > > > >> > level. The response I get via the Java client > > > > > >> > includes offsets at the end of the log that have not yet been > > > > > committed > > > > > >> and > > > > > >> > therefore can't be consumed. > > > > > >> > > > > > > >> > After digging around in the Java client code, it looks like > the > > > > > >> isolation > > > > > >> > level is being transmitted in the API request to the broker. > So > > my > > > > > >> question > > > > > >> > is does the broker use this information when crafting a > response > > > > and, > > > > > >> if it > > > > > >> > is taken into account, why does it return log end offsets for > > > > > >> transactions > > > > > >> > that have not yet been committed? > > > > > >> > > > > > > >> > For my own future reference I was struggling to find any > details > > > on > > > > > the > > > > > >> > broker API anywhere, if someone could point me in the right > > > > direction, > > > > > >> I'd > > > > > >> > really appreciate it. > > > > > >> > > > > > > >> > Thanks, > > > > > >> > > > > > > >> > > > > > > >> > Chris Jansen > > > > > >> > > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> -- Guozhang > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > -- > > > > -- Guozhang > > > > > > > > > > -- -- Guozhang
