Hi Guozhang,
Sorry, I should have been more clear. By "an unreadable end of the log", I
mean the `endOffsets` method returns an offset for a record that is never
surfaced to the caller of `poll`.
I've done some more digging and I think I understand why that is now. The
API `endOffsets` calls tells the client, at a low level, what offset it can
read up to in the log. This may include a transaction marker or an aborted
transaction that gets delivered, but the client skips over when returning
records via the `poll` method. I understand why the client must be told
where to read up to, as the decision to skip chunks of the log must be made
by the client.
What I was looking for when calling `endOffsets` was the last offset that
would be surfaced via the `poll` method rather than where the low level
client should read up to. If this is at all possible, it seems like this
might require a broker change.
Thanks
Chris
On Mon, Mar 21, 2022 at 5:38 PM Guozhang Wang <wangg...@gmail.com> wrote:
> Hi Chris,
>
> What do you mean by "an unreadable end of the log"? Could you elaborate?
>
> The version you used is recent enough, and the configs seems okay. So I
> think there are some issues elsewhere.
>
> On Mon, Mar 21, 2022 at 5:51 AM Chris Jansen <chris.jan...@permutive.com>
> wrote:
>
> > So an update on this, it seems that the consumer reports an unreadable
> end
> > of the log if the transaction has been aborted. Is this the intended
> > behaviour?
> >
> > Thanks again,
> >
> > Chris
> >
> > On Mon, Mar 21, 2022 at 12:25 PM Chris Jansen <
> chris.jan...@permutive.com>
> > wrote:
> >
> > > Hi Guozhang,
> > >
> > > Thanks for getting back to me. I'm using Confluent's distribution
> version
> > > 6.2.2, so Kafka 2.8.1. Could there be some consumer or broker
> > configuration
> > > I'm missing that would change this behaviour?
> > >
> > > Just to confirm, here are my consumer settings:
> > >
> > > auto.offset.reset -> earliest, isolation.level -> read_committed,
> > > group.instance.id -> <redacted>, group.id ->
> > > ephemeral-6c5f8338-3d94-4296-b64a-d3a5a331117e, bootstrap.servers ->
> > > PLAINTEXT://localhost:58300, enable.auto.commit -> false
> > >
> > > Thanks again,
> > >
> > >
> > > Chris
> > >
> > >
> > > On Sun, Mar 20, 2022 at 12:46 AM Guozhang Wang <wangg...@gmail.com>
> > wrote:
> > >
> > >> Hi Chris,
> > >>
> > >> The broker does take the isolation level in the ListOffset API (which
> is
> > >> used for the endoffset call) in to consideration:
> > >>
> > >> val lastFetchableOffset = isolationLevel match {
> > >> case Some(IsolationLevel.READ_COMMITTED) =>
> localLog.lastStableOffset
> > >> case Some(IsolationLevel.READ_UNCOMMITTED) => localLog.highWatermark
> > >> case None => localLog.logEndOffset
> > >> }
> > >>
> > >>
> > >> If it is READ_COMMITTED, it would only return the last stable offset.
> > >>
> > >> But this requires the broker version to be newer enough to actually
> > >> know the new format of the request, is your broker on the newer
> > >> version? Other than that, I cannot think of a reason explaining what
> > >> you saw.
> > >>
> > >>
> > >>
> > >> On Wed, Mar 16, 2022 at 5:12 AM Chris Jansen <
> > chris.jan...@permutive.com>
> > >> wrote:
> > >>
> > >> > Hello all,
> > >> >
> > >> > I have the need to query end offsets for a particular topic using a
> > >> > consumer that has been configured with the "READ_COMMITTED"
> isolation
> > >> > level. The response I get via the Java client
> > >> > includes offsets at the end of the log that have not yet been
> > committed
> > >> and
> > >> > therefore can't be consumed.
> > >> >
> > >> > After digging around in the Java client code, it looks like the
> > >> isolation
> > >> > level is being transmitted in the API request to the broker. So my
> > >> question
> > >> > is does the broker use this information when crafting a response
> and,
> > >> if it
> > >> > is taken into account, why does it return log end offsets for
> > >> transactions
> > >> > that have not yet been committed?
> > >> >
> > >> > For my own future reference I was struggling to find any details on
> > the
> > >> > broker API anywhere, if someone could point me in the right
> direction,
> > >> I'd
> > >> > really appreciate it.
> > >> >
> > >> > Thanks,
> > >> >
> > >> >
> > >> > Chris Jansen
> > >> >
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> >
>
>
> --
> -- Guozhang
>
