We are currently using Kafka 0.8.1.1 with log compaction in order to
provide streams of messages to our clients.
As well as constantly consuming the stream, one of our use cases is to
provide a snapshot, meaning the user will receive a copy of every message
at least once.
Each one of these messag
feature
> > would allow you to control how long the full log is retained and when
> > compaction can kick in. This would give a configurable SLA for the reader
> > process to catch up.
> > 2. Make the log end offset available more easily in the consumer.
> >
> > -Ja
> > > > > the timeout. This is admittedly kind of a hack.
> > > > >
> > > > > You are also correct and perceptive to notice that if you check the
> > end
> > > > of
> > > > > the log then begin consuming and read up to that point c
is just the end of the committed messages in the log
> (the last thing the consumer has access to). It isn't the same as the
> cleaner point but is always later than it so it would work just as well.
>
> -Jay
>
> On Thu, Feb 19, 2015 at 8:54 AM, Will Funnell
> wrot
ance I misunderstood your point.
> >
> > On 19 February 2015 at 18:02, Jay Kreps wrote:
> >
> > > The log end offset is just the end of the committed messages in the log
> > > (the last thing the consumer has access to). It isn't the same as the
> > >
elated with the consumer's offset).
>
>
> On Thu, Feb 19, 2015 at 08:58:10PM +, Will Funnell wrote:
> > So at what point does the log end offset change? When you commit?
> >
> > On 19 February 2015 at 18:47, Joel Koshy wrote:
> >
> > > > I
Hi,
Any update on the above patch?
Hoping you might be able to review it soon.
Thanks.
On 23 February 2015 at 21:21, Will Funnell wrote:
> Hey guys,
>
> I created a patch based on your feedback.
>
> Let me know what you think.
>
> https://issues.apache.org/jira/browse
I've created a patch to expose the high end watermark, having this exact
requirement.
Still waiting for it to be accepted, but are using this in production at
the moment and it works quite nicely:
https://issues.apache.org/jira/browse/KAFKA-1977
On Sat, 9 May 2015 at 18:43 Charlie Knudsen
wrot
