Here's a summary that can go under the section "What’s new in Kafka broker,
producer, and consumer" as an "improvement".  Feel free to rephrase as you
see fit.

When a partition is paused by the user in the consumer the partition is
> considered "unfetchable".  When the consumer has already fetched data for a
> partition, and then the partition is paused, then in the next consumer poll
> all data from "unfetchable" partitions will be discarded.  In use cases
> where pausing and resuming partitions is common during regular operation of
> the consumer this can result in discarding pre-fetched data when it's not
> necessary.  Once the partition is resumed then new fetch requests will be
> generated and sent to the broker to get the same partition data again.
> Depending on the frequency of pausing and resuming of partitions this can
> impact a number of different aspects of consumer polling including:
> broker/consumer throughput, number of consumer fetch requests, and
> NIO-related GC concerns for regularly dereferenced byte buffers of
> partition data.  This issue is now resolved by retaining completed fetch
> data for partitions that are paused so that it may be returned in a future
> consumer poll once the partition is resumed by the user.
>


See [KAFKA-7548](https://issues.apache.org/jira/browse/KAFKA-7548) for more
> details.


Regards,
Sean

On Mon, Nov 18, 2019 at 11:45 AM Ismael Juma <ism...@juma.me.uk> wrote:

> That makes sense to me.
>
> Ismael
>
> On Mon, Nov 18, 2019 at 8:40 AM Sean Glover <sean.glo...@lightbend.com>
> wrote:
>
> > Hi Manikumar,
> >
> > I'm putting together an akka.io blog post regarding [KAFKA-7548] -
> > KafkaConsumer should not throw away already fetched data for paused
> > partitions.  Since it doesn't change any user-facing APIs it has no KIP,
> > but it has a significant impact on consumer use cases that frequently
> pause
> > and resume partitions, such as in Alpakka Kafka.  I can provide a small
> > summary for you to include in your blog post if you think it's
> appropriate.
> >
> > Regards,
> > Sean
> >
> > On Mon, Nov 18, 2019 at 11:25 AM Manikumar <manikumar.re...@gmail.com>
> > wrote:
> >
> > > Thanks Chris. will update the blog content.
> > >
> > > On Fri, Nov 15, 2019 at 12:34 AM Chris Egerton <chr...@confluent.io>
> > > wrote:
> > >
> > > > Hi Manikumar,
> > > >
> > > > It looks like the header for KIP-440 is accurate ("KIP-440: Extend
> > > Connect
> > > > Converter to support headers") but the content appears to correspond
> to
> > > > KIP-481 ("SerDe Improvements for Connect Decimal type in JSON")
> > instead.
> > > > Could we double-check and make sure that the summary for KIP-440
> > matches
> > > > what was contributed for it (and it nothing was, alter the summary to
> > > more
> > > > closely reflect what KIP-440 accomplished)?
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Thu, Nov 14, 2019 at 10:41 AM Manikumar <
> manikumar.re...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I've prepared a preliminary blog post about the upcoming Apache
> Kafka
> > > > 2.4.0
> > > > > release.
> > > > > Please take a look and let me know if you want to add/modify
> details.
> > > > > Thanks to all who contributed to this blog post.
> > > > >
> > > > >
> > > >
> > >
> >
> https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > >
> > >
> >

Reply via email to