Here's a summary that can go under the section "What’s new in Kafka broker, producer, and consumer" as an "improvement". Feel free to rephrase as you see fit.
When a partition is paused by the user in the consumer the partition is > considered "unfetchable". When the consumer has already fetched data for a > partition, and then the partition is paused, then in the next consumer poll > all data from "unfetchable" partitions will be discarded. In use cases > where pausing and resuming partitions is common during regular operation of > the consumer this can result in discarding pre-fetched data when it's not > necessary. Once the partition is resumed then new fetch requests will be > generated and sent to the broker to get the same partition data again. > Depending on the frequency of pausing and resuming of partitions this can > impact a number of different aspects of consumer polling including: > broker/consumer throughput, number of consumer fetch requests, and > NIO-related GC concerns for regularly dereferenced byte buffers of > partition data. This issue is now resolved by retaining completed fetch > data for partitions that are paused so that it may be returned in a future > consumer poll once the partition is resumed by the user. > See [KAFKA-7548](https://issues.apache.org/jira/browse/KAFKA-7548) for more > details. Regards, Sean On Mon, Nov 18, 2019 at 11:45 AM Ismael Juma <ism...@juma.me.uk> wrote: > That makes sense to me. > > Ismael > > On Mon, Nov 18, 2019 at 8:40 AM Sean Glover <sean.glo...@lightbend.com> > wrote: > > > Hi Manikumar, > > > > I'm putting together an akka.io blog post regarding [KAFKA-7548] - > > KafkaConsumer should not throw away already fetched data for paused > > partitions. Since it doesn't change any user-facing APIs it has no KIP, > > but it has a significant impact on consumer use cases that frequently > pause > > and resume partitions, such as in Alpakka Kafka. I can provide a small > > summary for you to include in your blog post if you think it's > appropriate. > > > > Regards, > > Sean > > > > On Mon, Nov 18, 2019 at 11:25 AM Manikumar <manikumar.re...@gmail.com> > > wrote: > > > > > Thanks Chris. will update the blog content. > > > > > > On Fri, Nov 15, 2019 at 12:34 AM Chris Egerton <chr...@confluent.io> > > > wrote: > > > > > > > Hi Manikumar, > > > > > > > > It looks like the header for KIP-440 is accurate ("KIP-440: Extend > > > Connect > > > > Converter to support headers") but the content appears to correspond > to > > > > KIP-481 ("SerDe Improvements for Connect Decimal type in JSON") > > instead. > > > > Could we double-check and make sure that the summary for KIP-440 > > matches > > > > what was contributed for it (and it nothing was, alter the summary to > > > more > > > > closely reflect what KIP-440 accomplished)? > > > > > > > > Cheers, > > > > > > > > Chris > > > > > > > > On Thu, Nov 14, 2019 at 10:41 AM Manikumar < > manikumar.re...@gmail.com> > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I've prepared a preliminary blog post about the upcoming Apache > Kafka > > > > 2.4.0 > > > > > release. > > > > > Please take a look and let me know if you want to add/modify > details. > > > > > Thanks to all who contributed to this blog post. > > > > > > > > > > > > > > > > > > > > https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1 > > > > > > > > > > Thanks, > > > > > Manikumar > > > > > > > > > > > > > >
