Hi Sean, Thanks for the summary. Will include in the blogpost.
On Mon, Nov 18, 2019 at 11:04 PM Sean Glover <sean.glo...@lightbend.com> wrote: > Here's a summary that can go under the section "What’s new in Kafka broker, > producer, and consumer" as an "improvement". Feel free to rephrase as you > see fit. > > When a partition is paused by the user in the consumer the partition is > > considered "unfetchable". When the consumer has already fetched data > for a > > partition, and then the partition is paused, then in the next consumer > poll > > all data from "unfetchable" partitions will be discarded. In use cases > > where pausing and resuming partitions is common during regular operation > of > > the consumer this can result in discarding pre-fetched data when it's not > > necessary. Once the partition is resumed then new fetch requests will be > > generated and sent to the broker to get the same partition data again. > > Depending on the frequency of pausing and resuming of partitions this can > > impact a number of different aspects of consumer polling including: > > broker/consumer throughput, number of consumer fetch requests, and > > NIO-related GC concerns for regularly dereferenced byte buffers of > > partition data. This issue is now resolved by retaining completed fetch > > data for partitions that are paused so that it may be returned in a > future > > consumer poll once the partition is resumed by the user. > > > > > See [KAFKA-7548](https://issues.apache.org/jira/browse/KAFKA-7548) for > more > > details. > > > Regards, > Sean > > On Mon, Nov 18, 2019 at 11:45 AM Ismael Juma <ism...@juma.me.uk> wrote: > > > That makes sense to me. > > > > Ismael > > > > On Mon, Nov 18, 2019 at 8:40 AM Sean Glover <sean.glo...@lightbend.com> > > wrote: > > > > > Hi Manikumar, > > > > > > I'm putting together an akka.io blog post regarding [KAFKA-7548] - > > > KafkaConsumer should not throw away already fetched data for paused > > > partitions. Since it doesn't change any user-facing APIs it has no > KIP, > > > but it has a significant impact on consumer use cases that frequently > > pause > > > and resume partitions, such as in Alpakka Kafka. I can provide a small > > > summary for you to include in your blog post if you think it's > > appropriate. > > > > > > Regards, > > > Sean > > > > > > On Mon, Nov 18, 2019 at 11:25 AM Manikumar <manikumar.re...@gmail.com> > > > wrote: > > > > > > > Thanks Chris. will update the blog content. > > > > > > > > On Fri, Nov 15, 2019 at 12:34 AM Chris Egerton <chr...@confluent.io> > > > > wrote: > > > > > > > > > Hi Manikumar, > > > > > > > > > > It looks like the header for KIP-440 is accurate ("KIP-440: Extend > > > > Connect > > > > > Converter to support headers") but the content appears to > correspond > > to > > > > > KIP-481 ("SerDe Improvements for Connect Decimal type in JSON") > > > instead. > > > > > Could we double-check and make sure that the summary for KIP-440 > > > matches > > > > > what was contributed for it (and it nothing was, alter the summary > to > > > > more > > > > > closely reflect what KIP-440 accomplished)? > > > > > > > > > > Cheers, > > > > > > > > > > Chris > > > > > > > > > > On Thu, Nov 14, 2019 at 10:41 AM Manikumar < > > manikumar.re...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > I've prepared a preliminary blog post about the upcoming Apache > > Kafka > > > > > 2.4.0 > > > > > > release. > > > > > > Please take a look and let me know if you want to add/modify > > details. > > > > > > Thanks to all who contributed to this blog post. > > > > > > > > > > > > > > > > > > > > > > > > > > > https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1 > > > > > > > > > > > > Thanks, > > > > > > Manikumar > > > > > > > > > > > > > > > > > > >
