Not sure atm why you see a performance degradation. Would need to dig into the details.
However, did you consider to upgrade to 2.0 instead or 0.11? Also note that we added a new operator `suppress()` in upcoming 2.1 release, that allows you to do rate control without caching: https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables Hope this helps. -Matthias On 11/6/18 9:49 AM, Jonathan Gordon wrote: > I have a Kafka Streams app that I'm trying to upgrade from 0.10.2.1 to > 0.11.0.3 but when I do I notice that CPU goes way up and consumption goes > down. A thread profile indicates that the most expensive task is during our > aggregation, fetching from the cache. > > Thread profile with caching: > https://imgur.com/l5VEsC2 > > If I disable the cache both performance and consumption are good but we are > producing every single aggregation modification, which is not what we want. > > Thread profile without caching: > https://imgur.com/a/JK3nkou > > I read this thread, which seems relevant e > > https://lists.apache.org/thread.html/2b44e74eaec7172b107bcff96861cf8b4837f55a44714f69d033cc2e@%3Cusers.kafka.apache.org%3E > > Notably: "Note, that caching was _not_ introduced to reduce the writes to > RocksDB, but to reduce the write the the changelog topic and to reduce the > number of records send downstream." > > So how can we reduce the number of records sent downstream while > maintaining the same performance characteristics that we have with caching > turned off? Or put another way, how can I upgrade my app without taking a > hit in performance or behavior? > > Thanks! >
signature.asc
Description: OpenPGP digital signature
