Hi Hu,
On 10/2/19 8:54 PM, Xiyuan Hu wrote:
Hi All,
I'm doing smoke testing with my Kafka Streams app(V2.1.0). I noticed
that below behaviors:
1) Out throughput of changelog topic could go up to 70mb/s while the
in-traffic is around 10mb/s.
2) When traffic is bumpy, either due to producer/consumer throttle or
some other reasons I'm still debugging, changelog topic out throughput
could go up to 160mb/s while in-throughput is less than 1mb/s. And
from the monitor tool, no messages are actually put into this topic.
Could anyone help me to understand how changelog topic works? Given it
still has consumer lag and producer timeout issues(records
expired(5min) since batch created), is it just the cluster
bandwidth/capacity issue?
Thanks, appreciate all the help!!
In case you're using KeyValueStore (the simplest of stores), each update
of a (K, V) entry into the store produced by the processor of your
stream will result in one message written to the log topic of that
store. Multiple quick consecutive updates to the same entry (with same
key) are possibly collapsed into just one message holding the last
version of the entry. So it is possible that your input stream is having
keys that are not repeating very frequently in short time, therefore the
rate of messages/s is roughly the same in your input and log topics. The
difference in bytes/s therefore must be the result of input topic
message size vs. log topic message size. The log topic message is
basically the whole (K, V) entry in the KVStore.
Regards, Peter
