Re: Offsets/Lags for global state stores not shown

2018-11-18 Thread Patrik Kleindl
Thanks for the reply. It would be interesting who else is using IQ with or without GlobalKTables and what problems and solutions they have come up with. Best regards Patrik > Am 18.11.2018 um 20:21 schrieb Matthias J. Sax : > > Because each instance needs to consume all data, it's limited by wha

Re: Offsets/Lags for global state stores not shown

2018-11-18 Thread Matthias J. Sax
Because each instance needs to consume all data, it's limited by what a single instance can consume -- a hard bound is the network. Note, network is shared, so don't take the maximum network speed into account. Also, it's not the number of unique messaged, but the number of updates that is importan

Re: Offsets/Lags for global state stores not shown

2018-11-07 Thread Patrik Kleindl
Thanks for the response. How "low" is the expected low throughput? We are are using GlobalKTables for IQ on several Topics, but with single-digit million unique messages and usually fewer changes per day. Just to verify, for this IQ setup (streams app which only builds a single table to be queried

Re: Offsets/Lags for global state stores not shown

2018-11-06 Thread Matthias J. Sax
The topics of global stores are not included by design. The "problem" is, that each instance needs to consume *all* topic-partitions from and thus topis, we thus they cannot be include into the consumer group that would assign each partition to exactly one instance. Hence, an additional consumer i