Re: Kafka (Streams) scalability

2016-07-26 Thread Guozhang Wang
As for KS, you can think of each single-threaded KS instance as a normal producer plus a normal consumer client, so you can do the math for capacity planning purposes assuming you understand your application traffic, and your state store write amplifications (if you use the default persistent key-v

Re: Kafka (Streams) scalability

2016-07-23 Thread Jagat Singh
My post if not directly referring to KS. The new free book by Orielly has very good explanation about Kafka Topic counts. You can download it from below link ( See Chapter 4) http://shop.oreilly.com/product/0636920049463.do In short quoting from there >>> These problems are likely substantiall

Kafka (Streams) scalability

2016-07-22 Thread Alex Glikson
Hi all, I wonder whether limitations mentioned in [1] regarding Kafka scalability in number of topics are still valid. For example, did the recent changes in the design around usage of ZooKeeper versus internal membership protocol affected the scalability - one way or the other? Also, it seems