Re: Kafka Topic Volume and (possibly ACL) question

2019-02-19 Thread Evelyn Bayes
Hi, I would use ACLs or something similar. For instance, you might assign the records which are limited to a subset of clients to a specific topic with an associated ACL. I expect you’ll find having 8k extra topics very problematic in a range of ways, such as: * Replication issues; * Poor bat

Re: kafka scaling

2019-04-03 Thread Evelyn Bayes
Hi Ramz, A good rule of thumb has been no more than 4,000 partitions per broker and no more than 100,000 in a cluster. This includes all replicas and it's related more to Kafka internals then it is resource usage so I strongly advise not pushing these limits. Otherwise, the usual reasons for sc