My post if not directly referring to KS.

The new free book by Orielly has very good explanation about Kafka Topic
counts.

You can download it from below link ( See Chapter 4)

http://shop.oreilly.com/product/0636920049463.do

In short quoting from there

>>> These problems are likely substantially tied to the fundamental
implementation decisions that underpin how Kafka works. In particular, as
the number of topics increases, the amount of random I/O that is imposed on
the broker increases dramatically because each topic partition write is
essentially a separate file append operation. This becomes more and more
problematic as the number of partitions increases and is very difficult to
fix without Kafka taking over the scheduling of I/O. Just above the current
limits on number of partitions, there are likely other limits waiting, some
fairly serious. In particular, the number of file descriptors that a single
process can open is typically limited.

Kafka can store offsets in both Kafka and in Zookeeper. In newer releases
it is recommended store consumer offsets inside Kafka.

>From high producer throughput see the properties of batch.size and linger.ms
On Consumer side fetch.min.bytes and max.partition.fetch.bytes



On 23 July 2016 at 16:40, Alex Glikson <glik...@il.ibm.com> wrote:

> Hi all,
>
> I wonder whether limitations mentioned in [1] regarding Kafka scalability
> in number of topics are still valid. For example, did the recent changes
> in the design around usage of ZooKeeper versus internal membership
> protocol affected the scalability - one way or the other?
> Also, it seems that the introduction of Kafka Streams may increase the
> number of topics (including those created by the app, and those created
> internally by KS), and maybe also change a bit the usage pattern in
> general (if large portion of the load is generated by KS). Are there any
> performance numbers (e.g., for table-based APIs), known bottlenecks
> (specific to KS), tuning recommendations?
>
> Thanks,
> Alex
>
>
> [1] https://www.quora.com/How-many-topics-can-be-created-in-Apache-Kafka
>
>

Reply via email to