Hello Greg, Thanks for mentioning batch sizes. Could you please elaborate on the impact of producer batch size? My understanding is that producer just writes messages (in batches of configurable size) to Kafka, but these batches does not impact data stored on disks. The consumer reading messages should be totally decupled from how producer is writing them, but maybe I am missing something. For the batch reading we are aware of that and will set fetch.min.bytes and fetch.max.bytes to force Kafka not to prefetch more than one message. Our understanding (maybe wrong) is that Kafka have many IO threads (I am not sure if there are/can be separate reading and writing pools) and each one can process IO assignments with different parameters (you can have one thread returning data after 10 bytes read and second one waiting for 10MB). By this understanding (and, as said, we may be wrong here) batch sizes can be an issue, but they are configurable on the individual producer and consumer level.
Best regards, Jan -----Original Message----- From: Greg Harris <greg.har...@aiven.io.INVALID> Sent: Friday, January 24, 2025 9:02 PM To: Users <users@kafka.apache.org> Subject: Re: Random access to kafka messages Hello Jan, I also have not heard of a use case like this for Kafka. One statistic that I think you might need to manage is batch size, and its effect on compression and read amplification. Larger batches on the producer side can make your producers more performant and compression more effective. But large batches will also increase the amount of data delivered to consumers that is then discarded to read a single message. This additional data transfer wastes disk bandwidth on the brokers and network bandwidth on the broker and consuming application. So while a lot of existing tuning advice and optimizations in Kafka work with larger batches, you will need to spend some time profiling and making batch size tradeoffs. Hope this helps, Greg On Fri, Jan 24, 2025, 3:05 AM Ömer Şiar Baysal <osiarbay...@gmail.com> wrote: > Hi, > > The data you gathered shows promising results, one thing the consider > is testing how the Page Cache that Kafka utilizes affect the response > times, which greatly improves response time for the fetch requests > that are already in the cache since it is stored in memory and may > give an impression that all the fetch requests performance would be > the same, it is in fact would be different for non-cached data. > > Good luck and let me know if you need more information about page cache. > Omer Siar Baysal > > > On Fri, Jan 24, 2025, 11:48 Jan Wypych > <jan.wyp...@billennium.com.invalid> > wrote: > > > Hello, > > > > We are currently designing a system that will ingest some XML > > messages > and > > then it will store them into some kind of long-term storage (years). > > The access pattern to data shows that new messages (1-2 weeks old) > > will be frequent, older data will be accessed rarely. > > We currently chose Kafka as an ingest part, some kind of S3 for cold > > long-term, but we are still thinking how we should approach hot > > storage > > (1-2 weeks). We established that our S3 for hot data is too slow. > > We have a few options for this hot part of a storage, but one of > > them is Kafka (it will greatly simplify the whole system and Kafka > > reliability if extremely high). > > Each Kafka message can be accessed using the offset/partition pair > > (we need some metadata from messages anyway, so getting this pair is > > free for us). Kafka stores its data in segments, each of them has > > its own index, > so > > we do not do a full scan of a topic. Consumer configs can be > > tweaked, so > we > > do not prefetch more than one message, do not commit offsets for > > consumer group etc. Our initial tests show very promising results > > with high throughput and low latency (3 brokers, 300GB in 50 > > partitions, 10k messages/s, average latency under 3ms). Everything > > we have seen so far tells us that it should work. > > However, this goes against the common understanding of Kafka usage, > > as a streaming solution. We searched the internet and could not find > > such use case deployed. > > On the other hand, every time we found someone discouraging such use > case, > > there was no technical explanation behind it. Just a vague "Kafka > > was not crated for this, better to use X". > > So, my question to you is: > > Does anybody see any technical reason why our approach (fetch > > messages by offset/partition in random order) should not work? Is > > there some > limitation > > we do not see, that could bite us in production (10-20 TB of data in > > topics, more than 3 brokers obviously)? > > > > Best regards, > > Jan Wypych > > >
