Hello Jan, Thanks for your question.
It is my understanding that the producer batching strongly informs the on-disk layout, because appending to the log is done on a per-batch level, not a per-record level. This is done for any number of reasons, one of them is being able to append data without uncompressing and recompressing it on the broker side. There are situations when Kafka must fall back to recompressing batches before append, but I don't think they're relevant to this discussion. The consumer batching is also somewhat informed by the on-disk layout. Data fetched by the consumers should always be one or more batches, never a fraction of a batch. This is done to avoid uncompressing and recompressing the data similar to the Produce side, but also to enable sendfile and block cache optimizations. So while consumer batching may be configurable per-client, this is done by including multiple batches together. Consumer fetches should have a lower bound, which is the producer batch size set at append time. I would encourage you to test this and verify the behavior for yourself. I am somewhat new in this area of the codebase so I could also be missing something. Thanks, Greg On Tue, Jan 28, 2025 at 12:15 AM Jan Wypych <jan.wyp...@billennium.com.invalid> wrote: > Hello Greg, > > Thanks for mentioning batch sizes. > Could you please elaborate on the impact of producer batch size? > My understanding is that producer just writes messages (in batches of > configurable size) to Kafka, but these batches does not impact data stored > on disks. The consumer reading messages should be totally decupled from how > producer is writing them, but maybe I am missing something. > For the batch reading we are aware of that and will set fetch.min.bytes > and fetch.max.bytes to force Kafka not to prefetch more than one message. > Our understanding (maybe wrong) is that Kafka have many IO threads (I am > not sure if there are/can be separate reading and writing pools) and each > one can process IO assignments with different parameters (you can have one > thread returning data after 10 bytes read and second one waiting for 10MB). > By this understanding (and, as said, we may be wrong here) batch sizes can > be an issue, but they are configurable on the individual producer and > consumer level. > > Best regards, > Jan > > -----Original Message----- > From: Greg Harris <greg.har...@aiven.io.INVALID> > Sent: Friday, January 24, 2025 9:02 PM > To: Users <users@kafka.apache.org> > Subject: Re: Random access to kafka messages > > Hello Jan, > > I also have not heard of a use case like this for Kafka. One statistic > that I think you might need to manage is batch size, and its effect on > compression and read amplification. > > Larger batches on the producer side can make your producers more > performant and compression more effective. But large batches will also > increase the amount of data delivered to consumers that is then discarded > to read a single message. This additional data transfer wastes disk > bandwidth on the brokers and network bandwidth on the broker and consuming > application. > > So while a lot of existing tuning advice and optimizations in Kafka work > with larger batches, you will need to spend some time profiling and making > batch size tradeoffs. > > Hope this helps, > Greg > > On Fri, Jan 24, 2025, 3:05 AM Ömer Şiar Baysal <osiarbay...@gmail.com> > wrote: > > > Hi, > > > > The data you gathered shows promising results, one thing the consider > > is testing how the Page Cache that Kafka utilizes affect the response > > times, which greatly improves response time for the fetch requests > > that are already in the cache since it is stored in memory and may > > give an impression that all the fetch requests performance would be > > the same, it is in fact would be different for non-cached data. > > > > Good luck and let me know if you need more information about page cache. > > Omer Siar Baysal > > > > > > On Fri, Jan 24, 2025, 11:48 Jan Wypych > > <jan.wyp...@billennium.com.invalid> > > wrote: > > > > > Hello, > > > > > > We are currently designing a system that will ingest some XML > > > messages > > and > > > then it will store them into some kind of long-term storage (years). > > > The access pattern to data shows that new messages (1-2 weeks old) > > > will be frequent, older data will be accessed rarely. > > > We currently chose Kafka as an ingest part, some kind of S3 for cold > > > long-term, but we are still thinking how we should approach hot > > > storage > > > (1-2 weeks). We established that our S3 for hot data is too slow. > > > We have a few options for this hot part of a storage, but one of > > > them is Kafka (it will greatly simplify the whole system and Kafka > > > reliability if extremely high). > > > Each Kafka message can be accessed using the offset/partition pair > > > (we need some metadata from messages anyway, so getting this pair is > > > free for us). Kafka stores its data in segments, each of them has > > > its own index, > > so > > > we do not do a full scan of a topic. Consumer configs can be > > > tweaked, so > > we > > > do not prefetch more than one message, do not commit offsets for > > > consumer group etc. Our initial tests show very promising results > > > with high throughput and low latency (3 brokers, 300GB in 50 > > > partitions, 10k messages/s, average latency under 3ms). Everything > > > we have seen so far tells us that it should work. > > > However, this goes against the common understanding of Kafka usage, > > > as a streaming solution. We searched the internet and could not find > > > such use case deployed. > > > On the other hand, every time we found someone discouraging such use > > case, > > > there was no technical explanation behind it. Just a vague "Kafka > > > was not crated for this, better to use X". > > > So, my question to you is: > > > Does anybody see any technical reason why our approach (fetch > > > messages by offset/partition in random order) should not work? Is > > > there some > > limitation > > > we do not see, that could bite us in production (10-20 TB of data in > > > topics, more than 3 brokers obviously)? > > > > > > Best regards, > > > Jan Wypych > > > > > >
