Hi, The data you gathered shows promising results, one thing the consider is testing how the Page Cache that Kafka utilizes affect the response times, which greatly improves response time for the fetch requests that are already in the cache since it is stored in memory and may give an impression that all the fetch requests performance would be the same, it is in fact would be different for non-cached data.
Good luck and let me know if you need more information about page cache. Omer Siar Baysal On Fri, Jan 24, 2025, 11:48 Jan Wypych <jan.wyp...@billennium.com.invalid> wrote: > Hello, > > We are currently designing a system that will ingest some XML messages and > then it will store them into some kind of long-term storage (years). The > access pattern to data shows that new messages (1-2 weeks old) will be > frequent, older data will be accessed rarely. > We currently chose Kafka as an ingest part, some kind of S3 for cold > long-term, but we are still thinking how we should approach hot storage > (1-2 weeks). We established that our S3 for hot data is too slow. > We have a few options for this hot part of a storage, but one of them is > Kafka (it will greatly simplify the whole system and Kafka reliability if > extremely high). > Each Kafka message can be accessed using the offset/partition pair (we > need some metadata from messages anyway, so getting this pair is free for > us). Kafka stores its data in segments, each of them has its own index, so > we do not do a full scan of a topic. Consumer configs can be tweaked, so we > do not prefetch more than one message, do not commit offsets for consumer > group etc. Our initial tests show very promising results with high > throughput and low latency (3 brokers, 300GB in 50 partitions, 10k > messages/s, average latency under 3ms). Everything we have seen so far > tells us that it should work. > However, this goes against the common understanding of Kafka usage, as a > streaming solution. We searched the internet and could not find such use > case deployed. > On the other hand, every time we found someone discouraging such use case, > there was no technical explanation behind it. Just a vague "Kafka was not > crated for this, better to use X". > So, my question to you is: > Does anybody see any technical reason why our approach (fetch messages by > offset/partition in random order) should not work? Is there some limitation > we do not see, that could bite us in production (10-20 TB of data in > topics, more than 3 brokers obviously)? > > Best regards, > Jan Wypych >
