Kafka didn't write to the pagecache. Actually no-one can only the underlying OS.
-- Raph On 06/03/2025 14:03, Sreyan Chakravarty wrote: > Hi Guang, > > Thanks so much for the quick response. > > However I have one question, a lot of material on the web says Kafka > writes to the pagecache. > > Then does that mean java.nio.GatheringByteChannel.write(buffer) actually > writes to the pagecache ? > > I only ask because this is not mentioned in the docs: > https://docs.oracle.com/javase/8/docs/api/java/nio/channels/GatheringByteChannel.html#write-java.nio.ByteBuffer:A > - > https://docs.oracle.com/javase/8/docs/api/java/nio/channels/FileChannel.html#write-java.nio.ByteBuffer > - > > Hence not guaranteed, am I correct? > > So if Kafka depends on just writing to the pagecache for performance > shouldn't it use some sort of API or mechanism to do just that ? > > Or are the docs on the web wrong about how Kafka operates ? > https://kafka.apache.org/documentation/#design_filesystem > Just confused about what is really happening. > > On Thu, Mar 6, 2025 at 5:31 AM Zhao, Guang > [<guang.z...@netapp.com.invalid>](mailto:guang.z...@netapp.com.invalid) > wrote: > >> (Sorry, missing links) -- the appending operation in FileRecords: >> https://github.com/apache/kafka/blob/56fc8e9d2a04172fd014b89428f1c97da93f4894/clients/src/main/java/org/apache/kafka/common/record/FileRecords.java#L184 >> Cheers, >> Guang >> -- >> Guang Zhao, NetApp >> gu...@netapp.com[<mailto:gu...@netapp.com>](mailto:gu...@netapp.com) >> From: Zhao, Guang >> [<guang.z...@netapp.com>](mailto:guang.z...@netapp.com) >> Date: Thursday, 6 March 2025 at 10:57 am >> To: >> users@kafka.apache.org >> >> [<users@kafka.apache.org>](mailto:users@kafka.apache.org) >> Subject: Re: How does Kafka write to the pagecache/pagetable ? >> Hi Sreyan, >> >> Thanks for the question. The page cache is managed by the OS, so I think >> it is transparent to Kafka writes, which are to files. Just at a higher >> abstraction level. >> >> Some pointers in code: >> >> * LogSegment: >> https://github.com/apache/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java >> * FileRecrods: >> https://github.com/apache/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java >> You can see appending records actually write to a file through Java’s >> FileChannel. >> >> Hope this helps! >> >> Cheers, >> Guang >> -- >> Guang Zhao, NetApp >> gu...@netapp.com[<mailto:gu...@netapp.com>](mailto:gu...@netapp.com) >> From: Sreyan Chakravarty >> [<sreya...@gmail.com>](mailto:sreya...@gmail.com) >> Date: Thursday, 6 March 2025 at 2:01 am >> To: >> users@kafka.apache.org >> >> [<users@kafka.apache.org>](mailto:users@kafka.apache.org) >> Subject: How does Kafka write to the pagecache/pagetable ? >> [You don't often get email from >> sreya...@gmail.com >> . Learn why this is >> important at >> https://aka.ms/LearnAboutSenderIdentification >> ] >> >> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments >> >> Hi devs, >> >> I am curious in trying to understand the internals of how Kafka actually >> writes to disk. I am wondering as to how it writes to the >> pagecache/pagetable ? >> >> From what I understand, Kafka never writes to the files directly instead it >> writes to the pagecache and lets the OS do the flush to disk. Please >> correct me if I am wrong. >> >> Where in the source code can I find code that does this ? >> >> Any help would be greatly appreciated. >> >> -- >> Regards, >> Sreyan Chakravarty > > -- > Regards, > Sreyan Chakravarty
