Hi Guang, Thanks so much for the quick response.
However I have one question, a lot of material on the web says Kafka writes to the pagecache. Then does that mean java.nio.GatheringByteChannel.write(buffer) actually writes to the pagecache ? I only ask because this is not mentioned in the docs: https://docs.oracle.com/javase/8/docs/api/java/nio/channels/GatheringByteChannel.html#write-java.nio.ByteBuffer:A- https://docs.oracle.com/javase/8/docs/api/java/nio/channels/FileChannel.html#write-java.nio.ByteBuffer- Hence not guaranteed, am I correct? So if Kafka depends on just writing to the pagecache for performance shouldn't it use some sort of API or mechanism to do just that ? Or are the docs on the web wrong about how Kafka operates ? https://kafka.apache.org/documentation/#design_filesystem Just confused about what is really happening. On Thu, Mar 6, 2025 at 5:31 AM Zhao, Guang <guang.z...@netapp.com.invalid> wrote: > (Sorry, missing links) -- the appending operation in FileRecords: > https://github.com/apache/kafka/blob/56fc8e9d2a04172fd014b89428f1c97da93f4894/clients/src/main/java/org/apache/kafka/common/record/FileRecords.java#L184 > > Cheers, > Guang > -- > Guang Zhao, NetApp > gu...@netapp.com<mailto:gu...@netapp.com> > > > From: Zhao, Guang <guang.z...@netapp.com> > Date: Thursday, 6 March 2025 at 10:57 am > To: users@kafka.apache.org <users@kafka.apache.org> > Subject: Re: How does Kafka write to the pagecache/pagetable ? > Hi Sreyan, > > Thanks for the question. The page cache is managed by the OS, so I think > it is transparent to Kafka writes, which are to files. Just at a higher > abstraction level. > > Some pointers in code: > > * LogSegment: > https://github.com/apache/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java > * FileRecrods: > https://github.com/apache/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java > > You can see appending records actually write to a file through Java’s > FileChannel. > > Hope this helps! > > Cheers, > Guang > -- > Guang Zhao, NetApp > gu...@netapp.com<mailto:gu...@netapp.com> > > > From: Sreyan Chakravarty <sreya...@gmail.com> > Date: Thursday, 6 March 2025 at 2:01 am > To: users@kafka.apache.org <users@kafka.apache.org> > Subject: How does Kafka write to the pagecache/pagetable ? > [You don't often get email from sreya...@gmail.com. Learn why this is > important at https://aka.ms/LearnAboutSenderIdentification ] > > EXTERNAL EMAIL - USE CAUTION when clicking links or attachments > > > > > Hi devs, > > I am curious in trying to understand the internals of how Kafka actually > writes to disk. I am wondering as to how it writes to the > pagecache/pagetable ? > > From what I understand, Kafka never writes to the files directly instead it > writes to the pagecache and lets the OS do the flush to disk. Please > correct me if I am wrong. > > Where in the source code can I find code that does this ? > > Any help would be greatly appreciated. > > -- > Regards, > Sreyan Chakravarty > -- Regards, Sreyan Chakravarty
