(Sorry, missing links) -- the appending operation in FileRecords: https://github.com/apache/kafka/blob/56fc8e9d2a04172fd014b89428f1c97da93f4894/clients/src/main/java/org/apache/kafka/common/record/FileRecords.java#L184
Cheers, Guang -- Guang Zhao, NetApp gu...@netapp.com<mailto:gu...@netapp.com> From: Zhao, Guang <guang.z...@netapp.com> Date: Thursday, 6 March 2025 at 10:57 am To: users@kafka.apache.org <users@kafka.apache.org> Subject: Re: How does Kafka write to the pagecache/pagetable ? Hi Sreyan, Thanks for the question. The page cache is managed by the OS, so I think it is transparent to Kafka writes, which are to files. Just at a higher abstraction level. Some pointers in code: * LogSegment: https://github.com/apache/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java * FileRecrods: https://github.com/apache/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java You can see appending records actually write to a file through Java’s FileChannel. Hope this helps! Cheers, Guang -- Guang Zhao, NetApp gu...@netapp.com<mailto:gu...@netapp.com> From: Sreyan Chakravarty <sreya...@gmail.com> Date: Thursday, 6 March 2025 at 2:01 am To: users@kafka.apache.org <users@kafka.apache.org> Subject: How does Kafka write to the pagecache/pagetable ? [You don't often get email from sreya...@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Hi devs, I am curious in trying to understand the internals of how Kafka actually writes to disk. I am wondering as to how it writes to the pagecache/pagetable ? From what I understand, Kafka never writes to the files directly instead it writes to the pagecache and lets the OS do the flush to disk. Please correct me if I am wrong. Where in the source code can I find code that does this ? Any help would be greatly appreciated. -- Regards, Sreyan Chakravarty
