Hi, On Fri, Sep 25, 2026 at 04:15:04PM +0800, Lance Yang wrote: > +Cc DeepSeek and Muchun > > On Fri, Sep 25, 2026 at 09:04:41AM +0200, David Hildenbrand (Arm) wrote: > >On 9/25/26 07:44, Lance Yang wrote: > >> Hi all, > >> > >> I was reading DeepSeek's new DSec paper[1] and found a nice use of DAMON > >> and virtio-balloon: > >> > >> With this kind of workload, an agent may read a file once and never touch > >> it again, while those pages remain in the guest page cache. Without memory > >> pressure in the guest, they can stay cached even though the host would > >> like that memory back ... > >> > >> The trick is DAMON + virtio-balloon free-page reporting :) DAMON reclaims > > > >Heh, I read "virtio-balloon" and thought "balloon inflation/deflation, what > >year > >is it?!". Free-page reporting makes much more sense. > > > >> cold file pages from the guest page cache; buddy gets a chance to coalesce > >> them into reportable blocks, and virtio-balloon passes those blocks to > >> Firecracker. Firecracker can then drop the host backing with MADV_DONTNEED. > > > >When I was at RH we were looking at this issue as well. virtio-pmem was one > >way > >of avoiding the page cache in VM entirely. But it has its own limitations. > > YES, they enable virtio-pmem with DAX for the read-only EROFS base-image > and toolkit layers, while using DAMON with balloon free-page reporting to > reclaim cold file pages from the guest page cache on larger writable disks. > > They also point out that the guest must allocate struct page metadata for > the entire pmem-backed address range. So virtio-pmem is not free either :) > > BTW, Muchun recently posted a pretty cool series for exactly that: > > https://lore.kernel.org/linux-mm/[email protected]/ > > (It shares vmemmap backing until a DAX fault needs private metadata, > avoiding the full per-PFN cost up front.) > > I have a feeling the DeepSeek team will be watching this one closely :P
There are several points virtio-pmem RW from my own viewpoints: - It makes the write async I/O synchronously, note that write I/Os are not quite the same as read I/Os (read I/Os are mostly sync). Storage also support multi-queues which can better leverage that, and that is why sometimes brd block device is not good at high-performance nvme for example. - Note that sandbox usually has a memory limit, but dax RW makes the whole rootfs addressable, e.g. if you have 128GiB rootfs, which means you could fault 128GiB on the host, instead of the sandbox memory size, so it might cause some security concern (as long as users shouldn't expect the host memory can be used up to 128GiB + memsize). - It can cause sync 4K faults on the host in the worst case (maybe large folios on the host can improve a bit yet not quite), in constant to the guest memory + THP usage. I don't know how the reclaim overhead is measured currently, but it seems the dsec paper also mentioned in this case. - The guest workload will still use mmap() for many sandbox apps, so `struct page` optimization is just for the optimized case, but not for the worst cases, the malicious VM sandboxes can still take much more `struct page` in the guest. There would be better to have some benchmark here for typical RL training RW virtio-pmem: but block storage semantics cannot already be replaced with the memory semantics. I think virtio-pmem RO is useful simply because it can reuse the same page cache among multiple sandboxes on the host, which can even warm-up other sandbox workloads, although it still has some security concern but I guess for RL training it doesn't matter and read is almost synchronous unlike writes. Thanks, Gao Xiang > > >Thanks for sharing! > > Cheers!
