On 7/2/2025 2:48 AM, Jaroslav Pulchart wrote: >> >> On 6/30/2025 11:48 PM, Jaroslav Pulchart wrote: >>>> On 6/30/2025 2:56 PM, Jacob Keller wrote: >>>>> Unfortunately it looks like the fix I mentioned has landed in 6.14, so >>>>> its not a fix for your issue (since you mentioned 6.14 has failed >>>>> testing in your system) >>>>> >>>>> $ git describe --first-parent --contains --match=v* --exclude=*rc* >>>>> 743bbd93cf29f653fae0e1416a31f03231689911 >>>>> v6.14~251^2~15^2~2 >>>>> >>>>> I don't see any other relevant changes since v6.14. I can try to see if >>>>> I see similar issues with CONFIG_MEM_ALLOC_PROFILING on some test >>>>> systems here. >>>> >>>> On my system I see this at boot after loading the ice module from >>>> >>>> $ grep -F "/ice/" /proc/allocinfo | sort -g | tail | numfmt --to=iec> >>>> 26K 230 drivers/net/ethernet/intel/ice/ice_irq.c:84 [ice] >>>> func:ice_get_irq_res >>>>> 48K 2 drivers/net/ethernet/intel/ice/ice_arfs.c:565 [ice] >>>>> func:ice_init_arfs >>>>> 57K 226 drivers/net/ethernet/intel/ice/ice_lib.c:397 [ice] >>>>> func:ice_vsi_alloc_ring_stats >>>>> 57K 226 drivers/net/ethernet/intel/ice/ice_lib.c:416 [ice] >>>>> func:ice_vsi_alloc_ring_stats >>>>> 85K 226 drivers/net/ethernet/intel/ice/ice_lib.c:1398 [ice] >>>>> func:ice_vsi_alloc_rings >>>>> 339K 226 drivers/net/ethernet/intel/ice/ice_lib.c:1422 [ice] >>>>> func:ice_vsi_alloc_rings >>>>> 678K 226 drivers/net/ethernet/intel/ice/ice_base.c:109 [ice] >>>>> func:ice_vsi_alloc_q_vector >>>>> 1.1M 257 drivers/net/ethernet/intel/ice/ice_fwlog.c:40 [ice] >>>>> func:ice_fwlog_alloc_ring_buffs >>>>> 7.2M 114 drivers/net/ethernet/intel/ice/ice_txrx.c:493 [ice] >>>>> func:ice_setup_rx_ring >>>>> 896M 229264 drivers/net/ethernet/intel/ice/ice_txrx.c:680 [ice] >>>>> func:ice_alloc_mapped_page >>>> >>>> Its about 1GB for the mapped pages. I don't see any increase moment to >>>> moment. I've started an iperf session to simulate some traffic, and I'll >>>> leave this running to see if anything changes overnight. >>>> >>>> Is there anything else that you can share about the traffic setup or >>>> otherwise that I could look into? Your system seems to use ~2.5 x the >>>> buffer size as mine, but that might just be a smaller number of CPUs. >>>> >>>> Hopefully I'll get some more results overnight. >>> >>> The traffic is random production workloads from VMs, using standard >>> Linux or OVS bridges. There is no specific pattern to it. I haven’t >>> had any luck reproducing (or was not patient enough) this with iperf3 >>> myself. The two active (UP) interfaces are in an LACP bonding setup. >>> Here are our ethtool settings for the two member ports (em1 and p3p1) >>> >> >> I had iperf3 running overnight and the memory usage for >> ice_alloc_mapped_pages is constant here. Mine was direct connections >> without bridge or bonding. From your description I assume there's no XDP >> happening either. > > Yes, no XDP in use. > > BTW the allocinfo after 6days uptime: > # uptime ; sort -g /proc/allocinfo| tail -n 15 > 11:46:44 up 6 days, 2:18, 1 user, load average: 9.24, 11.33, 15.07 > 102489024 533797 fs/dcache.c:1681 func:__d_alloc > 106229760 25935 mm/shmem.c:1854 func:shmem_alloc_folio > 117118192 103097 fs/ext4/super.c:1388 [ext4] func:ext4_alloc_inode > 134479872 32832 kernel/events/ring_buffer.c:811 > func:perf_mmap_alloc_page > 162783232 7656 mm/slub.c:2452 func:alloc_slab_page > 189906944 46364 mm/memory.c:1056 func:folio_prealloc > 499384320 121920 mm/percpu-vm.c:95 func:pcpu_alloc_pages > 530579456 129536 mm/page_ext.c:271 func:alloc_page_ext > 625876992 54186 mm/slub.c:2450 func:alloc_slab_page > 838860800 400 mm/huge_memory.c:1165 func:vma_alloc_anon_folio_pmd > 1014710272 247732 mm/filemap.c:1978 func:__filemap_get_folio > 1056710656 257986 mm/memory.c:1054 func:folio_prealloc > 1279262720 610 mm/khugepaged.c:1084 func:alloc_charge_folio > 1334530048 325763 mm/readahead.c:186 func:ractl_alloc_folio > 3341238272 412215 drivers/net/ethernet/intel/ice/ice_txrx.c:681 > [ice] func:ice_alloc_mapped_page > I have a suspicion that the issue is related to the updating of page_count in ice_get_rx_pgcnt(). The i40e driver has a very similar logic for page reuse but doesn't do this. It also has a counter to track failure to re-use the Rx pages.
Commit 11c4aa074d54 ("ice: gather page_count()'s of each frag right
before XDP prog call") changed the logic to update page_count of the Rx
page just prior to the XDP call instead of at the point where we get the
page from ice_get_rx_buf(). I think this change was originally
introduced while we were trying out an experimental refactor of the
hotpath to handle fragments differently, which no longer happens since
743bbd93cf29 ("ice: put Rx buffers after being done with current
frame"), which ironically was part of this very same series..
I think this updating of page count is accidentally causing us to
miscount when we could perform page-reuse, and ultimately causes us to
leak the page somehow. I'm still investigating, but I think this might
trigger if somehow the page pgcnt - pagecnt_bias becomes >1, we don't
reuse the page.
The i40e driver stores the page count in i40e_get_rx_buffer, and I think
our updating it later can somehow get things out-of-sync.
Do you know if your traffic pattern happens to send fragmented frames? I
think iperf doesn't do that, which might be part of whats causing this
issue. I'm going to try to see if I can generate such fragmentation to
confirm. Is your MTU kept at the default ethernet size?
At the very least I'm going to propose a patch for ice similar to the
one from Joe Damato to track the rx busy page count. That might at least
help track something..
Thanks,
Jake
OpenPGP_signature.asc
Description: OpenPGP digital signature
