Great, please send me a link to the related patch set. I can apply them in our kernel build and try them ASAP!
st 25. 6. 2025 v 16:03 odesílatel Przemek Kitszel < [email protected]> napsal: > On 6/25/25 14:17, Jaroslav Pulchart wrote: > > Hello > > > > We are still facing the memory issue with Intel 810 NICs (even on latest > > 6.15.y). > > > > Our current stabilization and solution is to move everything to a new > > INTEL-FREE server and get rid of last Intel sights there (after Intel's > > CPU vulnerabilities fuckups NICs are next step). > > > > Any help welcomed, > > Jaroslav P. > > > > > > Thank you for urging us, I can understand the frustration. > > We have identified some (unrelated) memory leaks, will soon ship fixes. > And, as there were no clear issue with any commit/version you have > posted to be a culprit, there is a chance that our random findings could > help. Anyway going to zero kmemleak reports is good in itself, that is > a good start. > > Will ask my VAL too to increase efforts in this area too. > > Przemek > > > > > st 4. 6. 2025 v 10:42 odesílatel Jaroslav Pulchart > > <[email protected] <mailto:[email protected]>> > > > napsal: > > > > > > > > čt 17. 4. 2025 v 19:52 odesílatel Keller, Jacob E > > > <[email protected] <mailto:[email protected]>> > napsal: > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Jakub Kicinski <[email protected] <mailto:[email protected] > >> > > > > > Sent: Wednesday, April 16, 2025 5:13 PM > > > > > To: Keller, Jacob E <[email protected] > > <mailto:[email protected]>> > > > > > Cc: Jaroslav Pulchart <[email protected] > > <mailto:[email protected]>>; Kitszel, Przemyslaw > > > > > <[email protected] > > <mailto:[email protected]>>; Damato, Joe > > <[email protected] <mailto:[email protected]>>; intel-wired- > > > > > [email protected] <mailto:[email protected]>; > > [email protected] <mailto:[email protected]>; Nguyen, > > Anthony L > > > > > <[email protected] > > <mailto:[email protected]>>; Igor Raits <[email protected] > > <mailto:[email protected]>>; Daniel Secik > > > > > <[email protected] > > <mailto:[email protected]>>; Zdenek Pesek > > <[email protected] <mailto:[email protected]>>; > > > > > Dumazet, Eric <[email protected] > > <mailto:[email protected]>>; Martin Karsten > > > > > <[email protected] <mailto:[email protected]>>; Zaki, > > Ahmed <[email protected] <mailto:[email protected]>>; Czapnik, > > > > > Lukasz <[email protected] > > <mailto:[email protected]>>; Michal Swiatkowski > > > > > <[email protected] > > <mailto:[email protected]>> > > > > > Subject: Re: [Intel-wired-lan] Increased memory usage on NUMA > > nodes with ICE > > > > > driver after upgrade to 6.13.y (regression in commit > > 492a044508ad) > > > > > > > > > > On Wed, 16 Apr 2025 22:57:10 +0000 Keller, Jacob E wrote: > > > > > > > > And you're reverting just and exactly 492a044508ad13 ? > > > > > > > > The memory for persistent config is allocated in > > alloc_netdev_mqs() > > > > > > > > unconditionally. I'm lost as to how this commit could > > make any > > > > > > > > difference :( > > > > > > > > > > > > > > Yes, reverted the 492a044508ad13. > > > > > > > > > > > > Struct napi_config *is* 1056 bytes > > > > > > > > > > You're probably looking at 6.15-rcX kernels. Yes, the > > affinity mask > > > > > can be large depending on the kernel config. But report is > > for 6.13, > > > > > AFAIU. In 6.13 and 6.14 napi_config was tiny. > > > > > > > > Regardless, it should still be ~64KB even in that case which is > > a far cry from eating all available memory. Something else must be > > going on.... > > > > > > > > Thanks, > > > > Jake > > > > > > Hello > > > > > > Some observation, this "problem" still exists with the latest > 6.14.y > > > and there must be multiple issues, the memory utilization is > slowly > > > going down, from 3GB to 100MB in 10-20days. at home NUMA nodes > where > > > intel x810 NIC are (looks like some memory leak related to > > > networking). > > > > > > So without the revert the kawadX usage is observed asap like till > > > 1-2d, with revert of mentioned commit kswadX starts to consume > > > resources later like in ~10d-20d later. It is almost impossible > > to use > > > servers with Intel X810 cards (ice driver) with recent linux > kernels. > > > > > > Were you able to reproduce the memory problems in your testbed? > > > > > > Best, > > > Jaroslav > > > > Hello > > > > I deployed linux 6.15.0 to our servers 7d ago and observed the > > behaviour of memory utilization of NUMA home nodes of Intel X810 > > 1/ there is no need to revert the commit as before, > > 2/ the memory is continuously consumed (like memory leak), > > see attached "7d_memory_usage_per_numa_linux6.15.0.png" screenshot 8x > > numa nodes, (NUMA0 + NUMA1 are local for X810 nics). BTW: We do not > > see this memory utilization pattern on server s using Broadcom > > Netxtreme-E NICs > > > > > > > > -- > > Jaroslav Pulchart > > Sr. Principal SW Engineer > > GoodData > > -- Jaroslav Pulchart Sr. Principal SW Engineer GoodData
