Ran into a major system performance issue after updated Rocky LInux to
9.4 / and OpenvSwitch to 3.3.0 and DPDK to 23.11, below is the context
of the issue, along with what was discovered and how it was corrected.
Issue:
Updated a Virtualization Host Server to RL9.4 yesterday and 24 hours
later, TOP, /proc/meminfo, System Monitor are reporting 99% memory
utilization. Even after shutting down all the virtualization clients
(turn off auto-start) and rebooting, system instantly reports 99% memory
utilization before virtualization starts via TOP, /proc/meminfo, System
Monitor.
top - 18:51:24 up 49 min, 1 user, load average: 19.62, 19.09, 16.17
Tasks: 1097 total, 5 running, 1092 sleeping, 0 stopped, 0 zombie
%Cpu(s): 6.0 us, 4.4 sy, 0.1 ni, 84.8 id, 4.0 wa, 0.5 hi, 0.2 si, 0.0 st
MiB Mem : 257356.0 total, 665.1 free, 257356.0 used, 471.1 buff/cache
MiB Swap: 524288.0 total, 432395.2 free, 91892.8 used. 709.8 avail Mem
After:
Linux sys7 5.14.0-427.16.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Wed May 8
17:48:14 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Before:
Linux sys7 5.14.0-362.24.1.el9_3.0.1.x86_64 #1 SMP PREEMPT_DYNAMIC Thu
Apr 4 22:31:43 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Discovery/Resolution:
Discovered error via fire drill method, via the documentation
"https://docs.openvswitch.org/en/latest/intro/install/dpdk/";, it talks
about setting “echo ‘vm.nr_hugepages=2048’ >
/etc/sysctl.d/hugepages.conf” or “sysctl -w vm.nr_hugepages=N # where N
= No. of 2M huge pages”. If you follow the “sysctl.d/hugepages.conf”
method or “grubby --args hugepages=2048 --update-kernel DEFAULT” method
it applies as 204% of memory but if you apply via the “sysctl -w
vm.nr_hugepages=N # where N = No. of 2M huge pages” method and say 8 or
16 or 32 it applies correctly and doesn't over-allocate system memory,
shown below is with settings at "8" and everything back to running.
top - 09:43:20 up 1:17, 2 users, load average: 14.11, 15.26, 14.61
Tasks: 1012 total, 2 running, 1010 sleeping, 0 stopped, 0 zombie
%Cpu(s): 13.6 us, 3.0 sy, 0.0 ni, 82.8 id, 0.1 wa, 0.4 hi, 0.2 si,
0.0 st
MiB Mem : 257356.6 total, 73659.2 free, 130307.0 used, 55139.8 buff/cache
MiB Swap: 524288.0 total, 524259.4 free, 28.6 used. 127049.7 avail Mem
Even rebooting to previous 9.3 kernel, resulted in same over utilization
of memory with 3.3.0 and DPDK 23.11 until above change was made.
Previous to upgrade was running OpenvSwitch 3.2.1 and DPDK 21
--
Thanks,
Michael Miller
VP of Operations
Assorted Business Services
Office 925-625-7847 x 1040
Cell: 925-998-0087
_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
