On 2024-12-09 12:03, David Marchand wrote:
Hello,
On Fri, Dec 6, 2024 at 12:02 PM Mattias Rönnblom <hof...@lysator.liu.se> wrote:
On 2024-12-05 18:57, David Marchand wrote:
As I had reported in rc2, the lcore variables allocation have a
noticeable impact on applications consuming DPDK, even when such
applications does not use DPDK, or use features associated to
some lcore variables.
While the amount has been reduced in a rush before rc2,
there are still cases when the increased memory footprint is noticed
like in scaling tests.
See https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/2090931
What this bug report fails to mention is that it only affects
applications using locked memory.
- By locked memory, are you referring to mlock() and friends?
No ovsdb binary calls them, only the datapath cares about mlocking.
- At a minimum, I understand the lcore var change introduced an
increase in memory of 4kB * 128 (getpagesize() * RTE_MAX_LCORES),
since lcore_var_alloc() calls memset() of the lcore var size, for
every lcore.
Yes, that is my understanding. It's also consistent with the
measurements I've posted on this list.
In this unit test where 1000 processes are kept alive in parallel,
this means memory consumption increased by 512k * 1000, so ~500M at
least.
This amount of memory is probably significant in a resource-restrained
env like a (Ubuntu) CI.
I wouldn't expect thousands of concurrent processes in a
resource-constrained system. Sounds wasteful indeed. But sure, there may
well be scenarios where this make sense.
- I went and traced this unit tests on my laptop by monitoring
kmem:mm_page_alloc, though there may be a better metrics when it comes
to memory consumption.
# dir=build; perf stat -e kmem:mm_page_alloc -- tests/testsuite -C
$dir/tests
AUTOTEST_PATH=$dir/utilities:$dir/vswitchd:$dir/ovsdb:$dir/vtep:$dir/tests:$dir/ipsec::
2154
Which gives:
- 1 635 489 kmem:mm_page_alloc for v23.11
- 5 777 043 kmem:mm_page_alloc for v24.11
Interesting. What is vm.overcommit_memory set to?
How much does process RSS differ between the two version?
This is what I get with a hello world type test program, which links to
DPDK, but have not yet called rte_eal_init():
DPDK Version Linking RSS [kB]
22.11 Static 8448
23.11 Static 8192
9bbd44d638* Static 8192
18b5049ab4** Static 8704
24.11 Static 9472
22.11 Dynamic 2304
9bbd44d638* Dynamic 2304
18b5049ab4** Dynamic 2816
24.11 Dynamic 2816
No DPDK - 1024
* 24.11rc commit before lcore variables patch set.
* Last commit in the initial lcore variables patch set.
Ubuntu 24.04 w/ all defaults. --no-huge was used.
In the dynamic case, you see the expected ~0.5 MB increase in foot print
for yet-to-be-used lcore variables.
In the static case, you also see a ~0.5 MB bump with the lcore
variables, and then another ~0.5 MB bump (for some other reason) until
24.11.
(The dynamic hello world program only ends up being dynamically linked
to very few of the DPDK shared objects, so ymmv).
There is a 4M difference, where I would expect 128k.
So something more happens, than a simple page allocation per lcore,
though I fail to understand what.
Btw, just focusing on lcore var, I did two more tests:
- 1 606 998 kmem:mm_page_alloc for v24.11 + revert all lcore var changes.
- 1 634 606 kmem:mm_page_alloc for v24.11 + current series with
postponed allocations.
If one move initialization to shared object constructors (from having
been at some later time), and then end up not running that
initialization code at all (e.g., DPDK is not used), those code pages
will increase RSS. That might well hurt more than the lcore variable
memory itself, depending on how much code is run.
However, such read-only pages can be replaced with something more useful
if the system is under memory pressure, so they aren't really a big
issue as far as (real) memory footprint is concerned.
Just linking to DPDK (and its dependencies) already came with a 1-7 MB
RSS penalty, prior to lcore variables. I wonder how much of that goes
away if all RTE_INIT() type constructors are removed.
