[dpdk-dev] Performance impact of "declaring" more CPU cores

[Tom Barbette]

Hi all,

We're experiencing a very strange problem. The code of our application 
is modified to always use 8 cores. However, when running with "-l 
0-MAXCPU", with MAXCPU varying between 7 and 15 (therefore "allocating" 
more *unused* cores), the performance of the application change. It can 
be multiple Gbps at very high speed (100G) with a large number of cores, 
and it is not linear in the sense that more cores will not necessarily 
increase, nor degrade performance.

An example can be seen here : 
https://kth.box.com/s/v5v1hyidd51ebd7b8ixmcw513lfqwj4a . Note the 
errorbars (10 runs per points) show that it is not "global" variance 
that create the difference between cores. Once you end up in a "class" 
of performance, you stay there even when changing some parameters (such 
as the number of cores you actually use). From a research perspective 
this is problematic as we cannot trust results with X cores are indeed a 
certain %age better than Y core because it could be due to this problem, 
not what we "improved" in the application.

That application is a NAT and a FW, but we could observe that with other 
VNFs, at different scales. The link is saturated at 100G with real 
packets (avg size 1040B).

We have Mellanox ConnectX 5, DPDK 19.02. This could be observed on 
between Skylake and Cascade Lake machines. The example is a 
single-socket machine with 8 cores and HT. But this happened with a NUMA 
machine, or using only variation of the 18 cores of one sockets.

The only useful observation we made is that when we are in a "bad case", 
the LLC has more cache misses.

We could not really reproduce with XL710, but at 40G the problem can be 
barely seen with MLX, so that path is not very conclusive. We do not 
have other 100G NICs (hurry up Intel! :p). Similarly reproducing with 
testpmd is hard, as the problem really shows with something like 8 
cores, and we need the load to be less than the NIC wire speed, but high 
enough to observe the effect...

To rule out that problem, we hardcoded the number of cores to be used 
everywhere inside the application. So except for DPDK internals, all 
resources are allocated for 8 cores.

And that is still the best lead. Could it be that we simply get 
lucky/unlucky buffer allocations for "something per lcore" and therefore 
packet evict each others because of the limited associativity of the 
cache? At those speeds, failure of DDIO is death...
Would there be a way to fix/verify the memory allocation? base-virtaddr 
did not work though. ASLR disabled (enabled does not change anything).


Thanks for the help,

Tom