On 15-Jul-22 2:12 PM, Anatoly Burakov wrote:
Currently, there is no way to measure lcore busyness in a passive way, without any modifications to the application. This patch adds a new EAL API that will be able to passively track core busyness.The busyness is calculated by relying on the fact that most DPDK API's will poll for packets. Empty polls can be counted as "idle", while non-empty polls can be counted as busy. To measure lcore busyness, we simply call the telemetry timestamping function with the number of polls a particular code section has processed, and count the number of cycles we've spent processing empty bursts. The more empty bursts we encounter, the less cycles we spend in "busy" state, and the less core busyness will be reported. In order for all of the above to work without modifications to the application, the library code needs to be instrumented with calls to the lcore telemetry busyness timestamping function. The following parts of DPDK are instrumented with lcore telemetry calls: - All major driver API's: - ethdev - cryptodev - compressdev - regexdev - bbdev - rawdev - eventdev - dmadev - Some additional libraries: - ring - distributor To avoid performance impact from having lcore telemetry support, a global variable is exported by EAL, and a call to timestamping function is wrapped into a macro, so that whenever telemetry is disabled, it only takes one additional branch and no function calls are performed. It is also possible to disable it at compile time by commenting out RTE_LCORE_BUSYNESS from build config. This patch also adds a telemetry endpoint to report lcore busyness, as well as telemetry endpoints to enable/disable lcore telemetry. Signed-off-by: Kevin Laatz <kevin.la...@intel.com> Signed-off-by: Conor Walsh <conor.wa...@intel.com> Signed-off-by: David Hunt <david.h...@intel.com> Signed-off-by: Anatoly Burakov <anatoly.bura...@intel.com> --- Notes: We did a couple of quick smoke tests to see if this patch causes any performance degradation, and it seemed to have none that we could measure. Telemetry can be disabled at compile time via a config option, while at runtime it can be disabled, seemingly at a cost of one additional branch.That said, our benchmarking efforts were admittedly not very rigorous, socomments welcome! config/rte_config.h | 2 + lib/bbdev/rte_bbdev.h | 17 +- lib/compressdev/rte_compressdev.c | 2 + lib/cryptodev/rte_cryptodev.h | 2 + lib/distributor/rte_distributor.c | 21 +- lib/distributor/rte_distributor_single.c | 14 +- lib/dmadev/rte_dmadev.h | 15 +- lib/eal/common/eal_common_lcore_telemetry.c | 274 ++++++++++++++++++++ lib/eal/common/meson.build | 1 + lib/eal/include/rte_lcore.h | 80 ++++++ lib/eal/meson.build | 3 + lib/eal/version.map | 7 + lib/ethdev/rte_ethdev.h | 2 + lib/eventdev/rte_eventdev.h | 10 +- lib/rawdev/rte_rawdev.c | 5 +- lib/regexdev/rte_regexdev.h | 5 +- lib/ring/rte_ring_elem_pvt.h | 1 + 17 files changed, 437 insertions(+), 24 deletions(-) create mode 100644 lib/eal/common/eal_common_lcore_telemetry.c diff --git a/config/rte_config.h b/config/rte_config.h index 46549cb062..583cb6f7a5 100644 --- a/config/rte_config.h +++ b/config/rte_config.h @@ -39,6 +39,8 @@ #define RTE_LOG_DP_LEVEL RTE_LOG_INFO #define RTE_BACKTRACE 1 #define RTE_MAX_VFIO_CONTAINERS 64 +#define RTE_LCORE_BUSYNESS 1 +#define RTE_LCORE_BUSYNESS_PERIOD 4000000ULL
One possible improvement here would be to specify period in microseconds, and use rte_get_tsc_hz() to adjust the telemetry period. This would require adding code to EAL init, because we can't use that API until EAL has called `rte_eal_timer_init()`, but it would make the telemetry period CPU frequency independent.
-- Thanks, Anatoly
