On 11/17/2022 6:57 AM, Kaisen You wrote:
> In some cases, the DPDK does not allocate hugepage heap memory to
> some sockets due to the user setting parameters
> (e.g. -l 40-79, SOCKET 0 has no memory).
> When the interrupt thread runs on the corresponding core of this
> socket, each allocation/release will execute a whole set of heap
> allocation/release operations,resulting in poor performance.
> Instead we call malloc() to get memory from the system's
> heap space to fix this problem.
>
Hi Kaisen,
Using libc malloc can improve performance for this case, but I would
like to understand root cause of the problem.
As far as I can see, interrupt callbacks are run by interrupt thread
("eal-intr-thread"),
and interrupt thread created by 'rte_ctrl_thread_create()' API.
'rte_ctrl_thread_create()' comment mentions that "CPU affinity retrieved
at the time 'rte_eal_init()' was called,"
And 'rte_eal_init()' is run on main lcore, which is the first lcore in
the core list (unless otherwise defined with --main-lcore).
So, the interrupts should be running on a core that has hugepages
allocated for it, am I missing something here?
And what about using 'rte_malloc_socket()' API (instead of rte_malloc),
which gets 'socket' as parameter, and provide the socket that devices is
on as parameter to this API? Is it possible to test this?
> Fixes: cb5c1b91f76f ("net/iavf: add thread for event callbacks")
> Cc: sta...@dpdk.org
>
> Signed-off-by: Kaisen You <kaisenx....@intel.com>
> ---
> drivers/net/iavf/iavf_vchnl.c | 8 +++-----
> 1 file changed, 3 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/net/iavf/iavf_vchnl.c b/drivers/net/iavf/iavf_vchnl.c
> index f92daf97f2..a05791fe48 100644
> --- a/drivers/net/iavf/iavf_vchnl.c
> +++ b/drivers/net/iavf/iavf_vchnl.c
> @@ -36,7 +36,6 @@ struct iavf_event_element {
> struct rte_eth_dev *dev;
> enum rte_eth_event_type event;
> void *param;
> - size_t param_alloc_size;
> uint8_t param_alloc_data[0];
> };
>
> @@ -80,7 +79,7 @@ iavf_dev_event_handle(void *param __rte_unused)
> TAILQ_FOREACH_SAFE(pos, &pending, next, save_next) {
> TAILQ_REMOVE(&pending, pos, next);
> rte_eth_dev_callback_process(pos->dev, pos->event,
> pos->param);
> - rte_free(pos);
> + free(pos);
> }
> }
>
> @@ -94,14 +93,13 @@ iavf_dev_event_post(struct rte_eth_dev *dev,
> {
> struct iavf_event_handler *handler = &event_handler;
> char notify_byte;
> - struct iavf_event_element *elem = rte_malloc(NULL, sizeof(*elem) +
> param_alloc_size, 0);
> + struct iavf_event_element *elem = malloc(sizeof(*elem) +
> param_alloc_size);
> if (!elem)
> return;
>
> elem->dev = dev;
> elem->event = event;
> elem->param = param;
> - elem->param_alloc_size = param_alloc_size;
> if (param && param_alloc_size) {
> rte_memcpy(elem->param_alloc_data, param, param_alloc_size);
> elem->param = elem->param_alloc_data;
> @@ -165,7 +163,7 @@ iavf_dev_event_handler_fini(void)
> struct iavf_event_element *pos, *save_next;
> TAILQ_FOREACH_SAFE(pos, &handler->pending, next, save_next) {
> TAILQ_REMOVE(&handler->pending, pos, next);
> - rte_free(pos);
> + free(pos);
> }
> }
>
