> -----Original Message----- > From: David Marchand <david.march...@redhat.com> > Sent: 2022年12月20日 17:34 > To: You, KaisenX <kaisenx....@intel.com> > Cc: Ferruh Yigit <ferruh.yi...@amd.com>; dev@dpdk.org; Burakov, Anatoly > <anatoly.bura...@intel.com>; sta...@dpdk.org; Yang, Qiming > <qiming.y...@intel.com>; Zhou, YidingX <yidingx.z...@intel.com>; Wu, > Jingjing <jingjing...@intel.com>; Xing, Beilei <beilei.x...@intel.com>; Zhang, > Qi Z <qi.z.zh...@intel.com>; Luca Boccassi <bl...@debian.org>; Mcnamara, > John <john.mcnam...@intel.com>; Kevin Traynor <ktray...@redhat.com> > Subject: Re: [PATCH] net/iavf:fix slow memory allocation > > On Tue, Dec 20, 2022 at 7:52 AM You, KaisenX <kaisenx....@intel.com> > wrote: > > > >> As to the reason for not using rte_malloc_socket. I thought > > > >> rte_malloc_socket() could solve the problem too. And the > > > >> appropriate parameter should be the socket_id that created the > > > >> memory pool for DPDK initialization. Assuming that> the socket_id > > > >> of the initially allocated memory = 1, first let the > > > > eal_intr_thread > > > >> determine if it is on the socket_id, then record this socket_id > > > >> in the eal_intr_thread and pass it to the iavf_event_thread. But > > > >> there seems no way to link this parameter to the > > > >> iavf_dev_event_post() > > > function. That is why rte_malloc_socket is not used. > > > >> > > > > > > > > I was thinking socket id of device can be used, but that won't > > > > help if the core that interrupt handler runs is in different socket. > > > > And I also don't know if there is a way to get socket that > > > > interrupt thread is on. @David may help perhaps. > > > > > > > > So question is why interrupt thread is not running on main lcore. > > > > > > > > > > OK after some talk with David, what I am missing is > 'rte_ctrl_thread_create()' > > > does NOT run on main lcore, it can run on any core except data plane > cores. > > > > > > Driver "iavf-event-thread" thread (iavf_dev_event_handle()) and > > > interrupt thread (so driver interrupt callback > > > iavf_dev_event_post()) can run on any core, making it hard to manage. > > > And it seems it is not possible to control where interrupt thread to run. > > > > > > One option can be allocating hugepages for all sockets, but this > > > requires user involvement, and can't happen transparently. > > > > > > Other option can be to control where "iavf-event-thread" run, like > > > using 'rte_thread_create()' to create thread and provide attribute > > > to run it on main lcore (rte_lcore_cpuset(rte_get_main_lcore()))? > > > > > > Can you please test above option? > > > > > > > > The first option can solve this issue. but to borrow from your > > previous saying, "in a dual socket system, if all used cores are in > > socket 1 and the NIC is in socket 1, no memory is allocated for socket 0. > This is to optimize memory consumption." > > I think it's unreasonable to do so. > > > > About other option. In " rte_eal_intr_init" function, After the thread > > is created, I set the thread affinity for eal-intr-thread, but it does not > > solve > this issue. > > Jumping in this thread. > > I tried to play a bit with a E810 nic on a dual numa and I can't see anything > wrong for now. > Can you provide a simple and small reproducer of your issue? > > Thanks. > This is my environment: Enter "lscpu" on the command line: NUMA: NUMA node(s): 2 NUMA node0 CPU(S) : 0-27,56-83 NUMA node1 CPU(S) : 28-55,84-111
List the steps to reproduce the issue: 1. create vf and blind to dpdk echo 1 > /sys/bus/pci/devices/0000\:ca\:00.0/sriov_ numvfs ./usertools/dpdk-devbind. py -b vfio-pci 0000:ca:01.0 2. launch testpmd ./x86_ 64-native-linuxapp-clang/app/dpdk-testpmd -l 28-48 -n 4 -a 0000:ca:01.0 --file-prefix=dpdk_ 525342_ 20221104042659 -- -i --rxq=256 --txq=256 --total-num-mbufs=500000 Parameter Description: "-l 28-48":The range of parameter values after "-l" must be on "NUMA node1 CPU(S)" "0000:ca:01.0":inset on node1 > -- > David Marchand