> On Aug 29, 2018, at 12:19 PM, Saber Rezvani <irsa...@zoho.com> wrote: > > > > On 08/29/2018 01:39 AM, Wiles, Keith wrote: >> >>> On Aug 28, 2018, at 2:16 PM, Saber Rezvani <irsa...@zoho.com> wrote: >>> >>> >>> >>> On 08/28/2018 11:39 PM, Wiles, Keith wrote: >>>> Which version of Pktgen? I just pushed a patch in 3.5.3 to fix a >>>> performance problem. >>> I use Pktgen verion 3.0.0, indeed it is O.k as far as I have one core. (10 >>> Gb/s) but when I increase the number of core (one core per queue) then I >>> loose some performance (roughly 8.5 Gb/s for 8-core). In my scenario Pktgen >>> shows it is generating at line rate, but receiving 8.5 Gb/s. >>> Is it because of Pktgen??? >> Normally Pktgen can receive at line rate up to 10G 64 byte frames, which >> means Pktgen should not be the problem. You can verify that by looping the >> cable from one port to another on the pktgen machine to create a external >> loopback. Then send traffic what ever you can send from one port you should >> be able to receive those packets unless something is configured wrong. >> >> Please send me the command line for pktgen. >> >> >> In pktgen if you have this config -m “[1-4:5-8].0” then you have 4 cores >> sending traffic and 4 core receiving packets. >> >> In this case the TX cores will be sending the packets on all 4 lcores to the >> same port. On the rx side you have 4 cores polling 4 rx queues. The rx >> queues are controlled by RSS, which means the RX traffic 5 tuples hash must >> divide the inbound packets across all 4 queues to make sure each core is >> doing the same amount of work. If you are sending only a single packet on >> the Tx cores then only one rx queue be used. >> >> I hope that makes sense. > I think there is a misunderstanding of the problem. Indeed the problem is not > the Pktgen. > Here is my command --> ./app/app/x86_64-native-linuxapp-gcc/pktgen -c ffc0000 > -n 4 -w 84:00.0 -w 84:00.1 --file-prefix pktgen_F2 --socket-mem > 1000,2000,1000,1000 -- -T -P -m "[18-19:20-21].0, [22:23].1" > > The problem is when I run the symmetric_mp example for $numberOfProcesses=8 > cores, then I have less throughput (roughly 8.4 Gb/s). but when I run it for > $numberOfProcesses=3 cores throughput is 10G. > for i in `seq $numberOfProcesses`; > do > .... some calculation goes here..... > symmetric_mp -c $coremask -n 2 --proc-type=auto -w 0b:00.0 -w > 0b:00.1 --file-prefix sm --socket-mem 4000,1000,1000,1000 -- -p 3 > --num-procs=$numberOfProcesses --proc-id=$procid"; > ..... > done
Most NICs have a limited amount of memory on the NIC and when you start to segment that memory because you are using more queues it can effect performance. In one of the NICs if you go over say 6 or 5 queues the memory per queue for Rx/Tx packets starts to become a bottle neck as you do not have enough memory in the Tx/Rx queues to hold enough packets. This can cause the NIC to drop Rx packets because the host can not pull the data from the NIC or Rx ring on the host fast enough. This seems to be the problem as the amount of time to process a packet on the host has not changed only the amount of buffer space in the NIC as you increase queues. I am not sure this is your issue, but I figured I would state this point. > > I am trying find out what makes this loss! > > >>>>> On Aug 28, 2018, at 12:05 PM, Saber Rezvani <irsa...@zoho.com> wrote: >>>>> >>>>> >>>>> >>>>> On 08/28/2018 08:31 PM, Stephen Hemminger wrote: >>>>>> On Tue, 28 Aug 2018 17:34:27 +0430 >>>>>> Saber Rezvani <irsa...@zoho.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> >>>>>>> I have run multi_process/symmetric_mp example in DPDK example directory. >>>>>>> For a one process its throughput is line rate but as I increase the >>>>>>> number of cores I see decrease in throughput. For example, If the number >>>>>>> of queues set to 4 and each queue assigns to a single core, then the >>>>>>> throughput will be something about 9.4. if 8 queues, then throughput >>>>>>> will be 8.5. >>>>>>> >>>>>>> I have read the following, but it was not convincing. >>>>>>> >>>>>>> http://mails.dpdk.org/archives/dev/2015-October/024960.html >>>>>>> >>>>>>> >>>>>>> I am eagerly looking forward to hearing from you, all. >>>>>>> >>>>>>> >>>>>>> Best wishes, >>>>>>> >>>>>>> Saber >>>>>>> >>>>>>> >>>>>> Not completely surprising. If you have more cores than packet line rate >>>>>> then the number of packets returned for each call to rx_burst will be >>>>>> less. >>>>>> With large number of cores, most of the time will be spent doing reads of >>>>>> PCI registers for no packets! >>>>> Indeed pktgen says it is generating traffic at line rate, but receiving >>>>> less than 10 Gb/s. So, it that case there should be something that causes >>>>> the reduction in throughput :( >>>>> >>>>> >>>> Regards, >>>> Keith >>>> >>> >>> >> Regards, >> Keith >> > > Best regards, > Saber Regards, Keith
