Hello, perhaps you've already gone through these, but this site summarizes also a few important tweaks, e.g. disabling entropy harvest, etc..
Also, there are detailed and well documented benchmarks about FreeBSD Routing Performance, IIRC, also with Chelsio Cards.. https://bsdrp.net/documentation/technical_docs/performance Sydney > On 28 Feb 2017, at 07:35, Ben RUBSON <ben.rub...@gmail.com> wrote: > > Hi, > > Try disabling NUMA in your BIOS settings ? > I had perf issue on a 2-CPU (24 cores) server, I was not able to run a 40G > NIC at its max throughput. > We investigated a lot, disabling NUMA in the BIOS was the solution, as NUMA > is not fully supported yet (as of stable/11). > > Ben > > > >> On 28 Feb 2017, at 03:13, Caraballo-vega, Jordan A. (GSFC-6062)[COMPUTER >> SCIENCE CORP] <jordancaraball...@gmail.com> wrote: >> >> As a summarywe have a Dell R530 with a Chelsio T580 cardwith -CURRENT. >> >> In an attempt to reduce the time the system was taking to look for the >> cpus; we changed the BIOS setting to let the system have 8 visible cores >> and tested cxl* and vcxl* chelsio interfaces. Scores are still way lower >> than what we expected: >> >> Cxl interface >> >> root@router1:~ # netstat -w1 -h >> input (Total) output >> packets errs idrops bytes packets errs bytes colls >> 4.1M 0 3.4M 2.1G 725k 0 383M 0 >> 3.7M 0 3.1M 1.9G 636k 0 336M 0 >> 3.9M 0 3.2M 2.0G 684k 0 362M 0 >> 4.0M 0 3.3M 2.1G 702k 0 371M 0 >> 3.8M 0 3.2M 2.0G 658k 0 348M 0 >> 3.9M 0 3.2M 2.0G 658k 0 348M 0 >> 3.9M 0 3.2M 2.0G 721k 0 381M 0 >> 3.3M 0 2.6M 1.7G 681k 0 360M 0 >> 3.2M 0 2.5M 1.7G 666k 0 352M 0 >> 2.6M 0 2.0M 1.4G 620k 0 328M 0 >> 2.8M 0 2.1M 1.4G 615k 0 325M 0 >> 3.2M 0 2.6M 1.7G 612k 0 323M 0 >> 3.3M 0 2.7M 1.7G 664k 0 351M 0 >> >> >> Vcxl interface >> input (Total) output >> packets errs idrops bytes packets errs bytes colls drops >> 590k 7.5k 0 314M 590k 0 314M 0 0 >> 526k 6.6k 0 280M 526k 0 280M 0 0 >> 588k 7.1k 0 313M 588k 0 313M 0 0 >> 532k 6.6k 0 283M 532k 0 283M 0 0 >> 578k 7.2k 0 307M 578k 0 307M 0 0 >> 565k 7.0k 0 300M 565k 0 300M 0 0 >> 558k 7.0k 0 297M 558k 0 297M 0 0 >> 533k 6.7k 0 284M 533k 0 284M 0 0 >> 588k 7.3k 0 313M 588k 0 313M 0 0 >> 553k 6.9k 0 295M 554k 0 295M 0 0 >> 527k 6.7k 0 281M 527k 0 281M 0 0 >> 585k 7.4k 0 311M 585k 0 311M 0 0 >> >> Related to pmcstat scores are: >> >> root@router1:~/PMC_Stats/Feb22 # pmcstat -R sample.out -G - | head >> @ CPU_CLK_UNHALTED_CORE [2091 samples] >> >> 15.35% [321] lock_delay @ /boot/kernel/kernel >> 94.70% [304] _mtx_lock_spin_cookie >> 100.0% [304] __mtx_lock_spin_flags >> 57.89% [176] pmclog_loop @ /boot/kernel/hwpmc.ko >> 100.0% [176] fork_exit @ /boot/kernel/kernel >> 41.12% [125] pmclog_reserve @ /boot/kernel/hwpmc.ko >> 100.0% [125] pmclog_process_callchain >> 100.0% [125] pmc_process_samples >> >> root@router1:~/PMC_Stats/Feb22 # pmcstat -R sample0.out -G - | head >> @ CPU_CLK_UNHALTED_CORE [480 samples] >> >> 37.29% [179] acpi_cpu_idle_mwait @ /boot/kernel/kernel >> 100.0% [179] acpi_cpu_idle >> 100.0% [179] cpu_idle_acpi >> 100.0% [179] cpu_idle >> 100.0% [179] sched_idletd >> 100.0% [179] fork_exit >> >> 12.92% [62] cpu_idle @ /boot/kernel/kernel >> >> When trying to run pmcstat with the vcxl interfaces enabled the system >> just went to a state of not responding. >> >> Based on previous scores with Centos 7 (over 3M pps), we can assume that >> it is not the hardware. However, we are still looking for a reason of >> why are we getting these scores. >> >> Any feedback or suggestion would be highly appreciated. >> >> - Jordan >> >> On 2/9/17 11:34 AM, Navdeep Parhar wrote: >>> The vcxl interfaces should work under current or 11-STABLE. Let me know >>> if you run into any trouble when trying to use netmap with cxgbe driver. >>> >>> Regards, >>> Navdeep >>> >>> On Thu, Feb 09, 2017 at 10:29:08AM -0500, John Jasen wrote: >>>> It's not the hardware. >>>> >>>> Jordan booted up CentOS on the box, and untuned, were able to obtain >>>> over 3 mpps. >>>> >>>> He has some pmcstat output from freebsd-current, but basically, it >>>> appears the system spends most of its time looking for a CPU to service >>>> the interrupts and keeps landing on one or two of them, as opposed to >>>> any of the other 16 cores on the physical silicon. >>>> >>>> We also tried swapping out the T5 card for a Mellanox, tried different >>>> PCIe slots, adjusted cpuset for the low and the high CPUs, no matter >>>> what we try, the results have been bad. >>>> >>>> Our network test environment is under reconstruction at the moment, but >>>> our plans afterwards are to: >>>> >>>> a) test netmap-fwd again (the VCXL enabling works under -CURRENT?) >>>> >>>> b) test without netmap-fwd, and with reduced cores/physical cpus (BIOS >>>> setting) >>>> >>>> c) potentially, test with netmap-fwd and reduced core count. >>>> >>>> Any other ideas out there? >>>> >>>> Thanks! >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On 02/05/2017 12:55 PM, Navdeep Parhar wrote: >>>>> I've been following the email thread on freebsd-net on this. The >>>>> numbers you're getting are well below what the hardware is capable of. >>>>> >>>>> Have you tried netmap-fwd or something that bypasses the kernel? That >>>>> will be a very quick way to make sure that the hardware is doing ok. >>>>> >>>>> In case you try netmap: >>>>> cxgbe has virtual interfaces now and those are used for netmap (instead >>>>> of the main interface). Add this line to /boot/loader.conf and you'll >>>>> see a 'vcxl' interface for every cxl interface. >>>>> hw.cxgbe.num_vis=2 >>>>> It has its own MAC address and can be used like any other interface, >>>>> except it has native netmap support too. You can run netmap-fwd between >>>>> these vcxl ports. >>>>> >>>>> Regards, >>>>> Navdeep >>>>> >>>>> On Tue, Jan 31, 2017 at 01:57:37PM -0400, Jordan Caraballo wrote: >>>>>> Navdeep, Troy, >>>>>> >>>>>> I forwarded you this email to see if we could get feedback from both of >>>>>> you. I talked with Troy during November about >>>>>> >>>>>> this R530 system and the use of a 40G Chelsio T-580-CR card. So far, we >>>>>> have not experienced results above 1.4 million or so. >>>>>> >>>>>> Any help would be appreciated. >>>>>> >>>>>> - Jordan >>>>>> >>>>>> -------- Forwarded Message -------- >>>>>> >>>>>> Subject: Re: Disappointing packets-per-second performance results on a >>>>>> >>>>>> Dell,PE R530 >>>>>> >>>>>> Date: Tue, 31 Jan 2017 13:53:15 -0400 >>>>>> >>>>>> From: Jordan Caraballo <jordancaraball...@gmail.com> >>>>>> >>>>>> To: Slawa Olhovchenkov <s...@zxy.spb.ru> >>>>>> >>>>>> CC: freebsd-net@freebsd.org >>>>>> >>>>>> >>>>>> This are the most recent stats. No advances so far. The system has >>>>>> -Current right now. >>>>>> >>>>>> Any help or feedback would be appreciated. >>>>>> Hardware Configuration: >>>>>> Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E52695 CPU's, 18 cores per >>>>>> cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. >>>>>> >>>>>> BIOS tweaks: >>>>>> Hyperthreading (or Logical Processors) is turned off. >>>>>> loader.conf >>>>>> # Chelsio Modules >>>>>> t4fw_cfg_load="YES" >>>>>> t5fw_cfg_load="YES" >>>>>> if_cxgbe_load="YES" >>>>>> rc.conf >>>>>> # Gateway Configuration >>>>>> ifconfig_cxl0="inet 172.16.1.1/24" >>>>>> ifconfig_cxl1="inet 172.16.2.1/24" >>>>>> gateway_enable="YES" >>>>>> >>>>>> Last Results: >>>>>> packets errs idrops bytes packets errs bytes colls drops >>>>>> 2.7M 0 2.0M 1.4G 696k 0 368M 0 0 >>>>>> 2.7M 0 2.0M 1.4G 686k 0 363M 0 0 >>>>>> 2.6M 0 2.0M 1.4G 668k 0 353M 0 0 >>>>>> 2.7M 0 2.0M 1.4G 661k 0 350M 0 0 >>>>>> 2.8M 0 2.1M 1.5G 697k 0 369M 0 0 >>>>>> 2.8M 0 2.1M 1.4G 684k 0 361M 0 0 >>>>>> 2.7M 0 2.1M 1.4G 674k 0 356M 0 0 >>>>>> >>>>>> root@router1:~ # vmstat -i >>>>>> >>>>>> interrupt total rate >>>>>> irq9: acpi0 73 0 >>>>>> irq18: ehci0 ehci1 1155973 3 >>>>>> cpu0:timer 3551157 10 >>>>>> cpu29:timer 9303048 27 >>>>>> cpu9:timer 71693455 207 >>>>>> cpu16:timer 9798380 28 >>>>>> cpu18:timer 9287094 27 >>>>>> cpu26:timer 9342495 27 >>>>>> cpu20:timer 9145888 26 >>>>>> cpu8:timer 9791228 28 >>>>>> cpu22:timer 9288116 27 >>>>>> cpu35:timer 9376578 27 >>>>>> cpu30:timer 9396294 27 >>>>>> cpu23:timer 9248760 27 >>>>>> cpu10:timer 9756455 28 >>>>>> cpu25:timer 9300202 27 >>>>>> cpu27:timer 9227291 27 >>>>>> cpu14:timer 10083548 29 >>>>>> cpu28:timer 9325684 27 >>>>>> cpu11:timer 9906405 29 >>>>>> cpu34:timer 9419170 27 >>>>>> cpu31:timer 9392089 27 >>>>>> cpu33:timer 9350540 27 >>>>>> cpu15:timer 9804551 28 >>>>>> cpu32:timer 9413182 27 >>>>>> cpu19:timer 9231505 27 >>>>>> cpu12:timer 9813506 28 >>>>>> cpu13:timer 10872130 31 >>>>>> cpu4:timer 9920237 29 >>>>>> cpu2:timer 9786498 28 >>>>>> cpu3:timer 9896011 29 >>>>>> cpu5:timer 9890207 29 >>>>>> cpu6:timer 9737869 28 >>>>>> cpu7:timer 9790119 28 >>>>>> cpu1:timer 9847913 28 >>>>>> cpu21:timer 9192561 27 >>>>>> cpu24:timer 9300259 27 >>>>>> cpu17:timer 9786186 28 >>>>>> irq264: mfi0 151818 0 >>>>>> irq266: bge0 30466 0 >>>>>> irq272: t5nex0:evt 4 0 >>>>>> Total 402604945 1161 >>>>>> top -PHS >>>>>> last pid: 18557; load averages: 2.58, 1.90, 0.95 up 4+00:39:54 18:30:46 >>>>>> 231 processes: 40 running, 126 sleeping, 65 waiting >>>>>> CPU 0: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 2: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 4: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 5: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 6: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle >>>>>> CPU 7: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 8: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 9: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 10: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 11: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 12: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 13: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 14: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 15: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 16: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 17: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 18: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 19: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 20: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 21: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 22: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 23: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 24: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 25: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 26: 0.0% user, 0.0% nice, 0.0% system, 59.6% interrupt, 40.4% idle >>>>>> CPU 27: 0.0% user, 0.0% nice, 0.0% system, 96.3% interrupt, 3.7% idle >>>>>> CPU 28: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 29: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 30: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 31: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 32: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 33: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> CPU 34: 0.0% user, 0.0% nice, 0.0% system, 100% interrupt, 0.0% idle >>>>>> CPU 35: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle >>>>>> Mem: 15M Active, 224M Inact, 1544M Wired, 393M Buf, 29G Free >>>>>> Swap: 3881M Total, 3881M Free >>>>>> >>>>>> pmcstat -R sample.out -G - | head >>>>>> @ CPU_CLK_UNHALTED_CORE [159 samples] >>>>>> >>>>>> 39.62% [63] acpi_cpu_idle_mwait @ /boot/kernel/kernel >>>>>> 100.0% [63] acpi_cpu_idle >>>>>> 100.0% [63] cpu_idle_acpi >>>>>> 100.0% [63] cpu_idle >>>>>> 100.0% [63] sched_idletd >>>>>> 100.0% [63] fork_exit >>>>>> >>>>>> 17.61% [28] cpu_idle @ /boot/kernel/kernel >>>>>> >>>>>> root@router1:~ # pmcstat -R sample0.out -G - | head >>>>>> @ CPU_CLK_UNHALTED_CORE [750 samples] >>>>>> >>>>>> 31.60% [237] acpi_cpu_idle_mwait @ /boot/kernel/kernel >>>>>> 100.0% [237] acpi_cpu_idle >>>>>> 100.0% [237] cpu_idle_acpi >>>>>> 100.0% [237] cpu_idle >>>>>> 100.0% [237] sched_idletd >>>>>> 100.0% [237] fork_exit >>>>>> >>>>>> 10.67% [80] cpu_idle @ /boot/kernel/kernel >>>>>> >>>>>> On 03/01/17 13:46, Slawa Olhovchenkov wrote: >>>>>> >>>>>> On Tue, Jan 03, 2017 at 12:35:42PM -0400, Jordan Caraballo wrote: >>>>>> >>>>>> >>>>>> We recently tested a Dell R530 with a Chelsio T580 card, under FreeBSD >>>>>> 10.3, 11.0, -STABLE and -CURRENT, and Centos 7. >>>>>> >>>>>> Based on our research, including netmap-fwd and with the routing >>>>>> improvements project (https://wiki.freebsd.org/ProjectsRoutingProposal), >>>>>> we hoped for packets-per-second (pps) in the 5+ million range, or even >>>>>> higher. >>>>>> >>>>>> Based on prior testing (http://marc.info/?t=140604252400002&r=1&w=2), we >>>>>> expected 3-4 Million to be easily obtainable. >>>>>> >>>>>> Unfortunately, our current results top out at no more than 1.5 M (64 >>>>>> bytes length packets) with FreeBSD, and >>>>>> surprisingly around 3.2 M (128 bytes length packets) with Centos 7, and >>>>>> we are at a loss as to why. >>>>>> >>>>>> Server Description: >>>>>> Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E52695 CPU's, 18 cores per >>>>>> cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot. >>>>>> >>>>>> ** Can this be a lack in support issue related to the R530's hardware? ** >>>>>> >>>>>> Any help appreciated! >>>>>> >>>>>> What hardware configuration? >>>>>> What BIOS setting? >>>>>> What loader.conf/sysctl.conf setting? >>>>>> What `vmstat -i`? >>>>>> What `top -PHS`? >>>>>> what >>>>>> ==== >>>>>> pmcstat -S CPU_CLK_UNHALTED_CORE -l 10 -O sample.out >>>>>> pmcstat -R sample.out -G out.txt >>>>>> pmcstat -c 0 -S CPU_CLK_UNHALTED_CORE -l 10 -O sample0.out >>>>>> pmcstat -R sample0.out -G out0.txt >>>>>> ==== >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
