Which ones of these hit the line rate comfortably?
-a On 11 August 2016 at 15:35, Ben RUBSON <ben.rub...@gmail.com> wrote: > >> On 11 Aug 2016, at 18:36, Adrian Chadd <adrian.ch...@gmail.com> wrote: >> >> Hi! >> >> mlx4_core0: <mlx4_core> mem >> 0xfbe00000-0xfbefffff,0xfb000000-0xfb7fffff irq 64 at device 0.0 >> numa-domain 1 on pci16 >> mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 >> (Aug 11 2016) >> >> so the NIC is in numa-domain 1. Try pinning the worker threads to >> numa-domain 1 when you run the test: >> >> numactl -l first-touch-rr -m 1 -c 1 ./test-program >> >> You can also try pinning the NIC threads to numa-domain 1 versus 0 (so >> the second set of CPUs, not the first set.) >> >> vmstat -ia | grep mlx (get the list of interrupt thread ids) >> then for each: >> >> cpuset -d 1 -x <irq id> >> >> Run pcm-memory.x each time so we can see the before and after effects >> on local versus remote memory access. >> >> Thanks! > > Adrian, here are the results : > > > > Idle system : > http://pastebin.com/raw/K1iMVHVF > > > > No pinning : > http://pastebin.com/raw/w5KuexQ3 > CPU : http://pastebin.com/raw/8zgRaazN > > numactl -l fixed-domain-rr -m 1 -c 1 : > http://pastebin.com/raw/VWweYF9H > CPU : http://pastebin.com/raw/QjaVH32X > > numactl -l fixed-domain-rr -m 0 -c 0 : > http://pastebin.com/raw/71hfGJdw > CPU : http://pastebin.com/raw/hef058Na > > numactl -l fixed-domain-rr -m 1 -c 1 > + cpuset -l <DOM1_CPU> -x <IRQ> : > http://pastebin.com/raw/nEQkgMK2 > CPU : http://pastebin.com/raw/R652KAdJ > > numactl -l fixed-domain-rr -m 0 -c 0 > + cpuset -l <DOM0_CPU> -x <IRQ> : > http://pastebin.com/raw/GdYJHyae > CPU : http://pastebin.com/raw/Ggfx9uF9 > > > > No pinning, default kernel (no NUMA option) : > http://pastebin.com/raw/iQ2u8d8k > CPU : http://pastebin.com/raw/Xr77KpcM > > default kernel (no NUMA option) > + cpuset -l <DOM0_CPU> > + cpuset -l <DOM0_CPU> -x <IRQ> : > http://pastebin.com/raw/VBWg4SZs > > default kernel (no NUMA option) > + cpuset -l <DOM1_CPU> > + cpuset -l <DOM1_CPU> -x <IRQ> : > http://pastebin.com/raw/SrJLZxuT > > > > No pinning, default kernel (no NUMA option), NUMA BIOS disabled : > http://pastebin.com/raw/P5LrUASN > > > > I would say : > - FreeBSD <= 10.3 : disable NUMA in BIOS > - FreeBSD >= 11 : disable NUMA in BIOS or enable NUMA in kernel. > But let's wait your analysis :) > > > > Ben > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
