On Wed, Apr 06, 2016 at 10:01:00PM +0200, Jesper Dangaard Brouer wrote: > On Wed, 6 Apr 2016 21:48:48 +0200 > Jesper Dangaard Brouer <bro...@redhat.com> wrote: > > If I do multiple flows, via ./pktgen_sample05_flow_per_thread.sh > > then I hit this strange 14.5Mpps limit (proto 17: 14505558 drops/s). > > And the RX 4x CPUs are starting to NOT use 100% in softirq, they have > > some cycles attributed to %idle. (I verified generator is sending at > > 24Mpps). ... > > If I change the program to not touch packet data (don't call > > load_byte()) then the performance increase to 14.6Mpps (single > > flow/cpu). And the RX CPU is mostly idle... mlx4_en_process_rx_cq() > > and page alloc/free functions taking the time.
Please try it with module param log_num_mgm_entry_size=-1 It should get to 20Mpps when bpf doesn't touch the packet. > Before someone else point out the obvious... I forgot to enable JIT. > Enable it:: > > # echo 1 > /proc/sys/net/core/bpf_jit_enable > > Performance increased to: 10.8Mpps (proto 17: 10819446 drops/s) > > Samples: 51K of event 'cycles', Event count (approx.): 56775706510 > Overhead Command Shared Object Symbol > + 55.90% ksoftirqd/7 [kernel.vmlinux] [k] sk_load_byte_positive_offset > + 10.71% ksoftirqd/7 [mlx4_en] [k] mlx4_en_alloc_frags ... > It is a very likely cache-miss in sk_load_byte_positive_offset(). yes, likely due to missing ddio as you said.
