Hello,
I have performed some more thorough testing on 3 different machines to
illustrate the strange results with XL710.
Please note that all 3 systems were able to forward the traffic of Test
1 and Test 2 without packet loss when a 82599ES NIC was installed in the
same PCI slot as the XL710 in the tests below.
Here is the test setup and the test results:
## Test traffic
In all tests the t-rex traffic generator was used to generate traffic on
a XL710 card with the following parameters:
### Test 1
./t-rex-64 -f cap2/imix_1518.yaml -c 4 -d 60 -m 25 --flip
This resulted in a 60 second run with ~1.21 Gbps traffic on each of the
two interfaces with ~100000 packets per
second on each interface.
### Test 2
./t-rex-64 -f cap2/imix_1518.yaml -c 4 -d 60 -m 100 --flip
This resulted in a 60 second run with ~4.85 Gbps traffic on each of the
two interfaces with ~400000 packets per
second on each interface.
### Test 3
./t-rex-64 -f cap2/imix_1518.yaml -c 4 -d 60 -m 400 --flip
This resulted in a 60 second run with ~19.43 Gbps traffic on each of the
two interfaces with ~1600000 packets per
second on each interface.
## DPDK
On all systems a vanilla DPDK v16.11 testpmd was used with the following
parameters (PCI IDs differed between systems):
./build/app/testpmd -l 1,2 -w 0000:06:00.0 -w 0000:06:00.1 -- -i
## System 1
* Board: Supermicro X10SDV-TP8F
* CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 86
Model name: Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz
Stepping: 3
CPU MHz: 800.250
CPU max MHz: 2200.0000
CPU min MHz: 800.0000
BogoMIPS: 4399.58
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic
sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht
tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq
dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid
dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx
f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi
flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms
invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc
cqm_mbm_total cqm_mbm_local dtherm arat pln pts
* Memory channels: 2
* Memory: 2 * 8192 MB DDR4 @ 2133 MHz
* NIC firmware: FW 5.0 API 1.5 NVM 05.00.04 eetrack 80002505
* i40e version: 1.4.25-k
* OS: Ubuntu 16.04.1 LTS
* Kernel: 4.4.0-57-generic
* Kernel parameters: isolcpus=1,2,3,5,6,7 default_hugepagesz=1G
hugepagesz=1G hugepages=1
### Test 1
Mostly no packet loss. Sometimes ~10 packets missed of ~600000 on each
interface when testpmd was not started in
interactive mode.
### Test 2
100-300 packets of ~24000000 missed on each interface.
### Test 3
4000-5000 packets of ~96000000 missed on each interface.
## System 2
* Board: Supermicro X10SDV-7TP8F
* CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 86
Model name: 06/56
Stepping: 4
CPU MHz: 1429.527
CPU max MHz: 2300.0000
CPU min MHz: 800.0000
BogoMIPS: 3400.37
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 24576K
NUMA node0 CPU(s): 0-31
Flags: fpu vme de pse tsc msr pae mce cx8 apic
sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht
tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq
dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid
dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx
f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi
flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms
invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc
cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts
* Memory channels: 2
* Memory: 4 * 16384 MB DDR4 @ 2133 MHz
* NIC firmware: FW 5.0 API 1.5 NVM 05.00.04 eetrack 80002505
* i40e version: 1.4.25-k
* OS: Ubuntu 16.04.1 LTS
* Kernel: 4.4.0-57-generic
* Kernel parameters:
isolcpus=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31
default_hugepagesz=1G hugepagesz=1G hugepages=1
### Test 1
Mostly no packet loss of ~600000.
### Test 2
400000-500000 packets of ~24000000 missed on each interface.
### Test 3
1200000-1400000 packets of ~96000000 missed on each interface.
## System 3
* Board: Supermicro X9SRW-F
* CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 12
On-line CPU(s) list: 0-11
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Model name: Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
Stepping: 4
CPU MHz: 1200.253
CPU max MHz: 3900.0000
CPU min MHz: 1200.0000
BogoMIPS: 7000.29
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 12288K
NUMA node0 CPU(s): 0-11
Flags: fpu vme de pse tsc msr pae mce cx8 apic
sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht
tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq
dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca
sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand
lahf_lm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
xsaveopt dtherm arat pln pts
* Memory channels: 4
* Memory: 4 * 8192 MB DDR3 @ 1600 MHz
* NIC firmware: FW 5.0 API 1.5 NVM 05.00.04 eetrack 80002537
* i40e version: 1.4.25-k
* OS: Ubuntu 16.04.1 LTS
* Kernel: 4.4.0-57-generic
* Kernel parameters: default_hugepagesz=1G hugepagesz=1G hugepages=1
isolcpus=1-5,7-11
### Test 1
No packets lost.
### Test 2
No packets lost.
### Test 3
No packets lost.
Best regards,
Martin
On 03.01.17 13:18, Martin Weiser wrote:
> Hello,
>
> we are also seeing this issue on one of our test systems while it does
> not occur on other test systems with the same DPDK version (we tested
> 16.11 and current master).
>
> The system that we can reproduce this issue on also has a X552 ixgbe NIC
> which can forward the exact same traffic using the same testpmd
> parameters without a problem.
> Even if we install a 82599ES ixgbe NIC in the same PCI slot that the
> XL710 was in the 82599ES can forward the traffic without any drops.
>
> Like in the issue reported by Ilya all packet drops occur on the testpmd
> side and are accounted as 'imissed'. Increasing the number of rx
> descriptors only helps a little at low packet rates.
>
> Drops start occurring at pretty low packet rates like 100000 packets per
> second.
>
> Any suggestions would be greatly appreciated.
>
> Best regards,
> Martin
>
>
>
> On 22.08.16 14:06, Ilya Maximets wrote:
>> Hello, All.
>>
>> I've faced with a really bad situation with packet drops on a small
>> packet rates (~45 Kpps) while using XL710 NIC with i40e DPDK driver.
>>
>> The issue was found while testing PHY-VM-PHY scenario with OVS and
>> confirmed on PHY-PHY scenario with testpmd.
>>
>> DPDK version 16.07 was used in all cases.
>> XL710 firmware-version: f5.0.40043 a1.5 n5.04 e2505
>>
>> Test description (PHY-PHY):
>>
>> * Following cmdline was used:
>>
>> # n_desc=2048
>> # ./testpmd -c 0xf -n 2 --socket-mem=8192,0 -w 0000:05:00.0 -v \
>> -- --burst=32 --txd=${n_desc} --rxd=${n_desc} \
>> --rxq=1 --txq=1 --nb-cores=1 \
>> --eth-peer=0,a0:00:00:00:00:00 --forward-mode=mac
>>
>> * DPDK-Pktgen application was used as a traffic generator.
>> Single flow generated.
>>
>> Results:
>>
>> * Packet size: 128B, rate: 90% of 10Gbps (~7.5 Mpps):
>>
>> On the generator's side:
>>
>> Total counts:
>> Tx : 759034368 packets
>> Rx : 759033239 packets
>> Lost : 1129 packets
>>
>> Average rates:
>> Tx : 7590344 pps
>> Rx : 7590332 pps
>> Lost : 11 pps
>>
>> All of this dropped packets are RX-dropped on testpmd's side:
>>
>> +++++++++++++++ Accumulated forward statistics for all
>> ports+++++++++++++++
>> RX-packets: 759033239 RX-dropped: 1129 RX-total:
>> 759034368
>> TX-packets: 759033239 TX-dropped: 0 TX-total:
>> 759033239
>>
>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> At the same time 10G NIC with IXGBE driver works perfectly
>> without any packet drops in the same scenario.
>>
>> Much worse situation with PHY-VM-PHY scenario with OVS:
>>
>> * testpmd application used inside guest to forward incoming packets.
>> (almost same cmdline as for PHY-PHY)
>>
>> * For packet size 256 B on rate 1% of 10Gbps (~45 Kpps):
>>
>> Total counts:
>> Tx : 1358112 packets
>> Rx : 1357990 packets
>> Lost : 122 packets
>>
>> Average rates:
>> Tx : 45270 pps
>> Rx : 45266 pps
>> Lost : 4 pps
>>
>> All of this 122 dropped packets can be found in rx_dropped counter:
>>
>> # ovs-vsctl get interface dpdk0 statistics:rx_dropped
>> 122
>>
>> And again, no issues with IXGBE on the exactly same scenario.
>>
>>
>> Results of my investigation:
>>
>> * I found that all of this packets are 'imissed'. This means that rx
>> descriptor ring was overflowed.
>>
>> * I've modified i40e driver to check the real number of free descriptors
>> that was not still filled by the NIC and found that HW fills
>> rx descriptors with uneven rate. Looks like it fills them using
>> a huge batches.
>>
>> * So, root cause of packet drops with XL710 is somehow uneven rate of
>> filling of the hw rx descriptors by the NIC. This leads to exhausting
>> of rx descriptors and packet drops by the hardware. 10G IXGBE NIC
>> works
>> more smoothly and driver is able to refill hw ring with rx descriptors
>> in time.
>>
>> * The issue becomes worse with OVS because of much bigger latencies
>> between 'rte_eth_rx_burst()' calls.
>>
>> The easiest solution for this problem is to increase number of RX
>> descriptors.
>> Increasing up to 4096 eliminates packet drops but decreases the performance
>> a lot:
>>
>> For OVS PHY-VM-PHY scenario by 10%
>> For OVS PHY-PHY scenario by 20%
>> For tespmd PHY-PHY scenario by 17% (22.1 Mpps --> 18.2 Mpps for 64B
>> packets)
>>
>> As a result we have a trade-off between zero drop rate on small packet rates
>> and
>> the higher maximum performance that is very sad.
>>
>> Using of 16B descriptors doesn't really help with performance.
>> Upgrading the firmware from version 4.4 to 5.04 didn't help with drops.
>>
>> Any thoughts? Can anyone reproduce this?
>>
>> Best regards, Ilya Maximets.
