Ah sorry, didn’t see your reply. I’ll take a look. It doesn’t appear to 
cherry-pick cleanly to stable/1908, so it probably was among those that 500 :(

--a

> On 5 Jun 2020, at 19:54, Andrew Yourtchenko via lists.fd.io 
> <[email protected]> wrote:
> 
> Hi Mike,
> 
> Any chances you might do “git bisect” on this ? Should take about 8-9 
> iterations I suppose...
> 
> You could use v20.01-rc0 label as the “git bisect bad” start and the 
> v20.09-rc0 as the “git bisect good” start.
> 
> I am currently working on integrating a bunch of “fix” commits that happened 
> between the 19.08.1 and the latest, but only about 1/3 of them (about 250) 
> can apply automatically, so I am analyzing the other 500. It is taking some 
> time, so if you could bisect - then I could just push that specific fix into 
> stable/1908 and get you going.
> 
> As for the tests - this type of testing won’t be covered in “make test”, that 
> is the per-patch test.
> But maybe there is a simpler way to catch it - so knowing which commit fixed 
> it would surely help... 
> 
> --a
> 
>>> On 4 Jun 2020, at 22:16, Bly, Mike via lists.fd.io 
>>> <[email protected]> wrote:
>>> 
>> 
>> Hello,
>>
>> We are observing a small percentage of frames being discarded in simple 
>> 2-port L2 xconnect setup when a constant, same frame, single (full duplex) 
>> traffic profile is offered to the system. The frames are being discarded due 
>> to a failed VLAN classification when all frames offered have the same VLAN 
>> present, i.e. send two sets of 1B of the same frame in two directions (A <-> 
>> B), see x% discarded due to random VLAN classification issues.
>>
>> We did not see this issue in v18.07.1. At the start of the year we upgraded 
>> to 19.08 and started seeing this issue during scale testing. We have been 
>> trying to root cause it and are at a point where we need some assistance. 
>> Moving from our integrated VPP solution to using stock VPP built in an 
>> Ubuntu container, we have found this issue to be present in all releases 
>> between 19.08 – 20.01, but appears fixed in 20.05. We are not in a position 
>> where we can immediately upgrade to v20.05, so we need a solution for the 
>> v19.08 code base, based on key changes v20.01 -> v20.05. As such, we are 
>> looking for guidance on potentially relevant changes made between v20.01 and 
>> v20.05.
>>
>> VPP configuration used:
>> create sub-interfaces TenGigabitEthernet19/0/0 100 dot1q 100
>> create sub-interfaces TenGigabitEthernet19/0/1 100 dot1q 100
>> set interface state TenGigabitEthernet19/0/0 up
>> set interface state TenGigabitEthernet19/0/0.100 up
>> set interface state TenGigabitEthernet19/0/1 up
>> set interface state TenGigabitEthernet19/0/1.100 up
>> set interface l2 xconnect TenGigabitEthernet19/0/0.100 
>> TenGigabitEthernet19/0/1.100
>> set interface l2 xconnect TenGigabitEthernet19/0/1.100 
>> TenGigabitEthernet19/0/0.100
>>
>> Traffic/setup:
>> Two traffic generator connections to 10G physical NICs, each connection 
>> having a single traffic stream, where all frames are the same
>> No NIC offloading being used, no RSS, single worker thread separate from 
>> master
>> 64B frames with fixed/cross-matching unicast L2 MAC addresses, non-IP Etype, 
>> incrementing payload
>> 1 billion frames full duplex, offered at max “lossless” throughput, e.g. 
>> approx. 36% of 10Gb/s for v20.05
>> “lossless” is maximum throughput allowed without observing “show interface” 
>> -> “rx-miss” statistics
>>
>> Resulting statistics:
>>
>> Working v18.07.1 with proper/expected “error” statistics:
>> vpp# show version
>> vpp v18.07.1
>>
>> vpp# show errors
>>    Count                    Node                  Reason
>> 2000000000                l2-output               L2 output packets
>> 2000000000                l2-input                L2 input packets
>>
>> Non-Working v20.01 with unexpected “error” statistics:
>> vpp# show version   
>> vpp v20.01-release
>>
>> vpp# show errors                                     
>>    Count                    Node                  Reason
>> 1999974332                l2-output               L2 output packets
>> 1999974332                l2-input                L2 input packets
>>      25668             ethernet-input             l3 mac mismatch         
>> <-- we should NOT be seeing these
>>
>> Working v20.05 with proper/expected “error” statistics:
>> vpp# show version
>> vpp v20.05-release
>>
>> vpp# show errors
>>    Count                    Node                  Reason
>> 2000000000                l2-output               L2 output packets
>> 2000000000                l2-input                L2 input packets
>>
>> Issue found:
>>
>> In eth_input_process_frame() calls to eth_input_get_etype_and_tags() are 
>> sometimes failing to properly parse/store the “etype” and/or “tag” values, 
>> which then results later on in failed VLAN classification and resultant “l3 
>> mac mismatch” discards due to parent L3 mode.
>>
>> Here is a sample debug profiling of the discards. We implement some 
>> down-n-dirty debug statistics as shown:
>> bad_l3_frm_offset[256] is showing which frame in “n_left” sequence of a 
>> given batch was discarded
>> bad_l3_batch_size[256] is showing the size of each batch of frames being 
>> processed when a discard occurs
>>
>> (gdb) p bad_l3_frm_offset
>> $1 = {1078, 1078, 1078, 1078, 0 <repeats 12 times>, 383, 383, 383, 383, 0 
>> <repeats 236 times>}
>>
>> (gdb) p bad_l3_batch_size
>> $2 = {0 <repeats 251 times>, 1424, 0, 0, 1356, 3064}
>>
>> I did manage to find the following thread, which seems to be possibly 
>> related to our issue: https://lists.fd.io/g/vpp-dev/message/15488 Sharing 
>> just in case it is in fact relevant.
>>
>> Finally, are VPP performance regressions monitoring/checking “vpp show 
>> errors” content? We are looking to understand how this may have gone 
>> unnoticed between v18.07.1 and 20.05 release efforts given the simplicity of 
>> the configuration and test stimulus.
>>
>> -Mike
>> 
> 
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#16665): https://lists.fd.io/g/vpp-dev/message/16665
Mute This Topic: https://lists.fd.io/mt/74679715/21656
Mute #vpp: https://lists.fd.io/mk?hashtag=vpp&subid=1480452
Mute #vnet: https://lists.fd.io/mk?hashtag=vnet&subid=1480452
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Reply via email to