Thanks for this.. turned off netflow export.. and it dropped our qfp load from 44% to 18%. ugh..
--- Colin Legendre On Thu, Dec 9, 2021 at 4:22 AM Brian Turnbow via NANOG <nanog@nanog.org> wrote: > > > > On 11/26/2021 1:09 PM, Colin Legendre wrote: > > > Hi, > > > > > > We have ... > > > > > > ASR1006 that has following cards... > > > 1 x ESP40 > > > 1 x SIP40 > > > 4 x SPA-1x10GE-L-V2 > > > 1 x 6TGE > > > 1 x RP2 > > > > > > We've been having latency and packet loss during peak periods... > > > > > > We notice all is good until we reach 50% utilization on output of... > > > > > > 'show platform hardware qfp active datapath utilization summary' > > > > > > Literally ... 47% good... 48% good... 49% latency to next hop goes > > > from 1ms to 15-20ms... 50% we see 1-2% packet-loss and 30-40ms > > > latency... 53% we see 60-70ms latency and 8-10% packet loss. > > > > > > Is this expected... the ESP40 can only really push 20G and then starts > > > to have performance issues? > > > > > He had a similar issue about 4 years ago. > We were showing packet loss and drops getting progressively worse and the > router was falling over when reaching about 70% of usage. > We could see the interface reliability go down and input errors due to > overruns on the interfaces. > Cisco blamed it on microburtst not being able to be handled under load. > > > "We were able to replicate this scenario in our lab as well. > QFP under high load generated input errors and overruns which in turn led > to unicast failures/ drops/ latency. > The issue is not consistent with QFP % utilization as sometimes with even > 80%+ traffic, we do not see the drops:" > > And recommended removing traffic or upgrading esp. > > One of our guys disabled nbar on the router and the problem disappeared. > I would suggest taking a look at what features you are using and if you > can try and disable them to see if it makes any impact. > We then upgraded esps and all has been fine since. > > Brian > >
