(solution below) The time to send that extra byte over the wire is 8 nsec. It seems unlikely that would produce such a clean-cut difference, but I've been surprised before. :)
The "idle=poll" parameter sounds like a good idea for any application which tries to maintain real-time behavior. I'm putting that into place. Result: About the same. I believe I have C-states disabled, as the directories /sys/devices/system/cpu/cpuX/cpufreq/ are not in the filesystem. Just to be sure, I removed "idle=poll" and added "processor.max_cstate=1". Result: About the same as "idle=poll". I set the interrupt moderation parameter to 10 on both interfaces involved in the bridge with "ethtool -C <dev> rx-usecs 10"". Result: HUGE difference. This looks much better. I'm going to explore the adjustments available and I think this will solve the problem. Thank you, Alex, -Todd On Thu, May 21, 2015 at 7:55 AM, Alexander Duyck <alexander.du...@gmail.com> wrote: > On 05/20/2015 09:50 PM, Todd Bezenek wrote: >> >> I'm pushing 10,000 frames per second of UDP traffic through a Linux >> system with a bridge configured between two 1GbE ports. >> >> Iptables is installed and running, but the default rule is ACCEPT with >> no other rules. >> >> When I make the packets 1179 bytes in size (total size includes >> Ethernet header, etc.), I see the latency of most packets between 60 >> and 200 usec. When I change the size to 1180 bytes, I start seeing >> about 1% of the packets with latencies larger than 300 usec, and some >> as high as 500+ usec. >> >> Any ideas what buffer size, cache line count, TLB reach, etc. might be >> causing this dramatic change? >> >> Cut-and-paste to my terminal is not working, so limited info here: >> >> Kernel: 3.12.20 x86-64 >> Processor: Intel E3845 (Atom, 4 cores) >> Network interface driver: Intel igb >> >> Thank you for any helpful pointers. >> >> -Todd > > > The igb driver defaults to an adaptive interrupt moderation scheme. It is > possible that what you are seeing could be due to that, or it could possibly > be that the time between packets is becoming large enough that your CPU is > starting to go into deeper sleep states. > > My first suggestion would be to try disabling sleeps states beyond C1. To > do that you could boot your kernel with the parameter > "processor.max_cstate=1" or "idle=poll". Either one should prevent the CPU > from going into a sleep state but it will consume more power. > > You also might try modifying the interrupt moderation to a fixed value of 10 > or more via "ethtool -C <dev> rx-usecs <x>" to see if the behavior goes away > after that. If the device is the cause it might point to a possible glitch > in the interrupt moderation scheme or possibly lost interrupts for the > device or driver. > > - Alex > > -- Todd Bezenek beze...@gmail.com A people hire A people, B people hire C people. --Jim Gray -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
