On 02/24/2014 02:23 PM, Alexander Duyck wrote: > On 02/24/2014 10:51 AM, Prarit Bhargava wrote: >> The ixgbe driver makes some assumptions about the layout of cpus in the >> system which are not always correct given a particular system layout. The >> ixgbe driver allocates one MSI/cpu for queue usage but the code does not take >> into account that devices are located on NUMA nodes and that the cpus in a >> node >> are not contiguous. >> >> These issues were found while doing cpu hotplug testing, however, both of >> these >> issues can lead to obvious system performance issues as they defeat the >> purpose of having one MSI processing a queue per cpu. >> >> Cc: Jeff Kirsher <[email protected]> >> Cc: Jesse Brandeburg <[email protected]> >> Cc: Bruce Allan <[email protected]> >> Cc: Carolyn Wyborny <[email protected]> >> Cc: Don Skidmore <[email protected]> >> Cc: Greg Rose <[email protected]> >> Cc: Alex Duyck <[email protected]> >> Cc: John Ronciak <[email protected]> >> Cc: Mitch Williams <[email protected]> >> Cc: "David S. Miller" <[email protected]> >> Cc: [email protected] >> Cc: [email protected] >> Cc: [email protected] >> >> Prarit Bhargava (2): >> ixgbe, make interrupt allocations NUMA aware >> ixgbe, don't assume mapping of numa node cpus >> >> drivers/net/ethernet/intel/ixgbe/ixgbe.h | 2 ++ >> drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c | 44 >> ++++++++++++++++++------ >> drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 6 ++-- >> drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.c | 5 +-- >> 4 files changed, 42 insertions(+), 15 deletions(-) >> > > This is a step in the right direction but totally defeats the purpose of > ATR. With this change we might as well defeature ATR all together since
ATR? First hit on google is Americans for Tax Reform ;) > things are now back to RSS w/ NUMA specific allocations which is what we > had a couple of years ago. The code as it is written now would be a > better for for igb which doesn't have ATR than ixgbe. The big(ger) problem here is that the ixgbe (and other drivers IIUC) do not do a good job of handling MSIs, making sure they are launched on the right cpus, and cleaning up during cpu hotplug operations. This code looks like it needs a bit of work so your advice is appreciated. > > ATR is supposed to map 1:1 queues to CPUs. The problem is RSS is also a > factor and not especially smart or NUMA aware. The ideal solution would > be to allocate the first N CPUs, where N is the number in the local node > for ATR/RSS. Okay ... I'll look into that. Then map all other queues as ATR with a 1:1 mapping to CPUs. > Hmmm ... but what if off-node CPUs cannot reach the device? Part of the puzzle here is that ACPI may be not only telling us that the device is on a specific node, but that the device is physically separated on a root bus. P. ------------------------------------------------------------------------------ Flow-based real-time traffic analytics software. Cisco certified tool. Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer Customize your own dashboards, set traffic alerts and generate reports. Network behavioral analysis & security monitoring. All-in-one tool. http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
