On Thu, 23 Jul 2015 11:14:22 +0100 Gregory Farnum wrote: > Your note that dd can do 2GB/s without networking makes me think that > you should explore that. As you say, network interrupts can be > problematic in some systems. The only thing I can think of that's been > really bad in the past is that some systems process all network > interrupts on cpu 0, and you probably want to make sure that it's > splitting them across CPUs. >
An IRQ overload would be very visible with atop. Splitting the IRQs will help, but it is likely to need some smarts. As in, irqbalance may spread things across NUMA nodes. A card with just one IRQ line will need RPS (Receive Packet Steering), irqbalance can't help it. For example, I have a compute node with such a single line card and Quad Opterons (64 cores, 8 NUMA nodes). The default is all interrupt handling on CPU0 and that is very little, except for eth2. So this gets a special treatment: --- echo 4 >/proc/irq/106/smp_affinity_list --- Pinning the IRQ for eth2 to CPU 4 by default --- echo f0 > /sys/class/net/eth2/queues/rx-0/rps_cpus --- giving RPS CPUs 4-7 to work with. At peak times it needs more than 2 cores, otherwise with this architecture just using 4 and 5 (same L2 cache) would be better. Regards, Christian -- Christian Balzer Network/Systems Engineer ch...@gol.com Global OnLine Japan/Fusion Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
