On 13.11.24 18:25, Elliott Mitchell wrote:
On Tue, Jul 09, 2024 at 08:36:18AM +0000, Andrei Semenov wrote:After some investigations we notices a huge performances drop (perfs divided by factor of 5) starting from 5.10.88 Linux kernel version on the AMD EPYC platforms. The patch introduced in this kernel version that allows to pinpoint the buggy behavior is : “xen/netfront: harden netfront against event channel storms” d31b3379179d64724d3bbfa87bd4ada94e3237de The patch basically binds the network frontend to the `xen_lateeoi_chip` irq_chip (insead of `xen_dynamic_chip`) which allows to its clients to inform the chip if spurious interrupts are detected and so the delay in interrupt treatment is introduced by the chip.I worry I'm being knave here. For the heck of it, I took a glance at b27d47950e48. If my understanding is correct, b27d47950e48 is making a very basic (and wrong) assumption about timing/latency. In particular any time either side receive an event, it will handle X # of incoming payloads and Y # of acknowledged outgoing payloads. As such if X + Y > 1, then up to X + Y - 1 spurious events may be detected. The issue is there is no synchronization between the event channel and the work queues. In particular the network back end could legitimately generate: work0 signal0 work1 signal1 work2 signal2 work3 signal3 Whereas the network front end may handle this as: event0 work0 work1 work2 work3 event1 event2 event3 Where b27d47950e48 would interpret events 1-3 as spurious, even though they're perfectly legitimate. The same phenomenon could occur in both directions and also with the Xen block devices.
No. For one, as long as event0 isn't EOI'd, the other events would just be merged into a single one. Additionally, as long as work0 isn't acknowledged by incrementing the consumer index, additional queued work items should NOT result in additional events being sent. An event is only sent if a work item is queued to a ring buffer with consumer == producer.
Ultimately how is the network portion of XSA-391 any different from any other network DoS? If an interrupt is generated for every single packet of a series of runt frames, there will be heavy processor use for little network traffic.
The problem is that a steady stream of events could keep the other side in IRQ handling for arbitrary amount of times, leading to hangups.
AMD systems may fair worse than Intel systems due to differing cache coherence behavior/latency. Perhaps AMD's NUMA implementation adds some latency. (huh, suddenly the RAID1 issue comes to mind) Hopefully I'm not making knave speculation here. Might this be the simplest of issues, just it was missed due to being too obvious?
I don't agree with your analysis, see above. Juergen
OpenPGP_0xB0DE9DD628BF132F.asc
Description: OpenPGP public key
OpenPGP_signature.asc
Description: OpenPGP digital signature
