Am 13.10.2023 um 20:56 schrieb Paul Leiber:
Hi Xen developers list,
TL;DR:
------
Causing certain web server traffic on a secondary VLAN on Raspberry Pi
under vanilla Debian/UEFI in combination with Xen leads to complete
system reboot (watchdog triggering for Dom0). Other strange things are
happening.
Description:
----------
I recently set up Xen (self compiled, Version 4.18-rc) on a Raspberry Pi
4B (on vanilla Debian Bookworm, UEFI boot mode). Until some time ago,
everything worked well with Dom0, one DomU and one bridge.
Then I wanted to actually make use of the virtualization and started to
set up a second Debian Bookworm DomU (using xen-create-image) for
monitoring my systems with zabbix (a webserver based system monitoring
solution). The bridge used for this setup was the device bridging the
hardware NIC. I installed zabbix, set it up, and everything went well, I
could access the web interface without any problem.
Then I set up VLANs (initally using VLAN numbers 1 and 2) to separate
network traffic between the DomUs. I made the existing device bridge
VLAN 1 (bridge 1) and created a secondary device for bridging VLAN 2
(bridge 2). Using only bridge 1 / VLAN 1 everything works well, I can
access the zabbix web interface without any noticeable issue. After
switching the zabbix DomU to VLAN 2 / bridge 2, everything seemingly
keeps on working well, I can ping different devices in my network from
the zabbix DomU and vice versa, I can ssh into the machine.
However, as soon as I remotely access the zabbix web interface, the
complete system (DomUs and Dom0) becomes unresponsive and reboots after
some time (usually seconds, sometimes 1-2 minutes). The reboot is
reliably reproducable.
I didn't see any error message in any log (zabbix, DomU syslog, Dom0
syslog) except for the following lines immediately before the system
reboots on the Xen serial console:
(XEN) Watchdog timer fired for domain 0
(XEN) Hardware Dom0 shutdown: watchdog rebooting machine
As soon as I change the bridge to bridge 1 (with or without VLAN setup),
the web interface is accessible again after booting the zabbix DomU, no
reboots.
So I assume that causing specific traffic on the virtual NIC when using
a VLAN setup with more than one VLAN under Xen makes the Dom0 system
hard crash. Of course, there might be other causes that I'm not aware
of, but to me, this seems to be the most likely explanation right now.
What I tried:
-------------
1. I changed the VLAN numbers. First to 101, 102, 103 etc. This was when
I noticed another strange thing: VLANs with numbers >99 simply don't
work on my Raspberry Pi under Debian, with or without Xen. VLAN 99
works, VLAN 100 (or everything else >99 that I tried) doesn't work. If I
choose a number >99, the VLAN is not configured, "ip a" doesn't list it.
Other Debian systems on x64 architecture don't show this behavior,
there, it was no problem to set up VLANs > 99. Therefore, I've changed
the VLANs to 10, 20, 30 etc., which worked. But it didn't solve the
initial problem of the crashing Dom0 and DomUs.
2. Different bridge options, without noticable effect:
bridge_stp off # dont use STP (spanning tree proto)
bridge_waitport 0 # dont wait for port to be available
bridge_fd 0 # no forward delay
3. Removing IPv6: No noticable effect.
4. Network traffic analysis: Now, here it becomes _really_ strange. I
started tcpdumps on Dom0, and depending on on which interface/bridge
traffic was logged, the problem went away, meaning, the DomU was running
smoothly for hours, even when accessing the zabbix web interface.
Stopping the log makes the system crash (as above, after seconds up to
1-2 minutes) reproducably if I access the zabbix web interface.
Logging enabcm6e4ei0 (NIC): no crashes
Logging enabcm6e4ei0.10 (VLAN 10): instant crash
Logging enabcm6e4ei0.20 (VLAN 20): no crashes
Logging xenbr0 (on VLAN 10): instant crash
Logging xenbr1 (on VLAN 20): no crashes
I am clinging to the thought that there must be a rational explanation
for why logging the traffic on certain interfaces/bridges should avoid
the crash of the complete system, while logging other interfaces/bridges
doesn't. I myself can't think of one.
I checked the dumps of enabcm6e4ei0.10 and xenbr0 (where the system
crashes) with wireshark, nothing sticks out to me (but I am really no
expert in analyzing network traffic). Dumps can be provided.
5. Watchdog: I tried to dig deeper into the cause for the watchdog
triggering. However, I didn't find any useful documentation on the web
on how the watchdog works or how to enable logging.
6. Eliminating Xen as cause: I booted the Debian system (which in Xen
setup would be Dom0) without Xen and set it up to use the VLAN 20 bridge
(the same that leads to a reboot when using it in the DomU) as primary
network interface. Everything seemed to be working, I could download
large files from the internet without any problem. Setting up Zabbix on
the base Debian system showed that the same setup (VLANs 10 and 20,
bridges 1 and 2, using bridge 2 as interface for Zabbix) without Xen is
working reliably, no reboots. This points to some Xen related component
being the root cause, I think.
7. Eliminating Apache as root cause: Reloading the Apache starting page
hosted on DomU several times per second didn't lead to a reboot.
8. Recompiling Xen: Independent of which Xen master branch version I was
using (all 4.18), the behavior was the same. I didn't get Xen working on
ARM64/UEFI in version 4.17.
Current situation:
------------------
I am out of ideas what to do next. Everything that was recommended to me
on xen-users didn't lead to significant insight or solve the problem.
I'd appreciate any hints how to troubleshoot this and/or how to proceed
otherwise.
O.k., let's try to break that issue down.
Firstly, how can I get more information on why the Xen watchdog
triggers? Is there documentation? Are there any logs? I couldn't find
anything useful with my search skills.
Thanks,
Paul
