Initiating an xm shutdown command to a centos 7 can VM cause a
Hypervisor kernel panic.
Fortunately I can reproduce this bug for every second shutdown
(statistically), this HV was pulled out from production environment, so
I can test anything on it from now.
Running: xm shutdown p39glp9m68muq2
after some seconds, HV kernel panic
Dom0 Operating system:
Linux c2-node08 3.10.55-11.el6.centos.alt.x86_64 #1 SMP Fri Sep 26
19:08:24 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
DomU Operating system (guest centos7):
Linux centos7memtest 3.10.0-123.20.1.el7.onapp.x86_64 #1 SMP Fri Feb 6
14:54:22 EET 2015 x86_64 x86_64 x86_64 GNU/Linux
HW:
CPU: Intel Xeon E5645 @ 2.40 Ghz
Chassis/Motherboard: Dell PowerEdge R410
Memory: 48GB (4x16GB HMT42GR7BMR4A-G7)
Disk: INTEL SSDSA2CT04 - 40GB
NIC: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
We're using a SAN network, 2x2 multipath iscsi, with a EMC VNX 5300
storage. iSCSI is connected via 2x10 gbit, through two Cisco Nexus 5548
switches.
Grub config:
kernel /boot/xen.gz dom0_mem=3145728 dom0_max_vcpus=6 log_lvl=all
guest_loglvl=all noreboot=true
module /boot/vmlinuz-3.10.55-11.el6.centos.alt.x86_64 ro
root=UUID=796d645a-c97a-4574-a4fc-716cbeb7247e rd_NO_LUKS rd_NO_LVM
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto
KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM pcie_asmp=off
module /boot/initramfs-3.10.55-11.el6.centos.alt.x86_64.img
We are using these extra settings at startup
ethtool -K tge1 tso off gso off lro off
ethtool -K tge2 tso off gso off lro off
sysctl -w vm.min_free_kbytes=262144
sysctl -w kernel.sem="250 32000 100 512"
Dmesg output from dom0 is attached as dom0_dmesg.txt
xl dmesg output from dom0 is attached as dom0_xl_dmesg.txt
Instead of searial we are using netconsole, but it gives us a good
output. Attached as netconsole_log.txt. But I'm putting here the top of
the call stack:
[<ffffffffa02932c8>] ? iscsi_tcp_recv_skb+0x1b8/0x3c0 [libiscsi_tcp]
[<ffffffffa02adfe9>] iscsi_sw_tcp_recv+0x49/0xe0 [iscsi_tcp]
[<ffffffff8156ed75>] tcp_read_sock+0x95/0x1e0
[<ffffffff81063b87>] ? local_bh_enable+0x27/0xa0
[<ffffffffa02adfa0>] ? iscsi_sw_tcp_state_change+0xd0/0xd0 [iscsi_tcp]
[<ffffffffa02ae3d7>] iscsi_sw_tcp_data_ready+0x57/0xd8 [iscsi_tcp]
[<ffffffff8157739d>] ? tcp_try_rmem_schedule+0x6d/0x130
[<ffffffff81577e6f>] tcp_data_queue+0x37f/0x5c0
[<ffffffff8157aaa9>] tcp_rcv_established+0x319/0
If it's required I can configure a serial link to a HV, to catch a
better output.
Dmesg output from domU is attached as centos7_dmesg.txt
Xen version: 4.2.4-33.el6, full xen info output is attached as xen_info.txt
Probably it's related is iSCSI, so I'm putting here some info about the
network and iSCSI settings:
Interfaces:
4: tge1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP
qlen 1000
link/ether a0:36:9f:2a:70:90 brd ff:ff:ff:ff:ff:ff
5: tge2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP
qlen 1000
link/ether a0:36:9f:2a:70:92 brd ff:ff:ff:ff:ff:ff
6: tge1.20@tge1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP
link/ether a0:36:9f:2a:70:90 brd ff:ff:ff:ff:ff:ff
7: tge1.12@tge1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc
noqueue state UP
link/ether a0:36:9f:2a:70:90 brd ff:ff:ff:ff:ff:ff
8: tge1.4@tge1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue
state UP
link/ether a0:36:9f:2a:70:90 brd ff:ff:ff:ff:ff:ff
9: tge2.5@tge2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue
state UP
link/ether a0:36:9f:2a:70:92 brd ff:ff:ff:ff:ff:ff
iSCSI traffic is sent via the tge1.4 and tge2.5 VLAN interfaces
8: tge1.4@tge1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue
state UP
link/ether a0:36:9f:2a:70:90 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.18/24 brd 10.0.1.255 scope global tge1.4
valid_lft forever preferred_lft forever
inet6 fe80::a236:9fff:fe2a:7090/64 scope link
valid_lft forever preferred_lft forever
9: tge2.5@tge2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue
state UP
link/ether a0:36:9f:2a:70:92 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.18/24 brd 10.0.2.255 scope global tge2.5
valid_lft forever preferred_lft forever
inet6 fe80::a236:9fff:fe2a:7092/64 scope link
valid_lft forever preferred_lft forever
Multipathd is working with the following config:
devices {
device {
vendor "DGC"
product "*"
product_blacklist "LUNZ"
path_grouping_policy "group_by_prio"
getuid_callout "/sbin/scsi_id -g -u -d /dev/%n"
path_selector "round-robin 0"
features "1 queue_if_no_path"
hardware_handler "1 alua"
prio "alua"
path_checker "emc_clariion"
no_path_retry 60
failback immediate
rr_weight uniform
rr_min_io 1000
}
}
iSCSI config:
iscsid.startup = /etc/rc.d/init.d/iscsid force-start
node.startup = automatic
node.leading_login = No
node.session.timeo.replacement_timeout = 10
node.conn[0].timeo.login_timeout = 5
node.conn[0].timeo.logout_timeout = 5
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.session.err_timeo.abort_timeout = 10
node.session.err_timeo.lu_reset_timeout = 10
node.session.err_timeo.tgt_reset_timeout = 10
node.session.initial_login_retry_max = 8
node.session.cmds_max = 512
node.session.queue_depth = 32
node.session.xmit_thread_priority = -20
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
node.conn[0].iscsi.HeaderDigest = None
node.session.nr_sessions = 1
node.session.iscsi.FastAbort = Yes
XM config for the virtual machine:
bootloader = "/usr/bin/pygrub"
vcpus = "1"
memory = "400"
name = "p39glp9m68muq2"
vif = [ "mac=00:16:3e:84:XX:XX, bridge=x0evss6g1ztoa4, ip=XX.XX.XX.XX,
vifname=gh4txstv16yoaw, rate=100Mb/s" ]
disk = [ "phy:/dev/onapp-p65uo6l3rgns6n/x5c58a6aj8cgiw,xvda1,w",
"phy:/dev/onapp-p65uo6l3rgns6n/s89aa4a3m3uzi2,xvda2,w" ]
vfb = [ "type=vnc,vncpasswd=lysuad" ]
Thanks for all,
- Armin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
