Hey,
Since we upgraded to the 4.11 branch (currently 4.11.3) and virtual routers
have become HVM on XenServer/XCP-ng we have had problems with the virtual
routers randomly rebooting themselves. We still have some running in the older
paravirtualized mode and they seem to be fine (it may be that the Management
server can't communicate to these virtual routers since they are an older
template?). Other running Windows/Linux VMs are fine.
CloudStack Cluster: XCP-ng 7.5 (previously XenServer 6.5, same issue was
happening)
CloudStack: 4.11.3 (same issue in 4.11.2, was working fine in V4.9.3
When digging through the management-server.log, I have found the following;
>grep -n "Error while collecting network stats from router"
>/var/log/cloudstack/management/management-server.log
919060:2019-07-16 10:28:43,756 WARN
[c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-1931d792)
(logid:6e236728) Error while collecting network stats from router: r-2280-VM
from host: 71; details: Exception: java.lang.Exception
919328:2019-07-16 10:28:50,940 WARN
[c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterMonitor-1:ctx-1931d792)
(logid:6e236728) Error while collecting network stats from router: r-2280-VM
from host: 71; details: Exception: java.lang.Exception
920533:2019-07-16 10:29:54,768 WARN
[c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-63:ctx-6d1f0235
job-24930/job-24944 ctx-a15a2697) (logid:2b94fd5d) Error while collecting
network stats from router: r-2280-VM from host: 71; details: Exception:
java.lang.Exception
920621:2019-07-16 10:30:01,952 WARN
[c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-63:ctx-6d1f0235
job-24930/job-24944 ctx-a15a2697) (logid:2b94fd5d) Error while collecting
network stats from router: r-2280-VM from host: 71; details: Exception:
java.lang.Exception
>less +920621 /var/log/cloudstack/management/management-server.log
2019-07-16 10:30:01,952 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-63:ctx-6d1f0235 job-24930/job-24944 ctx-a15a2697)
(logid:2b94fd5d) Seq 71-7505811728966836555: Received: { Ans: , MgmtId:
226842157555374, via: 71(hostname), Ver: v1, Flags: 10, { NetworkUsageAnswer } }
2019-07-16 10:30:01,952 DEBUG [c.c.a.m.AgentManagerImpl]
(Work-Job-Executor-63:ctx-6d1f0235 job-24930/job-24944 ctx-a15a2697)
(logid:2b94fd5d) Details from executing class
com.cloud.agent.api.NetworkUsageCommand: Exception: java.lang.Exception
Message: vpc network usage plugin call failed
Stack: java.lang.Exception: vpc network usage plugin call failed
at
com.cloud.hypervisor.xenserver.resource.wrapper.xen56.XenServer56NetworkUsageCommandWrapper.executeNetworkUsage(XenServer56NetworkUsageCommandWrapper.java:84)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xen56.XenServer56NetworkUsageCommandWrapper.execute(XenServer56NetworkUsageCommandWrapper.java:41)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xen56.XenServer56NetworkUsageCommandWrapper.execute(XenServer56NetworkUsageCommandWrapper.java:33)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1737)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-07-16 10:30:01,952 WARN [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(Work-Job-Executor-63:ctx-6d1f0235 job-24930/job-24944 ctx-a15a2697)
(logid:2b94fd5d) Error while collecting network stats from router: r-2280-VM
from host: 71; details: Exception: java.lang.Exception
Message: vpc network usage plugin call failed
Stack: java.lang.Exception: vpc network usage plugin call failed
at
com.cloud.hypervisor.xenserver.resource.wrapper.xen56.XenServer56NetworkUsageCommandWrapper.executeNetworkUsage(XenServer56NetworkUsageCommandWrapper.java:84)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xen56.XenServer56NetworkUsageCommandWrapper.execute(XenServer56NetworkUsageCommandWrapper.java:41)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xen56.XenServer56NetworkUsageCommandWrapper.execute(XenServer56NetworkUsageCommandWrapper.java:33)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1737)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-07-16 10:30:01,973 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-63:ctx-6d1f0235 job-24930/job-24944 ctx-a15a2697)
(logid:2b94fd5d) Seq 71-7505811728966836557: Sending { Cmd , MgmtId:
226842157555374, via: 71(hostname), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.0.200","forceStop":true,"volumesToDisconnect":[],"vmName":"r-2280-VM","executeInSequence":false,"wait":0}}]
}
2019-07-16 10:30:01,973 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-63:ctx-6d1f0235 job-24930/job-24944 ctx-a15a2697)
(logid:2b94fd5d) Seq 71-7505811728966836557: Executing: { Cmd , MgmtId:
226842157555374, via: 71(hostname), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.0.200","forceStop":true,"volumesToDisconnect":[],"vmName":"r-2280-VM","executeInSequence":false,"wait":0}}]
}
2019-07-16 10:30:01,973 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-430:ctx-77c57645) (logid:8733e444) Seq 71-7505811728966836557:
Executing request
2019-07-16 10:30:01,995 DEBUG [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
(DirectAgent-430:ctx-77c57645) (logid:2b94fd5d) 9. The VM r-2280-VM is in
Stopping state
2019-07-16 10:30:02,303 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-390:ctx-3b9a78cd) (logid:27f6ec94) Seq 67-8459448950062117779:
Response Received
Any thoughts would be greatly appreciated.
Cheers,
Nick.
