Hi, all:
We just did an upgrade from CS 4.3.2 -> 4.5.1. Our environment is rhel 6 +
Adv. Zone with SecurtyGroup + XenServer 6.2.
After the upgrade, all xenserver hosts were in alert state and MS can’t connect
to them with following errors:
INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce) Host
10.0.100.25 OpaqueRef:996c575a-ad04-5f3f-cd6d-56b7daa16844: Host 10.0.100.25 is
already setup.
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce) Failed to
configure brige firewall
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce) Check host
10.0.100.25 for CSP is installed or not and check network mode for bridge
WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-13:ctx-b4517020) Unable to
setup agent 5 due to Failed to configure brige firewall
INFO [c.c.u.e.CSExceptionErrorCode] (AgentTaskPool-13:ctx-b4517020) Could not
find exception: com.cloud.exception.ConnectionException in error code list for
exceptions
WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-13:ctx-b4517020) Monitor
XcpServerDiscoverer says there is an error in the connect process for 5 due to
Reinitialize agent after setup.
INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-13:ctx-b4517020) Host 5 is
disconnecting with event AgentDisconnected
WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-13:ctx-b4517020) Unable to
connect due to
com.cloud.exception.ConnectionException: Reinitialize agent after setup.
at
com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer.processConnect(XcpServerDiscoverer.java:621)
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:539)
at
com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1447)
at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1794)
at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1920)
at sun.reflect.GeneratedMethodAccessor142.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy149.createHostAndAgent(Unknown Source)
at
com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.runInContext(AgentManagerImpl.java:1078)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
The error implies that it was expecting linux bridge on XenServer and couldn’t
find it. After checking on XenServer hosts, we have been using openvswitch
instead of linux bridge as network backend all this time.
So the question comes down to:
1. If CS advanced zone with SecurityGroup only supported on XenServer using
linux bridge backend, why did 4.3.x allow us to work with unsupported
configurations for so long without any apparent issues, and does CS 4.5.1 now
enforce this requirement ?
2. If we switch network backend from openvswitch to linux bridge, would this
fix the problem? We are hoping to avoid this step, as it requires rebooting
all XenServer hosts and shuffling around hundreds of VM instances.
3. Is there any other solutions to make newly upgraded cloudstack 4.5.1
management server to reconnect to our XenServer hosts ?
Thanks in advance,
Yiping
