Hi,
I am facing issue with xenserver 6.2 SP1, on ACS 4.3.1 and 4.4 both.
While creating Zone with NFS as primary & secondary storage
2015-01-05 19:29:45,715 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-2:ctx-23cfb66b) Seq 1-3875347479352311816: Executing request
2015-01-05 19:29:47,079 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-2:ctx-23cfb66b) Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed
due to com.cloud.utils.exception.CloudRuntimeException: Unable to create
NFS SR Pool[1|192.168.161.110:2049|/mnt/data/VMS-TT/primary] on
host:ba237c07-71cb-48a8-8a6d-6e6c3aff65b2 pool:
192.168.161.110/mnt/data/VMS-TT/primary
com.cloud.utils.exception.CloudRuntimeException: Unable to create NFS SR
Pool[1|192.168.161.110:2049|/mnt/data/VMS-TT/primary]
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getNfsSR(CitrixResourceBase.java:6133)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:5074)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:475)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:307)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: SR_BACKEND_FAILURE_73NFS mount error [opterr=mount failed with
return code 32]
at com.xensource.xenapi.Types.checkResponse(Types.java:2006)
at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456)
at com.xensource.xenapi.SR.create(SR.java:801)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getNfsSR(CitrixResourceBase.java:6129)
... 17 more
2015-01-05 19:29:47,080 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-2:ctx-23cfb66b) Seq 1-3875347479352311816: Response Received:
2015-01-05 19:29:47,080 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-2:ctx-23cfb66b) Seq 1-3875347479352311816: Response Received:
2015-01-05 19:29:47,080 DEBUG [c.c.a.t.Request]
(DirectAgent-2:ctx-23cfb66b) Seq 1-3875347479352311816: Processing: { Ans:
, MgmtId: 158445874166998, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed
due to com.cloud.utils.exception.CloudRuntimeException: Unable to create
NFS SR Pool[1|192.168.161.110:2049|/mnt/data/VMS-TT/primary] on
host:ba237c07-71cb-48a8-8a6d-6e6c3aff65b2 pool:
192.168.161.110/mnt/data/VMS-TT/primary","wait":0}}] }
2015-01-05 19:29:47,081 DEBUG [c.c.a.t.Request]
(catalina-exec-13:ctx-c00f6f19 ctx-37181993) Seq 1-3875347479352311816:
Received: { Ans: , MgmtId: 158445874166998, via: 1, Ver: v1, Flags: 10, {
Answer } }
2015-01-05 19:29:47,081 DEBUG [c.c.a.m.AgentManagerImpl]
(catalina-exec-13:ctx-c00f6f19 ctx-37181993) Details from executing class
com.cloud.agent.api.CreateStoragePoolCommand: Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed
due to com.cloud.utils.exception.CloudRuntimeException: Unable to create
NFS SR Pool[1|192.168.161.110:2049|/mnt/data/VMS-TT/primary] on
host:ba237c07-71cb-48a8-8a6d-6e6c3aff65b2 pool:
192.168.161.110/mnt/data/VMS-TT/primary
2015-01-05 19:29:47,126 WARN
[o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl]
(catalina-exec-13:ctx-c00f6f19 ctx-37181993) Can not create storage pool
through host 1 due to Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed
due to com.cloud.utils.exception.CloudRuntimeException: Unable to create
NFS SR Pool[1|192.168.161.110:2049|/mnt/data/VMS-TT/primary] on
host:ba237c07-71cb-48a8-8a6d-6e6c3aff65b2 pool:
192.168.161.110/mnt/data/VMS-TT/primary
2015-01-05 19:29:47,126 DEBUG [c.c.s.StorageManagerImpl]
(catalina-exec-13:ctx-c00f6f19 ctx-37181993) Failed to add data store: Can
not create storage pool through host 1 due to Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed
due to com.cloud.utils.exception.CloudRuntimeException: Unable to create
NFS SR Pool[1|192.168.161.110:2049|/mnt/data/VMS-TT/primary] on
host:ba237c07-71cb-48a8-8a6d-6e6c3aff65b2 pool:
192.168.161.110/mnt/data/VMS-TT/primary
com.cloud.utils.exception.CloudRuntimeException: Can not create storage
pool through host 1 due to Catch Exception
com.cloud.utils.exception.CloudRuntimeException, create StoragePool failed
due to com.cloud.utils.exception.CloudRuntimeException: Unable to create
NFS SR Pool[1|192.168.161.110:2049|/mnt/data/VMS-TT/primary] on
host:ba237c07-71cb-48a8-8a6d-6e6c3aff65b2 pool:
192.168.161.110/mnt/data/VMS-TT/primary
at
org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.createStoragePool(CloudStackPrimaryDataStoreLifeCycleImpl.java:384)
at
org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:407)
at
com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:669)
at
com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:178)
at
org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:163)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:682)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:511)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:330)
at com.cloud.api.ApiServlet.access$000(ApiServlet.java:54)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:118)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:115)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-01-05 19:29:47,129 INFO [c.c.a.ApiServer]
(catalina-exec-13:ctx-c00f6f19 ctx-37181993) null
2015-01-05 19:29:47,130 DEBUG [c.c.a.ApiServlet]
(catalina-exec-13:ctx-c00f6f19 ctx-37181993) ===END=== 192.168.160.95 --
GET
command=createStoragePool&zoneid=123cf76d-077a-4d92-a4e1-708aca573b1a&podId=5041c821-0d10-47df-a363-01f87cccfc00&clusterid=19698fe8-2921-48ea-9158-608cac69b5ac&name=Primary&scope=cluster&url=nfs%3A%2F%2F192.168.161.110%2Fmnt%2Fdata%2FVMS-TT%2Fprimary&response=json&sessionkey=qT%2FMwkWgbzKOyRSPltz6ETNUkeE%3D&_=1420466385645
2015-01-05 19:30:01,600 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-c4bb0888) Skip capacity scan due to there is no Primary
Storage UPintenance mode
Same time Xenserver host showing following log..
xenserver-01 xapi: [error|xenserver-01|926 INET 0.0.0.0:80|SR.create
R:adea9138f9b1|storage_access] Re-raising as SR_BACKEND_FAILURE_73 [ ; NFS
mount error [opterr=mount failed with return code 32]; ]
Jan 5 19:29:47 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:c87d8d71bfdd|xapi] get_since_for_events: no in_memory_cache!
Jan 5 19:29:47 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:11d92819dc30|xapi] get_since_for_events: no in_memory_cache!
Jan 5 19:30:17 xenserver-01 last message repeated 2 times
Jan 5 19:30:17 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:d575526e384a|xapi] get_since_for_events: no in_memory_cache!
Jan 5 19:30:17 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:d575526e384a|xapi] get_since_for_events: no in_memory_cache!
Jan 5 19:30:43 xenserver-01 heartbeat: Problem with heartbeat, no iSCSI or
NFS mount defined in /opt/cloud/bin/heartbeat!
Jan 5 19:30:47 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:d575526e384a|xapi] get_since_for_events: no in_memory_cache!
Jan 5 19:30:47 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:6d4a45ec4259|xapi] get_since_for_events: no in_memory_cache!
Jan 5 19:31:17 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:6d4a45ec4259|xapi] get_since_for_events: no in_memory_cache!
Jan 5 19:31:17 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:0ecee91bf125|xapi] get_since_for_events: no in_memory_cache!
Jan 5 19:31:43 xenserver-01 heartbeat: Problem with heartbeat, no iSCSI or
NFS mount defined in /opt/cloud/bin/heartbeat!
Jan 5 19:31:47 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:0ecee91bf125|xapi] get_since_for_events: no in_memory_cache!
Jan 5 19:31:47 xenserver-01 xapi: [ warn|xenserver-01|866 INET
0.0.0.0:80|event.from
D:f473b74de5fd|xapi] get_since_for_events: no in_memory_cache!
Tried to stop iptables on XS host
I checked /opt/cloud/bin/heartbeat file, on XS host and file is empty.
Need help on this..
Regards,
Tejas
