Hi all, I’m moving this conversation here, to be picked up by a separate PR since we’ve merged the original 1854.
Pasting original content from github: @mike-tutkowski<https://github.com/mike-tutkowski> @syed<https://github.com/syed>, I've been investigating the test failure from test_primary_storage.py: test_01_primary_storage_iscsi, and we came to a conclusion that iscsi attaches but it's not writeable. By any chance do you recall any change that might cause this? Here's the management log failing to create a hearbeat volume: 2016-12-23 11:30:58,826 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-404:ctx-c95759e3) (logid:f96038aa) Setting up the heartbeat sr for host 10.2.2.33 and sr ca748d82-3668-9559-07ba-19ff94844ed0 2016-12-23 11:30:59,104 WARN [c.c.h.x.r.w.x.CitrixModifyStoragePoolCommandWrapper] (DirectAgent-404:ctx-c95759e3) (logid:f96038aa) ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat sr on SR ca748d82-3668-9559-07ba-19ff94844ed0 due to #6# Unable to create heartbeat volume hb-db2a4468-ef57-4936-9ccd-7b8f4c06d822 host:db2a4468-ef57-4936-9ccd-7b8f4c06d822 pool: 10.5.0.4/iqn.2016-10.l ocal.marvin:tgt1/0 com.cloud.utils.exception.CloudRuntimeException: Unable to setup heartbeat sr on SR ca748d82-3668-9559-07ba-19ff94844ed0 due to #6# Unable to create heartbeat volume hb-db2a4468-ef57-4936-9ccd-7b8f4c06d 822 at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.setupHeartbeatSr(CitrixResourceBase.java:4596) at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixModifyStoragePoolCommandWrapper.execute(CitrixModifyStoragePoolCommandWrapper.java:54) at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixModifyStoragePoolCommandWrapper.execute(CitrixModifyStoragePoolCommandWrapper.java:40) at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1687) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-12-23 11:30:59,111 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-404:ctx-c95759e3) (logid:f96038aa) Seq 1-4225502350380368681: Response Received: 2016-12-23 11:30:59,111 DEBUG [c.c.a.t.Request] (DirectAgent-404:ctx-c95759e3) (logid:f96038aa) Seq 1-4225502350380368681: Processing: { Ans: , MgmtId: 7003044906821, via: 1(pr1854-t722-xenserver-65sp1 -xs1), Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":false,"details":"ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat sr on SR ca748d82-3668-9559-07ba-19ff94844ed0 due to #6# Unable to create heartbeat volume hb-db2a4468-ef57-4936-9ccd-7b8f4c06d822 host:db2a4468-ef57-4936-9ccd-7b8f4c06d822 pool: 10.5.0.4/iqn.2016-10.local.marvin:tgt1/0","wait":0}}] } 2016-12-23 11:30:59,111 DEBUG [c.c.a.t.Request] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) Seq 1-4225502350380368681: Received: { Ans: , MgmtId: 7003044906821, via: 1(pr 1854-t722-xenserver-65sp1-xs1), Ver: v1, Flags: 10, { Answer } } 2016-12-23 11:30:59,111 DEBUG [c.c.a.m.AgentManagerImpl] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) Details from executing class com.cloud.agent.api.ModifyStoragePoolComm and: ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat sr on SR ca748d82-3668-9559-07ba-19ff94844ed0 due to #6# Unable to create heartbeat volume hb-db2a4468-ef57-4936-9ccd-7b8f4c06 d822 host:db2a4468-ef57-4936-9ccd-7b8f4c06d822 pool: 10.5.0.4/iqn.2016-10.local.marvin:tgt1/0 2016-12-23 11:30:59,112 WARN [o.a.c.alerts] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) alertType:: 7 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: U nable to attach storage pool11 to the host1 2016-12-23 11:30:59,120 WARN [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) Unable to establish a connection between Ho st[-1-Routing] and org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@70894d68 com.cloud.utils.exception.CloudRuntimeException: Unable establish connection from storage head to storage pool 11 due to ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat sr on SR c a748d82-3668-9559-07ba-19ff94844ed0 due to #6# Unable to create heartbeat volume hb-db2a4468-ef57-4936-9ccd-7b8f4c06d822 host:db2a4468-ef57-4936-9ccd-7b8f4c06d822 pool: 10.5.0.4/iqn.2016-10.local.marvin :tgt1/011 at org.apache.cloudstack.storage.datastore.provider.DefaultHostListener.hostConnect(DefaultHostListener.java:75) at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:955) at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:414) at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:709) at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:199) at org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:163) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) at com.cloud.api.ApiServer.queueCommand(ApiServer.java:709) at com.cloud.api.ApiServer.handleRequest(ApiServer.java:533) at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:299) at com.cloud.api.ApiServlet$1.run(ApiServlet.java:129) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:126) at com.cloud.api.ApiServlet.doGet(ApiServlet.java:88) at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) at javax.servlet.http.HttpServlet.service(HttpServlet.java:723) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:299) at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:732) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-12-23 11:30:59,121 WARN [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) No host can access storage pool org.apache. cloudstack.storage.datastore.PrimaryDataStoreImpl@70894d68 on cluster 1 2016-12-23 11:30:59,127 DEBUG [c.c.s.StorageManagerImpl] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) Failed to add data store: Failed to access storage pool com.cloud.utils.exception.CloudRuntimeException: Failed to access storage pool at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:427) at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:709) at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:199) at org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:163) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) at com.cloud.api.ApiServer.queueCommand(ApiServer.java:709) at com.cloud.api.ApiServer.handleRequest(ApiServer.java:533) at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:299) at com.cloud.api.ApiServlet$1.run(ApiServlet.java:129) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:126) at com.cloud.api.ApiServlet.doGet(ApiServlet.java:88) at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) at javax.servlet.http.HttpServlet.service(HttpServlet.java:723) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:299) at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:732) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-12-23 11:30:59,129 DEBUG [c.c.s.StorageManagerImpl] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) Failed to clean up storage pool: null 2016-12-23 11:30:59,130 INFO [c.c.a.ApiServer] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) Failed to add data store: Failed to access storage pool 2016-12-23 11:30:59,130 DEBUG [c.c.a.ApiServlet] (catalina-exec-18:ctx-ecf415b7 ctx-29dead47 ctx-1fd26575) (logid:f96038aa) ===END=== 10.5.0.4 -- GET apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&name=Test+iSCSI&url=iscsi%3A%2F%2F10.5.0.4%2Fiqn.2016-10.local.marvin%3Atgt1%2F0&podid=8b409e63-f298-4113-a23b-0700a8dfc76d&clusterid=4b2458e0-9da9-4a99-987e-3d2cd6f2f0a9&zoneid=8d414040-93d1-47d9-8706-7a15f3c1dfb6&command=createStoragePool&signature=eLjbko60Wfjj4iny2w0ktjvKcK4%3D&response=json 2016-12-23 11:30:59,143 DEBUG [c.c.a.ApiServlet] (catalina-exec-16:ctx-9fd311e8) (logid:2d0d3d40) ===START=== 10.5.0.4 -- GET apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5 Marvin test failure: 2016-12-23 11:30:58,520 - CRITICAL - EXCEPTION: test_01_primary_storage_iscsi: ['Traceback (most recent call last):\n', ' File "/usr/lib64/python2.7/unittest/case.py", line 369, in run\n testMethod()\n', ' File "/marvin/tests/smoke/test_primary_storage.py", line 199, in test_01_primary_storage_iscsi\n podid=self.pod.id\n', ' File "/usr/lib/python2.7/site-packages/marvin/lib/base.py", line 2732, in create\n return StoragePool(apiclient.createStoragePool(cmd).__dict__)\n', ' File "/usr/lib/python2.7/site-packages/marvin/cloudstackAPI/cloudstackAPIClient.py", line 2537, in createStoragePool\n response = self.connection.marvinRequest(command, response_type=response, method=method)\n', ' File "/usr/lib/python2.7/site-packages/marvin/cloudstackConnection.py", line 379, in marvinRequest\n raise e\n', 'CloudstackAPIException: Execute cmd: createstoragepool failed, due to: errorCode: 530, errorText:Failed to add data store: Failed to access storage pool\n'] boris.stoya...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
