Hi Vivek/Nux,
Thanks for the response.
@vivek I was able to manually mount the secondary storage to xen server, But
strange factor is that it was already mounted by cloudstack with Version 4.1
As you have mentioned Changing the NFS version to v3 in global settings worked,
Now Volume snapshot and Vm creation works fine. Thank you
Midhun Jose
----- Original Message -----
From: "Nux" <n...@li.nux.ro>
To: "users" <users@cloudstack.apache.org>
Cc: "Vivek Kumar" <vivek.ku...@indiqus.com.invalid>
Sent: Tuesday, April 5, 2022 12:01:45 PM
Subject: Re: ACS 14.16.1::XCP-ng 8.2.1 - Secondary storage mount issue|Volume
Snapshot
Hi,
What Vivek said, also make sure there is no firewall between hypervisor
and sec storage, or at least disable it temporarily to rule things out.
NFSv3 and v4 are slightly different at network level, some port
difference etc.
---
Nux!
www.nux.ro
On 2022-04-05 07:25, Vivek Kumar wrote:
> Hey Midhun,
>
> This error is you see is coming from the XenServer, looks like it’s
> not able to mount the NFS on the destination directory, I know you
> have verified the mount point but can you try to manually mount your
> secondary storage on XenServer, also try checking the global settings
> - secstorage.nfs.version to check and enforce your nfs version.
>
> Regards,
> Vivek Kumar
>
>
>
>
>> On 05-Apr-2022, at 11:25 AM, Midhun Jose
>> <mid...@assistanz.com.INVALID> wrote:
>>
>>
>> Hi @all,
>>
>> I am using Apache cloudstack 4.16.1 integrated with XCP-ng 8.2.1. I am
>> not able to take VM volume snapshot and sometimes vm creation is also
>> not working. I am getting the error as attached screenshot and while
>> checking the logs we are getting below error,
>>
>> P.S: I have veriifed that nfs server is working fine and secondary
>> storage is mounted in xen server. Can someome piont out the reason on
>> this..
>>
>> ~~~~~~
>> 2022-04-04 13:13:44,163 WARN [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-17:ctx-f6381170) (logid:4835671f) Task failed! Task
>> record: uuid: e907d695-825f-24e4-5ce3-afd74eb0ece4
>> nameLabel: Async.host.call_plugin
>> nameDescription:
>> allowedOperations: []
>> currentOperations: {}
>> created: Mon Apr 04 13:13:43 IST 2022
>> finished: Mon Apr 04 13:13:43 IST 2022
>> status: failure
>> residentOn: com.xensource.xenapi.Host@5fa0255a
>> progress: 1.0
>> type: <none/>
>> result:
>> errorInfo: [XENAPI_PLUGIN_FAILURE,
>> mountNfsSecondaryStorage, SROSError, Error reporting error, unknown
>> key Unexpected error while trying to mount
>> 10.31.9.226:/storage/cloudstack/secondary/ to
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3]
>> otherConfig: {}
>> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>> subtasks: []
>> 2022-04-04 13:13:44,173 WARN [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-17:ctx-f6381170) (logid:4835671f) callHostPlugin failed
>> for cmd: mountNfsSecondaryStorage with args remoteDir:
>> 10.31.9.226:/storage/cloudstack/secondary/, localDir:
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3, nfsVersion:
>> null, due to Task failed! Task record: uuid:
>> e907d695-825f-24e4-5ce3-afd74eb0ece4
>> nameLabel: Async.host.call_plugin
>> nameDescription:
>> allowedOperations: []
>> currentOperations: {}
>> created: Mon Apr 04 13:13:43 IST 2022
>> finished: Mon Apr 04 13:13:43 IST 2022
>> status: failure
>> residentOn: com.xensource.xenapi.Host@5fa0255a
>> progress: 1.0
>> type: <none/>
>> result:
>> errorInfo: [XENAPI_PLUGIN_FAILURE,
>> mountNfsSecondaryStorage, SROSError, Error reporting error, unknown
>> key Unexpected error while trying to mount
>> 10.31.9.226:/storage/cloudstack/secondary/ to
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3]
>> otherConfig: {}
>> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>> subtasks: []
>> Task failed! Task record: uuid:
>> e907d695-825f-24e4-5ce3-afd74eb0ece4
>> nameLabel: Async.host.call_plugin
>> nameDescription:
>> allowedOperations: []
>> currentOperations: {}
>> created: Mon Apr 04 13:13:43 IST 2022
>> finished: Mon Apr 04 13:13:43 IST 2022
>> status: failure
>> residentOn: com.xensource.xenapi.Host@5fa0255a
>> progress: 1.0
>> type: <none/>
>> result:
>> errorInfo: [XENAPI_PLUGIN_FAILURE,
>> mountNfsSecondaryStorage, SROSError, Error reporting error, unknown
>> key Unexpected error while trying to mount
>> 10.31.9.226:/storage/cloudstack/secondary/ to
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3]
>> otherConfig: {}
>> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>> subtasks: []
>> Task failed! Task record: uuid:
>> e907d695-825f-24e4-5ce3-afd74eb0ece4
>> nameLabel: Async.host.call_plugin
>> nameDescription:
>> allowedOperations: []
>> currentOperations: {}
>> created: Mon Apr 04 13:13:43 IST 2022
>> finished: Mon Apr 04 13:13:43 IST 2022
>> status: failure
>> residentOn: com.xensource.xenapi.Host@5fa0255a
>> progress: 1.0
>> type: <none/>
>> result:
>> errorInfo: [XENAPI_PLUGIN_FAILURE,
>> mountNfsSecondaryStorage, SROSError, Error reporting error, unknown
>> key Unexpected error while trying to mount
>> 10.31.9.226:/storage/cloudstack/secondary/ to
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3]
>> otherConfig: {}
>> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>> subtasks: []
>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:500)
>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:422)
>> at
>> com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:80)
>> at
>> com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.createFileSr(Xenserver625StorageProcessor.java:249)
>> at
>> com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.copyTemplateToPrimaryStorage(Xenserver625StorageProcessor.java:286)
>> at
>> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:95)
>> at
>> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:59)
>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:36)
>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:30)
>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1758)
>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> at
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> at
>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> at
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:829)
>> 2022-04-04 13:13:44,180 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-17:ctx-f6381170) (logid:4835671f) unable to destroy
>> task(com.xensource.xenapi.Task@3cd54231) on
>> host(bc164e35-7efe-4ff7-9460-1c92cdb44396) due to You gave an invalid
>> object reference. The object may have recently been deleted. The
>> class parameter gives the type of reference given, and the handle
>> parameter echoes the bad value given.
>> 2022-04-04 13:13:44,180 WARN [c.c.h.x.r.XenServerStorageProcessor]
>> (DirectAgent-17:ctx-f6381170) (logid:4835671f) Could not mount
>> secondary storage 10.31.9.226:/storage/cloudstack/secondary/ on host
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3
>> 2022-04-04 13:13:44,181 WARN [c.c.h.x.r.XenServerStorageProcessor]
>> (DirectAgent-17:ctx-f6381170) (logid:4835671f) Catch Exception
>> com.cloud.utils.exception.CloudRuntimeException for template due to
>> com.cloud.utils.exception.CloudRuntimeException: Could not mount
>> secondary storage 10.31.9.226:/storage/cloudstack/secondary/ on host
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3
>> com.cloud.utils.exception.CloudRuntimeException: Could not mount
>> secondary storage 10.31.9.226:/storage/cloudstack/secondary/ on host
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3
>> at
>> com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:84)
>> at
>> com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.createFileSr(Xenserver625StorageProcessor.java:249)
>> at
>> com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.copyTemplateToPrimaryStorage(Xenserver625StorageProcessor.java:286)
>> at
>> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:95)
>> at
>> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:59)
>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:36)
>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:30)
>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1758)
>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttach
>> 2022-04-04 13:13:44,163 WARN [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-17:ctx-f6381170) (logid:4835671f) Task failed! Task
>> record: uuid: e907d695-825f-24e4-5ce3-afd74eb0ece4
>> nameLabel: Async.host.call_plugin
>> nameDescription:
>> allowedOperations: []
>> currentOperations: {}
>> created: Mon Apr 04 13:13:43 IST 2022
>> finished: Mon Apr 04 13:13:43 IST 2022
>> status: failure
>> residentOn: com.xensource.xenapi.Host@5fa0255a
>> progress: 1.0
>> type: <none/>
>> result:
>> errorInfo: [XENAPI_PLUGIN_FAILURE,
>> mountNfsSecondaryStorage, SROSError, Error reporting error, unknown
>> key Unexpected error while trying to mount
>> 10.31.9.226:/storage/cloudstack/secondary/ to
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3]
>> otherConfig: {}
>> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>> subtasks: []
>> 2022-04-04 13:13:44,173 WARN [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-17:ctx-f6381170) (logid:4835671f) callHostPlugin failed
>> for cmd: mountNfsSecondaryStorage with args remoteDir:
>> 10.31.9.226:/storage/cloudstack/secondary/, localDir:
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3, nfsVersion:
>> null, due to Task failed! Task record: uuid:
>> e907d695-825f-24e4-5ce3-afd74eb0ece4
>> nameLabel: Async.host.call_plugin
>> nameDescription:
>> allowedOperations: []
>> currentOperations: {}
>> created: Mon Apr 04 13:13:43 IST 2022
>> finished: Mon Apr 04 13:13:43 IST 2022
>> status: failure
>> residentOn: com.xensource.xenapi.Host@5fa0255a
>> progress: 1.0
>> type: <none/>
>> result:
>> errorInfo: [XENAPI_PLUGIN_FAILURE,
>> mountNfsSecondaryStorage, SROSError, Error reporting error, unknown
>> key Unexpected error while trying to mount
>> 10.31.9.226:/storage/cloudstack/secondary/ to
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3]
>> otherConfig: {}
>> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>> subtasks: []
>> Task failed! Task record: uuid:
>> e907d695-825f-24e4-5ce3-afd74eb0ece4
>> nameLabel: Async.host.call_plugin
>> nameDescription:
>> allowedOperations: []
>> currentOperations: {}
>> created: Mon Apr 04 13:13:43 IST 2022
>> finished: Mon Apr 04 13:13:43 IST 2022
>> status: failure
>> residentOn: com.xensource.xenapi.Host@5fa0255a
>> progress: 1.0
>> type: <none/>
>> result:
>> errorInfo: [XENAPI_PLUGIN_FAILURE,
>> mountNfsSecondaryStorage, SROSError, Error reporting error, unknown
>> key Unexpected error while trying to mount
>> 10.31.9.226:/storage/cloudstack/secondary/ to
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3]
>> otherConfig: {}
>> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>> subtasks: []
>> Task failed! Task record: uuid:
>> e907d695-825f-24e4-5ce3-afd74eb0ece4
>> nameLabel: Async.host.call_plugin
>> nameDescription:
>> allowedOperations: []
>> currentOperations: {}
>> created: Mon Apr 04 13:13:43 IST 2022
>> finished: Mon Apr 04 13:13:43 IST 2022
>> status: failure
>> residentOn: com.xensource.xenapi.Host@5fa0255a
>> progress: 1.0
>> type: <none/>
>> result:
>> errorInfo: [XENAPI_PLUGIN_FAILURE,
>> mountNfsSecondaryStorage, SROSError, Error reporting error, unknown
>> key Unexpected error while trying to mount
>> 10.31.9.226:/storage/cloudstack/secondary/ to
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3]
>> otherConfig: {}
>> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
>> subtasks: []
>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:500)
>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:422)
>> at
>> com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:80)
>> at
>> com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.createFileSr(Xenserver625StorageProcessor.java:249)
>> at
>> com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.copyTemplateToPrimaryStorage(Xenserver625StorageProcessor.java:286)
>> at
>> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:95)
>> at
>> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:59)
>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:36)
>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:30)
>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1758)
>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> at
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> at
>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> at
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:829)
>> 2022-04-04 13:13:44,180 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-17:ctx-f6381170) (logid:4835671f) unable to destroy
>> task(com.xensource.xenapi.Task@3cd54231) on
>> host(bc164e35-7efe-4ff7-9460-1c92cdb44396) due to You gave an invalid
>> object reference. The object may have recently been deleted. The
>> class parameter gives the type of reference given, and the handle
>> parameter echoes the bad value given.
>> 2022-04-04 13:13:44,180 WARN [c.c.h.x.r.XenServerStorageProcessor]
>> (DirectAgent-17:ctx-f6381170) (logid:4835671f) Could not mount
>> secondary storage 10.31.9.226:/storage/cloudstack/secondary/ on host
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3
>> 2022-04-04 13:13:44,181 WARN [c.c.h.x.r.XenServerStorageProcessor]
>> (DirectAgent-17:ctx-f6381170) (logid:4835671f) Catch Exception
>> com.cloud.utils.exception.CloudRuntimeException for template due to
>> com.cloud.utils.exception.CloudRuntimeException: Could not mount
>> secondary storage 10.31.9.226:/storage/cloudstack/secondary/ on host
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3
>> com.cloud.utils.exception.CloudRuntimeException: Could not mount
>> secondary storage 10.31.9.226:/storage/cloudstack/secondary/ on host
>> /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3
>> at
>> com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:84)
>> at
>> com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.createFileSr(Xenserver625StorageProcessor.java:249)
>> at
>> com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.copyTemplateToPrimaryStorage(Xenserver625StorageProcessor.java:286)
>> at
>> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:95)
>> at
>> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:59)
>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:36)
>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:30)
>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1758)
>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttach
>>
>> ~~~~~~
>>
>>
>> Best Regards,
>> Midhun
>>
