Re: KVM snapshots fail on GlusterFS shared mount point (NFS too)
On 20.12.2013 23:25, Nux! wrote:
On 20.12.2013 23:04, Nux! wrote:
Hi,
I know there's a vote for RC, maybe this has some importance. I'm
running KVM with a GlusterFS shared mount point on
In fact the snapshots fail with NFS primary storage as well, though
no more complaints from libvirt in this case.
Tried on another test setup with Basic zone and NFS primary storage,
same outcome, even today's 4.2 from github fails with the same error.
2013-12-21 10:17:02,913 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-6:null) Seq 1-1784545298: No more commands found
2013-12-21 10:17:02,913 DEBUG [agent.transport.Request]
(Job-Executor-1:job-12 = [ 7a568f3d-35a3-4c6c-ae11-691e6f4ab328 ]) Seq
1-1784545298: Received: { Ans: , MgmtId: 165340524328661, via: 1, Ver:
v1, Flags: 110, { CopyCmdAnswer } }
2013-12-21 10:17:02,923 DEBUG [storage.snapshot.SnapshotManagerImpl]
(Job-Executor-1:job-12 = [ 7a568f3d-35a3-4c6c-ae11-691e6f4ab328 ])
Failed to create snapshot
com.cloud.utils.exception.CloudRuntimeException: Failed to backup
98f25c16-fb79-4b0e-8443-4e7093d60555 for disk
/mnt/0740a8bf-c482-38a6-ad95-a2ef2ce4c1d5/739db255-5fc3-42c2-ae46-c7cd5047b8e1
to /mnt/e166eefb-d51e-33fe-80ab-9fec36914865/snapshots/2/3
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1007)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
at
com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2773)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2013-12-21 10:17:02,950 DEBUG [storage.volume.VolumeServiceImpl]
(Job-Executor-1:job-12 = [ 7a568f3d-35a3-4c6c-ae11-691e6f4ab328 ]) Take
snapshot: 3 failed
com.cloud.utils.exception.CloudRuntimeException: Failed to create
snapshot
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1034)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
at
com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2773)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to
backup 98f25c16-fb79-4b0e-8443-4e7093d60555 for disk
/mnt/0740a8bf-c482-38a6-ad95-a2ef2ce4c1d5/739db255-5fc3-42c2-ae46-c7cd5047b8e1
to /mnt/e166eefb-d51e-33fe-80ab-9fec36914865/snapshots/2/3
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1007)
... 16 more
2013-12-21 10:17:02,950 DEBUG [user.snapshot.CreateSnapshotCmd]
(Job-Executor-1:job-12 = [ 7a568f3d-35a3-4c6c-ae11-691e6f4ab328 ])
Failed to create snapshot
org.apache.cloudstack.api.ServerApi
Re: KVM snapshots fail on GlusterFS shared mount point
On 20.12.2013 23:04, Nux! wrote: Hi, I know there's a vote for RC, maybe this has some importance. I'm running KVM with a GlusterFS shared mount point on cloudstack-4.2-223b272, snapshots consistently fail to be backed up. They get stuck in "CreatedOnPrimary" status. The shared mount point is more than functional and the 2ndary storage can be mounted manually on both hypervisor and SSVM without problems. GUI says: "Failed to create snapshot due to an internal error creating snapshot for volume 15" Libvirt says: 2083: warning : qemuDomainObjBeginJobInternal:839 : Cannot start job (query, none) for domain i-2-12-VM; current job is (modify, none) owned by (2081, 0) 2083: error : qemuDomainObjBeginJobInternal:843 : Timed out during operation: cannot acquire state change lock Management logs says http://tmp.nux.ro/T94-ACSgluster.txt Has anyone seen this before? It could be this bug. Unless I'm doing something really stupid this is quite a release blocker. https://issues.apache.org/jira/browse/CLOUDSTACK-5393 Happy to test if someone comes can patch it. Lucian -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro
[SUMMARY][DISCUSS][PROPOSAL] Move docs to .rst format and readthedocs
Looks like we have a consensus, I will take this on and make sure it gets implemented as fast as possible. If any one wants to start to contribute docs over christmas, please send pr to: https://github.com/runseb/acsdocs Thanks to Shankar for already sending a trouble shooting guide. You can also send a rst or markdown file to review board, I will pull, convert and merge everything. Once the move is made I will document the entire process. Merry Christmas everyone, -Sebastien On Dec 19, 2013, at 4:47 PM, Animesh Chaturvedi wrote: > > > -Original Message- > From: Animesh Chaturvedi [mailto:animesh.chaturv...@citrix.com] > Sent: Tuesday, December 10, 2013 9:24 AM > To: dev@cloudstack.apache.org > Cc: Radhika Puthiyetath; msweet@gmail.com; Mike Tutkowski > Subject: RE: [DISCUSS][PROPOSAL] Move docs to .rst format and readthedocs > > > > -Original Message- > From: Sebastien Goasguen [mailto:run...@gmail.com] > Sent: Monday, December 09, 2013 11:31 PM > To: dev@cloudstack.apache.org > Cc: Radhika Puthiyetath; msweet@gmail.com; Mike Tutkowski > Subject: Re: [DISCUSS][PROPOSAL] Move docs to .rst format and readthedocs > > > On Dec 9, 2013, at 9:37 PM, Animesh Chaturvedi > wrote: > >> I am +1 to make documentation easier but we just passed code freeze for 4.3 >> and I feel we need to do it after 4.3. >> > > docs are not in the 4.3 code branch. it's different releases... > > Animesh> But they will be released together. > [Animesh] Sebastien what is the planned ETA for the conversion to RST. ACS > 4.3 planned RC date is 1/10 and I would like to have the documentation ready > by 1/3. > >> >> Animesh >> >> -Original Message- >> From: sebgoa [mailto:run...@gmail.com] >> Sent: Monday, December 09, 2013 3:54 AM >> To: dev@cloudstack.apache.org >> Cc: Radhika Puthiyetath; msweet@gmail.com; Mike Tutkowski >> Subject: [DISCUSS][PROPOSAL] Move docs to .rst format and readthedocs >> >> Hi, >> >> There has been lots of discussion about docs over the last 3/4 months, in >> summary the issues are: >> >> -Difficult to maintain and keep website up to date (issues with lang and >> issues with pdf formatting lately) -Difficult to contribute to easily, >> docbook is fine but tedious to work on. >> -Errors in the docs don't get properly fixed -Mix of OS information -Lack of >> content for certain features -Docs release cycle. Docs have bugs that will >> never get fixed in that specific release (because we see it as code release) >> >> To remedy some of those issues and work on a new release process specific to >> docs we moved the docs to its own repo: >> >> https://git-wip-us.apache.org/repos/asf?p=cloudstack-docs.git >> >> *I propose that we move away from docbook and use a more readable format: >> restructuredtext* >> >> I have worked on a prototype that uses restructured text: >> http://docutils.sourceforge.net/rst.html >> >> This format makes it extremely easy to write docs. Existing docbook content >> could be converted to .rst using a tool like pandoc: >> http://johnmacfarlane.net/pandoc/ >> >> *In addition to changing the format I propose that we use readthedocs.org* >> >> This will help with the release and build of the docs. readthedocs grabs the >> docs from a git repo, builds html, pdf and epub. >> It can also maintain several releases. We can apply a specific -theme- to >> our docs. >> >> See a prototype here: >> >> http://cloudstack.readthedocs.org/en/latest/ >> >> *I propose that we move to this as early as 4.3 documentation* >> >> Assuming this proposal passes, we would need to: >> >> -re-architect the repo >> -create the proper cnames to be in accordance with trademark guidelines -we >> can decide what content to keep or not and convert what we keep. >> -decide how we organize the content >> -start accepting pull requests (noting that pages can be edited directly >> from github) -make a first release of this new doc site at the same time >> than 4.3 release. >> >> >> Thoughts, flames ? >> >> >> -Sebastien >
Re: NFS secondary storage in multi zone config
Looks like this only happens for bootstrapping the ssvm in the new zones. On Thu, Dec 19, 2013 at 9:39 PM, Marcus Sorensen wrote: > Today I set up a second zone in one of our 4.2.1 CS environments. To my > surprise, it loaded the SSVM and consoleproxy for the zone from the NFS > secondary storage of the first zone. > > This is great for us, since we have 30Gbit of transport between sites, but > I'm guessing its not supposed to work that way. I thought NFS secondary > storage was limited to zone level. I had actually defined an NFS store for > the second zone, but hadn't provisioned it with a system template. It failed > and moved on to the other zone's template zone. > > I don't want to rely on this behavior if its a bug, but since there is no > upgrade path to using our S3 storage it's more ideal than having to download > the templates to each zone's secondary storage.
