Snapshots on KVM corrupting disk images

Hi all, We had some instances where VM disks are becoming corrupted when using KVM snapshots. We are running CloudStack 4.9.3 with KVM on CentOS 7. The first time was when someone mass-enabled scheduled snapshots on a lot of large number VMs and secondary storage filled up. We had to restore

Re: Snapshots on KVM corrupting disk images

Sean, What underlying primary storage are you using and how is it being utilized by ACS (e.g. NFS, shared mount et al)? - Si From: Sean Lair Sent: Tuesday, January 22, 2019 10:30 AM To: us...@cloudstack.apache.org; dev@cloudstack.apache.org Subject: Snapsho

CloudStack 4.11.2 Snapshot Revert fail

HI ALL I use CloudStack 4.11.2 to manage Xenserver 7.1.2 (XenServer CU2). VM snapshot for revert failure (snapshot does not contain memorysnapshot). 2019-01-23 00:06:54,210 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-156:ctx-28f7465a job-2867/job-2869 ctx-a04e0ed9) (logid:a9ef7fe7

RE: Snapshots on KVM corrupting disk images

Hi Simon It is NFS mount. The underlying storage is NetApp that we run a lot of different environments on, it is rock-solid, the only issues we've had are with KVM snapshots. Thanks Sean -Original Message- From: Simon Weller [mailto:swel...@ena.com.INVALID] Sent: Tuesday, January 22,

Re: CloudStack 4.11.2 Snapshot Revert fail

Hi there, after VM was deployed and snapshots created - was there any changes to VM or template from which VM was created - did ACS version get upgraded ? Best On Tue, 22 Jan 2019 at 17:52, li jerry wrote: > HI ALL > > I use CloudStack 4.11.2 to manage Xenserver 7.1.2 (XenServer CU2). > > VM s

答复: CloudStack 4.11.2 Snapshot Revert fail

VM deployed through the template After the snapshot was created, the calculation scheme was not changed and there was no upgrade. Only shut down the VM and perform snapshot recovery 发件人: Andrija Panic 发送时间: 2019年1月23日 1:05 收件人: dev

RE: CloudStack 4.11.2 Snapshot Revert fail

Sorry, replied to wrong snapshot thread.. -Original Message- From: Sean Lair Sent: Tuesday, January 22, 2019 11:48 AM To: dev Cc: us...@cloudstack.apache.org Subject: RE: CloudStack 4.11.2 Snapshot Revert fail Luckily it was for a VM that is never touched in CloudStack. The snaps were

RE: CloudStack 4.11.2 Snapshot Revert fail

Luckily it was for a VM that is never touched in CloudStack. The snaps were scheduled ones. No, no changes to VM or template. We are due to upgrade from 4.9.3 but we have not yet. -Original Message- From: Andrija Panic [mailto:andrija.pa...@gmail.com] Sent: Tuesday, January 22, 2019 1

Help! Jobs stuck in pending state

Hi guys I have opened a bug in jira about my problem in CS: https://issues.apache.org/jira/browse/CLOUDSTACK-10401 CloudStack doesn't process jobs! My cloud in totally unusable. Thanks in advance for you help.

Re: Snapshots on KVM corrupting disk images

Hi Sean, The (recurring) volume snapshot on running vms should be disabled in cloudstack. According to some discussions (for example https://bugzilla.redhat.com/show_bug.cgi?id=920020), the image might be corrupted due to the concurrent read/write operations in volume snapshot (by qemu-img snapsh

Re: Help! Jobs stuck in pending state

If you know the instance id and mysql password, it should work after removing some records in mysql. ``` set @id=X; delete from vm_work_job where vm_instance_id=@id; delete from sync_queue where sync_objid=@id; ``` Alireza Eskandari 于2019年1月22日周二 下午10:59写道： > Hi guys > I have opened a bug

Re: Help! Jobs stuck in pending state

Here is my query on those tables: MySQL [cloud]> select * from vm_work_job; +---+--+--++ | id| step | vm_type | vm_instance_id | +---+--+--++ | 57262 | Prepare | Instance |691 | | 57268 | Starting | Insta

Re:答复: CloudStack 4.11.2 Snapshot Revert fail

This is a fresh installation (XenSever 7.1.2+ACS 4.11.2). We simply can not revert the VM snapshot though there's no changes at all to VM or template. We can consistently reproduce this issue in our environtment with the error log attached. And we believe the issue doesn't exisit in ano

Re: Snapshots on KVM corrupting disk images

I've met the situations when CLOUDSTACK+KVM+QCOW2+SNAPSHOTS led to corrupted images, mostly in 4.3 and NFS, but I've thought that CS stops VM just before it does the snapshot. At least the VM behavior when the VM snapshot is created looks like it happens (freezing). That's why it looks strange. But

RE: Snapshots on KVM corrupting disk images

Thanks Wei! We really appreciate the response and the link. Shouldn't we be doing something to stop the ability to use snapshots (scheduled and other snapshot operations) in CloudStack? -Original Message- From: Wei ZHOU [mailto:ustcweiz...@gmail.com] Sent: Tuesday, January 22, 2019 4

Re: Help! Jobs stuck in pending state

First I deleted two jobs which was existed in vm_work_job table and its related entry in sync_queue table but it doesn't help. Then I delete all the entries in sync_queue tables and again no success. Any idea? On Wed, Jan 23, 2019 at 1:50 AM Wei ZHOU wrote: > If you know the instance id and my

Re: Help! Jobs stuck in pending state

Hi Alireza, Could you elaborate on how you instantiated the jobs and any thing specific that went wrong in between? Usually deleting directly through SQL statements is very risky and first try should be through any API support. Also, you might want to use github page (https://github.com/apache