Hi I've had this issue a few times in 2018 and managed to get it fixed pretty easily, although had spent a number of hours initially trying to figure out WTF is going on. This issue looks like one of those artefacts that creeped up in one of the versions released in 2018 and hasn't been addressed by the dev team.
The way I fixed it was similar to what has been recommended earlier. However, the difference was that I am sure I've looked at more tables than just the two suggested. Basically, I've stopped the management server, created the sql backup, connected to the sql db and listed all tables. Grepped for the words like job/schedule/queue/sync. After that I've went through all the tables and pretty much removed all the past / active / awaiting execution jobs. I have started by looking at the vm related jobs (the vm that I've tried to start but wasn't able to). This has worked once, but the second time I had to remove a lot more jobs which relate to other vms. After that I've started the management server and all went well from there. What I have also noticed is that my snapshot jobs (I use KVM and Ceph) seem to be blocking jobs on the hypervisor hosts which are running these snapshots. So, if I am trying to perform various vm related jobs on a host server which is currently running a snapshot process, that job will not be executed until the snapshot process is done. I've tested this countless number of times and it's still the case. Again, this issued appeared in one of the 2018 releases as I've never seen between 2012 - 2017. Both issues are annoying as hell! Cheers ----- Original Message ----- > From: "Alireza Eskandari" <astro.alir...@gmail.com> > To: "dev" <dev@cloudstack.apache.org> > Sent: Wednesday, 23 January, 2019 12:40:48 > Subject: Re: Help! Jobs stuck in pending state > I'm following this issue in github: > https://github.com/apache/cloudstack/issues/3104 > Please leave your comments > Thanks > > On Wed, Jan 23, 2019 at 12:39 PM Wei ZHOU <ustcweiz...@gmail.com> wrote: > >> Hi Alireza, >> >> could you try again after restarting mgt server ? >> >> -Wei >> >> Alireza Eskandari <astro.alir...@gmail.com> 于2019年1月23日周三 上午6:22写道： >> >> > First I deleted two jobs which was existed in vm_work_job table and its >> > related entry in sync_queue table but it doesn't help. >> > Then I delete all the entries in sync_queue tables and again no success. >> > Any idea? >> > >> > On Wed, Jan 23, 2019 at 1:50 AM Wei ZHOU <ustcweiz...@gmail.com> wrote: >> > >> > > If you know the instance id and mysql password, it should work after >> > > removing some records in mysql. >> > > >> > > ``` >> > > set @id=XXXXX; >> > > >> > > delete from vm_work_job where vm_instance_id=@id; >> > > delete from sync_queue where sync_objid=@id; >> > > ``` >> > > >> > > Alireza Eskandari <astro.alir...@gmail.com> 于2019年1月22日周二 下午10:59写道： >> > > >> > > > Hi guys >> > > > I have opened a bug in jira about my problem in CS: >> > > > https://issues.apache.org/jira/browse/CLOUDSTACK-10401 >> > > > CloudStack doesn't process jobs! My cloud in totally unusable. >> > > > Thanks in advance for you help. >> > > > >> > > >> >
