Also these - https://github.com/myENA/cloudstack/pull/20/commits/1948ce5d24b87433ae9e8f4faebdfc20b56b751a
https://github.com/myENA/cloudstack/pull/12/commits ________________________________ From: Andrija Panic <andrija.pa...@gmail.com> Sent: Monday, February 19, 2018 5:23 AM To: dev Subject: Re: HA issues Hi Simon, a big thank you for this, will have our devs check this! Thanks! On 19 February 2018 at 09:02, Simon Weller <swel...@ena.com.invalid> wrote: > Andrija, > > > We pushed quite a few PRs on the exception and lockup issues related to > Ceph in the agent. > > > We have a PR for the deletion issue. See if you have it pulled into your > release - https://github.com/myENA/cloudstack/pull/9 [https://avatars1.githubusercontent.com/u/1444686?s=400&v=4]<https://github.com/myENA/cloudstack/pull/9> context cleanup by leprechau · Pull Request #9 · myENA/cloudstack<https://github.com/myENA/cloudstack/pull/9> github.com cleanup rbd image and rados context even if exceptions are thrown in deletePhysicalDisk routine > > > - Si > > > > > ________________________________ > From: Andrija Panic <andrija.pa...@gmail.com> > Sent: Saturday, February 17, 2018 1:49 PM > To: dev > Subject: Re: HA issues > > Hi Sean, > > (we have 2 threads interleaving on the libvirt lockd..) - so, did you > manage to understand what can cause the Agent Disconnect in most cases, for > you specifically? Is there any software (CloudStack) root cause > (disregarding i.e. networking issues etc) > > Just our examples, which you should probably not have: > > We had CEPH cluster running (with ACS), and there any exception in librbd > would crash JVM and the agent, but this has been fixed mostly - > Now get i.e. agent disconnect when ACS try to delete volume on CEPH (and > for some reason not succeed withing 30 minutes, volume deletion fails) - > then libvirt get's completety stuck (virsh list even dont work)...so agent > get's disconnect eventually. > > It would be good to get rid of agent disconnections in general, obviously > :) so that is why I'm asking (you are on NFS, so would like to see your > experience here). > > Thanks > > On 16 February 2018 at 21:52, Sean Lair <sl...@ippathways.com> wrote: > > > We were in the same situation as Nux. > > > > In our test environment we hit the issue with VMs not getting fenced and > > coming up on two hosts because of VM HA. However, we updated some of > the > > logic for VM HA and turned on libvirtd's locking mechanism. Now we are > > working great w/o IPMI. The locking stops the VMs from starting > elsewhere, > > and everything recovers very nicely when the host starts responding > again. > > > > We are on 4.9.3 and haven't started testing with 4.11 yet, but it may > work > > along-side IPMI just fine - it would just have affect the fencing. > > However, we *currently* prefer how we are doing it now, because if the > > agent stops responding, but the host is still up, the VMs continue > running > > and no actual downtime is incurred. Even when VM HA attempts to power on > > the VMs on another host, it just fails the power-up and the VMs continue > to > > run on the "agent disconnected" host. The host goes into alarm state and > > our NOC can look into what is wrong the agent on the host. If IPMI was > > enabled, it sounds like it would power off the host (fence) and force > > downtime for us even if the VMs were actually running OK - and just the > > agent is unreachable. > > > > I plan on submitting our updates via a pull request at some point. But I > > can also send the updated code to anyone that wants to do some testing > > before then. > > > > -----Original Message----- > > From: Marcus [mailto:shadow...@gmail.com] > > Sent: Friday, February 16, 2018 11:27 AM > > To: dev@cloudstack.apache.org > > Subject: Re: HA issues > > > > From your other emails it sounds as though you do not have IPMI > > configured, nor host HA enabled, correct? In this case, the correct thing > > to do is nothing. If CloudStack cannot guarantee the VM state (as is the > > case with an unreachable hypervisor), it should do nothing, for fear of > > causing a split brain and corrupting the VM disk (VM running on two > hosts). > > > > Clustering and fencing is a tricky proposition. When CloudStack (or any > > other cluster manager) is not configured to or cannot guarantee state > then > > things will simply lock up, in this case your HA VM on your broken > > hypervisor will not run elsewhere. This has been the case for a long time > > with CloudStack, HA would only start a VM after the original hypervisor > > agent came back and reported no VM is running. > > > > The new feature, from what I gather, simply adds the possibility of > > CloudStack being able to reach out and shut down the hypervisor to > > guarantee state. At that point it can start the VM elsewhere. If > something > > fails in that process (IPMI unreachable, for example, or bad > credentials), > > you're still going to be stuck with a VM not coming back. > > > > It's the nature of the thing. I'd be wary of any HA solution that does > not > > reach out and guarantee state via host or storage fencing before > starting a > > VM elsewhere, as it will be making assumptions. Its entirely possible a > VM > > might be unreachable or unable to access it storage for a short while, a > > new instance of the VM is started elsewhere, and the original VM comes > back. > > > > On Wed, Jan 17, 2018 at 9:02 AM Nux! <n...@li.nux.ro> wrote: > > > > > Hi Rohit, > > > > > > I've reinstalled and tested. Still no go with VM HA. > > > > > > What I did was to kernel panic that particular HV ("echo c > > > > /proc/sysrq-trigger" <- this is a proper way to simulate a crash). > > > What happened next is the HV got marked as "Alert", the VM on it was > > > all the time marked as "Running" and it was not migrated to another HV. > > > Once the panicked HV has booted back the VM reboots and becomes > > available. > > > > > > I'm running on CentOS 7 mgmt + HVs and NFS primary and secondary > storage. > > > The VM has HA enabled service offering. > > > Host HA or OOBM configuration was not touched. > > > > > > Full log http://tmp.nux.ro/W3s-management-server.log > > > > > > -- > > > Sent from the Delta quadrant using Borg technology! > > > > > > Nux! > > > www.nux.ro > > > > > > ----- Original Message ----- > > > > From: "Rohit Yadav" <rohit.ya...@shapeblue.com> > > > > To: "dev" <dev@cloudstack.apache.org> > > > > Sent: Wednesday, 17 January, 2018 12:13:33 > > > > Subject: Re: HA issues > > > > > > > I performed VM HA sanity checks and was not able to reproduce any > > > regression > > > > against two KVM CentOS7 hosts in a cluster. > > > > > > > > > > > > Without the "Host HA" feature, I deployed few HA-enabled VMs on a > > > > KVM > > > host2 and > > > > killed it (powered off). After few minutes of CloudStack attempting > > > > to > > > find why > > > > the host (kvm agent) timed out, CloudStack kicked investigators, > > > > that eventually led KVM fencers to work and VM HA job kicked to > > > > start those > > > few VMs > > > > on host1 and the KVM host2 was put to "Down" state. > > > > > > > > > > > > - Rohit > > > > > > > > <https://cloudstack.apache.org> > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > rohit.ya...@shapeblue.com > > > > www.shapeblue.com<http://www.shapeblue.com> > > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > > > > > > > > > > > > From: Rohit Yadav > > > > Sent: Wednesday, January 17, 2018 2:39:19 PM > > > > To: dev > > > > Subject: Re: HA issues > > > > > > > > > > > > Hi Lucian, > > > > > > > > > > > > The "Host HA" feature is entirely different from VM HA, however, they > > > may work > > > > in tandem, so please stop using the terms interchangeably as it may > > > cause the > > > > community to believe a regression has been caused. > > > > > > > > > > > > The "Host HA" feature currently ships with only "Host HA" provider > for > > > KVM that > > > > is strictly tied to out-of-band management (IPMI for fencing, i.e > power > > > off and > > > > recovery, i.e. reboot) and NFS (as primary storage). (We also have a > > > provider > > > > for simulator, but that's for coverage/testing purposes). > > > > > > > > > > > > Therefore, "Host HA" for KVM (+nfs) currently works only when OOBM is > > > enabled. > > > > The frameowkr allows interested parties may write their own HA > > providers > > > for a > > > > hypervisor that can use a different strategy/mechanism for > > > fencing/recovery of > > > > hosts (including write a non-IPMI based OOBM plugin) and host/disk > > > activity > > > > checker that is non-NFS based. > > > > > > > > > > > > The "Host HA" feature ships disabled by default and does not cause > any > > > > interference with VM HA. However, when enabled and configured > > correctly, > > > it is > > > > a known limitation that when it is unable to successfully perform > > > recovery or > > > > fencing tasks it may not trigger VM HA. We can discuss how to handle > > > such cases > > > > (thoughts?). "Host HA" would try couple of times to recover and > failing > > > to do > > > > so, it would eventually trigger a host fencing task. If it's unable > to > > > fence a > > > > host, it will indefinitely attempt to fence the host (the host state > > > will be > > > > stuck at fencing state in cloud.ha_config table for example) and > alerts > > > will be > > > > sent to admin who can do some manual intervention to handle such > > > situations (if > > > > you've email/smtp enabled, you should see alert emails). > > > > > > > > > > > > We can discuss how to improve and have a workaround for the case > you've > > > hit, > > > > thanks for sharing. > > > > > > > > > > > > - Rohit > > > > > > > > ________________________________ > > > > From: Nux! <n...@li.nux.ro> > > > > Sent: Tuesday, January 16, 2018 10:42:35 PM > > > > To: dev > > > > Subject: Re: HA issues > > > > > > > > Ok, reinstalled and re-tested. > > > > > > > > What I've learned: > > > > > > > > - HA only works now if OOB is configured, the old way HA no longer > > > applies - > > > > this can be good and bad, not everyone has IPMIs > > > > > > > > - HA only works if IPMI is reachable. I've pulled the cord on a HV > and > > > HA failed > > > > to do its thing, leaving me with a HV down along with all the VMs > > running > > > > there. That's bad. > > > > I've opened this ticket for it: > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-10234 > > > > > > > > Let me know if you need any extra info or stuff to test. > > > > > > > > Regards, > > > > Lucian > > > > > > > > -- > > > > Sent from the Delta quadrant using Borg technology! > > > > > > > > Nux! > > > > www.nux.ro > > > > > > > > ----- Original Message ----- > > > >> From: "Nux!" <n...@li.nux.ro> > > > >> To: "dev" <dev@cloudstack.apache.org> > > > >> Sent: Tuesday, 16 January, 2018 11:35:58 > > > >> Subject: Re: HA issues > > > > > > > >> I'll reinstall my setup and try again, just to be sure I'm working > on > > a > > > clean > > > >> slate. > > > >> > > > >> -- > > > >> Sent from the Delta quadrant using Borg technology! > > > >> > > > >> Nux! > > > >> www.nux.ro > > > >> > > > >> ----- Original Message ----- > > > >>> From: "Rohit Yadav" <rohit.ya...@shapeblue.com> > > > >>> To: "dev" <dev@cloudstack.apache.org> > > > >>> Sent: Tuesday, 16 January, 2018 11:29:51 > > > >>> Subject: Re: HA issues > > > >> > > > >>> Hi Lucian, > > > >>> > > > >>> > > > >>> If you're talking about the new HostHA feature (with KVM+nfs+ipmi), > > > please refer > > > >>> to following docs: > > > >>> > > > >>> > > > http://docs.cloudstack.apache.org/projects/cloudstack- > > administration/en/latest/hosts.html#out-of-band-management > > > >>> > > > >>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA > > > >>> > > > >>> > > > >>> We'll need to you look at logs perhaps create a JIRA ticket with > the > > > logs and > > > >>> details? If you saw ipmi based reboot, then host-ha indeed tried to > > > recover > > > >>> i.e. reboot the host, once hostha has done its work it would > schedule > > > HA for VM > > > >>> as soon as the recovery operation succeeds (we've simulator and kvm > > > based > > > >>> marvin tests for such scenarios). > > > >>> > > > >>> > > > >>> Can you see it making attempt to schedule VM ha in logs, or any > > > failure? > > > >>> > > > >>> > > > >>> - Rohit > > > >>> > > > >>> <https://cloudstack.apache.org> > > > >>> > > > >>> > > > >>> > > > >>> ________________________________ > > > >>> From: Nux! <n...@li.nux.ro> > > > >>> Sent: Tuesday, January 16, 2018 12:47:56 AM > > > >>> To: dev > > > >>> Subject: [4.11] HA issues > > > >>> > > > >>> Hi, > > > >>> > > > >>> I see there's a new HA engine for KVM and IPMI support which is > > really > > > nice, > > > >>> however it seems hit and miss. > > > >>> I have created an instance with HA offering, kernel panicked one of > > the > > > >>> hypervisors - after a while the server was rebooted via IPMI > > probably, > > > but the > > > >>> instance never moved to a running hypervisor and even after the o > > > <https://maps.google.com/?q=to+a+running+hypervisor+and+ > > even+after+the+o&entry=gmail&source=g> > > > riginal > > > >>> hypervisor came back it was still left in Stopped state. > > > >>> Is there any extra things I need to set up to have proper HA? > > > >>> > > > >>> Regards, > > > >>> Lucian > > > >>> > > > >>> -- > > > >>> Sent from the Delta quadrant using Borg technology! > > > >>> > > > >>> Nux! > > > >>> www.nux.ro > > > >>> > > > >>> rohit.ya...@shapeblue.com > > > >>> www.shapeblue.com<http://www.shapeblue.com> > > > >>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > > > > > @shapeblue > > > > > > > > > -- > > Andrija Panić > -- Andrija Panić
