KVM HostHA was developed to bring it to a closer parity with how VMware vSphere handles its HA.
Basically in old model - there were several corner cases where CloudStack did not know whether Hypervisor crashed or just lost connectivity to Management server. We’ve added a logic to make sure that hypervisor is truly dead - only then we can choose to use IPMI, power off or reboot hypervisor and bring up guest VMs elsewhere. There is a lot more that goes on behind the scenes - Rohit did a talk about it and it should be available on YouTube. On Wed, Mar 14, 2018 at 7:36 AM Parth Patel <[email protected]> wrote: > Hi Paul and Adrina, > > I don't know the functioning of Host-HA features but what Paul explained, > my ACS 4.11 does the same without even host HA or ipmi access. As I stated > earlier multiple times, without host HA and ipmi, my ha-enabled VMs > executing on a normal host get restarted on another suitable host in > cluster after approximately 3 minutes of event ping timeout. After which > the cloudstack agent with no connection to management server because of > unplugged NIC (all my machines currently have only one NIC / whole zone is > in a flat network) reboots itself (the reason was explained by Rohit in an > another thread). The management server marks the host down and only > Ha-enabled VMs executing on it get restarted on another host (without any > mention of host HA or ipmi or fencing in management server logs) while > normal VMs executing on it are stopped. > > I don't know if this was a desired outcome, but I think my current ACS 4.11 > installation has features (at least performs some ;) provided by Host HA > without configuring it or ipmi. > > Regards, > Parth Patel > > On Wed 14 Mar, 2018, 18:41 Boris Stoyanov, <[email protected]> > wrote: > > > yes, KVM + NFS shared storage. > > > > Boris. > > > > > > [email protected] > > www.shapeblue.com > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > @shapeblue > > > > > > > > > On 14 Mar 2018, at 14:51, Andrija Panic <[email protected]> > wrote: > > > > > > Hi Boris, > > > > > > ok thanks for the explanation - that makes sense, and covers my > > "exception > > > case" that I have. > > > > > > This is atm only available for NFS as I could read (KVM on NFS) ? > > > > > > Cheers > > > > > > On 14 March 2018 at 13:02, Boris Stoyanov < > [email protected]> > > > wrote: > > > > > >> Hi Andrija, > > >> > > >> There’s two types of checks Host-HA is doing to determine if host if > > >> healthy. > > >> > > >> 1. Health checks - pings the host as soon as there’s connection issues > > >> with the agent > > >> > > >> If that fails, > > >> > > >> 2. Activity checks - checks if there are any writing operations on the > > >> Disks of the VMs that are running on the hosts. This is to determine > if > > the > > >> VMs are actually alive and executing processes. Only if no disk > > operations > > >> are executed on the shared storage, only then it’s trying to Recover > the > > >> host with IPMI call, if that eventually fails, it migrates the VMs to > a > > >> healthy host and Fences the faulty one. > > >> > > >> Hope that explains your case. > > >> > > >> Boris. > > >> > > >> > > >> [email protected] > > >> www.shapeblue.com > > >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > >> @shapeblue > > >> > > >> > > >> > > >>> On 14 Mar 2018, at 13:53, Andrija Panic <[email protected]> > > wrote: > > >>> > > >>> Hi Paul, > > >>> > > >>> sorry to bump in the middle of the thread, but just curious about the > > >> idea > > >>> behing host-HA and why it behaves the way you exlained above: > > >>> > > >>> > > >>> Would it be more sense (or not?), that when MGMT detects agents is > > >>> unreachable or host unreachable (or after unsuccessful i.e. agent > > >> restart, > > >>> etc...,to be defined), to actually use IPMI to STONITH the node, thus > > >>> making sure no VMS running and then to really start all HA-enabled > VMs > > on > > >>> other hosts ? > > >>> > > >>> I'm just trying to make parallel to the corosync/pacemaker as > > clustering > > >>> suite/services in Linux (RHEL and others), where when majority of > nodes > > >>> detect that one node is down, a common thing (especially for shared > > >>> storage) is to STONITH that node, make sure it;s down, then move > > >> "resource" > > >>> (in our case VMs) to other cluster nodes ? > > >>> > > >>> I see it's actually much broader setup per > > >>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA but > > >> again - > > >>> whole idea (in my head at least...) is when host get's down, we make > > sure > > >>> it's down (avoid VM corruption, by doint STONITH to that node) and > then > > >>> start HA VMs on ohter hosts. > > >>> > > >>> I understand there might be exceptions as I have right now (4.8) - > > >> libvirt > > >>> get stuck (librbd exception or similar) so agent get's disconnected, > > but > > >>> VMs are still running fine... (except DB get messed up, all NICs > loose > > >>> isolation_uri, VR's loose MAC addresses and other IP addresses > etc...) > > >>> > > >>> > > >>> Thanks > > >>> Andrija > > >>> > > >>> > > >>> > > >>> > > >>> On 14 March 2018 at 10:57, Jon Marshall <[email protected]> > wrote: > > >>> > > >>>> That would make sense. > > >>>> > > >>>> > > >>>> I have another server being used for something else at the moment > so I > > >>>> will add that in and update this thread when I have tested > > >>>> > > >>>> > > >>>> Jon > > >>>> > > >>>> > > >>>> ________________________________ > > >>>> From: Paul Angus <[email protected]> > > >>>> Sent: 14 March 2018 09:16 > > >>>> To: [email protected] > > >>>> Subject: RE: KVM HostHA > > >>>> > > >>>> I'd need to do some testing, but I suspect that your problem is that > > you > > >>>> only have two hosts. At the point that one host is deemed out of > > >> service, > > >>>> you only have one host left. With only one host, CloudStack will > show > > >> the > > >>>> cluster as ineligible. > > >>>> > > >>>> It is extremely common for any system working as a cluster to > require > > a > > >>>> minimum starting point of 3 nodes to be able to function. > > >>>> > > >>>> > > >>>> Kind regards, > > >>>> > > >>>> Paul Angus > > >>>> > > >>>> [email protected] > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > >>>> @shapeblue > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> -----Original Message----- > > >>>> From: Jon Marshall <[email protected]> > > >>>> Sent: 14 March 2018 08:36 > > >>>> To: [email protected] > > >>>> Subject: Re: KVM HostHA > > >>>> > > >>>> Hi Paul > > >>>> > > >>>> > > >>>> My testing does indeed end up with the failed host in maintenance > mode > > >> but > > >>>> the VMs are never migrated. As I posted earlier the management > server > > >> seems > > >>>> to be saying there is no other host that the VM can be migrated to. > > >>>> > > >>>> > > >>>> Couple of questions if you have the time to respond - > > >>>> > > >>>> > > >>>> 1) this article seems to suggest a reboot or powering off a host > will > > >> end > > >>>> result in the VMs being migrated and this was on CS v 4.2.1 back in > > >> 2013 so > > >>>> does Host HA do something different > > >>>> > > >>>> > > >>>> 2) Whenever one of my two nodes is taken down in testing the active > > >>>> compute nodes HA status goes from Available to Ineligible. Should > this > > >>>> happen ie. is it going to Ineligible stopping the manager from > > migrating > > >>>> the VMs. > > >>>> > > >>>> > > >>>> Apologies for all the questions but I just can't get this to work at > > the > > >>>> moment. If I do eventually get it working I will do a write up for > > >> others > > >>>> with same issue :) > > >>>> > > >>>> > > >>>> ________________________________ > > >>>> From: Paul Angus <[email protected]> > > >>>> Sent: 14 March 2018 07:45 > > >>>> To: [email protected] > > >>>> Subject: RE: KVM HostHA > > >>>> > > >>>> Hi Parth, > > >>>> > > >>>> Two answer your questions, VM-HA does not restart VMs on an > alternate > > >> host > > >>>> if the original host goes down. The management server (without > > host-HA) > > >>>> cannot tell what happened to the host. It cannot tell if there was > a > > >>>> failure in the agent, loss of connectivity to the management NIC or > if > > >> the > > >>>> host is truly down. In the first two scenarios, the guest VMs can > > >> still be > > >>>> running perfectly well, and to restart them elsewhere would be very > > >>>> dangerous. Therefore, the correct thing to do is - nothing but > alert > > >> the > > >>>> operator. These scenarios are what Host-HA was introd > > < > https://maps.google.com/?q=These+scenarios+are+what+Host-HA+was+introd&entry=gmail&source=g > >uced > > for. > > >>>> > > >>>> Wrt to STONITH, if no disk activity is detected on the host, host-HA > > >> will > > >>>> try to restart (via IPMI) the host. If, after a configurable number > of > > >>>> attempts, the host agent still does not check in, then host-HA will > > shut > > >>>> down the host (via IPMA), trigger VM-HA and mark the host as > > >> in-maintenance. > > >>>> > > >>>> > > >>>> > > >>>> [email protected] > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> -----Original Message----- > > >>>> From: Parth Patel <[email protected]> > > >>>> Sent: 14 March 2018 05:05 > > >>>> To: [email protected] > > >>>> Subject: Re: KVM HostHA > > >>>> > > >>>> Hi Paul, > > >>>> > > >>>> Thanks for the clarification. I currently don't have an ipmi enabled > > >>>> hardware (in test environment), but it will be beneficial if you can > > >> help > > >>>> me clear out some basic concepts of it: > > >>>> - If HA-enabled VMs are autostarted on another host when current > host > > >> goes > > >>>> down, what is the need or purpose of HA-host? (other than management > > >> server > > >>>> able to remotely control it's power interfaces) > > >>>> - I understood the "Shoot-the-other-node-in-the-head" (STONITH) > > >> approach > > >>>> ACS uses to fence the host, but I couldn't find what mechanism or > > events > > >>>> trigger this? > > >>>> > > >>>> Thanks and regards, > > >>>> Parth Patel > > >>>> > > >>>> On Wed, 14 Mar 2018 at 02:22 Paul Angus <[email protected]> > > >> wrote: > > >>>> > > >>>>> The management server doesn't ping the host through IPMI. However > > if > > >>>>> IPMI is not available, you will not be able to use Host HA, as > there > > >>>>> is no way for CloudStack to 'fence' the host - that is shut it down > > to > > >>>>> be sure that a VM cannot start again on that host. > > >>>>> > > >>>>> I can explain why that is necessary if you wish. > > >>>>> > > >>>>> > > >>>>> Kind regards, > > >>>>> > > >>>>> Paul Angus > > >>>>> > > >>>>> [email protected] > > >>>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> -----Original Message----- > > >>>>> From: Parth Patel <[email protected]> > > >>>>> Sent: 13 March 2018 16:57 > > >>>>> To: [email protected] > > >>>>> Cc: Jon Marshall <[email protected]> > > >>>>> Subject: Re: KVM HostHA > > >>>>> > > >>>>> Hi Jon and Victor, > > >>>>> > > >>>>> I think the management server pings your host using ipmi (I really > > >>>>> don't hope this is the case). > > >>>>> In my case, I did not have OOBM enabled at all (my hardware didn't > > >>>>> support > > >>>>> it) > > >>>>> I think you could disable OOBM and/or HA-Host and give that a try > :) > > >>>>> > > >>>>> On Tue, 13 Mar 2018 at 20:40 victor <[email protected]> wrote: > > >>>>> > > >>>>>> Hello Guys, > > >>>>>> > > >>>>>> I have tried the following two cases. > > >>>>>> > > >>>>>> 1, "echo c > /proc/sysrq-trigger" > > >>>>>> > > >>>>>> 2, Pulled the network cable of one of the host > > >>>>>> > > >>>>>> In both cases, the following happened. > > >>>>>> > > >>>>>> ===== > > >>>>>> 2018-03-13 08:22:54,978 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] > > >>>>>> (AgentTaskPool-15:ctx-c8d9f5d2) (logid:c0a3d2da) Notifying other > > >>>>>> nodes of to disconnect > > >>>>>> 2018-03-13 08:22:54,983 INFO [c.c.a.m.AgentManagerImpl] > > >>>>>> (AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Host 4 is > > >>>>>> disconnecting with event AgentDisconnected > > >>>>>> 2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl] > > >>>>>> (AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Host 4 is already > > >>>>>> Alert > > >>>>>> 2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl] > > >>>>>> (AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Deregistering > link > > >>>>>> for > > >>>>>> 4 with state Alert > > >>>>>> 2018-03-13 08:22:54,985 DEBUG [c.c.a.m.AgentManagerImpl] > > >>>>>> (AgentTaskPool-16:ctx-d8204625) (logid:ffe4a426) Remove Agent : 4 > > >>>>>> ===== > > >>>>>> > > >>>>>> But nothing happened for the vm's in that node. I have waited for > > >>>>>> one hour and the VM's in that node has been migrated to the other > > >>>>>> available hosts. I think the issue is that the management server > > >>>>>> still thinks that the VM's in that host is running. Please check > the > > >>>>>> following logs > > >>>>>> > > >>>>>> ======= > > >>>>>> 2018-03-13 11:08:25,882 DEBUG [c.c.c.CapacityManagerImpl] > > >>>>>> (CapacityChecker:ctx-1d8378af) (logid:ae906a50) Found 1 VMs on > host > > >>>>>> 4 > > >>>>>> 2018-03-13 11:08:25,888 DEBUG [c.c.c.CapacityManagerImpl] > > >>>>>> (CapacityChecker:ctx-1d8378af) (logid:ae906a50) Found 0 VM, not > > >>>>>> running on host 4 ======== > > >>>>>> > > >>>>>> > > >>>>>> On 03/13/2018 04:20 PM, Jon Marshall wrote: > > >>>>>>> I tried "echo c > /proc/sysrq-trigger" which stopped me getting > > >>>>>>> into the > > >>>>>> server but it did not stop the server responding to an ipmitool > > >>>>>> request on the manager eg - > > >>>>>>> > > >>>>>>> > > >>>>>>> "ipmitool -I lanplus -H 172.16.7.29 -U admin3 -P letmein chassis > > >>>>> status" > > >>>>>>> > > >>>>>>> > > >>>>>>> from the management server got an answer saying the chassis power > > >>>>>>> was on > > >>>>>> so CS never registered the compute node as down. > > >>>>>>> > > >>>>>>> > > >>>>>>> I am obviously doing something wrong but cannot work it out. > > >>>>>>> > > >>>>>>> > > >>>>>>> The management server has one NIC - 172.16.7.4 > > >>>>>>> > > >>>>>>> > > >>>>>>> Each compute node has 3 NICs - > > >>>>>>> > > >>>>>>> > > >>>>>>> cnode1 > > >>>>>> cnode2 > > >>>>>>> > > >>>>>>> > > >>>>>>> mangement NIC 172.16.7.5 172.16.7.6 > > >>>>>>> > > >>>>>>> vm NIC 172.16.6.130 > > 172.16.6.131 > > >>>>>>> > > >>>>>>> storage - 172.16.250.4 > > 172.16.250.5 > > >>>>>>> > > >>>>>>> > > >>>>>>> Dell LOM (for Idrac) 172.16.7.29 172.16.7.30 > > >>>>>>> > > >>>>>>> > > >>>>>>> the dell LOM IPs are the ones used to configure OOBM in the UI > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> If I pull the storage NIC presumably nothing will happen as the > > >>>>>>> ipmitool > > >>>>>> check is running across the management NIC so I need to pull both > ? > > >>>>>>> > > >>>>>>> My understanding of host HA was the management server monitored > > >>>>>>> the > > >>>>>> compute nodes using ipmitool and if it did not get a response > > >>>>>> because the host was down it would fence off that host and move > the > > >>>>>> VMs to an active compute node. > > >>>>>>> > > >>>>>>> This is obviously too simplistic so could someone explain how it > > >>>>>>> is > > >>>>>> meant to work and what it is protecting against ? > > >>>>>>> > > >>>>>>> ________________________________ > > >>>>>>> From: Paul Angus <[email protected]> > > >>>>>>> Sent: 13 March 2018 07:01 > > >>>>>>> To: [email protected] > > >>>>>>> Subject: RE: KVM HostHA > > >>>>>>> > > >>>>>>> Hi all, > > >>>>>>> > > >>>>>>> One small note, unplugging the management NIC will only cause an > > >>>>>>> HA > > >>>>>> event if the storage is running over that NIC also. > > >>>>>>> > > >>>>>>> Is the storage is over a separate NIC then, the guest VMs will > > >>>>>>> continue > > >>>>>> to run when the mgmt. NIC is unplugged, Host HA will detect the > disk > > >>>>>> activity and conclude that there is nothing it can do, as the VMs > > >>>>>> are still running other than mark the hosts as degra > <https://maps.google.com/?q=ll+running+other+than+mark+the+hosts+as+degra&entry=gmail&source=g> > ded. > > >>>>>>> > > >>>>>>> > > >>>>>>> Kind regards, > > >>>>>>> > > >>>>>>> Paul Angus > > >>>>>>> > > >>>>>>> [email protected] > > >>>>>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png > > >>>> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png] > > >>>> > > >>>> ]< > > >>>>>> http://www.shapeblue.com/> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> > > >>>>>>> Shapeblue - The CloudStack Company > > >>>>> < > > https://maps.google.com/?q=ack+Company+%0D%0A%3E+%3E+w&entry=gmail&so > > >>>>> urce=g> > > >>>>> <http://www.shapeblue.com/> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > > >>>>>>> CSForge is > > >>>>>> a framework developed by ShapeBlue to deli > > >>>>>> < > > https://maps.google.com/?q=is+a+framework+developed+by+ShapeBlue+to > > >>>>>> +d eli&entry=gmail&source=g>ver the rapid deployment of a > > >>>>>> standardised ... > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> -----Original Message----- > > >>>>>>> From: Parth Patel <[email protected]> > > >>>>>>> Sent: 12 March 2018 17:35 > > >>>>>>> To: [email protected] > > >>>>>>> Subject: Re: KVM HostHA > > >>>>>>> > > >>>>>>>> Hi Jon, > > >>>>>>>> > > >>>>>>>> As I said, in my case, making the host HA didn't work but by > just > > >>>>>>>> having a HA VM running on host and executing - (WARNING) "echo c > > >>>>>>>>> /proc/sysrq-trigger" to simulate a kernel crash on host, the > > >>>>>>>> management server registered it as down and started the VM on > > >>>>>>>> another host. I know I've suggested this before but I insist you > > >>>>>>>> give this a try. Also, you don't need to completely power off > the > > >>>>>>>> machine manually but just plugging out the network cable works > > >>>>>>>> fine. The cloudstack agent after losing connection to management > > >>>>>>>> server auto reboots because of KVM heartbeat check shell script > > >>>>>>>> mentioned by Rohit Yadav to one of my earlier queries in other > > >>>> thread. > > >>>>>>>> > > >>>>>>>> On Mon 12 Mar, 2018, 21:23 Jon Marshall, <[email protected] > > > > >>>>> wrote: > > >>>>>>>> Hi Paul > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Thanks for the response. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> I think I am not understanding how it was meant to work then. My > > >>>>>>>> understanding was that the manager used ipmitool to just keep > > >>>>>>>> querying the compute nodes as to their status so I assumed it > > >>>>>>>> didn't matter how you shut the node down, once it was down the > > >>>>>>>> manager would get no response and mark it as down (which it > does). > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> I am in testing mode so I think I will just go and pull the > power > > >>>>>>>> and see what happens :) > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Thanks > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Jon > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> ________________________________ > > >>>>>>>> From: Paul Angus <[email protected]> > > >>>>>>>> Sent: 12 March 2018 15:31 > > >>>>>>>> To: [email protected] > > >>>>>>>> Subject: RE: KVM HostHA > > >>>>>>>> Hi Jon, > > >>>>>>>> > > >>>>>>>> I think that what you guys are finding, is that a controlled > host > > >>>>>>>> shutdown, which will cause the agent to shutdown cleanly; Is not > > >>>>>>>> considered an HA event. I wouldn't expect CloudStack to take any > > >>>>>>>> action if you shut down a host, only if the host (agent) stops > > >>>>>> responding. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Kind regards, > > >>>>>>>> > > >>>>>>>> Paul Angus > > >>>>>>>> > > >>>>>>>> [email protected] > > >>>>>>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png > > >>>> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png] > > >>>> > > >>>> ]< > > >>>>>> http://www.shapeblue.com/> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> > > >>>>>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > > >>>>>>> CSForge > > >>>>> is > > >>>>>> a framework developed by ShapeBlue to deliver the rapid deployment > > >>>>>> of a standardised ... > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png > > >>>>>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png > > >>>> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png] > > >>>> > > >>>> ] > > >>>>>>> > > >>>>>>> ]< > > >>>>>>>> http://www.shapeblue.com/> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png > > >>>> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png] > > >>>> > > >>>> ]< > > >>>>>> http://www.shapeblue.com/> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> > > >>>>>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > > >>>>>>> CSForge > > >>>>> is > > >>>>>> a framework developed by ShapeBlue to deliver the rapid deployment > > >>>>>> of a standardised ... > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack > <https://maps.google.com/?q=deployment+framework+for+Apache+CloudStack+&entry=gmail&source=g>IaaS > Clouds. CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png > > >>>> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png] > > >>>> > > >>>> ]< > > >>>>>> http://www.shapeblue.com/> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> > > >>>>>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > > >>>>>>> CSForge > > >>>>> is > > >>>>>> a framework developed by ShapeBlue to deliver the rapid deployment > > >>>>>> of a standardised ... > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png > > >>>> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png] > > >>>> > > >>>> ]< > > >>>>>> http://www.shapeblue.com/> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> > > >>>>>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> [http://www.shapeblue.com/wp-content/uploads/2017/06/logo.png]< > > >>>> http://www.shapeblue.com/> > > >>>> > > >>>> Shapeblue - The CloudStack Company<http://www.shapeblue.com/> > > >>>> www.shapeblue.com<http://www.shapeblue.com> > > >>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > CSForge > > >> is a > > >>>> framework developed by ShapeBlue to deliver the rapid deployment of > a > > >>>> standardised ... > > >>>> > > >>>> > > >>>> > > >>>>>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > > >>>>>>> CSForge > > >>>>> is > > >>>>>> a framework developed by ShapeBlue to deliver < > > >>>>> > > https://maps.google.com/?q=framework+developed+by+ShapeBlue+to+deliver > > >>>>> &entry=gmail&source=g > > >>>>>> > > >>>>>> the rapid deployment of a standardised ... > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>>> Rapid deployment framework for Apache CloudStack IaaS Clouds. > > >>>> CSForge > > >>>>>>>> is a framework developed by ShapeBlue to deliver the rapid > > >>>> deployment > > >>>>>>>> of a standardised ... > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> -----Original Message----- > > >>>>>>>> From: Jon Marshall <[email protected]> > > >>>>>>>> Sent: 12 March 2018 15:15 > > >>>>>>>> To: [email protected] > > >>>>>>>> Subject: Re: KVM HostHA > > >>>>>>>> > > >>>>>>>> I have the same issue here and am not entirely sure what the > > >>>> behaviour > > >>>>>>>> should be. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> I have one manager node and 2 compute nodes running 4.11 with > ipmi > > >>>>>> working > > >>>>>>>> correctly. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> From the UI under HA - > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> HA Enabled Yes > > >>>>>>>> HA State Available > > >>>>>>>> HA Provider kvmhaprovider > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> although interestingly from the "Details" tab it shows - > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> HA enabled No > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> which I assume is a cosmetic issue ? > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On each compute node I have one HA enabled VM and one non HA > > enabled > > >>>>> VM. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> I power off a compute node and the UI updates the host status > and > > >>>> the > > >>>>>> VMs > > >>>>>>>> on that node stop responding but they never fail over to the > other > > >>>>> node. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Couple of things I noticed - > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> 1) as soon as i power off the compute node the HA state on the > > other > > >>>>>> node > > >>>>>>>> shows "Ineligible" > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> 2) In the UI the instances all still show as green even though > two > > >>>> of > > >>>>>> them > > >>>>>>>> are not available > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Any help much appreciated > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> ________________________________ > > >>>>>>>> From: victor <[email protected]> > > >>>>>>>> Sent: 07 March 2018 17:01 > > >>>>>>>> To: [email protected] > > >>>>>>>> Subject: KVM HostHA > > >>>>>>>> > > >>>>>>>> Hello Guys, > > >>>>>>>> > > >>>>>>>> I have installed cloudstack 4.11. I have enabled HA for each > > hosts I > > >>>>>> have > > >>>>>>>> added. I have also added ipmi successfully (using ipmi driver). > > >>>>>>>> The hosts are showing like the following. > > >>>>>>>> > > >>>>>>>> ======= > > >>>>>>>> > > >>>>>>>> HA Enabled Yes > > >>>>>>>> HA State Available > > >>>>>>>> HA Provider kvmhaprovider > > >>>>>>>> > > >>>>>>>> ====== > > >>>>>>>> > > >>>>>>>> Also the host is showing the following correctly > > >>>>>>>> > > >>>>>>>> Resource state --> Enabled > > >>>>>>>> State --> UP > > >>>>>>>> Power state --> On > > >>>>>>>> > > >>>>>>>> So I have shutdown one of the hosts to see how the KVM hosts Ha > is > > >>>>>>>> working. I have waited for half an hour. But nothing has > happened. > > >>>>> What > > >>>>>>>> will happen to the VM's in that host, if the host failed to back > > up. > > >>>>>>>> There isn't much from logs. > > >>>>>>>> > > >>>>>>>> Regards > > >>>>>>>> Victor > > >>>>>>>> > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >>> > > >>> > > >>> -- > > >>> > > >>> Andrija Panić > > >> > > >> > > > > > > > > > -- > > > > > > Andrija Panić > > > > >
