Hi Jeoroen, Sorry I missed you yesterday – meant to catch up after the user group but had to run to catch my flight.
I think what you describe will work – however I have my doubts it will fail over gracefully. For pacemaker to fail over cleanly the failover has to be perfectly synched – i.e. all packets have to be written in both primary storage pools, traffic ideally quiesced – then pacemaker can move the NFS or iSCSI endpoint. If you are even a byte out you could end up with data corruption – and even if this does work I have my doubts the VMs would stay online afterwards. Anyway – as the proverb goes, the proof is in the pudding – so I can only suggest you test this out. Very interested in the result though – so please let us know how you get on (if it works it would be a good talk for the next user group ☺ ). Regards, Dag Sonstebo Cloud Architect ShapeBlue From: Jeroen Keerl <[email protected]> Reply-To: "[email protected]" <[email protected]>, "[email protected]" <[email protected]> Date: Wednesday, 23 November 2016 at 22:43 To: "[email protected]" <[email protected]> Subject: AW: 2node HA ACS Cluster with DRBD Hi Dag, Erik, thanks for your input so far. What I am aiming for is a "HyperConverged" infrastructure, if possible with just two servers. The reason why I didn't look into Ceph any further, is that they explicitly state that they'll need 3 hosts. Apart from that, the seems to be quite a lot of resource needs to get Ceph up & running. DRBD and GlusterFS look like they're not that heavy on load. GlusterFS has moved away from 2 hosts only as well, and it seems less flexible when it comes to expansion, if I recall correctly. Hence: DRBD, which runs in Master-Slave or Dual Master mode. Together with Pacemaker and NFS or iSCSI software, this could work, albeit -after overthinking it all- probably in a master-slave mode, since the shared / clustered IP address can only be available on one of two nodes. As written before: HA-Lizard does all this out of the box, including HA - if needed, and fairly well too. Since I'll be hopping over to visit the CS User Group tomorrow, I'll have no time to look into this any further until Tuesday. (Dag, will I have to chance to see you there as well?) Cheers JK [email protected] www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -----Ursprüngliche Nachricht----- Von: Dag Sonstebo [mailto:[email protected]<mailto:[email protected]>] Gesendet: Mittwoch, 23. November 2016 10:35 An: [email protected]<mailto:[email protected]> Betreff: Re: 2node HA ACS Cluster with DRBD Hi Jeroen, My twopence worth: First of all – I guess your plan is to run two nodes – each with CloudStack management, MySQL (master-slave), KVM and storage? This all depends on your user case. As a bit of an experiment or as a small scale lab I think this may work – but I would be very reluctant to rely on this for a production workload. I think you will potentially have stability and integrity issues at the storage level in a HA failover scenario, on top of this I don’t think this will scale well. You may also end up with considerable storage overhead depending on number of nodes + technology used. With two nodes you immediately only have 50% max space utilization. Putting all of that aside I think it could work, I’ve played with similar ideas in the past (without actually spending enough time to get it working). I think you could get around the heartbeating / split brain situations relatively easily. The CloudStack and MySQL installs as well as KVM should work OK, but your challenge will be storage, which both has to work in the synchronized setup you want + very importantly fail over gracefully. I guess you would probably look at Ceph - if Wido or any of the other Ceph users read this they are probably better placed to advise. Regards, Dag Sonstebo Cloud Architect ShapeBlue From: Jeroen Keerl <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Wednesday, 23 November 2016 at 00:05 To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: 2node HA ACS Cluster with DRBD All, I’ve been pondering a bit about a possible small-scale, HA setup: Hardware 2 Nodes, each with 2 1Gbe and 2 10Gbe NICs Both have hardware RAID controllers, redundant PS, Array controller with BBWC and hot plug SAS drives. If we’d “combine” the quick setup for CentOS6 ( http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.9/qig.html ) with the “additional management server” part ( http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.9/management-server/index.html#additional-management-servers ) then we’d have 2 management servers. To make the local storage of both nodes “HA”, one could use DRBD in a “primary-primary” setup, so VM migration is possible as well. (I am not yet clear on how to present this storage to the nodes) To avoid split brain-situations, I see two possible strategi es: 1) Direct cabling of the 10Gbe controllers => Loss of connection can only mean either one host is dead, or at least its NIC has died. 2) Usage of IPMI / Fencing Did anybody gather experience in a setup like that? Does anybody have any thoughts on this – improvements, comments, doubts: Hit me! Cheers, JK · I came to this “setup”, after a few issues with Xen Server 6.5 combined with “HA-Lizard”, which actually uses DRBD in combination with TGTD iSCSI. On the one side, I’ve had quite a few stability issues with Xen 6.5 and on the other side: The CS MS needs to be outside of this cluster … but still needs to be installed on a redundant / HA piece of kit. Jeroen Keerl Keerl IT Services GmbH Birkenstraße 1b . 21521 Aumühle +49 177 6320 317 www.keerl-it.com<http://www.keerl-it.com><http://www.keerl-it.com/> [email protected]<mailto:[email protected]><mailto:[email protected]<mailto:[email protected]>> Geschäftsführer. Jacobus J. Keerl Registergericht Lubeck. HRB-Nr. 14511 Unsere Allgemeine Geschäftsbedingungen finden Sie hier.<http://www.keerl-it.com/AGB.pdf> [cid:d3544f14.06fb964e.PNG.90ac9d6c] [email protected]<mailto:[email protected]> www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue Jeroen Keerl Keerl IT Services GmbH Birkenstraße 1b . 21521 Aumühle +49 177 6320 317 www.keerl-it.com<http://www.keerl-it.com/> [email protected]<mailto:[email protected]> Geschäftsführer. Jacobus J. Keerl Registergericht Lubeck. HRB-Nr. 14511 Unsere Allgemeine Geschäftsbedingungen finden Sie hier.<http://www.keerl-it.com/AGB.pdf> [cid:d3544f14.06fb964e.PNG.d5855133]
