Leen, thanks for explaining things. I does make sense now.
Unfortunately, it does look like this technology would not fulfill my requirements as I do need to have an ability to perform maintenance without shutting down vms. I will open another topic to discuss possible solutions. Thanks for all your help Andrei ----- Original Message ----- From: "Leen Besselink" <[email protected]> To: [email protected] Cc: "Andrei Mikhailovsky" <[email protected]> Sent: Sunday, 11 May, 2014 11:41:08 PM Subject: Re: [ceph-users] NFS over CEPH - best practice On Sun, May 11, 2014 at 09:24:30PM +0100, Andrei Mikhailovsky wrote: > Sorry if these questions will sound stupid, but I was not able to find an > answer by googling. > As the Astralians say: no worries, mate. It's fine. > 1. Does iSCSI protocol support having multiple target servers to serve the > same disk/block device? > No, I don't think so. What does work is active/standby failover. I suggest to have some kind of clustering, because as far as I can see, you never want to have 2 target servers active if they don't share state (as far as I know there is no Linux iSCSI-target server which can share state between 2 targets). When there is a failure there is time to have all targets offline for a brief moment, before the second target comes online. The initiators should be able to handle short interruptions. > In case of ceph, the same rbd disk image. I was hoping to have multiple > servers to mount the same rbd disk and serve it as an iscsi LUN. This LUN > would be used as a vm image storage on vmware / xenserver. > You'd have one server which handles a LUN, with it goes down, an other should take over the target IP-address and handle requests for that LUN. > 2.Does iscsi multipathing provide failover/HA capability only on the > initiator side? The docs that i came across all mention multipathing on the > client side, like using two different nics. I did not find anything about > having multiple nics on the initiator connecting to multiple iscsi target > servers. > Multipathing for iSCSI, as I see it, only does one thing: it can be used to create multiple network paths between the initiator and the target. They can be used for resiliance (read: failover) or for loadbalancing when you need more bandwidth. The way I would do it is to have 2 switches and connect each initiator and each target to both switches. Also you would have 2 IP-subnets. So both the target and initiator would have 2 IP-addresses, one from each subnet. So for example: the target would have: 10.0.1.1 and 10.0.2.1 and the initiator: 10.0.1.11 and 10.0.2.11 Then you run the IP-traffic for 10.0.1.x on switch 1 and the 10.0.2.x traffic on switch 2. Thus, you have created a resilient set up: The target has multiple connections to the network, the initiator has multiple connections to the network and you can also handle a switch failover. > I was hoping to have resilient solution on the storage side so that I can > perform upgrades and maintenance without needing to shutdown vms running on > vmware/xenserver. Is this possible with iscsi? > The failover set up is mostly to handle failures, not really great for maintenance because it does give a short interruption in service. Like 30 seconds or so of no writing to the LUN. That might not be a problem for you, I don't know, but it is at least something to be aware of. And also something you should test when you've build the setup. > Cheers > Hope that helps. > Andrei > ----- Original Message ----- > > From: "Leen Besselink" <[email protected]> > To: [email protected] > Sent: Saturday, 10 May, 2014 8:31:02 AM > Subject: Re: [ceph-users] NFS over CEPH - best practice > > On Fri, May 09, 2014 at 12:37:57PM +0100, Andrei Mikhailovsky wrote: > > Ideally I would like to have a setup with 2+ iscsi servers, so that I can > > perform maintenance if necessary without shutting down the vms running on > > the servers. I guess multipathing is what I need. > > > > Also I will need to have more than one xenserver/vmware host servers, so > > the iscsi LUNs will be mounted on several servers. > > > > So you have multiple machines talking to the same LUN at the same time ? > > You'll have to co-ordinate how changes are written to the backing store, > normally you'd have the virtualization servers use some kind of protocol. > > When it's SCSI there are the older Reserve/Release commands and the newer > SCSI-3 Persistent Reservation commands. > > (i)SCSI allows multiple changes to be in-flight, without coordination things > will go wrong. > > Below it was mentioned that you can disable the cache for rbd, if you have no > coordination protocol you'll need to do the same on the iSCSI-side. > > I believe when you do that it will be slower, but it might work. > > > Would the suggested setup not work for my requirements? > > > > It depends on VMWare if they allow such a setup. > > Then there is an other thing. How do the VMWare machines coordinate which VM > they should be running ? > > I don't know VMWare but usually if you have some kind of clustering setup > you'll need to have a 'quorum'. > > A lot of times the quorum is handled by a quorum disk with the SCSI > coordiation protocols mentioned above. > > An other way to have a quorum is to have a majority voting system with an > un-even number of machines talking over the network. This is what Ceph > monitor nodes do. > > As an example of a clustering system that allows it to be used without a > quorum disk with only 2 machines talking over the network is Linux Pacemaker. > When something bad happends, one machine will just turn off the power of the > other machine to prevent things going wrong (this is called STONITH). > > > Andrei > > ----- Original Message ----- > > > > From: "Leen Besselink" <[email protected]> > > To: [email protected] > > Sent: Thursday, 8 May, 2014 9:35:21 PM > > Subject: Re: [ceph-users] NFS over CEPH - best practice > > > > On Thu, May 08, 2014 at 01:24:17AM +0200, Gilles Mocellin wrote: > > > Le 07/05/2014 15:23, Vlad Gorbunov a écrit : > > > >It's easy to install tgtd with ceph support. ubuntu 12.04 for example: > > > > > > > >Connect ceph-extras repo: > > > >echo deb http://ceph.com/packages/ceph-extras/debian $(lsb_release > > > >-sc) main | sudo tee /etc/apt/sources.list.d/ceph-extras.list > > > > > > > >Install tgtd with rbd support: > > > >apt-get update > > > >apt-get install tgt > > > > > > > >It's important to disable the rbd cache on tgtd host. Set in > > > >/etc/ceph/ceph.conf: > > > >[client] > > > >rbd_cache = false > > > [...] > > > > > > Hello, > > > > > > > Hi, > > > > > Without cache on the tgtd side, it should be possible to have > > > failover and load balancing (active/avtive) multipathing. > > > Have you tested multipath load balancing in this scenario ? > > > > > > If it's reliable, it opens a new way for me to do HA storage with iSCSI ! > > > > > > > I have a question, what is your use case ? > > > > Do you need SCSI-3 persistent reservations so multiple machines can use the > > same LUN at the same time ? > > > > Because in that case I think tgtd won't help you. > > > > Have a good day, > > Leen. > > _______________________________________________ > > ceph-users mailing list > > [email protected] > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
