On Mon, May 12, 2014 at 10:52:33AM +0100, Andrei Mikhailovsky wrote: > Leen, > > thanks for explaining things. I does make sense now. > > Unfortunately, it does look like this technology would not fulfill my > requirements as I do need to have an ability to perform maintenance without > shutting down vms. >
Sorry for being cautious. I've seen certain iSCSI-initiators act that way. I do not know if that is representative for other iSCSI-initiators. So I don't know if that applies to VMWare. During failover reads/writes would be stalled of course. When properly configured, failover of the target could be done in seconds. > I will open another topic to discuss possible solutions. > > Thanks for all your help > > Andrei > ----- Original Message ----- > > From: "Leen Besselink" <l...@consolejunkie.net> > To: ceph-users@lists.ceph.com > Cc: "Andrei Mikhailovsky" <and...@arhont.com> > Sent: Sunday, 11 May, 2014 11:41:08 PM > Subject: Re: [ceph-users] NFS over CEPH - best practice > > On Sun, May 11, 2014 at 09:24:30PM +0100, Andrei Mikhailovsky wrote: > > Sorry if these questions will sound stupid, but I was not able to find an > > answer by googling. > > > > As the Astralians say: no worries, mate. > > It's fine. > > > 1. Does iSCSI protocol support having multiple target servers to serve the > > same disk/block device? > > > > No, I don't think so. What does work is active/standby failover. > > I suggest to have some kind of clustering, because as far as I can see, you > never want to have 2 target servers active if they don't share state > (as far as I know there is no Linux iSCSI-target server which can share state > between 2 targets). > > When there is a failure there is time to have all targets offline for a brief > moment, before the second target comes online. The initiators should be able > to handle short interruptions. > > > In case of ceph, the same rbd disk image. I was hoping to have multiple > > servers to mount the same rbd disk and serve it as an iscsi LUN. This LUN > > would be used as a vm image storage on vmware / xenserver. > > > > You'd have one server which handles a LUN, with it goes down, an other should > take over the target IP-address and handle requests for that LUN. > > > 2.Does iscsi multipathing provide failover/HA capability only on the > > initiator side? The docs that i came across all mention multipathing on the > > client side, like using two different nics. I did not find anything about > > having multiple nics on the initiator connecting to multiple iscsi target > > servers. > > > > Multipathing for iSCSI, as I see it, only does one thing: it can be used to > create multiple network paths between the initiator and the target. They can > be used for resiliance (read: failover) or for loadbalancing when you need > more bandwidth. > > The way I would do it is to have 2 switches and connect each initiator and > each target to both switches. Also you would have 2 IP-subnets. > > So both the target and initiator would have 2 IP-addresses, one from each > subnet. > > So for example: the target would have: 10.0.1.1 and 10.0.2.1 and the > initiator: 10.0.1.11 and 10.0.2.11 > > Then you run the IP-traffic for 10.0.1.x on switch 1 and the 10.0.2.x traffic > on switch 2. > > Thus, you have created a resilient set up: The target has multiple > connections to the network, the initiator has multiple connections to the > network and you can also handle a switch failover. > > > I was hoping to have resilient solution on the storage side so that I can > > perform upgrades and maintenance without needing to shutdown vms running on > > vmware/xenserver. Is this possible with iscsi? > > > > The failover set up is mostly to handle failures, not really great for > maintenance because it does give a short interruption in service. Like 30 > seconds or so of no writing to the LUN. > > That might not be a problem for you, I don't know, but it is at least > something to be aware of. And also something you should test when you've > build the setup. > > > Cheers > > > > Hope that helps. > > > Andrei > > ----- Original Message ----- > > > > From: "Leen Besselink" <l...@consolejunkie.net> > > To: ceph-users@lists.ceph.com > > Sent: Saturday, 10 May, 2014 8:31:02 AM > > Subject: Re: [ceph-users] NFS over CEPH - best practice > > > > On Fri, May 09, 2014 at 12:37:57PM +0100, Andrei Mikhailovsky wrote: > > > Ideally I would like to have a setup with 2+ iscsi servers, so that I can > > > perform maintenance if necessary without shutting down the vms running on > > > the servers. I guess multipathing is what I need. > > > > > > Also I will need to have more than one xenserver/vmware host servers, so > > > the iscsi LUNs will be mounted on several servers. > > > > > > > So you have multiple machines talking to the same LUN at the same time ? > > > > You'll have to co-ordinate how changes are written to the backing store, > > normally you'd have the virtualization servers use some kind of protocol. > > > > When it's SCSI there are the older Reserve/Release commands and the newer > > SCSI-3 Persistent Reservation commands. > > > > (i)SCSI allows multiple changes to be in-flight, without coordination > > things will go wrong. > > > > Below it was mentioned that you can disable the cache for rbd, if you have > > no coordination protocol you'll need to do the same on the iSCSI-side. > > > > I believe when you do that it will be slower, but it might work. > > > > > Would the suggested setup not work for my requirements? > > > > > > > It depends on VMWare if they allow such a setup. > > > > Then there is an other thing. How do the VMWare machines coordinate which > > VM they should be running ? > > > > I don't know VMWare but usually if you have some kind of clustering setup > > you'll need to have a 'quorum'. > > > > A lot of times the quorum is handled by a quorum disk with the SCSI > > coordiation protocols mentioned above. > > > > An other way to have a quorum is to have a majority voting system with an > > un-even number of machines talking over the network. This is what Ceph > > monitor nodes do. > > > > As an example of a clustering system that allows it to be used without a > > quorum disk with only 2 machines talking over the network is Linux > > Pacemaker. When something bad happends, one machine will just turn off the > > power of the other machine to prevent things going wrong (this is called > > STONITH). > > > > > Andrei > > > ----- Original Message ----- > > > > > > From: "Leen Besselink" <l...@consolejunkie.net> > > > To: ceph-users@lists.ceph.com > > > Sent: Thursday, 8 May, 2014 9:35:21 PM > > > Subject: Re: [ceph-users] NFS over CEPH - best practice > > > > > > On Thu, May 08, 2014 at 01:24:17AM +0200, Gilles Mocellin wrote: > > > > Le 07/05/2014 15:23, Vlad Gorbunov a écrit : > > > > >It's easy to install tgtd with ceph support. ubuntu 12.04 for example: > > > > > > > > > >Connect ceph-extras repo: > > > > >echo deb http://ceph.com/packages/ceph-extras/debian $(lsb_release > > > > >-sc) main | sudo tee /etc/apt/sources.list.d/ceph-extras.list > > > > > > > > > >Install tgtd with rbd support: > > > > >apt-get update > > > > >apt-get install tgt > > > > > > > > > >It's important to disable the rbd cache on tgtd host. Set in > > > > >/etc/ceph/ceph.conf: > > > > >[client] > > > > >rbd_cache = false > > > > [...] > > > > > > > > Hello, > > > > > > > > > > Hi, > > > > > > > Without cache on the tgtd side, it should be possible to have > > > > failover and load balancing (active/avtive) multipathing. > > > > Have you tested multipath load balancing in this scenario ? > > > > > > > > If it's reliable, it opens a new way for me to do HA storage with iSCSI > > > > ! > > > > > > > > > > I have a question, what is your use case ? > > > > > > Do you need SCSI-3 persistent reservations so multiple machines can use > > > the same LUN at the same time ? > > > > > > Because in that case I think tgtd won't help you. > > > > > > Have a good day, > > > Leen. > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
