I don't know what your disk performance in your OSD nodes are, but dual FDR would probably be more than enough that I wouldn't worry about doing a separate cluster network. The FDR card should have more capacity than your PCI bus anyway.
Since Ceph does not use RDMA or native IB verbs yet, you won't see a ton of improvement over your Ethernet implementation. Mellanox and others are working to have Ceph use AccelIO which will provide RDMA messages for Ceph, but there is still quite a bit of work to get it fully supported. I haven't used the SX6036G, so I don't know what to expect or offer you in that regard, but it may determine if you use the Mellanox driver set or if you can use the OFED drivers (packaged with your distro). I personally like using the distro packages even if they are a little more out of date, but to support the gateway, you might need something newer. We use connected mode on our cluster with MTU of 65520, but some people have better performance with datagram mode and MTU of 4096. I think if you are using RDMA, then datagram mode does perform better. You will need to test it in your environment and with the drivers you choose to use. As far as Ceph, currently you would configure it the same as Ethernet with IPoIB. You just have to make sure you have a subnet manager up on the IB fabric so that you can bring up the IPoIB interfaces. For some reason CentOS7 doesn't bring up open-sm automatically, so I have to remember to start it when I reboot one of the boxes, luckily it can be clustered very easily. I recommend searching through the mailing list archives, some tweaks for IB have been given for memory contention scenarios (this will be important if your OSDs don't have a lot of RAM since IB locks memory regions). Keep a look out for progress on XIO on the mailing list to see when native IB support will be in Ceph. On Tue, Mar 17, 2015 at 12:13 PM, German Anders <[email protected]> wrote: > Hi Robert, > > How are you? Thanks a lot for the quick response. I would like to > know if you could share some info on this. We have an existing Ceph cluster > in production, with the following: > > 3 x MON servers with 10GbE ADPT DP (one port on the PUB network) > 4 x OSD servers with 10GbE ADPT DP (one port on the PUB network and one > port on the CLUS network) > We are using Juniper EX4500 at 10GbE > > This cluster works fine but we need to improve performance on the network > side, and we are going for other reason too, to implement Infiniband FDR > 56Gb/s, I've spoke with Mellanox people and we are going to use (2) SX6036G > gateway switches, so we can connect our existing ETH network (2 x 40G > connections on each gateway), the gateways are going to be in an > active/active configuration, also we are going to put one ADPT FDR DP on > each node. > > And then we are going to have two more switches SX6018F so use for CLUS > Ceph network, and the PUB network of the Ceph nodes is going to be > connected directly to the IB gateways switches. > What considerations do we need to have in order to use IB with Ceph?, > also, do you have any procedure to configure IB with Ceph?, like > dependencies to install on the hosts, etc. > > Any help will really be appreciated. > > Thanks in advance, > > > > *German Anders* > > Storage System Engineer Leader > > *Despegar* | IT Team > > *office* +54 11 4894 3500 x3408 > > *mobile* +54 911 3493 7262 > *mail* [email protected] > > > > > > > > > > > > > > > > > --- Original message --- > *Asunto:* Re: [ceph-users] Ceph + Infiniband CLUS & PUB Network > *De:* Robert LeBlanc <[email protected]> > *Para:* German Anders <[email protected]> > *Cc:* [email protected] <[email protected]> > *Fecha:* Tuesday, 17/03/2015 15:06 > > We have a test cluster with IB. We have both networks over IPoIB on the > same IP subnet though (no cluster network configuration). > > On Tue, Mar 17, 2015 at 12:02 PM, German Anders <[email protected]> > wrote: > >> Hi All, >> >> Does anyone have Ceph implemented with Infiniband for Cluster and >> Public network? >> >> Thanks in advance, >> >> >> >> *German Anders* >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> ceph-users mailing list >> [email protected] >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
