On Fri, Mar 24, 2017 at 10:04 AM Alejandro Comisario <alejan...@nubeliu.com> wrote:
> thanks for the recommendations so far. > any one with more experiences and thoughts? > > best > On the network side, 25, 40, 56 and maybe soon 100 Gbps can now be fairly affordable, and simplify the architecture for the high throughput nodes. > On Mar 23, 2017 16:36, "Maxime Guyot" <maxime.gu...@elits.com> wrote: > > Hi Alexandro, > > As I understand you are planning NVMe for Journal for SATA HDD and > collocated journal for SATA SSD? > > Option 1: > - 24x SATA SSDs per server, will have a bottleneck with the storage > bus/controller. Also, I would consider the network capacity 24xSSDs will > deliver more performance than 24xHDD with journal, but you have the same > network capacity on both types of nodes. > - This option is a little easier to implement: just move nodes in > different CRUSHmap root > - Failure of a server (assuming size = 3) will impact all PGs > Option 2: > - You may have noisy neighbors effect between HDDs and SSDs, if HDDs are > able to saturate your NICs or storage controller. So be mindful of this > with the hardware design > - To configure the CRUSHmap for this you need to split each server in 2, I > usually use “server1-hdd” and “server1-ssd” and map the right OSD in the > right bucket, so a little extra work here but you can easily fix a “crush > location hook” script for it (see example > http://www.root314.com/2017/01/15/Ceph-storage-tiers/) > - In case of a server failure recovery will be faster than option 1 and > will impact less PGs > > Some general notes: > - SSD pools perform better with higher frequency CPUs > - the 1GB of RAM per TB is a little outdated, the current consensus for > HDD OSDs is around 2GB/OSD (see > https://www.redhat.com/cms/managed-files/st-rhcs-config-guide-technology-detail-inc0387897-201604-en.pdf > ) > - Network wise, if the SSD OSDs are rated for 500MB/s and use collocated > journal you could generate up to 250MB/s of traffic per SSD OSD (24Gbps for > 12x or 48Gbps for 24x) therefore I would consider doing 4x10G and > consolidate both client and cluster network on that > > Cheers, > Maxime > > On 23/03/17 18:55, "ceph-users on behalf of Alejandro Comisario" < > ceph-users-boun...@lists.ceph.com on behalf of alejan...@nubeliu.com> > wrote: > > Hi everyone! > I have to install a ceph cluster (6 nodes) with two "flavors" of > disks, 3 servers with SSD and 3 servers with SATA. > > Y will purchase 24 disks servers (the ones with sata with NVE SSD for > the SATA journal) > Processors will be 2 x E5-2620v4 with HT, and ram will be 20GB for the > OS, and 1.3GB of ram per storage TB. > > The servers will have 2 x 10Gb bonding for public network and 2 x 10Gb > for cluster network. > My doubts resides, ar want to ask the community about experiences and > pains and gains of choosing between. > > Option 1 > 3 x servers just for SSD > 3 x servers jsut for SATA > > Option 2 > 6 x servers with 12 SSD and 12 SATA each > > Regarding crushmap configuration and rules everything is clear to make > sure that two pools (poolSSD and poolSATA) uses the right disks. > > But, what about performance, maintenance, architecture scalability, > etc ? > > thank you very much ! > > -- > Alejandrito > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- -- Alex Gorbachev Storcium
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
