Hi, I’m planning a similar cluster. Because it’s a new project I’ll start with only 2 node cluster witch each:
2x E5-2640v4 with 40 threads total @ 3.40Ghz with turbo 24x 1.92 TB Samsung SM863 128GB RAM 3x LSI 3008 in IT mode / HBA for OSD - 1 each 8 OSD/SDDs 2x SSD for OS 2x 40Gbit/s NIC What about this hardware configuration? Is that wrong or I’m missing something ? Regards Matteo > Il giorno 06 ott 2016, alle ore 13:52, Denny Fuchs <linuxm...@4lin.net> ha > scritto: > > God morning, > >>> * 2 x SN2100 100Gb/s Switch 16 ports >> Which incidentally is a half sized (identical HW really) Arctica 3200C. > > really never heart from them :-) (and didn't find any price €/$ region) > > >>> * 10 x ConnectX 4LX-EN 25Gb card for hypervisor and OSD nodes > [...] > >> You haven't commented on my rather lengthy mail about your whole design, >> so to reiterate: > > maybe accidentally skipped, so much new input :-) sorry > >> The above will give you a beautiful, fast (but I doubt you'll need the >> bandwidth for your DB transactions), low latency and redundant network >> (these switches do/should support MC-LAG). > > Jepp, they do MLAG (with the 25Gbit version of the cx4 NICs) > >> In more technical terms, your network as depicted above can handle under >> normal circumstances around 5GB/s, while your OSD nodes can't write more >> than 1GB/s. >> Massive, wasteful overkill. > > before we started with planing Ceph / new hypervisor design, we where sure > that our network would be more powerful, than we need in the near future. Our > applications / DB never used the full 1GBs in any way ... we loosing speed > on the plain (painful LANCOM) switches and the applications (mostly Perl > written in the beginning of the 2005). > But anyway, the network should be have enough capacity for the next years, > because it is much more complicated to change network (design) components, > than to kick a node. > >> With a 2nd NVMe in there you'd be at 2GB/s, or simple overkill. > > We would buy them ... so that in the end, every 12 disk has a separated NVMe > >> With decent SSDs and in-line journals (400GB DC S3610s) you'd be at 4.8 >> GB/s, a perfect match. > > What about the worst case, two nodes are broken, fixed and replaced ? I red > (a lot) that some Ceph users had massive problems, while the rebuild runs. > > >> Of course if your I/O bandwidth needs are actually below 1GB/s at all times >> and all your care about is reducing latency, a single NVMe journal will be >> fine (but also be a very obvious SPoF). > > Very happy to put the finger in the wound, SPof ... is a very hard thing ... > so we try to plan everything redundant :-) > > The bad side of life: the SSD itself. A consumer SSD lays round about 70/80€, > a DC SSD jumps up to 120-170€. My nightmare is: a lot of SSDs are jumping > over the bridge at the same time .... -> arghh > > But, we are working on it :-) > > I've searching an alternative for the Asus board with more PCIe slots and > maybe some components; better CPU with 3.5Ghz-> ; maybe a mix from the SSDs > ... > > At this time, I've found the X10DRi: > > https://www.supermicro.com/products/motherboard/xeon/c600/x10dri.cfm > <https://www.supermicro.com/products/motherboard/xeon/c600/x10dri.cfm> > > and I think we use the E5-2637v4 :-) > > cu denny > > > -- > Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato non > infetto. > Clicca qui per segnalarlo come spam. > <http://mx01.enter.it/cgi-bin/learn-msg.cgi?id=0E9124029A.A17D5> > Clicca qui per metterlo in blacklist > <http://mx01.enter.it/cgi-bin/learn-msg.cgi?blacklist=1&id=0E9124029A.A17D5>_______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
