God morning,
>> * 2 x SN2100 100Gb/s Switch 16 ports
> Which incidentally is a half sized (identical HW really) Arctica 3200C.
really never heart from them :-) (and didn't find any price EUR/$
region)
>> * 10 x ConnectX 4LX-EN 25Gb card for hypervisor and OSD nodes
[...]
> You haven't commented on my rather lengthy mail about your whole design,
> so to reiterate:
maybe accidentally skipped, so much new input :-) sorry
> The above will give you a beautiful, fast (but I doubt you'll need the
> bandwidth for your DB transactions), low latency and redundant network
> (these switches do/should support MC-LAG).
Jepp, they do MLAG (with the 25Gbit version of the cx4 NICs)
> In more technical terms, your network as depicted above can handle under
> normal circumstances around 5GB/s, while your OSD nodes can't write more
> than 1GB/s.
> Massive, wasteful overkill.
before we started with planing Ceph / new hypervisor design, we where
sure that our network would be more powerful, than we need in the near
future. Our applications / DB never used the full 1GBs in any way ...
we loosing speed on the plain (painful LANCOM) switches and the
applications (mostly Perl written in the beginning of the 2005).
But anyway, the network should be have enough capacity for the next
years, because it is much more complicated to change network (design)
components, than to kick a node.
> With a 2nd NVMe in there you'd be at 2GB/s, or simple overkill.
We would buy them ... so that in the end, every 12 disk has a separated
NVMe
> With decent SSDs and in-line journals (400GB DC S3610s) you'd be at 4.8
> GB/s, a perfect match.
What about the worst case, two nodes are broken, fixed and replaced ? I
red (a lot) that some Ceph users had massive problems, while the rebuild
runs.
> Of course if your I/O bandwidth needs are actually below 1GB/s at all times
> and all your care about is reducing latency, a single NVMe journal will be
> fine (but also be a very obvious SPoF).
Very happy to put the finger in the wound, SPof ... is a very hard
thing ... so we try to plan everything redundant :-)
The bad side of life: the SSD itself. A consumer SSD lays round about
70/80EUR, a DC SSD jumps up to 120-170EUR. My nightmare is: a lot of
SSDs are jumping over the bridge at the same time .... -> arghh
But, we are working on it :-)
I've searching an alternative for the Asus board with more PCIe slots
and maybe some components; better CPU with 3.5Ghz-> ; maybe a mix from
the SSDs ...
At this time, I've found the X10DRi:
https://www.supermicro.com/products/motherboard/xeon/c600/x10dri.cfm
and I think we use the E5-2637v4 :-)
cu denny
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
