Hello,

On Tue, 03 Jun 2014 18:52:00 +0200 Cedric Lemarchand wrote:

> Hello,
> 
> Le 03/06/2014 12:14, Christian Balzer a écrit :
> > A simple way to make 1) and 2) cheaper is to use AMD CPUs, they will do
> > just fine at half the price with these loads. 
> > If you're that tight on budget, 64GB RAM will do fine, too.
> I am interested about this specific thought, could you elaborate how did
> you determine if such hardware (CPU and RAM) will handle well cases
> where the cluster goes in rebalancing mode when a node or some OSD goes
> down ?
> 
Well, firstly we both read:
https://objects.dreamhost.com/inktankweb/Inktank_Hardware_Configuration_Guide.pdf

And looking at those values a single Opteron 4386 would be more
than sufficient for both 1) and 2). 
I'm saying and suggesting a single CPU here to keep things all in one NUMA
node. 
AFAIK (I haven't used anything Intel for years) some Intel boards require
both CPUs in place to use all available interfaces (PCIe buses), so the
above advice is only for AMD.
As for RAM, it would be totally overspec'ed with 64GB, but a huge
pagecache is an immense help for reads and RAM is fairly cheap these days,
so the more you can afford, the better. 

Secondly experience.
The above document is pretty much on spot when comes to CPU suggestions in
combination with OSDs backed by a single HDD (SSD journal or not).
I think it is overly optimistic when it comes to purely SSD based storage
nodes or something like my HW RAID backed OSD.
Remember, when using the 4k fio I could get Ceph to use about 2 cores
per OSD and then stall on whatever locking contention or other things that
are going on inside it before actually exhausting all available CPU
resources. 
OSDs (journal and backing storage) as well as the network were nowhere
near getting exhausted.

Compared to that fio run a cluster rebalancing is a breeze, at least when
it comes to CPU resources needed. 
It comes in a much more CEPH friendly IO block size and thus exhausts
either network or disk bandwidth first.

> Because, as Robert stated (and I totally agree with that!), designing a
> cluster is about the expected performances in optimal conditions, and
> expected recovery time and nodes loads in non optimal conditions
> (typically rebalancing), and I found this last point hard to consider
> and anticipate.
> 
This is why one builds test clusters and then builds production HW
clusters with the expectation that it will be twice as bad as anticipated
from what you saw on the test cluster. ^o^

> As a quick exercise (without taking in consideration FS size overhead
> ect ...), based on config "1.NG" from Christian (ratio SSD/HDD of 1:3,
> thus 9x4TB HDD/nodes, 24 nodes) and replication ratio of 2 :
I would never use a replication of 2 unless I were VERY confident in my
backing storage devices (either high end and well monitored SSDs or RAIDs).

> 
> - each nodes : ~36TB RAW /~18TB NET
> - the whole cluster, 864TB RAW / ~432TB NET
> 
> If a node goes down, ~36TB have to be re balanced between the 23
> existing, so ~1,6TB have to be read and write on each nodes. I think
> this is the expected workload of the cluster in rebalancing mode.
> 
> So 2 questions :
> 
> * did my maths are good until now ?
Math is hard, lets go shopping. ^o^
But yes, given your parameters that looks correct.
> * where will be the main bottleneck with such configuration and workload
> (CPU/IO/RAM/NET) ? how calculate it ?
> 
See above. 
In the configurations suggested by Benjamin disk IO will be the
bottleneck, as the network bandwidth is higher than write capacity of the
SSDs and HDDs. CPU and RAM will not be an issue.

The other thing to consider are the backfilling and/or recovery settings
in CEPH, these will of course influence how much of an impact a node
failure (and potential recovery of it) will have.
Depending on those settings and the cluster load (as in client side) at
the time of failure the most optimistic number for full recovery of
redundancy I can come up with is about an hour, in reality it is probably
going to be substantially longer. 
And during that time any further disk failure (with over 200 in the
cluster a pretty decent probability) can result in irrecoverable data loss.

Christian
> 
> --
> Cédric
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


-- 
Christian Balzer        Network/Systems Engineer                
ch...@gol.com           Global OnLine Japan/Fusion Communications
http://www.gol.com/
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to