Le 15/02/2015 16:48, hp cre a écrit :
Hello all, I'm currently studying the possibility of creating a small
ceph cluster on arm nodes.
The reasonably priced boards I found (like the banana pi/pro, Orange
pi/pro/h3, etc..) most have either dual core or quad core Allwinner
chips and 1GB RAM. They also use a micro sd card for os and a sata
drive connection.
Has anyone ever tried a deployment like this? I want to know if that
hardware is enough to create a usable cluster.
I have, at home :)
I have 3 Banana pi (each is an OSD), and 1 Odroid C1 (which is acting as
MON, and in the same time, hosts other LXC servers)
Each banana pi has a Sata 240 GB SSD + 1 USB 1,5 TB drive.
In a few words : It works. But it has lots of limitations, and it's
absolutely not advisable to put production on it.
I you want to experiment on this, some advices :
-> You need recent kernel on Banana pi & Odroid to have good Ethernet
bandwidth (over 600 Mbit/s)
-> Another reason to put a recent vanilla kernel (3.6.18 or more on
Bananapi) is stability, I had lots of glitches with 3.10 customs
Bananapi kernels ( with data corruption :) )
-> If you plan to use XFS, BEWARE !!! there is miscompilation with XFS
on arm if you use standard (-O2) optimization. Use (-Os) to have a
reliable XFS module.
-> banana pi has Sata, performance is OK for read (over 150 MB/s) but
it's definitively SLOW for writes (not more than 40 MB/s), so putting an
SSD on Sata for journal don't have much interest. In fact you have
better bandwidth using 2 separate USB 2.0 disks !!! (Bananapi has 2
independents USB channels)
-> Bananapi (and Odroid) are limited to 1 GB of RAM, it's adequate for
OSD in the 1/2 TB range, not more, or you'll end up swapping like mad
(on slow device)
-> Bananapi or Odroid are not servers anyway so you don't have any kind
of error protection !!!
And even if you avoid all those traps, you'll find that performance is
not that great.
Using RBD, I can barely reach 20 MB/s (which is not THAT bad). Even with
ONLY SSD pool (the root cause is : you need journal anyway and
limitation is Bananapi (in fact Allwinner A20) Sata chip or driver .
Odroid C1 has an eMMC channel, but it seems you won't have better
throughput using this....
I can give further details if you wish,
Cheers
I dont want loads of storage, just want to end up with a responsive
os and a 10 gb storage pool.
Beyond the basic minimum hardware listed on the ceph documents, i
haven't found much info on this. The pilot implementation with
startup company calxeda was on 64-bit arm boards, not 32-bit, and
they were a custom spec.
Any ideas welcome. Thanks.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
