Re: [ceph-users] Hardware: SFP+ or 10GBase-T

2013-10-24 Thread james
On 2013-10-24 15:08, Nathan Stratton wrote: 9 - Samsung 840 EVO 120 GB SSD (1 root 8 ceph) The EVO is a TLC drive with durability of about 1,100 write cycles. Whether that is or isn't a problem in your environment of course is a separate question - I'm just pointing it out :) If they are

Re: [ceph-users] Hardware: SFP+ or 10GBase-T

2013-10-24 Thread Mark Nelson
On 10/24/2013 11:38 AM, Nathan Stratton wrote: On Thu, Oct 24, 2013 at 11:19 AM, Kyle Bader wrote: If you are talking about the links from the nodes with OSDs to their ToR switches then I would suggest going with Twinax cables. Twinax doesn't go very far but it's really durable and uses less po

Re: [ceph-users] Hardware: SFP+ or 10GBase-T

2013-10-24 Thread Nathan Stratton
On Thu, Oct 24, 2013 at 11:19 AM, Kyle Bader wrote: > If you are talking about the links from the nodes with OSDs to their > ToR switches then I would suggest going with Twinax cables. Twinax > doesn't go very far but it's really durable and uses less power than > 10GBase-T. Here's a blog post tha

Re: [ceph-users] Hardware: SFP+ or 10GBase-T

2013-10-24 Thread Nathan Stratton
On Thu, Oct 24, 2013 at 9:48 AM, Mark Nelson wrote: > Ceph does work with IPoIB, We've got some people working on rsocket support, > and Mellanox just opensourced VMA, so there are some options on the > infiniband side if you want to go that route. With QDR and IPoIB we have > been able to push a

Re: [ceph-users] Hardware: SFP+ or 10GBase-T

2013-10-24 Thread Kyle Bader
> I know that 10GBase-T has more delay then SFP+ with direct attached > cables (.3 usec vs 2.6 usec per link), but does that matter? Some > sites stay it is a huge hit, but we are talking usec, not ms, so I > find it hard to believe that it causes that much of an issue. I like > the lower cost and

Re: [ceph-users] Hardware: SFP+ or 10GBase-T

2013-10-24 Thread Mark Nelson
On 10/24/2013 09:08 AM, Nathan Stratton wrote: I have tried to make Gluster FS work for last 2 years on different projects and have given up. With Gluster I have always used 10 gig Infiniband. Its dirt cheap (about $80 a port used including switch) and very low latency, however ceph does not supp

[ceph-users] Hardware: SFP+ or 10GBase-T

2013-10-24 Thread Nathan Stratton
I have tried to make Gluster FS work for last 2 years on different projects and have given up. With Gluster I have always used 10 gig Infiniband. Its dirt cheap (about $80 a port used including switch) and very low latency, however ceph does not support it so we are looking at ethernet. I know tha