On 04/14/2013 05:15 PM, Wim van den Berge wrote: > Hello, > > We have been running OpenIndiana (and its various predecessors) as storage > servers in production for the last couple of years. Over that time the > majority of our storage infrastructure has been moved to Open Indiana to the > point where we currently serve (iSCSI, NFS and CIFS) about 1.2PB from 10+ > servers in three datacenters . All of these systems are pretty much the > same, large pool of disks, SSD for root, ZIL and L2ARC, 64-128GB RAM, > multiple 10Gb uplinks. All of these work like a charm. > > However the next system is going to be a little different. It needs to be > the absolute fastest iSCSI target we can create/afford. We'll need about > 10-12TB of capacity and the working set will be 5-6TB and IO over time is > 90% reads and 10% writes using 32K blocks but this is a data analysis > scenario so all the writes are upfront. Contrary to previous installs, money > is a secondary (but not unimportant) issue for this one. I'd like to stick > with a SuperMicro platform and we've been thinking of trying the new Intel > S3700 800GB SSD's which seem to run about $2K. Ideally I'd like to keep > system cost below $60K. > > This is new ground for us. Before this one, the game has always been > primarily about capacity/data integrity and anything we designed based on > ZFS/Open Solaris has always more than delivered in the performance arena. > This time we're looking to fill up the dedicated 10Gbe connections to each > of the four to eight processing nodes as much as possible. The processing > nodes have been designed that they will consume whatever storage bandwidth > they can get. > > Any ideas/thoughts/recommendations/caveats would be much appreciated.
Hi Wim, Interesting project. You should definitely look at all-SSD pools here. With the 800GB DC S3700 running in 3-drive raidz1's you're looking at approximately $34k CAPEX (for the 10TB capacity point) just for the SSDs. That leaves you ~$25k you can spend on the rest of the box, which is *a lot*. Be sure to put lots of RAM (512GB+) into the box. Also consider ditching 10GE and go straight to IB. A dual-port QDR card can be had nowadays for about $1k (SuperMicro even makes motherboards with QDR-IB on-board) and a 36-port Mellanox QDR switch can be had for about $8k (this integrates the IB subnet manager, so this is all you need to set up an IB network): http://www.colfaxdirect.com/store/pc/viewPrd.asp?idcategory=7&idproduct=158 Cheers, -- Saso _______________________________________________ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
