On Tue, 2010-06-15 at 04:42 -0700, Arve Paalsrud wrote:
> Hi,
>
> We are currently building a storage box based on OpenSolaris/Nexenta using
> ZFS.
> Our hardware specifications are as follows:
>
> Quad AMD G34 12-core 2.3 GHz (~110 GHz)
> 10 Crucial RealSSD (6Gb/s)
> 42 WD RAID Ed. 4 2TB disks + 6Gb/s SAS expanders
> LSI2008SAS (two 4x ports)
> Mellanox InfiniBand 40 Gbit NICs
Just recognize that those "NICs" are IB only. Solaris currently does
not support 10GbE using Mellanox products, even though other operating
systems do. (There are folks working on resolving this, but I think
we're still a couple months from seeing the results of that effort.)
> 128 GB RAM
>
> This setup gives us about 40TB storage after mirror (two disks in spare),
> 2.5TB L2ARC and 64GB Zil, all fit into a single 5U box.
>
> Both L2ARC and Zil shares the same disks (striped) due to bandwidth
> requirements. Each SSD has a theoretical performance of 40-50k IOPS on 4k
> read/write scenario with 70/30 distribution. Now, I know that you should have
> mirrored Zil for safety, but the entire box are synchronized with an active
> standby on a different site location (18km distance - round trip of 0.16ms +
> equipment latency). So in case the Zil in Site A takes a fall, or the
> motherboard/disk group/motherboard dies - we still have safety.
I expect that you need more space for L2ARC and a lot less for Zil.
Furthmore, you'd be better served by an even lower latency/higher IOPs
ZIL. If you're going to spend this kind of cash, I think I'd recommend
at least one or two DDR Drive X1 units or something similar. While not
very big, you don't need much to get a huge benefit from the ZIL, and I
think the vastly superior IOPS of these units will pay off in the end.
>
> DDT requirements for dedupe on 16k blocks should be about 640GB when main
> pool are full (capacity).
Dedup is not always a win, I think. I'd look hard at your data and
usage to determine whether to use it.
-- Garrett
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
