On Thu, 31 May 2012, Oscar Hodgson wrote:
The subject is pretty much the question. Perhaps there's a better
place to be asking this question ...
We have (very briefly) discussed the possibility of using FreeBSD
pizza boxes as a storage heads direct attached to external JBOD arrays
with ZFS. In perusing the list, I haven't stumbled across indications
of people actually doing this. External JBODs would be running 24 to
48TB each, roughly. There would be a couple of units. The pizza
boxes would be used for computational tasks, and nominally would have
8 cores and 96G+ RAM.
Obvious questions are hardware compatibility and stability. I've set
up small FreeBSD 9 machines with ZFS roots and simple mirrors for
other tasks here, and those have been successful so far.
Observations would be appreciated.
mc:
real memory = 120259084288 (114688 MB)
FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs
FreeBSD/SMP: 4 package(s) x 16 core(s)
mc> zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
disk-1 14.5T 4.95T 9.55T 34% 1.00x ONLINE -
disk-2 270G 297M 270G 0% 1.00x ONLINE -
disk-1, RAIDz1, uses Hitachi 4TB drives.
iirc:
real memory = 68719476736 (65536 MB)
FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
FreeBSD/SMP: 2 package(s) x 16 core(s)
iirc> zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
disk-1 18.1T 6.70T 11.4T 36% 1.00x ONLINE -
disk-2 5.44T 3.05G 5.43T 0% 1.00x ONLINE -
disk-1, RAIDz1, uses a bunch of 2TB drives
I have another storage server named bd3 that has a RAIDz2 array of 2.5T
drives (11 of them, IIRC) but it is presently powered down for
maintenance.
btw:
real memory = 25769803776 (24576 MB)
FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs
FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 SMT threads
btw> zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
disk-1 9.06T 97.3G 8.97T 1% 1.00x ONLINE -
disk-2 9.06T 5.13T 3.93T 56% 1.00x ONLINE -
Those are smaller RAIDz1 arrays of 1TB and 2TB drives, IIRC.
I also have three other systems, over clocked to 4GHz with 16GB of RAM and
presently powered off, each with 3 or 4 2TB disks RAIDz1.
None of these systems have external arrays. The storage systems use common
technologies, such as NFS, to export their space but their primary mission
is manipulating (sort-of) big data and crypto attacks, though one is being
converted to a Hadoop node for experimentation.
I have only had four issues over the past year and a half:
1) It is important to keep your ZFS patches up to date and the firmware in
you controllers up to date. Failure to do this results in a => :(
2) Under heavy I/O my systems freeze for a few seconds. I haven't looked
into why but they are completely unresponsive. Note I am also using
compressed volumes (gzip), which puts a substantual load on the kernel.
3) I have had a number of disk failures -- not too many and not too few.
These are merely an annoyance with no loss of data.
4) In two systems I use OCZ Revo drives. After several months of operating
they go Tango Uniform, requiring a system boot where they return from the
dead. None of my other SSD technologies exhibit the same problem.
_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
