On 1/2/07, Anders Troberg <[EMAIL PROTECTED]> wrote:
I've ploughed through the documentation, but it's kind of vague on some points and I need to buy some hardware if I'm to test it, so I thought I'd ask first. I'll begin by describing what I want to achieve, and would apreciate if someone could tell me if this is possible or how close I can come. My current situation: I have a lot of data, currently som 10-12 TB spread over about 50 disks in five file servers, adding on the average a new disk every 1-2 months,
Ideally you should add 3-5 disks at a time so you can add raidz(like raid5) groups so the failure of a disk won't cause lost of data. and a new server every year or so. The old paradigm with volumes makes this
a major pain in the rear end, as it becomes very difficult
with ZFS its easier if you keep all disks on one server, just buy the biggest disk box you can and fill with 8x sata controllers. Spreading the disks over multiple servers means you have to use iscsi to share the disks which is just an added headache. Plus you will have to pay for more network infrastructure, plus powering extra CPU's and resources that isn't needed if all disks are attached to one box. to organize data. The large volume also makes backups impractical, as I'm
just an ordinary home user, so fancy tape robots and such are out of my price range. Most of my data is not compressable, and the size varies greatly, from large files (GB-sized) to huge numbers (millions) of tiny files (just a few kB). At the moment, all these are Windows servers, but I plan to switch to some Unix/Linux variant as soon as there is enough benefits (and ZFS sure looks like it could be the juicy bait in that trap). The clients (15 or so) are a mix of Linux, Windows and a bunch of Xboxes (as media players), with the Windows machines gradually being phased out and changed to Linux as fast as I can rewrite my own software (which I can't do without) for Linux. What I want: * The entire storage should be visible as one file system, with one logical file structure where volumes and servers are not even visible, as if it was one huge disk. No paths like /root/server1fs/volume1/dir... in other words.
Yes this is possible, but not advisable, but ZFS allows you to mount your file systems where ever so you won't have to deal with /root/server1fs/dir but you can use ones like mydata/january or mydata/movies or what ever you need. * Software RAID support, even across the network, so I can just add a bunch
of parity disks and survive if a few disks crash. To me, it's well worth it to pony up with the money for 5-10 extra disks if I know that that many disks can fail before I start to lose data. That would be good enough to dispense with the need for proper backups.
yes its possible but not a feature of ZFS, you will need to share disks using iscsi, and then put the shared disks in a zfs pool. But its much better just to put all the disks attached to one server, using external disk boxes and large server cases. * A RAID that allows me to use differently sized disks without losing lots
of disk space. I'm OK if some disk space is lost (ie a file is not striped over all disks, somewhat increasing the stripe size and thereby the size of the parity data), but I don't want to have my 400 GB disks only use the first 160 GB just because I have a shitload of 160 GB disks.
its best to add similar disks in a raidz group, so if you added 5x 160GB disks in a raidz group you would get 4x160GB of data with one drive being used for parity data protecting you from any drive in that group dieing. * Performance does not need to be stellar, but should not be snail-like
either. If it's enough to fill a 100 Mbit network cable, I'm perfectly happy, if it can't fill a 10 Mbit I'm starting to get worried.
if all disks are in one box you shouldn't have any performance problems, because ZFS spreads data amoung all drives in the pool. if you use iscsi to share disks you will have more performance bottle necks but if you have the network bandwidth, you should easily always have at least 100mbit of throughput. * A file system that handles huge numbers of tiny files somewhat
effectively. Many file systems use a full block even for a tiny file, which cause huge overheads when there are many files.
not a problem. ZFS uses variable sized blocks anything from 512bytes to 128k bytes per block, it is even flexible in raidz configurations where a 512 byte file uses just 1k of space, 512 bytes for data, and 512 bytes for parity. * Good interoperability with Linux, Windows and Xbox (actually, this is just
a question of Samba compliance and as such out of scope for this discussion).
Solaris supports, NFSv3 and v4, Samba, FTP, http, and just about any other network protocol you need. Is this doable? If not, how close can I get and what is it that I can't get? yes, its all doable but its best to as many disks in one main server as possible, even if you have to buy new cases and external disk boxes you should see savings simply because its cheaper to power one box, than 5. You will also need to budget differently for disks, adding 3-5 disks at a time instead of a single disk, but your data will be safer. James Dickens uadmin.blogspot.com This message posted from opensolaris.org
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
