On 4/21/07, Amos Shapira <[EMAIL PROTECTED]> wrote:
Hi,
Our servers have to deal with huge amounts of small files (tens, sometimes
hundreds of thousands of files IN ONE DIRECTORY).
Do you access them locally or remotely, if so, how?
Currently they use ext3 but I wonder wether this is the prefered FS.
I have found ReiserFS outperformed EXT3 on a similar site, this,
however was made irreverent, as access was via done mainly via NFS.
I used to be fond of ReiserFS v3 until I got beaten by it not recovering
from a partition resizing excercise.
Resizing a partition is not a good indicator. There are too many other
factors involved.
Trying to find the answer on the net I found:
http://librenix.com/?inode=3296 (Circa 2003, recommends ReiserFS v4, which
isn't in the mainstream kernel yet).
and
http://www.debian-administration.org/articles/388 (Circa
2006, recommends XFS).
The later compared handling of large trees (i.e. not necessarily single
directory with lots of files in it).
Does anyone have good and up to date recommendations for such situation?
The files are e-mail messages which are writen, transferred then deleted.
I'm also thinking about better ways to handle the files ( e.g. putting every
few thousands of them in a .zip file to transfer, spreading them across a
two-level directory tree etc) but I'd rathertry to keep the changes to the
existing software and scripts the the minimum which is required to speed
things up.
Benchmark your own environment. Hardware specs (RAID, RAM, CPU, etc)
can tilt the results. Repeat the benchmarks for about 3-5 time. I like
Bonnie++. Look for the results that best match your environment (Read,
Write, Create, etc).
Thanks,
--Amos
--
Gil Freund, Systems Analyst
-------------------------------------------
Sysnet consulting
[EMAIL PROTECTED], http://www.sysnet.co.il
voice: +972-54-2035888, Fax: +972-8-9356026
=================================================================
To unsubscribe, send mail to [EMAIL PROTECTED] with
the word "unsubscribe" in the message body, e.g., run the command
echo unsubscribe | mail [EMAIL PROTECTED]
