Re: [CentOS] wich filesystem to store > 250E6 small files in same or hashed dire

2011-03-14 Thread Adam Tauno Williams
On Mon, 2011-03-14 at 13:10 -0700, Dr. Ed Morbius wrote: > on 13:10 Sat 12 Mar, Alain Spineux (aspin...@gmail.com) wrote: > > Hi > > I need to store about 250.000.000 files. Files are less than 4k. > > On a ext4 (fedora 14) the system crawl at 10.000.000 in the same directory. > > I tried to creat

Re: [CentOS] wich filesystem to store > 250E6 small files in same or hashed dire

2011-03-14 Thread Dr. Ed Morbius
on 13:10 Sat 12 Mar, Alain Spineux (aspin...@gmail.com) wrote: > Hi > > I need to store about 250.000.000 files. Files are less than 4k. > > On a ext4 (fedora 14) the system crawl at 10.000.000 in the same directory. > > I tried to create hash directories, two level of 4096 dir = 16.000.000 > b

Re: [CentOS] wich filesystem to store > 250E6 small files in same or hashed dire

2011-03-14 Thread Les Mikesell
On 3/14/2011 12:33 PM, Alain Spineux wrote: > >> File aaa12345 goes in >> >>$DIR/a/a/a/12345 >> >> File abc6789 goes in >> >> $DIR/a/b/c/6789 > > Try to create this king of tree yourself and when done, remove it. > I took hours on my box and it is even faster to keep all files in the > same

Re: [CentOS] wich filesystem to store > 250E6 small files in same or hashed dire

2011-03-14 Thread Alain Spineux
On Sat, Mar 12, 2011 at 10:14 PM, Nico Kadel-Garcia wrote: > On Sat, Mar 12, 2011 at 4:04 PM, Alain Spineux wrote: >> Thanks to everybody for answering. >> >> I thing >250E6 is a lot and keep decent read and write access speed is unreal >> using mutli-purpose filesystems like ext? and other ?FS.

Re: [CentOS] wich filesystem to store > 250E6 small files in same or hashed dire

2011-03-12 Thread Nico Kadel-Garcia
On Sat, Mar 12, 2011 at 4:04 PM, Alain Spineux wrote: > Thanks to everybody for answering. > > I thing >250E6 is a lot and keep decent read and write access speed is unreal > using mutli-purpose filesystems like ext? and other ?FS. > I would need a dedicated filesystem for that. > > This problem w

Re: [CentOS] wich filesystem to store > 250E6 small files in same or hashed dire

2011-03-12 Thread Alain Spineux
Thanks to everybody for answering. I thing >250E6 is a lot and keep decent read and write access speed is unreal using mutli-purpose filesystems like ext? and other ?FS. I would need a dedicated filesystem for that. This problem was only a possible solution to another problem. I will solve the or

Re: [CentOS] wich filesystem to store > 250E6 small files in same or hashed dire

2011-03-12 Thread Simon Matter
> Hi > > I need to store about 250.000.000 files. Files are less than 4k. > > On a ext4 (fedora 14) the system crawl at 10.000.000 in the same > directory. > > I tried to create hash directories, two level of 4096 dir = 16.000.000 > but I had to stop the script to create these dir after hours > an

Re: [CentOS] wich filesystem to store > 250E6 small files in same or hashed dire

2011-03-12 Thread Emmanuel Noobadmin
I haven't tried it but could you possibly use a database to hold all those files instead? At less than 4K per "row", performance from an indexed database might be faster. On 3/12/11, Alain Spineux wrote: > Hi > > I need to store about 250.000.000 files. Files are less than 4k. > > On a ext4 (fed