Re: UFS large directory performance

2001-06-09 Thread Matt Dillon
:leaving the dirhash structure attached to the inode when its hash :array is freed. : :An updated patch is available at : : http://www.maths.tcd.ie/~iedowse/FreeBSD/dirhash.diff3 : :I haven't had a chance to do more than a minimal amount of testing, :so there may be many issues remaining. :

Re: UFS large directory performance

2001-06-08 Thread Ian Dowse
In message <[EMAIL PROTECTED]>, Terry Lambert writes: > >Use a chain allocator. I would suggest using the zone >allocator, but it has some fundamental problems that I >don't think are really resolvable without a rewrite. Heh, maybe, but I'm not sure I want to write a new allocator for this :-) B

Re: UFS large directory performance

2001-06-08 Thread Terry Lambert
Ian Dowse wrote: > Nice idea, but I'm not sure I see the benefit of partially reclaiming > second-level arrays. Because it is a hash array, there isn't really > the concept of a working set; a directory that is `in use' will > rarely see many create/rename/delete operations on a small fixed > set

Re: UFS large directory performance

2001-06-08 Thread Terry Lambert
Ian Dowse wrote: > >The only potential problem I see here is that you could > >end up seriously fragmenting the malloc pool you are > >using to allocate the slot arrays. And, of course, the > >two issues you brought up in regards to regularing memory > >use. > > Thanks for th

Re: UFS large directory performance

2001-06-03 Thread Matt Dillon
:Nice idea, but I'm not sure I see the benefit of partially reclaiming :second-level arrays. Because it is a hash array, there isn't really :the concept of a working set; a directory that is `in use' will :rarely see many create/rename/delete operations on a small fixed :set of filenames. The loo

Re: UFS large directory performance

2001-06-03 Thread Ian Dowse
In message <[EMAIL PROTECTED]>, Matt Dillon writes: > >I would further recommend a (dynamic) array of pointers at the first >level as part of the summary structure. Any given array entry would >either point to the second level array (the 512 byte allocations), >be NULL (no second

Re: UFS large directory performance

2001-06-02 Thread Matt Dillon
Oh yah, one more thing. At risk of making the second level block more complex you might want to make a real structure out of it rather then a simple array. Something like this: struct whateverYouNameTheSecondLevelBLock { short use; charfiller[sizeof(void *)

Re: UFS large directory performance

2001-06-02 Thread Matt Dillon
: :Thanks for the comments :-) Yes, malloc pool fragmentation is a :problem. I think that it can be addressed to some extent by using :a 2-level system (an array of pointers to fixed-size arrays) instead :of a single large array, but I'm open to any better suggestions. I do precisely this fo

Re: UFS large directory performance

2001-06-02 Thread Ian Dowse
In message <[EMAIL PROTECTED]>, Matt Dillon writes: >What are your commit plans? It looks extremely well contained, >it could be committed to -current and then -stable a few days later >without any destabilizing impact at all for when the option isn't >specified. ... >The on

Re: UFS large directory performance

2001-06-01 Thread Matt Dillon
:The new code is interesting; it will be enlightening to :see it's real world performance. I'd definitely suggest :using a zone for the allocations, however. : :FWIW: I guess if you are having problems with mail queue :perofrmance, you are running postfix or qmail or something, :instead of sendm

Re: UFS large directory performance

2001-06-01 Thread Terry Lambert
Matt Dillon wrote: > I can see this really helping mail queue performance, > especially when coupled with softupdates, and also > helping samba (windoz likes to scan directories), and > perhaps even squid to a degree. The new code is interesting; it will be enlightening to see it

Re: UFS large directory performance

2001-06-01 Thread Matt Dillon
:This is great -- once I finish moving back to Maryland (sometime :mid-next-week) I'd be very interested in running this code on a -CURRENT :mock-up of my Cyrus server, which regularly runs with 65,000+ file :directories. I assume this is a -CURRENT patch set? : :(Mind you, I've found that most

Re: UFS large directory performance

2001-06-01 Thread Matt Dillon
: :Prompted by the recent discussion about performance with large :directories, I had a go at writing some code to improve the situation :without requiring any filesystem changes. Large directories can :usually be avoided by design, but the performance hit is very :annoying when it occurs. The nam

Re: UFS large directory performance

2001-06-01 Thread John Baldwin
On 01-Jun-01 Robert Watson wrote: > > This is great -- once I finish moving back to Maryland (sometime > mid-next-week) I'd be very interested in running this code on a -CURRENT > mock-up of my Cyrus server, which regularly runs with 65,000+ file > directories. I assume this is a -CURRENT patch

Re: UFS large directory performance

2001-06-01 Thread Robert Watson
This is great -- once I finish moving back to Maryland (sometime mid-next-week) I'd be very interested in running this code on a -CURRENT mock-up of my Cyrus server, which regularly runs with 65,000+ file directories. I assume this is a -CURRENT patch set? (Mind you, I've found that most of th