On Tue, 29 Jan 2008, Bill Cole wrote:
At 6:32 PM -0800 1/29/08, Asheesh Laroia imposed structure on a stream
of electrons, yielding:
On Tue, 29 Jan 2008, Bill Cole wrote:
For Dovecot, the mailbox index also has to be updated for each
deletion, and while that in itself is not terribly expensive per
message, it has to be done one message at a time to maintain
consistency between the index and the real mail.
In general, this isn't really true - because Dovecot can deal with
other programs accessing the Maildir, it can't treat the index as
canonical.
I was speaking specifically of what Dovecot might be able to do to speed
up user experience, which would be to update (i.e. clear) the index,
then spawn off something to actually do the slow task of deleting the
files. To keep that behavior from cascading into pointless index
rebuilds during the cleanup time, there would need to be some big
changes in how Dovecot keeps index files in synch.
Ah, I see.
Or put another way: the consistency problem is not that Dovecot can't
regenerate its index files from the mailbox, but rather that it can and will
do so whenever it sees a directory change, and it always trusts the data over
the index (that's GOOD.) If the index were to be updated ahead of a slow
change of the data in order to provide the user the appearance of speed, it
violates the design of the index system.
Therefore if Dovecot knows it is deleting a whole folder, Dovecot could
just as easily skip over the index updates or equivalently "decide to do
them all at once at the end", at which point the index is empty.
Except that it wouldn't really help. The index updates are not the expensive
part, the changes to the filesystem are. In this case it looks like an issue
with XFS, a filesystem that is not normally recommended for usage that
involves the creation and deletion of a lot of small files. Changing a little
bit of data in an index file is very fast compared to the deletion of a file
from a directory with 35k entries.
I misunderstood the cause of the slowness then, if it is really true that
updating the index is not an important factor. I don't remember seeing
profiling, but since I don't have any myself I'll take your word for it.
-- Asheesh.
--
"Atomic batteries to power, turbines to speed."
-- Robin, The Boy Wonder
