On Aug 28, 2009, at 8:38 PM, Adam McDougall wrote:
Early next week I need to upload over 100,000 emails to an IMAP server as quickly as possible from an Outlook client. I am looking for any methods I can use to (temporarily?) speed up the rate at which dovecot can accept and store IMAP uploads, whether it be storing on local disk, ram disk, etc. I can setup a temporary server on a laptop for example and once the upload has finished I can use standard file copying methods to transfer the mail to stable, permanent storage. I haven't been able to see over about 7 msgs/sec upload speed from a local folder in any mail client to dovecot (only NFS or ZFS backend tested so far with Maildir). Is there something horribly wrong with the speed I am seeing or are there just tricks I can try? Any tips? I'll be working on it all weekend until I find something satisfactory. It seems like I can upload mails to an Exchange server quicker. I'll setup just about anything that my experience allows me to, I can be very resourceful with adhoc hardware and software.
From Dovecot's side the only thing you can do is fsync_disable=yes. The main problem is probably network latency, because Outlook doesn't support MULTIAPPEND extension (and perhaps not even LITERAL+ extension?) Did you already try running Dovecot on the same computer as Outlook (some virtual thingy or maybe it works in cygwin)?
Alternatively I'll take a fast way of converting Exchange email to a tree of local mbox files which I can then run mb2md on.
If the mails are in Exchange, can't you connect to it using IMAP?
