[Dovecot] dsync on large mailbox "fails"

[Jeff Kletsky]

I've been trying to convert an existing set of mbox mail to maildir 
using dsync, but it seems to fail when it gets to the large boxes.

$ dsync -Dv mirror maildir:~/Maildir
Everything seems to work reasonably well until the first of the large 
boxes hits:

dsync(jeff): Warning: Maildir /home/jeff/Maildir/.cron.2010-07-10: 
Synchronization took 1210 seconds (102289 new msgs, 0 flag change 
attempts, 0 expunge attempts)
dsync(jeff): Warning: Transaction log file 
/home/jeff/Maildir/.cron.2010-07-10/dovecot.index.log was locked for 
1210 seconds

After this, the doveadm process goes from ~75% of the CPU core 
utilization to 100% and stays there, even overnight.

I've also tried

$ dsync -Dv backup maildir:~/Maildir

with similar results.

While I did see some threads on dsync speed, they seemed to be minutes 
for a few GB of data, not hours.

There is plenty of space on the drive, and, from what I understand, 
there is not a file-count limit on FreeBSD's ufs (at least not one that 
I'm anywhere near)

Any suggestions?

Thanks!

Jeff



[jeff@mail ~]$ du -h -d 0 ~/mail
3.9G    /home/jeff/mail

[jeff@mail ~]$ ls -lh /var/mail/jeff
-rw-------  1 jeff  jeff   554M Apr 23 11:27 /var/mail/jeff

[jeff@mail ~]$ df -h
Filesystem      Size    Used   Avail Capacity  Mounted on
/dev/ada0s2g     38G     18G     16G    53%    /




(Note that the hostname and the location of the ssl certificate has been 
manually edited for this message in the following output)

From dmesg.boot:

Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-STABLE #0: Sat Apr 21 17:13:07 PDT 2012
    r...@my.host.name:/usr/obj/usr/src/sys/GENERIC i386
CPU: Intel(R) Atom(TM) CPU  330   @ 1.60GHz (1600.03-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x106c2  Family = 6  Model = 1c  
Stepping = 2
  
Features=0xbfe9fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x40e31d<SSE3,DTES64,MON,DS_CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE>
  AMD Features=0x20100000<NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 2147483648 (2048 MB)
avail memory = 2077757440 (1981 MB)




[jeff@mail ~]$ doveconf -n
# 2.1.3: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 9.0-STABLE i386
auth_mechanisms = plain login
disable_plaintext_auth = no
first_valid_gid = 1000
first_valid_uid = 1000
listen = *
mail_location = mbox:~/mail/:INBOX=/var/mail/%u
mail_privileged_group = mail
namespace {
  inbox = yes
  location =
  prefix =
  separator = /
}
passdb {
  args = session=yes dovecot
  driver = pam
}
protocols = imap
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0660
    user = postfix
  }
  unix_listener auth-master {
    mode = 0600
  }
  user = root
}
service imap-login {
  inet_listener imap {
    port = 0
  }
  process_limit = 200
}
ssl = required
ssl_cert = </path/to/certificate.pem
ssl_key = </path/to/certificate.pem
userdb {
  args = blocking=yes
  driver = passwd
}
verbose_proctitle = yes
protocol imap {
  imap_client_workarounds = tb-extra-mailbox-sep
}