Re: [Dovecot] Mails repopping

Tue, 13 Sep 2011 15:17:17 -0700 

Quoting Simon Brereton <simon.brere...@buongiorno.com>:


-----Original Message-----
From: dovecot-boun...@dovecot.org [mailto:dovecot-
boun...@dovecot.org] On Behalf Of Simon Brereton
> -----Original Message-----
> From: Timo Sirainen [mailto:t...@iki.fi] On Fri, 2011-09-09 at 13:07
> -0400, Simon Brereton wrote:
>
> > I have a server that's been running Courier for about 6 years and
> in
> > all that time I think I've only ever had 1 issues where an entire
> mail
> > box was repopped by a webmail client.  However, since moving to a
> new
> > server and dovecot 4 weeks ago, I've now had the webmail client
> repop
> > this account 4 times (there are about 230 mails in the account).
> >
> > Is there a setting I need to tighten to prevent/remedy this?  I
> have
> > no idea if it's happening on other accounts, but this is one that
I
> > see.  The format is maildir.  There has been no changes to the
> webmail
> > client.
>
> dovecot -n output would have been nice. Also do you see anything in
> error logs?

Ah.  My apologies of course.  Here it is..

mail:~# dovecot -n
# 1.2.15: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.2 ext3
log_timestamp: %Y-%m-%d %H:%M:%S
protocols: imap imaps pop3 pop3s
ssl_ca_file: /etc/ssl/keys/rhodes-ca.crt
ssl_cert_file: /etc/ssl/keys/mail.domain.net.crt
ssl_key_file: /etc/ssl/private/mail.domain.net.key
disable_plaintext_auth: no
login_dir: /var/run/dovecot/login
login_executable(default): /usr/lib/dovecot/imap-login
login_executable(imap): /usr/lib/dovecot/imap-login
login_executable(pop3): /usr/lib/dovecot/pop3-login
mail_privileged_group: mailsystem
mail_location: maildir:/var/spool/mail/virtual/%d/%n
maildir_very_dirty_syncs: yes
mbox_write_locks: fcntl dotlock
mail_executable(default): /usr/lib/dovecot/imap
mail_executable(imap): /usr/lib/dovecot/imap
mail_executable(pop3): /usr/lib/dovecot/pop3
mail_plugins(default): quota imap_quota
mail_plugins(imap): quota imap_quota
mail_plugins(pop3): quota
mail_plugin_dir(default): /usr/lib/dovecot/modules/imap
mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap
mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3
imap_client_workarounds(default): outlook-idle delay-newmail
imap_client_workarounds(imap): outlook-idle delay-newmail
imap_client_workarounds(pop3):
pop3_client_workarounds(default):
pop3_client_workarounds(imap):
pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh
lda:
  postmaster_address: postmas...@domain.net
  mail_plugins: quota
  log_path:
  info_log_path:
  deliver_log_format: msgid=%m: %f: %$
auth default:
  mechanisms: plain login
  user: mailsystem
  verbose: yes
  passdb:
    driver: sql
    args: /etc/dovecot/dovecot-sql.conf
  userdb:
    driver: prefetch
  userdb:
    driver: static
    args: uid=999 gid=115 home=/var/spool/mail/virtual/%d/%n
allow_all_users=yes
  socket:
    type: listen
    client:
      path: /var/spool/postfix/private/auth
      mode: 432
      user: postfix
      group: mailsystem
    master:
      path: /var/run/dovecot/auth-master
      mode: 432
      user: mailsystem
      group: mailsystem
plugin:
  quota: maildir

Could you make dovecot -n munge the certificate and postmaster email
addresses?  I'm not comfortable with that floating on the internet..

The only thing I have in the logs is 2 sessions where mail was popped
(note, it doesn't even add up to the 183 messages in the mail box).
But those sessions are vastly longer than the regular ones (tens of
minutes compared to a few seconds).  Since both IPs are on the back-
bone, that's quite a while to download 100 mails (none of which are
over

Sep 11 21:36:25 mail dovecot: pop3-login: Login:
user=<u...@domain.com>, method=PLAIN, rip=64.88.168.84,
lip=83.170.65.xxx, TLS Sep 11 21:36:34 mail dovecot:
POP3(u...@domain.com): Disconnected: Logged out top=0/0, retr=0/0,
del=0/183, size=14025971 Sep 11 21:43:44 mail dovecot: pop3-login:
Login: user=<u...@domain.com>, method=PLAIN, rip=64.88.168.84,
lip=83.170.65.xxx, TLS Sep 11 21:44:54 mail dovecot:
POP3(u...@domain.com): Disconnected: Logged out top=0/0, retr=0/0,
del=0/183, size=14025971 Sep 11 21:52:31 mail dovecot: pop3-login:
Login: user=<u...@domain.com>, method=PLAIN, rip=64.88.168.84,
lip=83.170.65.xxx, TLS Sep 11 22:56:01 mail dovecot:
POP3(u...@domain.com): Disconnected: Logged out top=0/0,
retr=100/9182678, del=0/183, size=14025971 Sep 11 23:08:58 mail
dovecot: pop3-login: Login: user=<u...@domain.com>, method=PLAIN,
rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 11 23:37:57 mail
dovecot: POP3(u...@domain.com): Disconnected: Logged out top=0/0,
retr=75/4748674, del=0/183, size=14025971 Sep 12 00:04:11 mail
dovecot: pop3-login: Login: user=<u...@domain.com>, method=PLAIN,
rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 12 00:04:26 mail
dovecot: POP3(u...@domain.com): Disconnected: Logged out top=0/0,
retr=0/0, del=0/183, size=14025971 Sep 12 00:07:40 mail dovecot:
pop3-login: Login: user=<u...@domain.com>, method=PLAIN,
rip=64.88.168.84, lip=83.170.65.xxx, TLS Sep 12 00:07:53 mail
dovecot: POP3(u...@domain.com): Disconnected: Logged out top=0/0,
retr=0/0, del=0/183, size=14025971


> If you're using the default pop3_uidl_format it'll rely on IMAP
UIDs
> to stay the same, and I guess it's possible that due to some other
> problem they change (that should be logged as an error/warning
> though).
>
> You could try setting pop3_uidl_format=%f, but it will cause
everyone
> to redownload mails. With newer Dovecot versions you could set
> pop3_save_uidl=yes and when you think everyone's downloaded mails
once
> you can safely change the pop3_uidl_format.

Sorry, I'm very new to dovecot and I'm not sure I understand.  I
presume because neither of those keys are in the dovecot -n output
that they are as the defaults, yes?  The account is indeed accessed
by IMAP as well (from a mobile device mostly), but I don't see
anything fishy there either.  How could I see if the IMAP UIDs have
changed?

Sep 11 21:20:32 mail dovecot: IMAP(u...@domain.com): Connection
closed bytes=1095/8292

Sep 11 21:26:03 mail dovecot: imap-login: Login:
user=<u...@domain.com>, method=PLAIN, rip=174.252.83.244,
lip=83.170.65.xxx, TLS Sep 11 22:11:20 mail dovecot:
IMAP(u...@domain.com): Disconnected for inactivity bytes=725/5638 Sep
11 22:17:10 mail dovecot: imap-login: Login: user=<u...@domain.com>,
method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 11
23:12:06 mail dovecot: IMAP(u...@domain.com): Disconnected for
inactivity bytes=1471/11025 Sep 11 23:23:22 mail dovecot: imap-login:
Login: user=<u...@domain.com>, method=PLAIN, rip=174.252.83.244,
lip=83.170.65.xxx, TLS Sep 11 23:52:52 mail dovecot:
IMAP(u...@domain.com): Connection closed bytes=1841/13679 Sep 12
00:08:47 mail dovecot: imap-login: Login: user=<u...@domain.com>,
method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 12
01:19:05 mail dovecot: imap-login: Login: user=<u...@domain.com>,
method=PLAIN, rip=174.252.83.244, lip=83.170.65.xxx, TLS Sep 12
02:57:01 mail dovecot: IMAP(u...@domain.com): Connection closed
bytes=2713/60026 Sep 12 02:57:01 mail dovecot: IMAP(u...@domain.com):
Connection closed bytes=2688/18635


There are no errors or warnings in the mail log (I have one shared
log file for postfix, amavis and dovecot).  Reading the notes for
pop3_save_uidl it doesn't seem to be a dangerous option  - should I
turn that on?  Why will it force everyone to redownload mails
(there's nothing about it on the wiki)?

Thanks!

Simon


Any help would be appreciated.
What do you mean by "repopped"? You mean downloading the entire data of the messages from the POP3 server? This is expected behavior when using a stateless (e.g. webmail) client. Kind of the whole reason you don't use POP3 in the first place.
Although caching can help. e.g., Here's what the first connection to the server looks like (this is using IMP 5 on a mailstore with 82 messages): 

S (1315951197.4976): +OK Dovecot ready.
C (1315951197.513): [AUTH PLAIN Command - username: slusarz]
S (1315951197.5319): +OK Logged in.
C (1315951197.5325): STAT
S (1315951197.5328): +OK 82 482351
C (1315951197.5348): UIDL
S (1315951197.5354): +OK
S (1315951197.5354): 1 000000014935d409
S (1315951197.5354): 2 000000024935d409
S (1315951197.5354): 3 000000114935d409
[...]
S (1315951197.5363): 82 000000824935d409
S (1315951197.5363): .
C (1315951197.9582): TOP 1 0
S (1315951198.0411): From u...@domain.com  Thu Jun 22 11:16:26 2006
[...]
S (1315951198.0416): .
[...]
C (1315951199.0607): LIST
S (1315951199.061): +OK 82 messages:
S (1315951199.061): 1 118630
[...]
S (1315951199.0619): .
We need to grab all headers so we can build the envelope information (needed to produce the mailbox listing). And the LIST command grabs the size information (also used in the mailbox listing).
But remember that the full headers will need to be redownloaded *EVERY* time you reload the page unless some sort of caching is enabled in the client. That's just the nature of POP3. (IMAP has the same sort of issues - if the stateless client does not cache, the envelope information must be downloaded on every access. However, with IMAP, the network traffic is reduced - you can download only the needed information, not all header text - and IMAP servers have the ability to cache this information behind the scenes due to the abstraction of the API.).
This is where caching is pretty much essential on the webmail side. If caching is enabled, the best-case scenario is that the the webmail server only needs to grab the list of UIDLs on every POP3 server access going forward - if the UIDL list has not changed, we know the mailbox hasn't changed and the cached information is still valid. (CONDSTORE/QRESYNC extensions for IMAP make this synchronization check even more efficient in IMAP) 

michael

Reply via email to