Re: [Dovecot] POP3 error
On 07 Mar 2011, at 19:15, Timo Sirainen wrote: > On Mon, 2011-03-07 at 19:03 +0200, Thierry de Montaudry wrote: >>> Mar 7 11:19:51 xxx dovecot: pop3-login: Error: net_connect_unix(pop3) >>> failed: Resource temporarily unavailable >>> .. >> As it is happening at least once a day, is there anything I can do to trace >> it? and whatever I'll do, will it slow down those machines? > > Set verbose_proctitle=yes (won't slow down) and get list of all Dovecot > processes when it happens. And check how much user and system CPU it's > using and what the load is. > Got the same problem this morning, here is the CPU usage and ps aux for dovecot. plus the different error I could pick up in the log, most of them are repeated a couple of times. I suspect it a problem with system resources, but can find any message to tell me what. Mail are stored on 17 NFS servers (CentOS), plus 3 servers for indexes only. CPU load is very high, but mainly from httpd running our webmail interface, which uses the local imap server. Mar 8 11:08:02 xxx dovecot: imap-login: Error: net_connect_unix(imap) failed: Resource temporarily unavailable Mar 8 11:08:02 xxx dovecot: pop3-login: Error: net_connect_unix(pop3) failed: Resource temporarily unavailable Mar 8 11:08:52 xxx dovecot: pop3-login: Error: master(pop3): Auth request timed out (received 0/12 bytes) Mar 8 11:12:54 xxx dovecot: pop3(xyz@wm): Error: net_connect_unix(/var/run/dovecot/dict) failed: Connection refused Mar 8 11:12:55 xxx dovecot: pop3-login: Error: read(pop3) failed: Connection reset by peer Mar 8 11:12:56 xxx dovecot: pop3-login: Error: net_connect_unix(pop3) failed: Connection refused Mar 8 11:12:59 xxx dovecot: pop3(xyz@wm): Error: net_connect_unix(/var/run/dovecot/dict) failed: Connection refused top - 11:10:14 up 14 days, 12:04, 2 users, load average: 55.04, 29.13, 14.55 Tasks: 474 total, 60 running, 414 sleeping, 0 stopped, 0 zombie Cpu(s): 99.6%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 16439812k total, 16353268k used,86544k free,33268k buffers Swap: 4192956k total, 140k used, 4192816k free, 8228744k cached vmail 313 0.0 0.0 24660 2260 ?S10:47 0:00 dovecot/imap [gabs002@wm 127.0.0.1 APPEND] vmail 1376 0.0 0.0 24432 2136 ?S10:48 0:00 dovecot/imap [phillippapi@wm 127.0.0.1 LOGOUT UID COPY] vmail 1738 0.0 0.0 24432 2196 ?S10:49 0:00 dovecot/imap [herlo@wm 127.0.0.1 APPEND] vmail 2053 0.0 0.0 24588 2188 ?S10:49 0:00 dovecot/imap [kifi@wm 127.0.0.1 APPEND] vmail 3224 0.0 0.0 24592 2192 ?S10:50 0:00 dovecot/imap [briankajengo@wm 127.0.0.1 APPEND] vmail 3267 0.0 0.0 24664 2268 ?S10:50 0:00 dovecot/imap [gabs002@wm 127.0.0.1 APPEND] vmail 4023 0.0 0.0 24572 2168 ?S10:50 0:00 dovecot/imap [mmakutloano@hm 127.0.0.1 APPEND] vmail 4025 0.0 0.0 24592 2188 ?S10:50 0:00 dovecot/imap [buhlungum@wm 127.0.0.1 APPEND] vmail 4066 0.0 0.0 24424 2192 ?S10:50 0:00 dovecot/imap [mowee@xm 127.0.0.1 APPEND] vmail 4181 0.0 0.0 24648 2212 ?S10:50 0:00 dovecot/imap [sophieh@wm 127.0.0.1 APPEND] vmail 4399 0.0 0.0 24620 2224 ?S10:51 0:00 dovecot/imap [tcc.dbn@wm 127.0.0.1 APPEND] vmail 4866 0.0 0.0 24592 2196 ?S10:51 0:00 dovecot/imap [kifi@wm 127.0.0.1 APPEND] vmail 5049 0.0 0.0 24584 2228 ?S10:51 0:00 dovecot/imap [malinga@sm 127.0.0.1 APPEND] vmail 5961 0.0 0.0 24588 2192 ?S10:52 0:00 dovecot/imap [briankajengo@wm 127.0.0.1 APPEND] vmail 6819 0.0 0.0 24624 2268 ?S10:52 0:00 dovecot/imap [ferns2004@wm 127.0.0.1 APPEND] vmail 6832 0.0 0.0 24636 2308 ?S10:52 0:00 dovecot/imap [lib@mm 127.0.0.1 APPEND] vmail 6854 0.0 0.0 24496 2216 ?S10:52 0:00 dovecot/imap [amawele@wm 127.0.0.1 UID] vmail 7164 0.0 0.0 24620 2224 ?S10:53 0:00 dovecot/imap [tcc.dbn@wm 127.0.0.1 APPEND] vmail 8441 0.0 0.0 24440 2124 ?S10:54 0:00 dovecot/imap [apheeha@wm 127.0.0.1 APPEND] root 8736 0.0 0.0 61736 2940 ?S07:05 0:00 dovecot/auth [0 wait, 0 passdb, 0 userdb] vmail 9559 0.0 0.0 24588 2192 ?S10:54 0:00 dovecot/imap [lib@mm 127.0.0.1 APPEND] vmail 9716 0.0 0.0 24628 2224 ?S10:55 0:00 dovecot/imap [buhlungum@wm 127.0.0.1 APPEND] vmail 9939 0.0 0.0 24624 2224 ?S10:55 0:00 dovecot/imap [tcc.dbn@wm 127.0.0.1 APPEND] vmail12112 0.0 0.0 24592 2200 ?S10:56 0:00 dovecot/imap [lib@mm 127.0.0.1 APPEND] vmail12558 0.0 0.0 24592 2196 ?S10:57 0:00 dovecot/imap [kifi@wm 127.0.0.1 APPEND] vmail13437 0.0 0.0 2 2128 ?S10:57 0:00 dovecot/imap [pmagqibelo@ut 127
[Dovecot] rkhunter alert dovecot using port 1984
Hi all, Debian Lenny, dovecot 1.0.15 My rkhunter script has picked up dovecot using port 1984 temporarily. When I run it now however, it is gone. Warning: Network TCP port 1984 is being used by /usr/lib/dovecot/imap. Possible rootkit: Fuckit Rootkit Use the 'lsof -i' or 'netstat -an' command to check this. Does dovecot use this port for any reason? anyone seen this before? Regards, Mark
[Dovecot] Curious problem: Plaintext authentication disallowed on non-secure (SSL/TLS) [read: all] connections.
Hello all, I've set up a new instance of dovecot 2.0.9 to use as a POP3/IMAP proxy. On trying to login I am told '-ERR Plaintext authentication disallowed on non-secure (SSL/TLS) connections'. I have set 'disable_plaintext_auth = no' (see output of doveconf attached). More curious still is that *this happens for SSL connections too*. Something seems very wrong here. Yelp? -- Andrew Lewis __ < Uncompensated overtime? Just Say No. > -- \ ^__^ \ (oo)\___ (__)\ )\/\ ||w | || || doveconf.conf Description: Binary data
Re: [Dovecot] Curious problem: Plaintext authentication disallowed on non-secure (SSL/TLS) [read: all] connections.
On Tue, 8 Mar 2011 13:55:03 +0200 Andrew Lewis wrote: > Something seems very wrong here. Ok I feel stupid... My *other* new Dovecot instance (ie the one behind the proxy) was generating this- so... that makes sense. Apologies for the noise! -- Andrew Lewis ___ < Do not write below this line. > --- \ ^__^ \ (oo)\___ (__)\ )\/\ ||w | || ||
[Dovecot] logging sent messages
Hello, i use plugin mail_log to see more activities like copy/delete/expunge caused by imap-users. unfortunately there is no log-line when a mail was sent and the copy was uploaded to "Sent-Folder". Isn't it a common imap-copy operation requested by client? How to log this? Thanks, Hajo
Re: [Dovecot] POP3 error
On 08 Mar 2011, at 13:24, Chris Wilson wrote: > Hi Thierry, > > On Tue, 8 Mar 2011, Thierry de Montaudry wrote: >> On 07 Mar 2011, at 19:15, Timo Sirainen wrote: >>> On Mon, 2011-03-07 at 19:03 +0200, Thierry de Montaudry wrote: > Mar 7 11:19:51 xxx dovecot: pop3-login: Error: > net_connect_unix(pop3) failed: Resource temporarily unavailable > .. As it is happening at least once a day, is there anything I can do to trace it? and whatever I'll do, will it slow down those machines? >>> >>> Set verbose_proctitle=yes (won't slow down) and get list of all >>> Dovecot processes when it happens. And check how much user and system >>> CPU it's using and what the load is. >> >> Got the same problem this morning, here is the CPU usage and ps aux for >> dovecot. plus the different error I could pick up in the log, most of >> them are repeated a couple of times. >> >> I suspect it a problem with system resources, but can find any message >> to tell me what. Mail are stored on 17 NFS servers (CentOS), plus 3 >> servers for indexes only. >> >> CPU load is very high, but mainly from httpd running our webmail >> interface, which uses the local imap server. > [...] >> top - 11:10:14 up 14 days, 12:04, 2 users, load average: 55.04, 29.13, >> 14.55 >> Tasks: 474 total, 60 running, 414 sleeping, 0 stopped, 0 zombie >> Cpu(s): 99.6%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.1%si, >> 0.0%st >> Mem: 16439812k total, 16353268k used,86544k free,33268k buffers >> Swap: 4192956k total, 140k used, 4192816k free, 8228744k cached > > You're lucky this server is still alive and that you could even run top > and ps on it. > > There's nothing to debug in dovecot here. Your server is overloaded by > about 55 times. Buy 55 times as many servers or do something about your > webmail interface (maybe a separate webmail cluster). > > Cheers, Chris. > As you can see the numbers (55.04, 29.13, 14.55) the load was busy getting higher when I took this snapshot and this was not a normal situation. Usually this machine's load is only between 1 and 4, which is quite ok for a quad core. It only happens when dovecot start generating errors, and pop3, imap and http get stuck. It went up to 200, and I was still able to stop web and mail daemons, then restart them, and everything was back to normal.
[Dovecot] compressed mailboxes ?
Hello Is it possible to use compressed mailboxes ( MBOX format ) with dovecot using a plugin ? Thanks
Re: [Dovecot] compressed mailboxes ?
On 03/08/2011 04:54 PM, Frank Bonnet wrote: Hello Is it possible to use compressed mailboxes ( MBOX format ) with dovecot using a plugin ? Thanks I found the doc about zlib plugin ... so it is only possible with read only mailboxes huh ?
Re: [Dovecot] auth on tcp socket?
Hello, sorry, iam digest reader, always creating a new message... > > there are some warnings in log: > > "when SASL type is "dovecot", SASL path "inet:localhost:1434" should > > be a > > socket pathname" > > ( Datei src/xsasl/xsasl_dovecot_server.c ) > > file xsasl_dovecot_server.c is in postfix sources but was written by > you, > Timo The initial version, yes, but it has been heavily modified since. I never added that warning. > Should there also made an update to support auth on tcp-socket without > warnings from postfix side? Looks like someone already added the inet: support, so I guess it should work. You could ask on Postfix list to get the warning removed.. wietse dont wants to change anything until it is not documented http://archives.neohapsis.com/archives/postfix/2011-03/0356.html may be the developers should talk directly to each other, i dont like to get forwarded to each other list to ask for reliability of features... Thanks, Hajo
Re: [Dovecot] POP3 error
On 2011-03-08 10:40 AM, Thierry de Montaudry wrote: > On 08 Mar 2011, at 13:24, Chris Wilson wrote: >> There's nothing to debug in dovecot here. Your server is overloaded >> by about 55 times. Buy 55 times as many servers or do something >> about your webmail interface (maybe a separate webmail cluster). > As you can see the numbers (55.04, 29.13, 14.55) the load was busy > getting higher when I took this snapshot and this was not a normal > situation. Usually this machine's load is only between 1 and 4, which > is quite ok for a quad core. It only happens when dovecot start > generating errors, and pop3, imap and http get stuck. It went up to > 200, and I was still able to stop web and mail daemons, then restart > them, and everything was back to normal. What is your webmail server (and version)? Maybe it is buggy? -- Best regards, Charles
Re: [Dovecot] POP3 error
Hi Thierry, On Tue, 8 Mar 2011, Thierry de Montaudry wrote: > On 08 Mar 2011, at 13:24, Chris Wilson wrote: > > > >> top - 11:10:14 up 14 days, 12:04, 2 users, load average: 55.04, 29.13, > >> 14.55 > >> Tasks: 474 total, 60 running, 414 sleeping, 0 stopped, 0 zombie > >> Cpu(s): 99.6%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.1%si, > >> 0.0%st > >> Mem: 16439812k total, 16353268k used,86544k free,33268k buffers > >> Swap: 4192956k total, 140k used, 4192816k free, 8228744k cached > > As you can see the numbers (55.04, 29.13, 14.55) the load was busy > getting higher when I took this snapshot and this was not a normal > situation. Usually this machine's load is only between 1 and 4, which is > quite ok for a quad core. It only happens when dovecot start generating > errors, and pop3, imap and http get stuck. It went up to 200, and I was > still able to stop web and mail daemons, then restart them, and > everything was back to normal. I don't have a definite answer, but I remember that there has been a long-running bug in the Linux kernel with schedulers behaving badly under heavy writes: "One of the problems commonly talked about in our forums and elsewhere is the poor responsiveness of the Linux desktop when dealing with significant disk activity on systems where there is insufficient RAM or the disks are slow. The GUI basically drops to its knees when there is too much disk activity..." [http://www.phoronix.com/scan.php?page=news_item&px=ODQ3Mw] (note, it's not just the GUI, all other tasks can starve when a disk I/O queue builds up). "There are a few options to tune the linux IO scheduler that can help a bunch... Typically CFQ stalls too long under heavy writes, especially if your disk subsystem sucks, so particularly if you have several spindles deadline is worth a try." [http://communities.vmware.com/thread/82544] "I run Ubuntu on a moderately powerful quad-core x86-64 system and the desktop response is basically crippled whenever something is reading or writing large files as fast as it can (at normal priority)... For example, cat /path/to/LARGE_FILE > /dev/null ... Everything else gets completely unusable because of the I/O latency." [https://bugs.launchpad.net/ubuntu/+source/linux/+bug/343371] "I was just running mkfs.ext4 -b 4096 -E stride=128 -E stripe-width=128 -O ^has_journal /dev/sdb2 on my SSD18M connected via USB1.1, and the result was, well, absolutely, positively _DEVASTATING_. The entire system became _FULLY_ unresponsive, not even switching back down to tty1 via Ctrl-Alt-F1 worked (took 20 seconds for even this key to be respected)." [http://lkml.org/lkml/2010/4/4/86] "This regression has been around since about the 2.6.18 timeframe and has eluded a lot of testing to isolate the root cause. The most promising fix is in the VM subsystem (mm) where the LRU scan has been changed to favor keeping executable pages active longer. Most of these symptoms come down to VM thrashing to make room for I/O pages. The key change/commit is ab4754d24a0f2e05920170c845bd84472814c6, "vmscan: make mapped executable pages the first class citizen"... This change was merged into the 2.6.31r1 kernel." [https://bugs.launchpad.net/ubuntu/+source/linux/+bug/131094/comments/235] One possible cause is that writing to a slow device can block the write queue for other devices, causing the machine to come to a standstill when there's plenty of useful work that it could be doing. This could cause a cascading failure in your server as soon as disk I/O write load goes over a certain point, a bit like a swap death. I'm not sure if the fact that you're using NFS makes a difference; perhaps only if you memory-map files? You could test this by booting with the NOOP or anticipatory scheduler instead of the default CFQ to see if it makes any difference. Cheers, Chris. -- Aptivate | http://www.aptivate.org | Phone: +44 1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES Aptivate is a not-for-profit company registered in England and Wales with company number 04980791.
Re: [Dovecot] compressed mailboxes ?
Frank Bonnet put forth on 3/8/2011 9:54 AM: > Hello > > Is it possible to use compressed mailboxes ( MBOX format ) > with dovecot using a plugin ? What is your motivation? -- Stan
[Dovecot] Filtering attachements ?
Hello again Is there a way to filter some attachments with dovecot ? I don't want my users to keep video files in the IMAP folders anymore ( disk is almost full ) . Thank you.
Re: [Dovecot] POP3 error
On 03/08/2011 09:26 AM, Chris Wilson wrote: Hi Thierry, On Tue, 8 Mar 2011, Thierry de Montaudry wrote: On 08 Mar 2011, at 13:24, Chris Wilson wrote: top - 11:10:14 up 14 days, 12:04, 2 users, load average: 55.04, 29.13, 14.55 Tasks: 474 total, 60 running, 414 sleeping, 0 stopped, 0 zombie Cpu(s): 99.6%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 16439812k total, 16353268k used,86544k free,33268k buffers Swap: 4192956k total, 140k used, 4192816k free, 8228744k cached As you can see the numbers (55.04, 29.13, 14.55) the load was busy getting higher when I took this snapshot and this was not a normal situation. Usually this machine's load is only between 1 and 4, which is quite ok for a quad core. It only happens when dovecot start generating errors, and pop3, imap and http get stuck. It went up to 200, and I was still able to stop web and mail daemons, then restart them, and everything was back to normal. I don't have a definite answer, but I remember that there has been a long-running bug in the Linux kernel with schedulers behaving badly under heavy writes: "One of the problems commonly talked about in our forums and elsewhere is the poor responsiveness of the Linux desktop when dealing with significant disk activity on systems where there is insufficient RAM or the disks are slow. The GUI basically drops to its knees when there is too much disk activity..." [http://www.phoronix.com/scan.php?page=news_item&px=ODQ3Mw] (note, it's not just the GUI, all other tasks can starve when a disk I/O queue builds up). "There are a few options to tune the linux IO scheduler that can help a bunch... Typically CFQ stalls too long under heavy writes, especially if your disk subsystem sucks, so particularly if you have several spindles deadline is worth a try." [http://communities.vmware.com/thread/82544] "I run Ubuntu on a moderately powerful quad-core x86-64 system and the desktop response is basically crippled whenever something is reading or writing large files as fast as it can (at normal priority)... For example, cat /path/to/LARGE_FILE> /dev/null ... Everything else gets completely unusable because of the I/O latency." [https://bugs.launchpad.net/ubuntu/+source/linux/+bug/343371] "I was just running mkfs.ext4 -b 4096 -E stride=128 -E stripe-width=128 -O ^has_journal /dev/sdb2 on my SSD18M connected via USB1.1, and the result was, well, absolutely, positively _DEVASTATING_. The entire system became _FULLY_ unresponsive, not even switching back down to tty1 via Ctrl-Alt-F1 worked (took 20 seconds for even this key to be respected)." [http://lkml.org/lkml/2010/4/4/86] "This regression has been around since about the 2.6.18 timeframe and has eluded a lot of testing to isolate the root cause. The most promising fix is in the VM subsystem (mm) where the LRU scan has been changed to favor keeping executable pages active longer. Most of these symptoms come down to VM thrashing to make room for I/O pages. The key change/commit is ab4754d24a0f2e05920170c845bd84472814c6, "vmscan: make mapped executable pages the first class citizen"... This change was merged into the 2.6.31r1 kernel." [https://bugs.launchpad.net/ubuntu/+source/linux/+bug/131094/comments/235] One possible cause is that writing to a slow device can block the write queue for other devices, causing the machine to come to a standstill when there's plenty of useful work that it could be doing. This could cause a cascading failure in your server as soon as disk I/O write load goes over a certain point, a bit like a swap death. I'm not sure if the fact that you're using NFS makes a difference; perhaps only if you memory-map files? You could test this by booting with the NOOP or anticipatory scheduler instead of the default CFQ to see if it makes any difference. Cheers, Chris. You can change it on the fly with: `echo noop > /sys/block/${DEVICE}/queue/scheduler` -- -Eric 'shubes'
Re: [Dovecot] compressed mailboxes ?
Em 08/03/11 12:57, Frank Bonnet escreveu: On 03/08/2011 04:54 PM, Frank Bonnet wrote: Hello Is it possible to use compressed mailboxes ( MBOX format ) with dovecot using a plugin ? Thanks I found the doc about zlib plugin ... so it is only possible with read only mailboxes huh ? the problem here is the mbox format the zlib plugin works flawlessly to store compressed files with Maildir mailboxes. I'm using it on several servers to serve some thousand mailboxes. -- Atenciosamente / Sincerily, Leonardo Rodrigues Solutti Tecnologia http://www.solutti.com.br Minha armadilha de SPAM, NÃO mandem email gertru...@solutti.com.br My SPAMTRAP, do not email it
Re: [Dovecot] POP3 error
On 08 Mar 2011, at 18:14, Charles Marcus wrote: > On 2011-03-08 10:40 AM, Thierry de Montaudry wrote: >> On 08 Mar 2011, at 13:24, Chris Wilson wrote: >>> There's nothing to debug in dovecot here. Your server is overloaded >>> by about 55 times. Buy 55 times as many servers or do something >>> about your webmail interface (maybe a separate webmail cluster). > >> As you can see the numbers (55.04, 29.13, 14.55) the load was busy >> getting higher when I took this snapshot and this was not a normal >> situation. Usually this machine's load is only between 1 and 4, which >> is quite ok for a quad core. It only happens when dovecot start >> generating errors, and pop3, imap and http get stuck. It went up to >> 200, and I was still able to stop web and mail daemons, then restart >> them, and everything was back to normal. > > What is your webmail server (and version)? Maybe it is buggy? > Using HastyMail2-1.0. But the problem only started when we moved to dovecot 2.0.9 (from 1.10.13), without changing anything else on any of our 7 machines, and now it's happening randomly on any of them. So that's why I suspect it has to do with dovecot.
Re: [Dovecot] Multiple Concurrent IMAP Connections For Same User
On 03/06/2011 03:46 PM, Timo Sirainen wrote: On 7.3.2011, at 1.39, Dave McGuire wrote: That said, I've been testing Timo's imap_capabilities suggestion for about the past 45mins, and it seems to solve the problem for me, at least so far. So it sounds like there's a bug in someone's QRESYNC code. Either Dovecot or Thunderbird, but could be a bit difficult to find out whose.. A reproducible test case would of course be best. Just a follow up. After adjusting the imap_capability as you suggested I see no left over emails in TB in the long term and only can occasionally get it to not delete one right away. I have turned GLODA back on and that seems to have no effect on the situation at all. I do not sync my messages locally and did not before trying this. Thanks very much, -- Knute Johnson knute2...@knutejohnson.com
Re: [Dovecot] POP3 error
On 08 Mar 2011, at 18:26, Chris Wilson wrote: > Hi Thierry, > > On Tue, 8 Mar 2011, Thierry de Montaudry wrote: >> On 08 Mar 2011, at 13:24, Chris Wilson wrote: >>> top - 11:10:14 up 14 days, 12:04, 2 users, load average: 55.04, 29.13, 14.55 Tasks: 474 total, 60 running, 414 sleeping, 0 stopped, 0 zombie Cpu(s): 99.6%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 16439812k total, 16353268k used,86544k free,33268k buffers Swap: 4192956k total, 140k used, 4192816k free, 8228744k cached >> >> As you can see the numbers (55.04, 29.13, 14.55) the load was busy >> getting higher when I took this snapshot and this was not a normal >> situation. Usually this machine's load is only between 1 and 4, which is >> quite ok for a quad core. It only happens when dovecot start generating >> errors, and pop3, imap and http get stuck. It went up to 200, and I was >> still able to stop web and mail daemons, then restart them, and >> everything was back to normal. > > I don't have a definite answer, but I remember that there has been a > long-running bug in the Linux kernel with schedulers behaving badly under > heavy writes: > > "One of the problems commonly talked about in our forums and elsewhere is > the poor responsiveness of the Linux desktop when dealing with significant > disk activity on systems where there is insufficient RAM or the disks are > slow. The GUI basically drops to its knees when there is too much disk > activity..." [http://www.phoronix.com/scan.php?page=news_item&px=ODQ3Mw] > (note, it's not just the GUI, all other tasks can starve when a disk I/O > queue builds up). > > "There are a few options to tune the linux IO scheduler that can help a > bunch... Typically CFQ stalls too long under heavy writes, especially if > your disk subsystem sucks, so particularly if you have several spindles > deadline is worth a try." [http://communities.vmware.com/thread/82544] > > "I run Ubuntu on a moderately powerful quad-core x86-64 system and the > desktop response is basically crippled whenever something is reading or > writing large files as fast as it can (at normal priority)... For example, > cat /path/to/LARGE_FILE > /dev/null ... Everything else gets completely > unusable because of the I/O latency." > [https://bugs.launchpad.net/ubuntu/+source/linux/+bug/343371] > > "I was just running mkfs.ext4 -b 4096 -E stride=128 -E stripe-width=128 -O > ^has_journal /dev/sdb2 on my SSD18M connected via USB1.1, and the result > was, well, absolutely, positively _DEVASTATING_. The entire system became > _FULLY_ unresponsive, not even switching back down to tty1 via Ctrl-Alt-F1 > worked (took 20 seconds for even this key to be respected)." > [http://lkml.org/lkml/2010/4/4/86] > > "This regression has been around since about the 2.6.18 timeframe and has > eluded a lot of testing to isolate the root cause. The most promising fix > is in the VM subsystem (mm) where the LRU scan has been changed to favor > keeping executable pages active longer. Most of these symptoms come down > to VM thrashing to make room for I/O pages. The key change/commit is > ab4754d24a0f2e05920170c845bd84472814c6, "vmscan: make mapped executable > pages the first class citizen"... This change was merged into the 2.6.31r1 > kernel." > [https://bugs.launchpad.net/ubuntu/+source/linux/+bug/131094/comments/235] > > One possible cause is that writing to a slow device can block the write > queue for other devices, causing the machine to come to a standstill when > there's plenty of useful work that it could be doing. > > This could cause a cascading failure in your server as soon as disk > I/O write load goes over a certain point, a bit like a swap death. I'm not > sure if the fact that you're using NFS makes a difference; perhaps only if > you memory-map files? > > You could test this by booting with the NOOP or anticipatory scheduler > instead of the default CFQ to see if it makes any difference. > > Cheers, Chris. Hi Chris, Thanks for your (long) comment and tech details, but having not changed anything on the 7 machines, but moving from dovecot 1.10.13 to 2.0.9, without increasing our traffic, I don't want to start changing tricky stuff in the system when it worked fine for almost 2 years. And the fact that all mails are stored on multiple NFS servers, all machine having 16G RAM, makes me think that it's not an IO problem. I though it might be the system running out of resources, but there nothing about it in the logs... For now, we might consider reversing to 1.10.13... but that would be with the loss of the new features that made us upgrade, so not good.
Re: [Dovecot] logging sent messages
W dniu 2011-03-08 16:03, Hajo Locke pisze: > i use plugin mail_log to see more activities like copy/delete/expunge caused > by imap-users. > unfortunately there is no log-line when a mail was sent and the copy was > uploaded to "Sent-Folder". It is Append action and as stated in documentation Appends are supported too (v1.2+ required). http://wiki.dovecot.org/Plugins/MailLog -- Aleksander 'A.L.E.C' Machniak http://alec.pl gg:2275252 LAN Management System Developer http://lms.org.pl Roundcube Webmail Developer http://roundcube.net
Re: [Dovecot] POP3 error
On 2011-03-08 11:49 AM, Thierry de Montaudry wrote: > Using HastyMail2-1.0. But the problem only started when we moved to > dovecot 2.0.9 (from 1.10.13), without changing anything else on any > of our 7 machines, and now it's happening randomly on any of them. So > that's why I suspect it has to do with dovecot. Or an interaction of the new version of Dovecot and HastyMail. The reason I asked about your webmail server is you had specifically said that it was the httpd process that was consuming all of the CPU... -- Best regards, Charles
Re: [Dovecot] POP3 error
On 2011-03-08 12:00 PM, Thierry de Montaudry wrote: > but moving from dovecot 1.10.13 to 2.0.9 First time I thought it was a typo and ignored it... There has never been a version 1.10.xxx Maybe you mean 1.0.13? -- Best regards, Charles
Re: [Dovecot] POP3 error
On 08 Mar 2011, at 19:11, Charles Marcus wrote: > On 2011-03-08 11:49 AM, Thierry de Montaudry wrote: >> Using HastyMail2-1.0. But the problem only started when we moved to >> dovecot 2.0.9 (from 1.10.13), without changing anything else on any >> of our 7 machines, and now it's happening randomly on any of them. So >> that's why I suspect it has to do with dovecot. > > Or an interaction of the new version of Dovecot and HastyMail. > > The reason I asked about your webmail server is you had specifically > said that it was the httpd process that was consuming all of the CPU... > Yes, because they were in the top of the top list.
Re: [Dovecot] POP3 error
On 08 Mar 2011, at 19:12, Charles Marcus wrote: > On 2011-03-08 12:00 PM, Thierry de Montaudry wrote: >> but moving from dovecot 1.10.13 to 2.0.9 > > First time I thought it was a typo and ignored it... > > There has never been a version 1.10.xxx > > Maybe you mean 1.0.13? > Sorry, my mistake, 1.1.13, version integrated in CentOS 5.
Re: [Dovecot] POP3 error
On 2011-03-08 12:30 PM, Thierry de Montaudry wrote: > On 08 Mar 2011, at 19:11, Charles Marcus wrote: >> The reason I asked about your webmail server is you had specifically >> said that it was the httpd process that was consuming all of the CPU... > Yes, because they were in the top of the top list. So... if the httpd process is the one consuming all of the CPU, doesn't it stand to reason that it might be something to do with one of your web apps, and not dovecot? -- Best regards, Charles
Re: [Dovecot] POP3 error
On 08 Mar 2011, at 19:37, Charles Marcus wrote: > On 2011-03-08 12:30 PM, Thierry de Montaudry wrote: >> On 08 Mar 2011, at 19:11, Charles Marcus wrote: >>> The reason I asked about your webmail server is you had specifically >>> said that it was the httpd process that was consuming all of the CPU... > >> Yes, because they were in the top of the top list. > > So... if the httpd process is the one consuming all of the CPU, doesn't > it stand to reason that it might be something to do with one of your web > apps, and not dovecot? > But then why was it fine with 1.1.13, which never had once this problem in 2 years? or is 2.0.9 slower, or consuming more resources to create the problem?
Re: [Dovecot] Multiple Concurrent IMAP Connections For Same User
On 3/8/2011 11:55 AM, Knute Johnson wrote: On 03/06/2011 03:46 PM, Timo Sirainen wrote: On 7.3.2011, at 1.39, Dave McGuire wrote: That said, I've been testing Timo's imap_capabilities suggestion for about the past 45mins, and it seems to solve the problem for me, at least so far. So it sounds like there's a bug in someone's QRESYNC code. Either Dovecot or Thunderbird, but could be a bit difficult to find out whose.. A reproducible test case would of course be best. Just a follow up. After adjusting the imap_capability as you suggested I see no left over emails in TB in the long term and only can occasionally get it to not delete one right away. I have turned GLODA back on and that seems to have no effect on the situation at all. I do not sync my messages locally and did not before trying this. And, shockingly, with the two capabilities removed, TB's notification popup actually reports the number of new messages in the inbox, instead of some random number in the tens of thousands. (dovecot 2.0.11 / TB 3.0.4) I think you're on to something here.
Re: [Dovecot] POP3 error
On 2011-03-08 12:30 PM, Thierry de Montaudry wrote: > On 08 Mar 2011, at 19:11, Charles Marcus wrote: >> The reason I asked about your webmail server is you had specifically >> said that it was the httpd process that was consuming all of the CPU... > Yes, because they were in the top of the top list. And they were on the top of the list because... they were consuming all of the CPU? -- Best regards, Charles
Re: [Dovecot] POP3 error
On 2011-03-08 12:42 PM, Thierry de Montaudry wrote: > On 08 Mar 2011, at 19:37, Charles Marcus wrote: >> So... if the httpd process is the one consuming all of the CPU, doesn't >> it stand to reason that it might be something to do with one of your web >> apps, and not dovecot? > But then why was it fine with 1.1.13, which never had once this > problem in 2 years? or is 2.0.9 slower, or consuming more resources > to create the problem? You don't see how it might be possible that 2.0.x does something that 1.1.x didn't do that your webmail app might not like, without it being a dovecot bug? I'm not saying it is or it isn't, but I'd look there first - see if an update is available for your webmail app... since you were running an ancient version of dovecot, maybe you're also running an ancient version of it too? -- Best regards, Charles
Re: [Dovecot] Multiple Concurrent IMAP Connections For Same User
On 2011-03-08 12:47 PM, Tom Talpey wrote: > (dovecot 2.0.11 / TB 3.0.4) > > I think you're on to something here. You do need to update TB too you know... The number of bugs squashed since 3.0.4 are too vast too even begin to contemplate... -- Best regards, Charles
Re: [Dovecot] Multiple Concurrent IMAP Connections For Same User
On 3/8/2011 12:52 PM, Charles Marcus wrote: On 2011-03-08 12:47 PM, Tom Talpey wrote: (dovecot 2.0.11 / TB 3.0.4) I think you're on to something here. You do need to update TB too you know... The number of bugs squashed since 3.0.4 are too vast too even begin to contemplate... It's actually Eudora 1.0, and workarounds are less painful than upgrades. Glad to have this one.
Re: [Dovecot] auth on tcp socket?
On Tue, Mar 08, 2011 at 05:07:49PM +0100, Hajo Locke wrote: > Hello, > > sorry, iam digest reader, always creating a new message... On http://dovecot.org/cgi-bin/mailman/options/dovecot you can login to change your membership options ;) [..] > >The initial version, yes, but it has been heavily modified since. I > >never added that warning. > > >> Should there also made an update to support auth on tcp-socket without > >> warnings from postfix side? > > >Looks like someone already added the inet: support, so I guess it should > >work. You could ask on Postfix list to get the warning removed.. > > wietse dont wants to change anything until it is not documented > http://archives.neohapsis.com/archives/postfix/2011-03/0356.html > may be the developers should talk directly to each other, i dont > like to get forwarded to each other list to ask for reliability of > features... You should read Wietses post again. He told you to not rely on not documented settings since they can silently disappear. With postfix you can rely on the documented behaviour. So it is natural that there will be no warning for officially unsupported features. Dennis
[Dovecot] ID command bug & patch
When I set imap_id_log=* in dovecot-2.0.11 and a client sends, say, tag id ("a" "b" "c" "d") then dovecot logs: ID sent: a=b, b=c, c=d The following patch makes it log instead: ID sent: a=b, c=d --- a/dovecot/src/lib-imap/imap-id.c(revision 113366) +++ b/dovecot/src/lib-imap/imap-id.c(working copy) @@ -164,6 +164,7 @@ str_append_c(reply, '='); str_append(reply, str_sanitize(value, 80)); } + args++; } return str_len(reply) == 0 ? NULL : str_c(reply); }
[Dovecot] warnings from 2.0.11
During a stress test dovecot-2.0.11 logs many warnings: Mar 8 06:19:52 gromit dovecot[59204]: imap(pid 68864 user user15): Warning: /Volumes/Mail/user15/dovecot-uidlist: Duplicate file entry at line 183: 1299556557.M571530P31883.gromit.example.com,S=21175,W=21549 (uid 67 -> 250) Mar 8 06:20:10 gromit dovecot[59204]: imap(pid 68865 user user34): Warning: /Volumes/Mail/user34/dovecot-uidlist: Duplicate file entry at line 230: 1299535329.M679065P61136.gromit.example.com,S=11040,W=11247 (uid 17 -> 240) Mar 8 06:20:10 gromit dovecot[59204]: imap(pid 68865 user user34): Warning: /Volumes/Mail/user34/dovecot-uidlist: Duplicate file entry at line 240: 1299535329.M902424P61136.gromit.example.com,S=12843,W=13075 (uid 30 -> 250) Mar 8 06:20:10 gromit dovecot[59204]: imap(pid 68865 user user34): Warning: /Volumes/Mail/user34/dovecot-uidlist: Duplicate file entry at line 266: 1299547358.M587728P69303.gromit.example.com,S=13637,W=13772 (uid 108 -> 276) Mar 8 06:20:10 gromit dovecot[59204]: imap(pid 68865 user user34): Warning: /Volumes/Mail/user34/dovecot-uidlist: Duplicate file entry at line 275: 1299558462.M498819P35210.gromit.example.com,S=15495,W=15663 (uid 160 -> 285) Mar 8 06:20:11 gromit dovecot[59204]: imap(pid 68864 user user36): Warning: /Volumes/Mail/user36/dovecot-uidlist: Duplicate file entry at line 281: 1299535339.M353973P61136.gromit.example.com,S=4498,W=4581 (uid 360 -> 604) Mar 8 06:21:00 gromit dovecot[59204]: imap(pid 68864 user user239): Warning: /Volumes/Mail/user239/dovecot-uidlist: Duplicate file entry at line 105: 1299547390.M813768P69393.gromit.example.com,S=4773,W=4861 (uid 22 -> 164) Mar 8 06:21:00 gromit dovecot[59204]: imap(pid 68864 user user239): Warning: /Volumes/Mail/user239/dovecot-uidlist: Duplicate file entry at line 106: 1299570075.M267312P49194.gromit.example.com,S=8041,W=8181 (uid 115 -> 165) Mar 8 06:28:24 gromit dovecot[59204]: imap(pid 69295 user user274): Warning: /Volumes/Mail/user274/dovecot-uidlist: Duplicate file entry at line 2020: 1299548064.M880781P67832.gromit.example.com,S=4976,W=5076 (uid 247 -> 2500) Mar 8 06:29:23 gromit dovecot[59204]: imap(pid 68973 user user115): Warning: /Volumes/Mail/user115/dovecot-uidlist: Duplicate file entry at line 526: 1299537039.M676050P61136.gromit.example.com,S=2485,W=2534 (uid 8 -> 1082) Mar 8 06:31:37 gromit dovecot[59204]: imap(pid 68971 user user114): Warning: /Volumes/Mail/user114/dovecot-uidlist: Duplicate file entry at line 667: 1299564058.M124508P40871.gromit.example.com,S=1187940,W=1207707 (uid 229 -> 780) - retrying by re-reading from beginning Mar 8 06:32:14 gromit dovecot[59204]: imap(pid 68971 user user114): Warning: Maildir /Volumes/Mail/user114: Expunged message reappeared, giving a new UID (old uid=229, file=1299564058.M124508P40871.gromit.example.com,S=1187940,W=1207707:2,S) Mar 8 06:39:22 gromit dovecot[59204]: imap(pid 68911 user user105): Warning: Maildir /Volumes/Mail/user105: Synchronization took 155 seconds (0 new msgs, 0 flag change attempts, 585 expunge attempts) Is there anything I need to reconfigure to prevent these? Thanks.
Re: [Dovecot] POP3 error
On 03/08/2011 06:51 PM, Charles Marcus wrote: On 2011-03-08 12:42 PM, Thierry de Montaudry wrote: On 08 Mar 2011, at 19:37, Charles Marcus wrote: So... if the httpd process is the one consuming all of the CPU, doesn't it stand to reason that it might be something to do with one of your web apps, and not dovecot? But then why was it fine with 1.1.13, which never had once this problem in 2 years? or is 2.0.9 slower, or consuming more resources to create the problem? You don't see how it might be possible that 2.0.x does something that 1.1.x didn't do that your webmail app might not like, without it being a dovecot bug? I'm not saying it is or it isn't, but I'd look there first - see if an update is available for your webmail app... since you were running an ancient version of dovecot, maybe you're also running an ancient version of it too? I can see similar problems (subject: "Restarting dovecot-auth stops authentication"), on a different OS, and nothing common in the webmail area. I think this is clearly related to Dovecot. It handles load very badly (well, it seems at least on common OS settings), doesn't just slow down, but starts to refuse clients. It seems to be obvious that the interprocess socket communication is where it fails, so this is what needs to be investigated. Sadly, doing this on a machine, which cries for a deep breath already is not always easy.
Re: [Dovecot] POP3 error
On 2011-03-08 3:38 PM, Attila Nagy wrote: > On 03/08/2011 06:51 PM, Charles Marcus wrote: >> You don't see how it might be possible that 2.0.x does something that >> 1.1.x didn't do that your webmail app might not like, without it being a >> dovecot bug? >> >> I'm not saying it is or it isn't, but I'd look there first - see if an >> update is available for your webmail app... since you were running an >> ancient version of dovecot, maybe you're also running an ancient version >> of it too? > I can see similar problems (subject: "Restarting dovecot-auth stops > authentication"), on a different OS, and nothing common in the webmail > area. Similar problem? I just read that entire thread, and there was absolutely no mention of high resource usage, and it was the 4th or 5th email before you finally provided system details (which should always be provided in the first email to save time) and Timo noticed that you had changed some defaults that you shouldn't have... so I don't think that thread qualifies as being anywhere near similar. > I think this is clearly related to Dovecot. It handles load very badly Whoa, pardner, fyi, there are many, many installations humming along smoothly. > (well, it seems at least on common OS settings), doesn't just slow down, > but starts to refuse clients. Maybe there is a bug somewhere that only becomes evident under certain circumstances, but it is also possibly due to config problems caused by... -- Best regards, Charles
[Dovecot] Interesting tool for debugging IMAP sessions
For anyone who is interested: http://www.aboutmyip.com/AboutMyXApp/ImapProxy.jsp They also have one for troubleshooting smtp (smtpProxy)... -- Best regards, Charles
Re: [Dovecot] POP3 error
Am 08.03.2011 21:38, schrieb Attila Nagy: > On 03/08/2011 06:51 PM, Charles Marcus wrote: >> On 2011-03-08 12:42 PM, Thierry de Montaudry wrote: >>> On 08 Mar 2011, at 19:37, Charles Marcus wrote: So... if the httpd process is the one consuming all of the CPU, doesn't it stand to reason that it might be something to do with one of your web apps, and not dovecot? >>> But then why was it fine with 1.1.13, which never had once this >>> problem in 2 years? or is 2.0.9 slower, or consuming more resources >>> to create the problem? >> You don't see how it might be possible that 2.0.x does something that >> 1.1.x didn't do that your webmail app might not like, without it being a >> dovecot bug? >> >> I'm not saying it is or it isn't, but I'd look there first - see if an >> update is available for your webmail app... since you were running an >> ancient version of dovecot, maybe you're also running an ancient version >> of it too? >> > I can see similar problems (subject: "Restarting dovecot-auth stops > authentication"), on a different OS, and nothing common in the webmail > area. > > I think this is clearly related to Dovecot. It handles load very badly > (well, it seems at least on common OS settings), doesn't just slow down, > but starts to refuse clients. > It seems to be obvious that the interprocess socket communication is > where it fails, so this is what needs to be investigated. > Sadly, doing this on a machine, which cries for a deep breath already is > not always easy. you might upgrade to the latest 2.x code as it might possible your using more stuff then you had in older versions, after all there was a long performance thread on this list , look for it in archives -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: [Dovecot] compressed mailboxes ?
Frank Bonnet put forth on 3/8/2011 2:07 PM: > disk space I need to gain a bit space > before migrate to the new server After you shut down the Dovecot daemons in preparation for copying the mailbox files to the new sever, simply delete all the index directories. That will free up a huge amount of space. The indexes will be automatically rebuilt when users log into the new server. -- Stan > Le 08/03/2011 17:39, Stan Hoeppner a écrit : >> Frank Bonnet put forth on 3/8/2011 9:54 AM: >>> Hello >>> >>> Is it possible to use compressed mailboxes ( MBOX format ) >>> with dovecot using a plugin ? >> >> What is your motivation? >>
Re: [Dovecot] POP3 error
On 03/08/2011 09:58 PM, Charles Marcus wrote: I think this is clearly related to Dovecot. It handles load very badly Whoa, pardner, fyi, there are many, many installations humming along smoothly. No offense. It may be more correct to say situations, where the OS can't deliver prompt resources to Dovecot, like saturated disk IO and similar stuff. I can't see such problems with moderate load, and maybe there aren't so many installations, which handle a lot of traffic. I don't know. I don't think it's a bug, currently to me it seems to be a tuning/configuration issue. But maybe it's a common design related issue, which is not yet fully explored. (well, it seems at least on common OS settings), doesn't just slow down, but starts to refuse clients. Maybe there is a bug somewhere that only becomes evident under certain circumstances, but it is also possibly due to config problems caused by... Sure.
Re: [Dovecot] POP3 error
On 03/08/2011 10:37 PM, Robert Schetterer wrote: Am 08.03.2011 21:38, schrieb Attila Nagy: On 03/08/2011 06:51 PM, Charles Marcus wrote: On 2011-03-08 12:42 PM, Thierry de Montaudry wrote: On 08 Mar 2011, at 19:37, Charles Marcus wrote: So... if the httpd process is the one consuming all of the CPU, doesn't it stand to reason that it might be something to do with one of your web apps, and not dovecot? But then why was it fine with 1.1.13, which never had once this problem in 2 years? or is 2.0.9 slower, or consuming more resources to create the problem? You don't see how it might be possible that 2.0.x does something that 1.1.x didn't do that your webmail app might not like, without it being a dovecot bug? I'm not saying it is or it isn't, but I'd look there first - see if an update is available for your webmail app... since you were running an ancient version of dovecot, maybe you're also running an ancient version of it too? I can see similar problems (subject: "Restarting dovecot-auth stops authentication"), on a different OS, and nothing common in the webmail area. I think this is clearly related to Dovecot. It handles load very badly (well, it seems at least on common OS settings), doesn't just slow down, but starts to refuse clients. It seems to be obvious that the interprocess socket communication is where it fails, so this is what needs to be investigated. Sadly, doing this on a machine, which cries for a deep breath already is not always easy. you might upgrade to the latest 2.x code as it might possible your using more stuff then you had in older versions, after all there was a long performance thread on this list , look for it in archives I'm running the latest 2.x code (well, sort of, I haven't upgraded to 2.0.10, because of the LDAP bug, so I have both .9 and .11), I've never run 1.x on these machines. I've run qmail and courier. They are pretty different in their architecture, where these kind of stuff (unix socket communication between persisently running daemons) is not touched, so there can't be a problem, where for example five thousand connections are made in the same moment to a single socket/process. There there will be five thousand forks/execs, which won't fail with connection refused, they will be served as fast as the machine can handle them (modulo available memory/file descriptors/etc of course).
[Dovecot] Fatal: postmaster_address setting not given
Hi, I'm getting: "Fatal: postmaster_address setting not given" errors in my log file dovecot ver: 1.2.12 Recently I tried to follow: http://postfixmail.com/blog/index.php/postfixadmin-on-ubuntu-9-10/ on ubuntu 10.10 (which works fine on ubunto 9.10) I note that in 10.10 their is no dovecot-postfix.conf so I made the appropriate changes to dovecot.conf instead and changed master.cf dovecot transport line since I saw someone recently had the same problem: dovecot unix - n n - - pipe flags=DRhu user=vmail:mail argv=/usr/lib/dovecot/deliver -c /etc/dovecot/dovecot.conf -f ${sender} -d ${recipient} I still have that problem however. Authentication seems ok but delivery fails. Any possibility of help on this one? vecot unix - n n - - pipe flags=DRhu user=vmail:mail argv=/usr/lib/dovecot/deliver -c /etc/dovecot/dovecot.conf -f ${sender} -d ${recipient} I still have that problem however. Authentication seems ok but delivery fails. Any possibility of help on this one?
Re: [Dovecot] compressed mailboxes ?
OK thank you Stan gonna try this Le 08/03/2011 22:39, Stan Hoeppner a écrit : > Frank Bonnet put forth on 3/8/2011 2:07 PM: >> disk space I need to gain a bit space >> before migrate to the new server > > After you shut down the Dovecot daemons in preparation for copying the > mailbox files to the new sever, simply delete all the index directories. > That will free up a huge amount of space. The indexes will be > automatically rebuilt when users log into the new server. >
Re: [Dovecot] auth on tcp socket?
Hello, You should read Wietses post again. He told you to not rely on not documented settings since they can silently disappear. With postfix you can rely on the documented behaviour. So it is natural that there will be no warning for officially unsupported features. yes, i know that. nobody should use undocumented features. auth-service on tcp-socket was added by timo after my Suggestion last november. so when asking this on postfix's-side iam not really interested in current docs but wanted to suggest the update on implemention and docs to get it as official feature for next releases. as an user all i can do is telling some facts and ask... Hajo