replication
Hello, I am currently investigating the possibility of adding a second cyrus server, the goal being to entirely duplicate the mailspool/userbase and have users POP in some sort of balanced means, either DNS load balancing or otherwise. We have a pool of inbound MX servers that handle email, delivering to Cyrus over lmtp. In the event of a crash, we would want to repair/rebuild the dead server and have it re-join the pairing. I have read on this list's archives that having user interaction on two replicated servers is iffy, and I wonder what the point is, or is there alternative documentation/methods I should be considering to do this. If it is true that popping email off of two servers set up in replication is a bad idea, I wonder what the point of it really is? Thanks, R. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
disable IMAP IDLE
Hello, I thought it was possible in Cyrus to disable the IDLE functionality, either with imapidlepoll: 0 in imapd.conf, or by commenting idled in cyrus.conf. However, having both disabled, clients still connect and maintain their socket open on tcp 143. Is it not possible or am I going about it wrong? R. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
reducing logging/duplicate messages
Hello, Is it possible to either reduce logging in Cyrus (CYRUS_VERBOSE=1 or 0 currently) to log less chatter? Everything seems to go to the MAIL facility, and there are alot of messages on a idle test system :-0 Also, the user_deny feature seems to log twice? Or 3 times even? Nov 22 20:09:30 node1 pop3[21154]: fetching user_deny.db entry for 'user' Nov 22 20:09:32 node1 pop3[21154]: fetching user_deny.db entry for 'user' Nov 22 20:09:32 node1 pop3[21154]: login: localhost.localdomain [127.0.0.1] user plaintext User logged in SESSIONID= Nov 22 20:09:32 node1 pop3[21154]: fetching user_deny.db entry for 'user' Nov 22 20:09:33 node1 pop3[21154]: USAGE user user: 0.008998 sys: 0.008998 Nov 22 20:09:33 node1 pop3[21154]: counts: retr=<0> top=<0> dele=<0> Nov 22 20:10:39 node1 pop3[21169]: accepted connection Nov 22 20:10:44 node1 pop3[21169]: fetching user_deny.db entry for 'cflav' Nov 22 20:10:44 node1 pop3[21169]: wild: 'imap ' service: 'pop3' Nov 22 20:10:44 node1 pop3[21169]: pat 0:'imap ' Nov 22 20:10:46 node1 pop3[21169]: fetching user_deny.db entry for 'cflav' Nov 22 20:10:46 node1 pop3[21169]: wild: 'imap ' service: 'pop3' Nov 22 20:10:46 node1 pop3[21169]: pat 0:'imap ' Nov 22 20:10:46 node1 pop3[21169]: login: localhost.localdomain [127.0.0.1] cflav plaintext User logged in SESSIONID= Nov 22 20:10:46 node1 pop3[21169]: fetching user_deny.db entry for 'cflav' Nov 22 20:10:46 node1 pop3[21169]: wild: 'imap ' service: 'pop3' Nov 22 20:10:46 node1 pop3[21169]: pat 0:'imap ' Nov 22 20:10:47 node1 pop3[21169]: USAGE cflav user: 0.009998 sys: 0.009998 Nov 22 20:10:47 node1 pop3[21169]: counts: retr=<0> top=<0> dele=<0> Nov 22 20:10:54 node1 pop3[21154]: counts: retr=<0> top=<0> dele=<0> So I seem to be getting alot of duplicate messaging. The "counts" twice, "user_deny.db" three times (including the wildcard and pattern?) for a single POP3 connection. I combed the man pages and found nowhere where I could specify alternate log files, debug levels or whatnot other than the CYRUS_VERBOSE setting, which I believe to have set to the minimum. I shudder to think how many messages will be generated on a busy server. Thanks, R. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: reducing logging/duplicate messages
I will check the .spec file of the RPM I used to install Cyrus to see if the logging facility was modified, however regarding the opening of user_deny.db, this is version 2.4.4 I am using. I don't see why it would be logging it so often then. > Date: Mon, 22 Nov 2010 19:34:26 -0600 > From: dwh...@olp.net > To: prout...@hotmail.com > CC: info-cyrus@lists.andrew.cmu.edu > Subject: Re: reducing logging/duplicate messages > > On 22/11/10 20:14 -0500, Ron Vachiyer wrote: > >Hello, > > > >Is it possible to either reduce logging in Cyrus (CYRUS_VERBOSE=1 or 0 > >currently) to log less chatter? Everything seems to go to the MAIL > >facility, and there are alot of messages on a idle test system :-0 > > > >Also, the user_deny feature seems to log twice? Or 3 times even? > > You can reduce your local6 syslog facility to reduce the amount of > messages: > > http://www.cyrusimap.org/docs/cyrus-imapd/2.4.4/install-configure.php > > Also, a change was introduced in version 2.4.0 which opens user_deny once, > at service startup time: > > http://www.cyrusimap.org/docs/cyrus-imapd/2.4.4/changes.php > > -- > Dan White Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: disable IMAP IDLE
>From how I understand the RFC, if no IMAP IDLE capability is advertised, the >client should not attempt to use it and should close the TCP socket after >every operation. Backberry connections to courier-imap and dovecot will not >maintain a TCP session, they seem to when Cyrus is the IMAP server. R. > Date: Tue, 23 Nov 2010 06:38:28 +0100 > Subject: Re: disable IMAP IDLE > From: simon.mat...@invoca.ch > To: prout...@hotmail.com > CC: info-cyrus@lists.andrew.cmu.edu > > > > > Hello, > > > > I thought it was possible in Cyrus to disable the IDLE functionality, > > either with imapidlepoll: 0 in imapd.conf, or by commenting idled in > > cyrus.conf. However, having both disabled, clients still connect and > > maintain their socket open on tcp 143. Is it not possible or am I going > > about it wrong? > > I may be completely wrong but as I understand it the IMAP client may > always keep connection on port 143 established, IDLE just changes the way > how the client learns about new messages and such. > > Regards, > Simon > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: disable IMAP IDLE
Date: Tue, 23 Nov 2010 08:57:52 -0400 From: bouti...@ednet.ns.ca To: info-cyrus@lists.andrew.cmu.edu Subject: Re: disable IMAP IDLE On 11/23/2010 08:52 AM, Ron Vachiyer wrote: > >From how I understand the RFC, if no IMAP IDLE capability is > advertised, the client should not attempt to use it and should close the > TCP socket after every operation. Backberry connections to courier-imap > and dovecot will not maintain a TCP session, they seem to when Cyrus is > the IMAP server. Is your server advertising IDLE ? telnet to port 143 and issue this command: . CAPABILITY Hello, No it isn't, I had already checked and strangely it is notwhich is really surprising me that the sessions remain; * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN SASL-IR] node1 Cyrus IMAP v2.4.4-Kolab-2.4.4-1 server ready . CAPABILITY * CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN SASL-IR ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE LIST-EXTENDED WITHIN QRESYNC SCAN XLIST URLAUTH URLAUTH=BINARY X-NETSCAPE COMPRESS=DEFLATE . OK Completed I might try and recompile using --with-idle=no and see if sessions stay open. R. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: disable IMAP IDLE
> Date: Tue, 23 Nov 2010 14:44:34 + > From: i...@sussex.ac.uk > To: prout...@hotmail.com; info-cyrus@lists.andrew.cmu.edu > Subject: Re: disable IMAP IDLE > > > > --On 22 November 2010 18:40:37 -0500 Ron Vachiyer > wrote: > > > > > Hello, > > > > I thought it was possible in Cyrus to disable the IDLE functionality, > > either with imapidlepoll: 0 in imapd.conf, or by commenting idled in > > cyrus.conf. However, having both disabled, clients still connect and > > maintain their socket open on tcp 143. Is it not possible or am I going > > about it wrong? > > I thought sessions remained open for efficiency, regardless of IDLE, until > closed by the client or 30 minutes have elapsed. > > IDLE just lets the server notify the client if new email arrives, doesn't > it? > > Even without IDLE, there are benefits in leaving the session open. > Hello, I won't argue since clearly I am in the minority ;) Using courier-imap on our Plesk servers, TCP/143 is closed after every new mail verification. A dovecot server I checked does the same. Cyrus seems to allow the session to be maintained, and yes, it does not advertise IDLE. Below is an example courier-imap capability; * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2004 Double Precision, Inc. See COPYING for distribution information. . CAPABILITY * CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA ACL ACL2=UNION STARTTLS this one is cyrus 2.4.4 . capability * CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN SASL-IR ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE LIST-EXTENDED WITHIN QRESYNC SCAN XLIST URLAUTH URLAUTH=BINARY X-NETSCAPE COMPRESS=DEFLATE . OK Completed I was asked by IT to not permit IDLE since the current server went down after 4-500 blackberries ate up all the (limited) capabilities of that machine. Perhaps I am looking in the wrong place, the point is the demand I am facing is to have IMAP that essentially behaves as a POP3 client when it comes to inbox scans. I believe there was an issue as well where POP clients using outlook would cause mailbox corruption when they popped a mailbox being maintained by a blackberry connected via IMAP. R. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: disable IMAP IDLE
Well after a number of tests, it seems that without IDLE, Blackberries are "smart" and disconnect. iPhones and mail clients (Outlook, Thunderbird) remain connected. This is consistent with what they are doing now so they will have to live with it. Unsure what the real reason behind IDLE-less capability is but I am under direction not to enable it. Thanks again all, R. > Date: Tue, 23 Nov 2010 14:36:47 -0800 > From: david.l...@digitalinsight.com > To: r...@fastmail.fm > CC: prout...@hotmail.com; i...@sussex.ac.uk; info-cyrus@lists.andrew.cmu.edu > Subject: RE: disable IMAP IDLE > > On Wed, 24 Nov 2010, Robert Mueller wrote: > > >> I was asked by IT to not permit IDLE since the current server went down > >> after 4-500 blackberries ate up all the (limited) capabilities of that > >> machine. > > > > I'd really be surprised if that was a problem these days. We have > > machines that have 1 connections quite fine. Yes they're fairly > > loaded servers with fast IO and lots of RAM, but even a pretty basic > > server should handle 500 connections no problem. > > > > Have you actually tested that long lived IDLE connections are a problem? > > if they are idle they eat up very few resources, if they are checking things > frequently, forcing them to reconnect each time will just eat up more > resources. > > David Lang Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
migration from 2.1.x to 2.4.5
Hello, I am pretty much done testing a 2.4.5 Cyrus setup to replace an aging (2.1.x) installation. I have about 36GB of spool, 11,000 accounts or so. I have done some small-scale testing of using imapsync to fetch the contents of the legacy imap folders which seem to work successfully. My question is, when it comes time to migrate, is it safe to allow lmtpd to forward mails to mailboxes that theoretically could be getting imapsync'ed at that time? Or is my only option to spool inbound emails until the imapsync is done? I'm guessing that, from the speed I saw imapsync run, that this might take a good number of hours to complete. As well, is it safe/unsafe to allow imap/pop3 connections to those same mailboxes during the sync? I am open to any ideas should I be looking at this migration the wrong way. I read about using rsync to copy the files directly, however I am wondering if the jump between 2.1 and 2.4 might be too large and cause unforseen issues. Thanks, Ron Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
IOERROR: index record 543384900 for user.pfoo past end of file
Hello, On my Cyrus 2.4.5 test system, I get this error on what seems to be an otherwise functioning mailbox; Dec 10 13:37:41 pop pop3[7635]: IOERROR: index record 543384900 for user.pfoo past end of file I have done a reconstruct that reports nothing, and still the error persists. I can POP mail or use IMAP and all seems well other than this strange message. Hints? R. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
FUD?
Hello,
We have a small tool we were using in an earlier version of Cyrus, using
fud-client to check the last-seen time of a mailbox;
user: ronvach
mbox: user.ronvach
Number of Recent 1
Last read: Wed Dec 31 19:00:00 1969
Last arrived: Tue May 10 12:27:16 2011
Since we upgraded to 2.4.8 (Invoca RPMS), we notice that fud-client/server
returns read to show epoch (?), whereas the cyradm does show something
realistic;
localhost> info user.ronvach
{user.ronvach}:
duplicatedeliver: false
lastpop: 11-May-2011 10:00:11 -0400
lastupdate: 11-May-2011 10:03:59 -0400
partition: default
pop3newuidl: true
sharedseen: false
size: 457
The 2.4.8 was installed on a clean SciLinux 6, what should we look for to
correct this behaviour?
Thanks,
Ron
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: FUD?
> Really? I've never gotten anything but epoch on any Cyrus version (2.1, > 2.2, 2.3) or OS (Fedora 3, 5, CentOS5) that I've tried. I brought it up > on the list a long time ago (2006) and it never got any interest, and > others have brought it up since (2009) with no interest, so I > implemented my own solution which tails the log file and updates a MySQL db. > > > 2006 - http://irbs.net/internet/info-cyrus/0612/0196.html > 2009 - > http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-December/030177.html Hmm, I was testing this deployment on a CentOS5.5 server with 2.4.x/2.4.3 and it seemed to work, I put that project aside and with a brand new SciLinux6+2.4.8 Cyrus I am getting epoch. Irony. Using MySQL sounds like a bit of overkill, maybe cyradm would be scriptable somehow. Or fix FUD ... Ron Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
migration and seen flag
I am to work on migrating a very (very) old install using 2.1.17. The seen information seems to be stored in a /var/lib/imap/user/xx/xxx/xxx.seen file. How can I migrate this seen info to a 2.4.13 install? I thought of using imapsync to move the mailspool, which works, however some mailboxes are 10 Gigs and this process is much slower than scp/rsync of the files followed by a reconstruct. The only lost data seems to be the seen information and I have searched to find a way to migrate it. Any tips? Thanks, Ron Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: migration and seen flag
> Do you know what format the seen files are in on the old server? I think > you should be able to copy them across with the rest of the mail spool > using rsync, but the default database format for seen might have changed. > The default in 2.4 appears to be skiplist. > > Check the setting seenstate_db in imapd.conf. > Hello, On the old server there is no setting for seenstate_db actually specified in the config, however it seems to be skiplist since on a restart it logs; imapd[27028]: skiplist: recovered /var/lib/imap/user/v/viktort.seen The thing is on the 2.4.13 server I have configured; seenstate_db: skiplist however it doesn't create any .seen files like the 2.1.x version, it seems to store seen information in the "cyrus.index" file in each users mailspool directory. I am using the invoca rpm build, is this a particularity or is there another setting I have to set/unset to imitate the previous behaviour? Thanks, Ron Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: migration and seen flag
> Seen data is only stored in .seen files for shared or other user > mailboxes. The seen data for the mailbox owner is stored in the > cyrus.index. This has two benefits: > > 1) lower IO for the most common case. We need to update the >cyrus.index file anyway to record the new MODSEQ. > 2) owner seen becomes shared seen if you switch to shared seen >mode on the mailbox, so that you don't wind up with one huge >file full of shared seen data if you have a lot of shared >mailboxes. > > Seen files were actually a lot of extra IO on 2.3, small files > with multiple fsyncs! > > Bron. Hello, I am beginning to see this. Is there a mecanism to read from these .seen files to recover this data during a migration? Because from what I am seeing, if I move a .seen file from the old server, the new server never reads from it and all my seen data is lost. Thanks in advance, Ron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: migration and seen flag
> Oh, if you've already moved the mailbox and the seen file hasn't > been moved yet... yeah, that's messy :( > > The main problem is that the seen file is indexed by UNIQUEID > rather than mailbox name, otherwise you could just copy-paste > the sequence out and run > > TAG UID STORE +Flags \Seen > > via IMAP after selecting each folder. > > But I suspect you have hundreds of these folders? You can get > the UNIQUEID from the cyrus.header file to script it. You can > use cyr_dbtool from the new cyrus to dump the file and then > use perl or something to munge the data. NOTE: you need to be > able to log in to imap as the user themselves, otherwise you'll > be setting the wrong seen flags. > > Possible cheap-n-nasty workaround is to use an admin connection > and switch each mailbox to sharedseen, apply the flags, then > switch it back. To be clear, here is my scenario; I have 8000 user accounts, 40+Gigs of email on a 2.1.17 server. They all seem to be using .seen files stored in /var/lib/imap/user/u/user.seen files. The mailspool is in /var/spool/imap/u/user/user If I copy the contents of /var/lib/imap/user/[a-z]/* and /var/spool/imap/[a-z]/* to a new server, create accounts beforehand on the new server, then reconstruct -r -f each account, on a 2.3.16 installation this works. I continue to read the .seen files as I am learning that this mecanism was still present in 2.3.x If I do this on a 2.4.13 server, however, the seen data is lost. Perhaps I have to upgrade to 2.3 first and then to 2.4? Ron Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
4096 file descriptors
Quick question about filedescriptors. On Centos6, cyrus 2.3.16 seems to be able to open 4096 FDs ; master[27121]: retrying with 4096 (current max) ulimit -a says 1024; open files (-n) 1024 I am looking to increase this, and have found some documentation saying to increse file-max in /proc. However, file-max already has a much larger number; cat /proc/sys/fs/file-max 1201105 The only way I have found so far is to add a ulimit -n 8192 in /etc/rc.d/init.d/cyrus-imapd Is there a more generic/cleaner way to do this? Thanks, Ron Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
optimized mode for empty maildrop...
Hello, I just installed a 2.3 system, and am looking to decrease the syslog verbosity. This system has mostly POP accounts, and the log message "optimized mode for empty maildrop" is all the eye can see, hundreds of them per minute. Is there a way to limit the logging to success/fail/delivered and reduce the rest? Thanks, Ron Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
simple php sieve script
Hello, I am looking for a rather simple php script to connect to the sieve server, list any active script, and download one if it is active, and print its contents as output. I have been looking at various libraries, but I have come up short of examples. Pear's NetSieve seems particularily powerful yet inversely proportionally documented :( Is anyone using something to do this task? Thanks in advance, Ron Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
