Migration: INBOX^Trash & friends
I have 'www', a Mac OS X 10.4 Server system, running Apple's build of
Cyrus, and want to move my mail over to 'pe', a CentOS 5.1 system
running CentOS 5.1's cyrus-imapd-2.3.7-1.1.el5.rpm (derived from Simon
Matter's Invoca RPM). I would very much like to rsync the mail over and
(when ready) change the hostnames and have email clients not notice the
difference. In particular, if Eudora decides the mailbox layout has
changed, it will reysnc all mail and lose much of its status
information, so I want to avoid this if possible.
I'm getting hung up on a directory naming discrepancy I don't
understand. Any guidance on where the discrepancy originates will be
much appreciated.
On the Mac, pepper's top-level directory structure looks like this
(folders, not files):
> www:~ root# ls -d /var/spool/imap/user/pepper/*/
> /var/spool/imap/user/pepper/Deleted Messages/
> /var/spool/imap/user/pepper/Drafts/
> /var/spool/imap/user/pepper/JUNK.20061225/
> /var/spool/imap/user/pepper/Junk/
> /var/spool/imap/user/pepper/Sent Messages/
> /var/spool/imap/user/pepper/Sent/
> /var/spool/imap/user/pepper/Trash/
> /var/spool/imap/user/pepper/bulk/
> /var/spool/imap/user/pepper/company-archive/
> /var/spool/imap/user/pepper/company/
> /var/spool/imap/user/pepper/debevoise/
> /var/spool/imap/user/pepper/debevoise2/
> /var/spool/imap/user/pepper/frb/
> /var/spool/imap/user/pepper/goldman-sachs/
> /var/spool/imap/user/pepper/hh2005/
> /var/spool/imap/user/pepper/hh2006/
> /var/spool/imap/user/pepper/hh2007/
> /var/spool/imap/user/pepper/hts^20060328^txt/
> /var/spool/imap/user/pepper/info-mac/
> /var/spool/imap/user/pepper/keepers-archive/
> /var/spool/imap/user/pepper/list-archive/
> /var/spool/imap/user/pepper/list/
> /var/spool/imap/user/pepper/macworld-expo/
> /var/spool/imap/user/pepper/mail/
> /var/spool/imap/user/pepper/misc/
> /var/spool/imap/user/pepper/people-archive/
> /var/spool/imap/user/pepper/people/
> /var/spool/imap/user/pepper/pepper^development/
> /var/spool/imap/user/pepper/reppep/
> /var/spool/imap/user/pepper/reppep^com/
> /var/spool/imap/user/pepper/ru-archive/
> /var/spool/imap/user/pepper/ru/
> /var/spool/imap/user/pepper/scratch/
> /var/spool/imap/user/pepper/tidbits-archive/
> /var/spool/imap/user/pepper/tidbits/
> /var/spool/imap/user/pepper/writing/
On the Linux system, pepper's top-level directory structure looks like
this:
> [EMAIL PROTECTED] imap]# ls -l /var/spool/imap/user/pepper/
> total 40
> -rw--- 1 cyrus mail4 Jan 5 17:37 cyrus.cache
> -rw--- 1 cyrus mail 154 Jan 5 17:36 cyrus.header
> -rw--- 1 cyrus mail 96 Jan 5 17:37 cyrus.index
> drwx-- 2 cyrus mail 4096 Jan 5 17:36 Drafts
> drwx-- 2 cyrus mail 4096 Jan 5 17:36 hh2007
> drwx-- 2 cyrus mail 4096 Jan 5 17:36 INBOX^Drafts
> drwx-- 2 cyrus mail 4096 Jan 5 17:36 INBOX^Sent
> drwx-- 2 cyrus mail 4096 Jan 5 17:36 INBOX^Trash
> drwx-- 2 cyrus mail 4096 Jan 5 17:36 Sent
> drwx-- 2 cyrus mail 4096 Jan 5 17:36 Trash
I have Cyrus set to auto-create "Trash | Sent | Junk", and that's
working, but I don't understand why they are prefixed by "INBOX^". I
tried renaming the folders to simply 'Drafts', 'Sent', and 'Trash' and
reconstructing, and imapd re-created the INBOX^* files to match the
contents of mailboxes.db.
So what causes the discrepancy? I don't know if there's a configuration
setting I missed, or a compilation option that Apple changed, but I have
verified that both systems have "altnamespace: yes" and
"unixhierarchysep: yes".
Thanks,
Chris Pepper
PS-I'm sorry if this is a FAQ. I've read all the docs I could find under
<http://cyrusimap.web.cmu.edu/imapd/>, and Google doesn't ignores '^',
so I'm having no luck at tracking this down.
--
Chris Pepper:<http://www.reppep.com/~pepper/>
<http://www.extrapepperoni.com/>
The Rockefeller University: <http://www.rockefeller.edu/>
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Migration: INBOX^Trash & friends
> I have 'www', a Mac OS X 10.4 Server system, running Apple's build of > Cyrus, and want to move my mail over to 'pe', a CentOS 5.1 system running > CentOS 5.1's cyrus-imapd-2.3.7-1.1.el5.rpm (derived from Simon Matter's > Invoca RPM). I would very much like to rsync the mail over and (when ready) > change the hostnames and have email clients not notice the difference. In > particular, if Eudora decides the mailbox layout has changed, it will reysnc > all mail and lose much of its status information, so I want to avoid this if > possible. > > I'm getting hung up on a directory naming discrepancy I don't understand. > Any guidance on where the discrepancy originates will be much appreciated. > So what causes the discrepancy? I don't know if there's a configuration > setting I missed, or a compilation option that Apple changed, but I have > verified that both systems have "altnamespace: yes" and "unixhierarchysep: > yes". For the record, it wasn't Cyrus IMAPd at all. SquirrelMail (which I was using to check Cyrus functionality) was recreating the undesired folders whenever I accessed the account. I tweaked the defaults (including manually overriding the default Cyrus delimiter) and now all looks right. Sorry for the noise. Chris Pepper -- Chris Pepper:<http://www.reppep.com/~pepper/> <http://www.extrapepperoni.com/> The Rockefeller University: <http://www.rockefeller.edu/> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Plaintext only for loopback?
Hello,
I want to allow plaintext auth only for SquirrelMail (running on the
Cyrus IMAPd server), and require encrypted authentication over all
physical network connections. I see several options governing plaintext
auth in the documentation for imap.conf:
> allowplaintext: 1
> Allow the use of cleartext passwords on the wire.
> plaintextloginpause: 0
> Number of seconds to pause after a successful plaintext login. For systems
> that support strong authentication, this permits users to perceive a cost of
> using plaintext passwords. (This does not affect the use of PLAIN in SASL
> authentications.)
> plaintextloginalert:
> Message to send to client after a successful plaintext login.
In addition, my Invoca 2.3.7 RPM includes:
> allowplainwithouttls: 0
> Allow plain login mechanism without an encrypted connection.
So I'm left wondering: a) if there is a way to do this that I'm not
getting (perhaps "on the wire" is more subtle than my simplistic
reading), and b) if not, what's the best way to request/suggest this as
an enhancement. Should I just open a bug in Bugzilla, or is there a
better way?
Thanks,
Chris Pepper
--
Chris Pepper:<http://www.reppep.com/~pepper/>
<http://www.extrapepperoni.com/>
The Rockefeller University: <http://www.rockefeller.edu/>
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Plaintext only for loopback?
Jorey Bump wrote: > Chris Pepper wrote, at 01/13/2008 01:59 AM: > >> I want to allow plaintext auth only for SquirrelMail (running on >> the Cyrus IMAPd server), and require encrypted authentication over all >> physical network connections. > > Why do you want plaintext auth only for SquirrelMail? It supports TLS, > alternate ports, CRAM-MD5, and DIGEST-MD5. For example, My Squirrelmail > is set up to use LOGIN/TLS on port 993 (settings inherited from a > historical setup, I can also support the other options). Are you trying > to avoid the overhead of TLS? Arrgh! SquirrelMail offers plain, cram-md5, and digest-md5, and only plain appears to work against /etc/shadow. I don't want the overhead of running TLS over loopback, so I think I will have to do without forcing secure auth for non-SSL IMAP/POP, and use the firewall to prevent Internet users from connecting over the Internet w/o SSL (so I don't have to worry about them unwisely using PLAIN or LOGIN over plaintext connection). Pity. It would be nice to have the option of doing IMAP on the IMAP port without worrying about unencrypted plaintext auth. Thanks, Chris PS-Bron, I don't want to deal with multiple instances, and I don't need too, since I can firewall IMAP (non-SSL) and only let SquirrelMail connect to port 143. I'm not looking forward to the SpamAssassin/ClamAV sandwich on the SMTP side. -- Chris Pepper:<http://www.reppep.com/~pepper/> <http://www.extrapepperoni.com/> The Rockefeller University: <http://www.rockefeller.edu/> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
IMAP not seeing old mail present on filesystem
All, I have run a small mail service based on Cyrus IMAP for a few years. The (CentOS 5) server I've used for the past couple years failed last week. I brought up a new CentOS 5 system on a new Linux server, installed cyrus-imapd-2.3.7-7.el5_4.3, mounted the old /var disk (actually one of 2 mdadm submirrors), and copied /var/spool/imap over to the new /var FS. It's running CentOS' cyrus-imapd-2.3.7-7.el5_4.3 RPM. Unfortunately, for many users (all but me?), mail clients (at least Apple's Mail.app and SquirrelMail) don't show any messages from before the migration in INBOX. When I grope around in /var/spool/imap/user, I see the old message (with high numbers) and the new messages (starting a new sequence from 1). I know there are last-message counters in the cyrus.* files, so would resetting those have caused IMAP to start storing new messages from 1. and vanish the old ones? I don't know how those could have been cleared for at least 6 users simultaneously. Presumably it happened for all, and I just fixed it for myself somehow. More importantly, I don't know how to make the old messages accessible to my users via IMAP (I can give them the files, but that's quite awkward). chk_cyrus agrees with IMAP clients about message counts (very low). I have tried reconstruct with various combinations of "-rfx", and "quota -f", but not found any way to make it show the old messages. Any suggestions? Thanks, Chris Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
On 10/4/10 9:17 AM, Simon Matter wrote: >> Simon, >> >> I did recover /var/lib/imap (although a bit later, FWIW) and > > I think "a bit later" is your problem. But I think reconstruct should fix > this. > BTW, did you check the subscription status of mailboxes? Maybe the folders > are there but not subscribed, and that's why some users can't see them? Simon, No, users see the folders, just not old messages. For most (all?) INBOXes but my own, new messages started arriving as 1. and continued from there. Users can see the new mail, but not the old. This makes me think it's not an internal permissions problem, because they see the mailboxes and (some) mail in them. All file permissions I checked appear correct "reconstruct -rfx" doesn't help. Is there anything else to try? Thanks, Chris >> /etc/imapd.conf. I just now compared /etc/cyrus.conf and added squatter >> & adjusted prefork numbers -- the rest all matched. >> >> Is there other configuration I should check? Both systems are 64-bit >> CentOS 5, so the db4 installations should be fully compatible. > > If both the old and the new system are basically the same I don't think > there is anything missing. > > Simon > >> >> Thanks, >> >> Chris >> >> On 10/4/10 8:36 AM, Simon Matter wrote: All, I have run a small mail service based on Cyrus IMAP for a few years. The (CentOS 5) server I've used for the past couple years failed last week. I brought up a new CentOS 5 system on a new Linux server, installed cyrus-imapd-2.3.7-7.el5_4.3, mounted the old /var disk (actually one of 2 mdadm submirrors), and copied /var/spool/imap over to the new /var FS. It's running CentOS' cyrus-imapd-2.3.7-7.el5_4.3 RPM. >>> >>> Did you also recover /var/lib/imap from the old server and make sure the >>> configs are the same? >>> >>> Simon >>> Unfortunately, for many users (all but me?), mail clients (at least Apple's Mail.app and SquirrelMail) don't show any messages from before the migration in INBOX. When I grope around in /var/spool/imap/user, I see the old message (with high numbers) and the new messages (starting a new sequence from 1). I know there are last-message counters in the cyrus.* files, so would resetting those have caused IMAP to start storing new messages from 1. and vanish the old ones? I don't know how those could have been cleared for at least 6 users simultaneously. Presumably it happened for all, and I just fixed it for myself somehow. More importantly, I don't know how to make the old messages accessible to my users via IMAP (I can give them the files, but that's quite awkward). chk_cyrus agrees with IMAP clients about message counts (very low). I have tried reconstruct with various combinations of "-rfx", and "quota -f", but not found any way to make it show the old messages. Any suggestions? >> > > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
Simon, I did recover /var/lib/imap (although a bit later, FWIW) and /etc/imapd.conf. I just now compared /etc/cyrus.conf and added squatter & adjusted prefork numbers -- the rest all matched. Is there other configuration I should check? Both systems are 64-bit CentOS 5, so the db4 installations should be fully compatible. Thanks, Chris On 10/4/10 8:36 AM, Simon Matter wrote: >> All, >> >> I have run a small mail service based on Cyrus IMAP for a few >> years. The (CentOS 5) server I've used for the past couple years failed >> last week. I brought up a new CentOS 5 system on a new Linux server, >> installed cyrus-imapd-2.3.7-7.el5_4.3, mounted the old /var disk >> (actually one of 2 mdadm submirrors), and copied /var/spool/imap over to >> the new /var FS. It's running CentOS' cyrus-imapd-2.3.7-7.el5_4.3 RPM. > > Did you also recover /var/lib/imap from the old server and make sure the > configs are the same? > > Simon > >> >> Unfortunately, for many users (all but me?), mail clients (at least >> Apple's Mail.app and SquirrelMail) don't show any messages from before >> the migration in INBOX. When I grope around in /var/spool/imap/user, I >> see the old message (with high numbers) and the new messages (starting a >> new sequence from 1). >> >> I know there are last-message counters in the cyrus.* files, so >> would resetting those have caused IMAP to start storing new messages >> from 1. and vanish the old ones? I don't know how those could have been >> cleared for at least 6 users simultaneously. Presumably it happened for >> all, and I just fixed it for myself somehow. >> >> More importantly, I don't know how to make the old messages >> accessible to my users via IMAP (I can give them the files, but that's >> quite awkward). chk_cyrus agrees with IMAP clients about message counts >> (very low). I have tried reconstruct with various combinations of >> "-rfx", and "quota -f", but not found any way to make it show the old >> messages. >> >> Any suggestions? Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
On 10/4/10 1:12 AM, Patrick Goetz wrote: > On 10/3/2010 6:57 AM, Chris Pepper wrote: >> >>More importantly, I don't know how to make the old messages >> accessible to my users via IMAP (I can give them the files, but that's >> quite awkward). chk_cyrus agrees with IMAP clients about message counts >> (very low). I have tried reconstruct with various combinations of >> "-rfx", and "quota -f", but not found any way to make it show the old >> messages. >> >> Any suggestions? >> > > You probably need to run cyrreconstruct on each user mailbox. On my system it's /usr/lib/cyrus-imapd/reconstruct, and I have. No joy, alas. Chris Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
On 10/4/10 10:23 AM, Patrick Goetz wrote: > On 10/04/2010 08:37 AM, Chris Pepper wrote: >> >> No, users see the folders, just not old messages. For most (all?) >> INBOXes but my own, new messages started arriving as 1. and continued >> from there. Users can see the new mail, but not the old. This makes me >> think it's not an internal permissions problem, because they see the >> mailboxes and (some) mail in them. All file permissions I checked appear >> correct >> >> "reconstruct -rfx" doesn't help. Is there anything else to try? >> > > > I wasn't clear about whether the old install was completely gone or > could still be booted. If you can still start cyrus on the old server, > you could try imapsync to transfer mail to the new one. Old system is not bootable, unfortunately. FYI: I have 943 directories & 298,409 mail files, so manually fixing things isn't feasible. Thanks for all the suggestions! Chris -- Chris Pepper:<http://cbio.mskcc.org/> <http://www.extrapepperoni.com/> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
On 10/4/10 11:36 AM, Bron Gondwana wrote: > On Mon, Oct 04, 2010 at 11:17:59AM -0400, Chris Pepper wrote: >> On 10/4/10 10:23 AM, Patrick Goetz wrote: >>> I wasn't clear about whether the old install was completely gone or >>> could still be booted. If you can still start cyrus on the old server, >>> you could try imapsync to transfer mail to the new one. >> >> Old system is not bootable, unfortunately. >> >> FYI: I have 943 directories& 298,409 mail files, >> so manually fixing things isn't feasible. > ^^ > > Well, of course not. It there were 5 directories and 20 mail > files I wouldn't consider doing it manually - I'd write a script > to automate it and then sit back and drink coffee. Manually is > how you do the first one to find out how it's done. > > And then the second one to make sure the process you settled on > after stack of trial and error is repeatable. Maybe a third one > if you screwed up number 2. > > After that, you automate the process you've decided on and let > the rest happen automatically. It always amazes me to see > admins repeating themselves manually over and over for a > frequent task. > > I'd love to see the output of your reconstruct command (including > syslog with the logging level turned up) to see why it's not > finding the files. And maybe an 'ls -la' of one of the imap > directories with this issue as well. I'm away on a "team building" > exercise for the next couple of days Oslo time - but I can certainly > look at it afterwards. I've been travelling (from Australia) which > is why I haven't been looking in on this earlier... it's a strange > set of symptoms. > > If you have a small folder that you don't mind sharing, a tar.gz of > the entire folder contents (including the metadata files) would be > fantastic, because then I could check the contents of the .index > and .cache files as well - and maybe even try a reconstruct on a > testbed here. > > But file permissions are the most interesting - I'm wondering if > reconstruct is unable to read the directory correctly or unable > to read the old files. Bron, Unfortunately I don't know how to write a script to do this. The old system was CentOS 5/x64, just like the new system. I believe it had the same cyrus-imapd & db4 RPMs, as I had patched a few weeks ago, but I cannot confirm. I will send you a tarball of this directory directly. I will try "reconstruct -rfx" with debug logging enabled in syslog tonight. Thanks, Chris > [r...@inspector ~]# ls -ltr /var/spool/imap/user/julia/ > total 1756 > -rw--- 1 cyrus mail 2931 May 20 12:56 158. > -rw--- 1 cyrus mail 2052 Jun 21 08:33 159. > -rw--- 1 cyrus mail 1536 Aug 11 17:15 162. > -rw--- 1 cyrus mail 2221 Aug 13 08:15 163. > -rw--- 1 cyrus mail 413395 Aug 14 18:40 164. > -rw--- 1 cyrus mail 79756 Aug 21 13:57 165. > -rw--- 1 cyrus mail 296759 Aug 22 15:18 166. > -rw--- 1 cyrus mail 156064 Sep 4 09:43 167. > -rw--- 1 cyrus mail 75003 Sep 4 10:06 168. > -rw--- 1 cyrus mail 164035 Sep 6 09:51 169. > -rw--- 1 cyrus mail 156853 Sep 7 12:24 170. > -rw--- 1 cyrus mail 340068 Sep 16 17:44 171. > drwx-- 2 cyrus mail 4096 Oct 1 21:31 Junk > drwx-- 2 cyrus mail 4096 Oct 2 23:40 Drafts > drwx-- 2 cyrus mail 4096 Oct 2 23:40 Sent > drwx-- 2 cyrus mail 4096 Oct 2 23:40 Apple Mail To Do > drwx-- 2 cyrus mail 4096 Oct 2 23:40 Deleted Messages > -rw--- 1 cyrus mail 1148 Oct 2 23:44 1. > drwx-- 2 cyrus mail 4096 Oct 3 00:09 Trash > drwx-- 2 cyrus mail 4096 Oct 3 13:16 Sent Messages > -rw--- 1 cyrus mail179 Oct 3 13:39 cyrus.header > -rw--- 1 cyrus mail176 Oct 4 09:04 cyrus.index > -rw--- 1 cyrus mail640 Oct 4 09:04 cyrus.cache -- Chris Pepper:<http://cbio.mskcc.org/> <http://www.extrapepperoni.com/> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: IMAP not seeing old mail present on filesystem
Thanks, all, for the generous help. Bron asked about output from reconstruct, which never provided any. It turns out that I was using the wrong delimiters (., per chk_cyrus output, rather than /), and reconstruct wasn't even trying. It looks like all the missing mail is accessible again. Thanks again! Chris -- Chris Pepper:<http://cbio.mskcc.org/> <http://www.extrapepperoni.com/> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Odd problem: IMAP/S suddenly not working, but no errors, and IMAP still works
mail.reppep.com (CentOS 5) is running cyrus-imapd-2.3.7-7.el5_4.3, along with SquirrelMail, postfix, etc. Last night, I noticed that when I sent mail from Thunderbird, it was not able to file copies in the Sent mailbox, although they did reach the recipients, so postfix was accepting mail on 587/tcp. I restarted Cyrus IMAPd but don't see any error messages in /var/log/maillog, and the cert & key look fine. SquirrelMail is fine using plain IMAP. I opened 143/tcp in the firewall, and am able to fetch mail via IMAP with STARTTLS, so it looks like the cert and key are fine. But "telnet mail.reppep.com 993" and openssl fail to get any response. Port 993 is open to the Internet, FWIW. Does anyone have any suggestions for what went wrong and/or how to fix? I'll try tcpdump next to see if it's responding at all. Alternatively, is there a way to make sure Cyrus requires STARTTLS on 143? I was blocking external access to it to make sure users always use encryption to connect, but port 143 with STARTTLS required would be an acceptable alternative. Thanks, Chris Pepper > pep...@imp:~$ !openssl > openssl s_client -connect www.reppep.com:993 > CONNECTED(0003) > 4284:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake > failure:/SourceCache/OpenSSL098/OpenSSL098-32/src/ssl/s23_lib.c:188: > [r...@inspector ~]# cat /etc/imapd.conf > admins: cyrus > altnamespace: yes > configdirectory: /var/lib/imap > duplicatesuppression: yes > hashimapspool: no > partition-default: /var/spool/imap > servername: mail.reppep.com > singleinstancestore: yes > #syslog_prefix: cyrus > unixhierarchysep: yes > > lmtp_downcase_rcpt: yes > maxmessagesize: 20971520 > sendmail: /usr/sbin/sendmail > #quotawarn: 80 > > #allowplaintext: yes > #allowplainwithouttls: yes > sasl_pwcheck_method: saslauthd > #imap_auth_login: yes > #imap_auth_cram_md5: yes > #imap_auth_plain: yes > > autocreateinboxfolders: Junk > autocreatequota: -1 > #autocreate_sieve_script: /etc/junk.sieve > autocreate_sieve_compiledscript: /etc/sieve.bc > autosievefolders: Junk > autosubscribeinboxfolders: Junk > createonpost: yes > #sievedir: /var/lib/imap/sieve > sieveusehomedir: true > > tls_ca_file: /etc/pki/tls/certs/mail.reppep.com.20100115.crt > tls_cert_file: /etc/pki/tls/certs/mail.reppep.com.20100115.crt > tls_key_file: /etc/pki/tls/private/mail.reppep.com.20080219.key > tls_cipher_list: SSLv3:TLSv1:!NULL:!EXPORT:!DES:!LOW:@STRENGTH > [r...@inspector ~]# ls -l /etc/pki/tls/certs/mail.reppep.com.20100115.crt > /etc/pki/tls/private/mail.reppep.com.20080219.key > -rw-r--r-- 1 root root 6466 Oct 1 17:13 > /etc/pki/tls/certs/mail.reppep.com.20100115.crt > -rw-r- 1 root mail 497 Feb 19 2008 > /etc/pki/tls/private/mail.reppep.com.20080219.key > [r...@inspector ~]# netstat -an|grep LIST|grep tcp|sort -n > tcp0 0 0.0.0.0:110 0.0.0.0:* > LISTEN > tcp0 0 0.0.0.0:111 0.0.0.0:* > LISTEN > tcp0 0 0.0.0.0:139 0.0.0.0:* > LISTEN > tcp0 0 0.0.0.0:143 0.0.0.0:* > LISTEN > tcp0 0 0.0.0.0:20000.0.0.0:* > LISTEN > tcp0 0 0.0.0.0:25 0.0.0.0:* > LISTEN > tcp0 0 0.0.0.0:33060.0.0.0:* > LISTEN > tcp0 0 0.0.0.0:445 0.0.0.0:* > LISTEN > tcp0 0 0.0.0.0:587 0.0.0.0:* > LISTEN > tcp0 0 0.0.0.0:993 0.0.0.0:* > LISTEN > tcp0 0 0.0.0.0:995 0.0.0.0:* > LISTEN > tcp0 0 10.0.104.200:53 0.0.0.0:* > LISTEN > tcp0 0 :::110 :::* > LISTEN > tcp0 0 127.0.0.1:10024 0.0.0.0:* > LISTEN > tcp0 0 127.0.0.1:10025 0.0.0.0:* > LISTEN > tcp0 0 127.0.0.1:530.0.0.0:* > LISTEN > tcp0 0 127.0.0.1:953 0.0.0.0:* > LISTEN > tcp0 0 :::143 :::* > LISTEN > tcp0 0 ::1:953 :::* > LISTEN > tcp0 0 :::2000 :::* > LISTEN > tcp0 0 :::22 :::*
Re: Odd problem: IMAP/S suddenly not working, but no errors, and IMAP still works
Bron, My Cyrus is from RPM, and I am just nursing it along until my users finish migrating off and FastMail manages to complete my own migration, so I don't want to build from source. Why would IMAP/S block on empty /dev/random, while IMAP+STARTTLS works? FWIW, SASL2 seems to use urandom. > [r...@inspector random]# strings /usr/lib/libsasl* |grep random > /dev/urandom > /dev/urandom But my /dev/random does seem quite low. Still surfing and looking for a good way to fill it on a mostly headless server -- I haven't found a good solution yet. Chris > [r...@inspector ~]# ls -l /dev/*random > crw-rw-rw- 1 root root 1, 8 Oct 31 02:05 /dev/random > cr--r--r-- 1 root root 1, 9 Oct 31 02:05 /dev/urandom > [r...@inspector ~]# cd /proc/sys/kernel/random > [r...@inspector random]# more *|cat > :: > boot_id > :: > d3724e19-7462-4224-960b-49d5d3a18d7a > :: > entropy_avail > :: > 17 > :: > poolsize > :: > 4096 > :: > read_wakeup_threshold > :: > 64 > :: > uuid > :: > a3ed2323-e04d-4034-a72a-76b5d4b697f7 > :: > write_wakeup_threshold > :: > 128 On 10/31/10 9:26 PM, Bron Gondwana wrote: > Sounds like your /dev/random is empty. You can compile with /dev/urandom or > add a source of entropy... > > "Chris Pepper" wrote: > >> mail.reppep.com (CentOS 5) is running cyrus-imapd-2.3.7-7.el5_4.3, >> along with SquirrelMail, postfix, etc. Last night, I noticed that when I >> sent mail from Thunderbird, it was not able to file copies in the Sent >> mailbox, although they did reach the recipients, so postfix was >> accepting mail on 587/tcp. >> >> I restarted Cyrus IMAPd but don't see any error messages in >> /var/log/maillog, and the cert& key look fine. SquirrelMail is fine >> using plain IMAP. I opened 143/tcp in the firewall, and am able to fetch >> mail via IMAP with STARTTLS, so it looks like the cert and key are fine. >> >> But "telnet mail.reppep.com 993" and openssl fail to get any response. >> Port 993 is open to the Internet, FWIW. >> >> Does anyone have any suggestions for what went wrong and/or how to fix? >> I'll try tcpdump next to see if it's responding at all. >> >> Alternatively, is there a way to make sure Cyrus requires STARTTLS on >> 143? I was blocking external access to it to make sure users always use >> encryption to connect, but port 143 with STARTTLS required would be an >> acceptable alternative. >> >> Thanks, >> >> Chris Pepper >> >>> pep...@imp:~$ !openssl >>> openssl s_client -connect www.reppep.com:993 >>> CONNECTED(0003) >>> 4284:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake >>> failure:/SourceCache/OpenSSL098/OpenSSL098-32/src/ssl/s23_lib.c:188: >> >> >>> [r...@inspector ~]# cat /etc/imapd.conf >>> admins: cyrus >>> altnamespace: yes >>> configdirectory: /var/lib/imap >>> duplicatesuppression: yes >>> hashimapspool: no >>> partition-default: /var/spool/imap >>> servername: mail.reppep.com >>> singleinstancestore: yes >>> #syslog_prefix: cyrus >>> unixhierarchysep: yes >>> >>> lmtp_downcase_rcpt: yes >>> maxmessagesize: 20971520 >>> sendmail: /usr/sbin/sendmail >>> #quotawarn: 80 >>> >>> #allowplaintext: yes >>> #allowplainwithouttls: yes >>> sasl_pwcheck_method: saslauthd >>> #imap_auth_login: yes >>> #imap_auth_cram_md5: yes >>> #imap_auth_plain: yes >>> >>> autocreateinboxfolders: Junk >>> autocreatequota: -1 >>> #autocreate_sieve_script: /etc/junk.sieve >>> autocreate_sieve_compiledscript: /etc/sieve.bc >>> autosievefolders: Junk >>> autosubscribeinboxfolders: Junk >>> createonpost: yes >>> #sievedir: /var/lib/imap/sieve >>> sieveusehomedir: true >>> >>> tls_ca_file: /etc/pki/tls/certs/mail.reppep.com.20100115.crt >>> tls_cert_file: /etc/pki/tls/certs/mail.reppep.com.20100115.crt >>> tls_key_file: /etc/pki/tls/private/mail.reppep.com.20080219.key >>> tls_cipher_list: SSLv3:TLSv1:!NULL:!EXPORT:!DES:!LOW:@STRENGTH >>> [r...@inspector ~]# ls -l /etc/pki/tls/certs/mail.reppep.com.20100115.crt >>> /etc/pki/tls/private/mail.reppep.com.20080219.key >>> -rw-r--r-- 1 root root 6466 Oct 1 17:13 >>> /etc/pki/tls/certs/mail.reppep.com.20100115.crt
Re: Odd problem: IMAP/S suddenly not working, but no errors, and IMAP still works
On 11/1/10 10:46 AM, Simon Matter wrote: >> Bron, >> >> My Cyrus is from RPM, and I am just nursing it along until my users >> finish migrating off and FastMail manages to complete my own migration, >> so I don't want to build from source. Why would IMAP/S block on empty >> /dev/random, while IMAP+STARTTLS works? FWIW, SASL2 seems to use urandom. > > If this is really stock CentOS 5 then I think everything Cyrus related > should use /dev/urandom and not /dev/random. But, could it be that other > software you installed uses /dev/random and makes it "empty"? Most things are CentOS RPMs (thanks for those! ;), with a few from RPMforge. > [r...@inspector ~]# rpm -q cyrus-imapd amavisd-new clamav spamassassin > postfix httpd mod_ssl > cyrus-imapd-2.3.7-7.el5_4.3 > amavisd-new-2.6.4-3.el5.rf > clamav-0.96.4-1.el5.rf > spamassassin-3.3.1-3.el5.rf > postfix-2.3.3-2.1.el5_2 > httpd-2.2.3-43.el5.centos.3 > mod_ssl-2.2.3-43.el5.centos.3 Which still leaves me thinking my port 993 problem isn't entropy, because STARTTLS works fine. Chris >>> [r...@inspector random]# strings /usr/lib/libsasl* |grep random >>> /dev/urandom >>> /dev/urandom >> >> >> But my /dev/random does seem quite low. Still surfing and looking for a >> good way to fill it on a mostly headless server -- I haven't found a >> good solution yet. >> >> Chris >> >>> [r...@inspector ~]# ls -l /dev/*random >>> crw-rw-rw- 1 root root 1, 8 Oct 31 02:05 /dev/random >>> cr--r--r-- 1 root root 1, 9 Oct 31 02:05 /dev/urandom >>> [r...@inspector ~]# cd /proc/sys/kernel/random >>> [r...@inspector random]# more *|cat >>> :: >>> boot_id >>> :: >>> d3724e19-7462-4224-960b-49d5d3a18d7a >>> :: >>> entropy_avail >>> :: >>> 17 >>> :: >>> poolsize >>> :: >>> 4096 >>> :: >>> read_wakeup_threshold >>> :::::: >>> 64 >>> :: >>> uuid >>> :: >>> a3ed2323-e04d-4034-a72a-76b5d4b697f7 >>> :: >>> write_wakeup_threshold >>> :: >>> 128 >> >> >> On 10/31/10 9:26 PM, Bron Gondwana wrote: >>> Sounds like your /dev/random is empty. You can compile with /dev/urandom >>> or add a source of entropy... >>> >>> "Chris Pepper" wrote: >>> >>>>mail.reppep.com (CentOS 5) is running cyrus-imapd-2.3.7-7.el5_4.3, >>>> along with SquirrelMail, postfix, etc. Last night, I noticed that when >>>> I >>>> sent mail from Thunderbird, it was not able to file copies in the Sent >>>> mailbox, although they did reach the recipients, so postfix was >>>> accepting mail on 587/tcp. >>>> >>>>I restarted Cyrus IMAPd but don't see any error messages in >>>> /var/log/maillog, and the cert& key look fine. SquirrelMail is fine >>>> using plain IMAP. I opened 143/tcp in the firewall, and am able to >>>> fetch >>>> mail via IMAP with STARTTLS, so it looks like the cert and key are >>>> fine. >>>> >>>>But "telnet mail.reppep.com 993" and openssl fail to get any response. >>>> Port 993 is open to the Internet, FWIW. >>>> >>>>Does anyone have any suggestions for what went wrong and/or how to >>>> fix? >>>> I'll try tcpdump next to see if it's responding at all. >>>> >>>>Alternatively, is there a way to make sure Cyrus requires STARTTLS on >>>> 143? I was blocking external access to it to make sure users always use >>>> encryption to connect, but port 143 with STARTTLS required would be an >>>> acceptable alternative. >>>> >>>> Thanks, >>>> >>>> Chris Pepper >>>> >>>>> pep...@imp:~$ !openssl >>>>> openssl s_client -connect www.reppep.com:993 >>>>> CONNECTED(0003) >>>>> 4284:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake >>>>> failure:/SourceCache/OpenSSL098/OpenSSL098-32/src/ssl/s23_lib.c:188: >>>> >>>> >>>>> [r...@inspector ~]# cat /etc/imapd.conf >>>>> admins: cyrus >>>>> altnamespace: yes >>>>> configdirectory: /var/lib/imap >>>>> duplicatesuppression: yes >>
Re: Odd problem: IMAP/S suddenly not working, but no errors, and IMAP still works
On 11/1/10 10:41 AM, Dan White wrote: > On 31/10/10 20:51 -0400, Chris Pepper wrote: >> Alternatively, is there a way to make sure Cyrus requires STARTTLS on >> 143? I was blocking external access to it to make sure users always use >> encryption to connect, but port 143 with STARTTLS required would be an >> acceptable alternative. > > You can set 'allowplaintext: 0' to disallow plaintext logins over port 143. > That would require clients to perform a STARTTLS, or negotiate a SASL > security layer which meets your 'sasl_minimum_layer:' setting. Excellent, thanks! > allowplaintext: 0 I am leaving sasl_minimum_layer at default for now. LOGINDISABLED before STARTTLS is encouraging, but I don't know why "Authentication failed. generic failure" *after* STARTTLS. On the other hand, with "allowplaintext: 0" and after restarting cyrus-imapd, I can still get mail, so I suspect this is exactly what I wanted. Thanks, Chris > [r...@inspector ~]# imtest -u pepper -t "" localhost > S: * OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS LOGINDISABLED > AUTH=DIGEST-MD5 AUTH=CRAM-MD5 SASL-IR] mail.reppep.com Cyrus IMAP4 > v2.3.7-Invoca-RPM-2.3.7-7.el5_4.3 server ready > C: C01 CAPABILITY > S: * CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS LOGINDISABLED > AUTH=DIGEST-MD5 AUTH=CRAM-MD5 SASL-IR ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS > NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT > SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE > CONDSTORE IDLE LISTEXT LIST-SUBSCRIBED X-NETSCAPE URLAUTH > S: C01 OK Completed > C: S01 STARTTLS > S: S01 OK Begin TLS negotiation now > verify error:num=19:self signed certificate in certificate chain > TLS connection established: TLSv1 with cipher AES256-SHA (256/256 bits) > C: C01 CAPABILITY > S: * CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID AUTH=PLAIN AUTH=DIGEST-MD5 > AUTH=CRAM-MD5 AUTH=LOGIN SASL-IR ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS > NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT > SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE > CONDSTORE IDLE LISTEXT LIST-SUBSCRIBED X-NETSCAPE URLAUTH > S: C01 OK Completed > Please enter your password: > C: A01 AUTHENTICATE PLAIN > S: A01 NO authentication failure > Authentication failed. generic failure > Security strength factor: 256 -- Chris Pepper:<http://cbio.mskcc.org/> <http://www.extrapepperoni.com/> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Odd problem: IMAP/S suddenly not working, but no errors, and IMAP still works
On 11/1/10 11:21 AM, Simon Matter wrote: >> On 11/1/10 10:46 AM, Simon Matter wrote: >>>> Bron, >>>> >>>>My Cyrus is from RPM, and I am just nursing it along until my users >>>> finish migrating off and FastMail manages to complete my own migration, >>>> so I don't want to build from source. Why would IMAP/S block on empty >>>> /dev/random, while IMAP+STARTTLS works? FWIW, SASL2 seems to use >>>> urandom. >>> >>> If this is really stock CentOS 5 then I think everything Cyrus related >>> should use /dev/urandom and not /dev/random. But, could it be that other >>> software you installed uses /dev/random and makes it "empty"? >> >> Most things are CentOS RPMs (thanks for those! ;), with a few from >> RPMforge. >> >>> [r...@inspector ~]# rpm -q cyrus-imapd amavisd-new clamav spamassassin >>> postfix httpd mod_ssl >>> cyrus-imapd-2.3.7-7.el5_4.3 >>> amavisd-new-2.6.4-3.el5.rf >>> clamav-0.96.4-1.el5.rf >>> spamassassin-3.3.1-3.el5.rf >>> postfix-2.3.3-2.1.el5_2 >>> httpd-2.2.3-43.el5.centos.3 >>> mod_ssl-2.2.3-43.el5.centos.3 >> >> Which still leaves me thinking my port 993 problem isn't entropy, >> because >> STARTTLS works fine. > > That's my impression from the beginning, because lack of entropy has not > been a known problem on the RHEL/CentOS configs. That's not much help of > course. > > If you already restarted master and you know it's not stuck somehow, then > the only thing I could think to check is your > /var/lib/imap/tls_sessions.db database. I don't know if a broken TLS db > could result in what you see but better check it out. Interesting. I moved tls_sessions.db aside & restarted IMAPd, and it's apparently in a new format -- perhaps the default format has changed since it was first created. But 993 is still open but not responsive. I am going to try disabling Cyrus' IMAP/SSL and swapping in stunnel, as Rob @ FastMail has suggested as a workaround. Thanks, Chris > [r...@inspector imap]# ls -l tls* > -rw--- 1 cyrus mail 8192 Nov 1 11:27 tls_sessions.db > -rw--- 1 cyrus mail 1976 Nov 1 11:27 tls_sessions.db.BAD > [r...@inspector imap]# file tls* > tls_sessions.db: Berkeley DB (Btree, version 9, native byte-order) > tls_sessions.db.BAD: Cyrus skiplist DB -- Chris Pepper:<http://cbio.mskcc.org/> <http://www.extrapepperoni.com/> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Odd problem: IMAP/S suddenly not working, but no errors, and IMAP still works
On 11/1/10 7:26 PM, Bron Gondwana wrote: > On Sun, Oct 31, 2010 at 10:40:13PM -0400, Chris Pepper wrote: >> Bron, >> >> My Cyrus is from RPM, and I am just nursing it along until my users >> finish migrating off and FastMail manages to complete my own >> migration, so I don't want to build from source. Why would IMAP/S >> block on empty /dev/random, while IMAP+STARTTLS works? FWIW, SASL2 >> seems to use urandom. > > I really don't know to be honest - we don't run any ssl enabled imapds, > we do all the ssl in nginx on the frontend. It sounds like Rob's > workaround might be all you need though :) Neither do I. I decided to re-enable pop3 (which I don't use or allow, and had recently commented out) in cyrus.conf and restarted cyrus-imapd, and IMAP/SSL is working again! I commented it out and restarted Cyrus, and port 993 is still working. I'd say I just needed to restart the daemon, except I rebooted Saturday night after port 993 stopped working, so I don't know what's up. One interesting & odd data point: after "service cyrus-imapd stop", I still had a couple active connections to an imap daemon which was listening on port 993. I killed the process, but again that couldn't have persisted across the reboot I performed 1d19h ago. Bizarre! Thanks for everyone's suggestions. Chris Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
