Re: SASL re-entrancy crisis (was: OpenLDAP 2.0.x + pam_ldap +cyrus-imapd-2.0.x)
note there are (or were) limits on the size of usernames and passwords that pwcheck can deal with. David Lang On Fri, 10 Aug 2001, Jeremy Howard wrote: > Date: Fri, 10 Aug 2001 05:59:54 +1000 > From: Jeremy Howard <[EMAIL PROTECTED]> > To: [EMAIL PROTECTED], Marco Colombo <[EMAIL PROTECTED]> > Cc: [EMAIL PROTECTED] > Subject: Re: SASL re-entrancy crisis (was: OpenLDAP 2.0.x + pam_ldap + > cyrus-imapd-2.0.x) > > Devdas Bhagat wrote: > > The problem with the current design of imapd is that it assumes that > > SASL will be available locally in some form, ignoring that it may not > > be available there. > > Do the pwcheck daemons provide support for this? > > Yes. The pwcheck 'API' is this simple: > - SASL sends username\0password\0 over a socket > - The daemon sends back 'OK\0' or 'Incorrect password\0' > > How the daemon decides on what response to return is completely open. For > instance, my pwcheck daemon contacts a MySQL server on a remote machine to > check the credentials. > >
Re: turning off AUTH=CRAM-MD5
Also if you have already installed SASL you will need to go to the directory it gets installed into and delete the authentications libraries that you don't want to use. it doesn't matter what you disable at compile time. if there is something in the directory the cyrus (and other SASL enabled stuff) will insist on trying to use it. David Lang On Mon, 20 Aug 2001, Amos Gouaux wrote: > Date: Mon, 20 Aug 2001 00:33:16 -0500 > From: Amos Gouaux <[EMAIL PROTECTED]> > To: [EMAIL PROTECTED] > Subject: Re: turning off AUTH=CRAM-MD5 > > >>>>> On Sun, 19 Aug 2001 21:51:33 -0700, > >>>>> David Wright <[EMAIL PROTECTED]> (dw) writes: > > dw> Cyrus-imapd (1.6.24) insists on advertising AUTH=CRAM-MD5, even > dw> though this is a lie. This is (I think) one of the (many bad) > dw> side-effects of SASL -- because of SASL cyrus advertises this AUTH, > dw> but in fact my sasldb is utterly empty (all authentication is via > dw> PAM) and so any client that takes cyrus up on the offer gets told > dw> the user doesn't exist. > > dw> So... how can I get cyrus to stop advertising AUTH=CRAM-MD5? > > Configure cyrus-sasl accordingly. Use the various --disable-* > options to configure. See --help for details. > > -- > Amos >
Re: limit of file descriptors
linux 2.0 and 2.2 have a FD limit ~512, this can be bumped up to 4092 with a source code edit, but cannot be pushed above that. 2.4 defaults to a much larger number (based on ram I think, on my 512M machines it's 8K) and can be bumped up to 32K or 64K (don't remember which at the moment) in a boot script David Lang On Thu, 6 Sep 2001, Jeremy Howard wrote: > Date: Thu, 6 Sep 2001 22:08:50 +1000 > From: Jeremy Howard <[EMAIL PROTECTED]> > To: Lawrence Greenfield <[EMAIL PROTECTED]>, > Horst Lederhaas <[EMAIL PROTECTED]>, > [EMAIL PROTECTED] > Subject: Re: limit of file descriptors > > Lawrence Greenfield wrote: > >From: "Jeremy Howard" <[EMAIL PROTECTED]> > >Date: Sat, 25 Aug 2001 08:15:42 +1000 > > > >Lawrence Greenfield wrote: > >> This message is usually harmless. > >> > >> Some systems limit how many file descriptors a process can use, and > >> the 'master' process tries bumping it up to be infinite. If it > fails, > >> it usually means that there's no default limit. > >> > >I too get this message, under Linux kernel 2.4.8. But I'm pretty sure > that > >Linux has an FD limit (1024 FDs according to `ulimit -a`). Do I have to > do > >something special to let Cyrus increase FDs under Linux? > > > > As long as root invokes master, there shouldn't be anything else. > > > Strange... I am on linux kernel 2.2.19 and root is invoking master. But I'm > still getting this error. I'm running 2.0.16. > > It's no big deal yet because I'm not hitting the limit, but I'm curious > now... What else could be causing the problem? How should I go about > debugging this one? > >
Re: Cyrus and very large folders
I was running cyrus as my company mailserver for a while, I saw things start to slowdown when there were more then ~7K messages in one folder (and start to be significant when it got to more then ~20K messages/folder). This was on linux 2.0.x on a pentium 200 with 64MB ram serving ~200 users. it's a problem, but it's far less of a problem then attempting to parse a unix mail file to get the message you need, that starts to slow down significantly at <1000 messages (on a much faster linux box) David Lang On Mon, 22 Oct 2001, Amos Gouaux wrote: > Date: Mon, 22 Oct 2001 09:03:16 -0500 > From: Amos Gouaux <[EMAIL PROTECTED]> > To: [EMAIL PROTECTED] > Subject: Re: Cyrus and very large folders > > >>>>> On Sun, 21 Oct 2001 23:24:30 -0700, > >>>>> Jurgen Botz <[EMAIL PROTECTED]> (jb) writes: > > jb> At one point in the past I used Netscape Messaging Server (now iPlanet) > jb> and it had this problem at versions less than 4.x. With a few hundred > jb> users, many of whom had mailboxes with a few thousand messages in them, > jb> opening a mailbox was painfully slow. The problem is that normal Unix > > Well, my inbox currently has 3568 messages in it and PINE pops it > open in a jiffy. You need to keep in mind that Cyrus caches things > like the headers. See the four "cyrus.*" files in each folder. > > In fact, I typically use the auto-expire capabilities in Gnus > (news/mail reader for Emacs/XEmacs) and rarely ever manually delete > a message. I could not do this if Cyrus didn't handle large folders > well. > > jb> Has anyone who uses Cyrus in a large organization environment found > jb> this to be a problem? > > How do you define "large"? ;-) I think if you spread your message > store across spindles, you should be okay. > > > -- > Amos >
Re: Unknown user problem.
you do this type of username mapping in sendmail (or equivalent) not in cyrus. in sendmail look at the virtusertable feture and with it you can set the type of mapping you are describing. now you may be able to do something like this in sieve, I haven't looked, but other then that there is no way to do the user mapping at this layer. David Lang On Sun, 24 Feb 2002, Chris Gilbert wrote: > Hi, > > I've just setup a system running cyrus for my own use (it's installed and > seems to be running fine 8). > > However I've got a problem with unknown users. Having come from picking up > mail via POP and I'm now switching to imap, I could could create new accounts > on the fly, as they all turned up in the same POP box on my isp's server. > Basically anything to @paradox.demon.co.uk would get to me, so I took > advantage of this. > > The problem is I can't see an easy way to allow unknown user mails to turn up > in a mailbox somewhere, rather than get bounced back to the sender. Does > anyone know of a way to do this in sendmail or cyrus? > > Note I've only just joined the list, but I did check the archives and > couldn't see anything relevant (most posts to do with unknown users were to > do with making sure the mails were bounced) > > Thanks, > Chris > > >
Re: Too many users with Cyrus IMAP
what you have run into is a limit in the ext2/3 fs on the max number of
directory entries you can have.
there are patches out there for cyrus to create a second tier of
directories rather then having all mailboxes in the user directory you
have user/a user/b user/c etc (or in your case /1 /2 /3 etc) to avoid not
only these problems, but also the problem that ext2/3 does sequential
seaches through the directory so with this many entries you will already
be very slow.
also take a look at reiserfs and XFS as possible candidates for you to use
for your mailboxes, both of them have very different structures that are
designed to handle the large numbers of directories problem better.
David Lang
On Sun, 24 Feb 2002, Andres Maduro wrote:
> Date: Sun, 24 Feb 2002 23:16:06 -0800
> From: Andres Maduro <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Subject: Too many users with Cyrus IMAP
>
> Hi,
>
> I have installed Cyrus 2.0.16 on Red Hat 7.2 with the standard SASL
> cyrus-sasl-1.5. I have been able to use it successfully and create perl web
> administration tools for managing mailboxes.
>
> I am currently doing a stress test, I need to be able to handle more than
> 100 thousand users on this server. I modified Cyrus Imap code so it can
> accept numeric accounts which I need as I am creating emails for a cellular
> company ([EMAIL PROTECTED]). I made a perl script to create 10
> accounts. I am using ext3 filesystem under linux. After the user number
> 31948 was created, no more accounts where created. Examining the imapd.log,
> I found that it was complaining about "too many links error", see below
> extract from imapd.log:
>
> Feb 24 22:45:54 mail imapd[22212]: abort_txn: aborting txn 2147683085
> Feb 24 22:45:54 mail imapd[22212]: myfetch: starting txn 2147683086
> Feb 24 22:45:54 mail imapd[22212]: myfetch: reusing txn 2147683086
> Feb 24 22:45:54 mail imapd[22212]: mystore: reusing txn 2147683086
> Feb 24 22:45:54 mail imapd[22212]: IOERROR: creating directory
> /var/spool/imap/user/0132123: Too many links
>
> Any help is greatly appreciated.
>
> It would be nice if we could split /var/spool/imap/user on several
> partitions, is this possible ? What options do I have ?
>
> Following I will show several configurations files I am using:
>
> /etc/imapd.conf -
> configdirectory: /var/imap
> partition-default: /var/spool/imap
> admins: cyrus root apache andres
> sasl_pwcheck_method: sasldb
> #sasl_auto_transition: yes
> sendmail: /usr/sbin/sendmail
> --
>
> /etc/cyrus.conf -
> # standard standalone server implementation
>
> START {
> # do not delete these entries!
> mboxlist cmd="ctl_mboxlist -r"
> deliver cmd="ctl_deliver -r"
>
> # this is only necessary if using idled for IMAP IDLE
> # idledcmd="idled"
> }
>
> # UNIX sockets start with a slash and are put into /var/imap/socket
> SERVICES {
> # add or remove based on preferences
> imap cmd="imapd" listen="imap" prefork=0
> imaps cmd="imapd -s" listen="imaps" prefork=0
> pop3 cmd="pop3d" listen="pop3" prefork=0
> pop3s cmd="pop3d -s" listen="pop3s" prefork=0
> sieve cmd="timsieved" listen="sieve" prefork=0
>
> # at least one LMTP is required for delivery
> # lmtp cmd="lmtpd" listen="lmtp" prefork=0
> lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0
> }
>
> EVENTS {
> # this is required
> checkpointcmd="ctl_mboxlist -c" period=30
>
> # this is only necessary if using duplicate delivery suppression
> delprune cmd="ctl_deliver -E 3" period=1440
> }
>
>
> Best regards,
> Andres Maduro
>
Re: Too many users with Cyrus IMAP
On Mon, 25 Feb 2002, Andres Maduro wrote: > When I installed Cyrus Imap I follow the installation instructions by the > book. Does any one know if you have to set the partition directories to > update synchronouslysome when using Ext3 or ReiserFS ? Since both of these are journaling file systems I think you will be able to get away without needing to do the sync trick for the mailboxes themselves. if you need to do anything you may want to make the journal syncronous to avoid the possibility that you accept the mail and crash before the journal gets written to disk. David Lang
Re: Cyrus and IMP
what hardware do you use to support this load? David Lang On Sun, 24 Mar 2002, Nick Ustinov wrote: > Date: Sun, 24 Mar 2002 11:52:32 +0200 > From: Nick Ustinov <[EMAIL PROTECTED]> > To: Jonas Jacobsson <[EMAIL PROTECTED]> > Cc: [EMAIL PROTECTED] > Subject: Re: Cyrus and IMP > > We are running cyrus 2.1.0 with imp 3.0 in a production environment. The > system has about 150,000 user accounts and over 600,000 cyrus mailboxes. > Apache server load is 50-100 reqs/second. Everything works excellent, > basically I don't even look after it -- it just works :) > > Nick > > > > Hi all, > > > > I'm a rather new Linux user and I have just started up > > my own server. The machine is right now running Debian 2.2 (potato), > > Exim, courier-imap and imp 2.2. > > > > My question is if anyone else on this list is running > > IMP (pref. 3.0) with Cyrus 1.5.19 or above? > > > > My goal is to upgrade to the much improved IMP version 3.0 > > and it depends on that other programs be upgraded first. Potato > > includes a version of Cyrus that is reported to work with IMP 3. > > So by switching to Cyrus would mean one program less to upgrade > > manually. > > > > Thanks in advance for any tips or tricks. > > > > /jonas, Sweden. > > > > > > > > > > Sincerely, > Nick > > > > --- > This message contains no viruses. > Guaranteed by Kaspersky Anti-Virus. > www.antivirus.lv >
RE: Connecting to imap using Outlook
D. you tell outlook that you have an IMAP server that you want it to connect to and it works (at least it works as well as outlook ever works ;-) you will have to look in your outlook documentation for where the option is to tell it where your mail servers are. David Lang On 26 Mar 2002, Chris Picton wrote: > Date: 26 Mar 2002 15:57:13 +0200 > From: Chris Picton <[EMAIL PROTECTED]> > To: OCNS Consulting <[EMAIL PROTECTED]> > Cc: [EMAIL PROTECTED] > Subject: RE: Connecting to imap using Outlook > > On Tue, 2002-03-26 at 15:47, OCNS Consulting wrote: > > Chris: > > > > Did you search the List Archive? I think there have been discussions on > > this topic. > > I have searched google a lot, but not found anything. Thats why I came > to the list. Maybe I'm using the wrong search terms on google :( > > Chris > > > > > > RB > > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED]]On Behalf Of Chris Picton > > Sent: Tuesday, March 26, 2002 8:33 AM > > To: Chris Picton > > Cc: [EMAIL PROTECTED] > > Subject: Re: Connecting to imap using Outlook > > > > > > I am curious. Does nobody reply because: > > > > a) Nobody wants to be associated with a Microsoft product? > > b) It can't be done? > > c) It can be done but nobody knows how? > > d) It can be done, somebody knows, but is feeling lazy? > > > > :) > > > > Cheers > > Chris > > > > On Wed, 2002-03-20 at 12:32, Chris Picton wrote: > > > Hi > > > > > > I have set up a cyrus-imapd server on redhat 7.2 > > > I have the following versions: > > > cyrus-sasl-md5-1.5.24-23 > > > cyrus-sasl-1.5.24-23 > > > cyrus-sasl-plain-1.5.24-23 > > > cyrus-imapd-2.0.16-5rm > > > cyrus-imapd-utils-2.0.16-5rm > > > > > > Everything is working fine from evolution (CRAM-MD5/DIGEST-MD5/PLAIN and > > > ssl). However, I can't use secure password authentication from > > > outlook. I get the following error: > > > > > > Your 'Inbox' folder was not polled for its unread count. CRAM-MD5 > > > authentication failed. None of the authentication methods supported by > > > your IMAP server (if any) are supported on this computer. Account: > > > 'biology.wits.ac.za', Server: 'biology.wits.ac.za', Protocol: IMAP, > > > Server Response: '', Port: 143, Secure(SSL): No, Error Number: > > > 0x800CCCDF > > > > > > I get the same error if I include the realm in the username or not. > > > > > > My logs say: > > > Mar 20 12:33:16 biology master[7517]: about to exec /usr/cyrus/bin/imapd > > > Mar 20 12:33:16 biology service-imap[7517]: executed > > > Mar 20 12:33:16 biology imapd[7517]: accepted connection > > > Mar 20 12:33:16 biology master[7025]: process 7517 exited, status 0 > > > > > > > > > Does anybody know what the problem is, and how to fix it? > > > > > > Regards > > > -- > > > Chris Picton > > > Tangent Systems > > > [EMAIL PROTECTED] > > > > > > > > > __ > > -- > > Chris Picton > > Tangent Systems > > [EMAIL PROTECTED] > > > > > > __ > -- > Chris Picton > Tangent Systems > [EMAIL PROTECTED] > > > __ >
Re: Connecting to imap using Outlook
have you attempted to configure SASL to just do plain passwords, it's likly that outlook can't do anythign more sophisticated. David Lang On 26 Mar 2002, Chris Picton wrote: > Date: 26 Mar 2002 15:32:44 +0200 > From: Chris Picton <[EMAIL PROTECTED]> > To: Chris Picton <[EMAIL PROTECTED]> > Cc: [EMAIL PROTECTED] > Subject: Re: Connecting to imap using Outlook > > I am curious. Does nobody reply because: > > a) Nobody wants to be associated with a Microsoft product? > b) It can't be done? > c) It can be done but nobody knows how? > d) It can be done, somebody knows, but is feeling lazy? > > :) > > Cheers > Chris > > On Wed, 2002-03-20 at 12:32, Chris Picton wrote: > > Hi > > > > I have set up a cyrus-imapd server on redhat 7.2 > > I have the following versions: > > cyrus-sasl-md5-1.5.24-23 > > cyrus-sasl-1.5.24-23 > > cyrus-sasl-plain-1.5.24-23 > > cyrus-imapd-2.0.16-5rm > > cyrus-imapd-utils-2.0.16-5rm > > > > Everything is working fine from evolution (CRAM-MD5/DIGEST-MD5/PLAIN and > > ssl). However, I can't use secure password authentication from > > outlook. I get the following error: > > > > Your 'Inbox' folder was not polled for its unread count. CRAM-MD5 > > authentication failed. None of the authentication methods supported by > > your IMAP server (if any) are supported on this computer. Account: > > 'biology.wits.ac.za', Server: 'biology.wits.ac.za', Protocol: IMAP, > > Server Response: '', Port: 143, Secure(SSL): No, Error Number: > > 0x800CCCDF > > > > I get the same error if I include the realm in the username or not. > > > > My logs say: > > Mar 20 12:33:16 biology master[7517]: about to exec /usr/cyrus/bin/imapd > > Mar 20 12:33:16 biology service-imap[7517]: executed > > Mar 20 12:33:16 biology imapd[7517]: accepted connection > > Mar 20 12:33:16 biology master[7025]: process 7517 exited, status 0 > > > > > > Does anybody know what the problem is, and how to fix it? > > > > Regards > > -- > > Chris Picton > > Tangent Systems > > [EMAIL PROTECTED] > > > > > > __ > -- > Chris Picton > Tangent Systems > [EMAIL PROTECTED] > > > __ >
Re: Connecting to imap using Outlook
the IMAP support in outlook is rather primitive (or was the last time I had to make it work) so I wouldn't be surprised if you are just stuck with LOGIN. sorry I can't help more David Lang On 26 Mar 2002, Chris Picton wrote: > I have sasl set up to do LOGIN PLAIN DIGEST-MD5 and CRAM-MD5 > > I have tested DIGEST-MD5, CRAM-MD5 and PLAIN using Evolution. Outlook > works with LOGIN, and attempts CRAM-MD5 for 'Secure Password > Authentication', but fails. > > Cheers > Chris > > On Tue, 2002-03-26 at 16:19, David Lang wrote: > > have you attempted to configure SASL to just do plain passwords, it's > > likly that outlook can't do anythign more sophisticated. > > > > David Lang > > > > On 26 Mar 2002, Chris Picton wrote: > > > > > Date: 26 Mar 2002 15:32:44 +0200 > > > From: Chris Picton <[EMAIL PROTECTED]> > > > To: Chris Picton <[EMAIL PROTECTED]> > > > Cc: [EMAIL PROTECTED] > > > Subject: Re: Connecting to imap using Outlook > > > > > > I am curious. Does nobody reply because: > > > > > > a) Nobody wants to be associated with a Microsoft product? > > > b) It can't be done? > > > c) It can be done but nobody knows how? > > > d) It can be done, somebody knows, but is feeling lazy? > > > > > > :) > > > > > > Cheers > > > Chris > > > > > > On Wed, 2002-03-20 at 12:32, Chris Picton wrote: > > > > Hi > > > > > > > > I have set up a cyrus-imapd server on redhat 7.2 > > > > I have the following versions: > > > > cyrus-sasl-md5-1.5.24-23 > > > > cyrus-sasl-1.5.24-23 > > > > cyrus-sasl-plain-1.5.24-23 > > > > cyrus-imapd-2.0.16-5rm > > > > cyrus-imapd-utils-2.0.16-5rm > > > > > > > > Everything is working fine from evolution (CRAM-MD5/DIGEST-MD5/PLAIN and > > > > ssl). However, I can't use secure password authentication from > > > > outlook. I get the following error: > > > > > > > > Your 'Inbox' folder was not polled for its unread count. CRAM-MD5 > > > > authentication failed. None of the authentication methods supported by > > > > your IMAP server (if any) are supported on this computer. Account: > > > > 'biology.wits.ac.za', Server: 'biology.wits.ac.za', Protocol: IMAP, > > > > Server Response: '', Port: 143, Secure(SSL): No, Error Number: > > > > 0x800CCCDF > > > > > > > > I get the same error if I include the realm in the username or not. > > > > > > > > My logs say: > > > > Mar 20 12:33:16 biology master[7517]: about to exec /usr/cyrus/bin/imapd > > > > Mar 20 12:33:16 biology service-imap[7517]: executed > > > > Mar 20 12:33:16 biology imapd[7517]: accepted connection > > > > Mar 20 12:33:16 biology master[7025]: process 7517 exited, status 0 > > > > > > > > > > > > Does anybody know what the problem is, and how to fix it? > > > > > > > > Regards > > > > -- > > > > Chris Picton > > > > Tangent Systems > > > > [EMAIL PROTECTED] > > > > > > > > > > > > __ > > > -- > > > Chris Picton > > > Tangent Systems > > > [EMAIL PROTECTED] > > > > > > > > > __ > > > > -- > Chris Picton > Tangent Systems > [EMAIL PROTECTED] > > > __ >
RE: Connecting to imap using Outlook
1. get a cert that is valid (otherwise you are vunerable to man-in-the-middle attacks anyway, and it's a bad idea to get users used to ignoring security warnings) 2. if they can disable SSL can't they disable 'secure passwords' and cause it to revert to plain logins anyway? David Lang On 26 Mar 2002, Chris Picton wrote: > Date: 26 Mar 2002 17:08:52 +0200 > From: Chris Picton <[EMAIL PROTECTED]> > To: Clifford Thurber <[EMAIL PROTECTED]> > Cc: T Churchward <[EMAIL PROTECTED]>, > [EMAIL PROTECTED] > Subject: RE: Connecting to imap using Outlook > > On Tue, 2002-03-26 at 16:48, Clifford Thurber wrote: > > But as long as you enable TLS/SSL I don't see why this would matter? Am I > > missing something here? > > Thanks > > > > At 02:35 PM 3/26/2002 +, T Churchward wrote: > > >correctly the only way I could get Outlook to successfully > > >connect was using plain text passwords . Yeah, I agree, not an ideal > > >solution! > > Because a luser would find that if they disable SSL, they don't get an > extra popup saying that the certificate can't be validated. So they > would disable SSL to get rid of the popup. They probably don't care > much about password security, but I do > > Also, I would like a server that works for all clients :) > > -- > Chris Picton > Tangent Systems > [EMAIL PROTECTED] > > > __ >
Re: removing banners from cyrus
as far as I've seen eliminating version banners causes more problems becouse it makes it harder for the sysadmins to check what version is running (you can try to keep records, but we all know that records don't always agree with reality) so you end up being more likly to be running a bad version then if you could check. David Lang On 2 Apr 2002, Jim Levie wrote: > Date: 02 Apr 2002 13:59:18 -0600 > From: Jim Levie <[EMAIL PROTECTED]> > To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> > Subject: Re: removing banners from cyrus > > On Tue, 2002-04-02 at 13:26, Ken Murchison wrote: > > > > > > Clifford Thurber wrote: > > > > > > Ken I am just interested in suppresing platform/version information when > > > someone telnet to port 143. Just one more layer of security. > > > > But by doing this, you're implying that there is a security hole in the > > Cyrus server which can be exploited if the hacker discovers the > > vendor/version info. Is there some known security hole in Cyrus that > > isn't in other servers. Even if there is, I don't think that the lack > > of info in the banner is going to stop a hacker from trying the exploit > > anyway. Furthermore, a good hacker intent on finding Cyrus servers > > could also detect them by look for known response strings from commands, > > etc. > > > Ah yes, the old "security through obscurity" game. From what I've seen > eliminating the server type and version has no affect on whether a > cracker can exploit any weakness that an application has. And that's > because the vast majority of attacks aren't done in what one would > consider an intelligent manner. The attacker doesn't examine services to > figure out if they are vulnerable or not. He/she simply runs a script > that attempts to exploit all known vulnerabilities. So hiding the fact > that you are running a certain version of Sendmail, or Cyrus, or > whatever doesn't generally help. The attack scripts that I've recovered > from cracked boxes (that were then used to try to crack other boxes) > just had a big list of things to try. > -- > =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ > Jim Levie email: > [EMAIL PROTECTED] > Dynetics Inc, Huntsville, Al Ph.256.964.4337 > The opinions expressed above are just that... >
can I use sieve for this?
I am looking at implementing a read-only (as far as the users are concerned, messages get posted through other means) web based message system by useing a modified webmail client and cyrus (via LMTP from the message generator server). Idealy I would like to set something up on the cyrus server that would send out a 'your got a message' mail when a message is put into a mailbox, but then not send another message until the person logs in. I can watch syslog to find when they login, is there any way to use sieve to detect a new message? (I seem to remember something about sieve haveing a unix domain socket it could work with for output-only stuff, but don't see anything about it on the sieve page). am I making a mistake in thinking of sieve at all? should I just implement this with a syslog watcher that looks for the lmtp delivery and the imap login instead? if I can't do the one message until they check it I need to at least be able to throttle the messages to one per (whatever time period). David Lang
Re: imapd timeout
the cost of forking can vary greatly depending on the OS. David Lang On Tue, 21 May 2002, Lawrence Greenfield wrote: > Date: Tue, 21 May 2002 22:38:43 -0400 > From: Lawrence Greenfield <[EMAIL PROTECTED]> > To: David Wright <[EMAIL PROTECTED]> > Cc: Cyrus-Info <[EMAIL PROTECTED]> > Subject: Re: imapd timeout > >Date: Tue, 21 May 2002 19:32:44 -0700 >From: David Wright <[EMAIL PROTECTED]> >Cc: Cyrus-Info <[EMAIL PROTECTED]> > >> Cyrus does recycle processes. Unix forking is amazingly slow compared >> to not forking and on servers that receive many connections a second >> this performance tweak is vital. > >That explains it; thanks for the explanation. > >(Still, even 10 forks/second seems entirely do-able. While I don't >dispute the principle, I'd think you'd need to get closer to 100 >forks/second before forking bottlenecks would become as important as >disk I/O bottlenecks.) > > Unfortunately, experience doesn't agree with your estimate. > > Larry > >
Re: multiple ssl certificates (for one service)
not all browsers accept *.domain certs so be careful the problem with different certs is that SSL hands out the cert as soon as the connection is established, before the sender tells you anything. TLS has an option to have the client tell the server what it's trying to connect to so that the server can hand back the proper cert, but this has almost no support currently and is the part of the TLS spec that isn't compatable with SSL. David Lang On Wed, 25 Sep 2002, twk wrote: > Date: Wed, 25 Sep 2002 09:45:50 -0400 > From: twk <[EMAIL PROTECTED]> > Cc: [EMAIL PROTECTED] > Subject: Re: multiple ssl certificates (for one service) > > > > Samuel Hug wrote: > > > Hi, > > > > is there a possibility to use more than one server certificate? The > > problem is that the mailserver has multiple domain names. The domain > > names don't specify different services as pop or imap, therefore > > tls_pop3 and tls_imap... wouldn't help me. > > > > Anybody got a hint? > > > > > Thawte has wild card certificates...so you can get a cert for *.moritzi.ch and > the cert is recognized for all servers whose domain name ends in ".moritzi.ch". > > If the domains are completely different, I don't know what you can do. > > Cheers, > Tom > > > > > -- > Tom Karchesemail : [EMAIL PROTECTED] > Web Systems Administrator phone : 919.515.5508 > NCSU Information Technology >
Re: squatter running longer than 24 hours
On Mon, 22 Oct 2007, Rob Mueller wrote: >> squatter would really benefit from incremental updates. At the moment a >> single new message in a mailbox containing 20k messages causes it to read >> in all the existing messages in order to regenerate the index. > > We spoke to Ken about this ages back, and even offered to pay for the work > to make it happen, but it was just around the time CMU hired him, so it > never actually happened pity. It would be nice to be able to dedicate a > couple of weeks to rummage around in there and actually make it happen... postgres has full-text search capabilities at acceptable performance on very large databases, their code is BSD so anything relavent coudl be merged into cyrus. it may be worth someone looking into their logic. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
On Tue, 20 Nov 2007, Ian G Batten wrote: > On 20 Nov 07, at 1332, Michael R. Gettes wrote: > >> I am wondering about the use of fsync() on journal'd file systems >> as described below. Shouldn't there be much less use of (or very >> little use) of fsync() on these types of systems? Let the journal >> layer due its job and not force it within cyrus? This would likely >> save a lot of system overhead. > > fsync() forces the data to be queued to the disk. A journaling > filesystem won't usually make any difference, because no one wants to > keep an intent log of every 1 byte write, or the 100 overwrites of > the same block. If you want every write() to go to disk, > immediately, the filesystem layout doesn't really matter: it's just a > matter of disk bandwidth. Journalling filesystems are more usually > concerned with metadata consistency, so that the filesystem isn't > actively corrupt if the music stops at the wrong point in a directory > create or something. however a fsync on a journaled filesystem just means the data needs to be written to the journal, it doesn't mean that the journal needs to be flushed to disk. on ext3 if you have data=journaled then your data is in the journal as well and all that the system needs to do on a fsync is to write things to the journal (a nice sequential write), and everything is perfectly safe. if you have data=ordered (the default for most journaled filesystems) then your data isn't safe when the journal is written and two writes must happen on a fsync (one for the data, one for the metadata) for cyrus you should have the same sort of requirements that you would have for a database server, including the fact that without a battery-backed disk cache (or solid state drive) to handle your updates, you end up being throttled by your disk rotation rate (you can only do a single fsync write per rotation, and that good only if you don't have to seek), RAID 5/6 arrays are even worse, as almost all systems will require a read of the entire stripe before writing a single block (and it's parity block) back out, and since the stripe is frequently larger then the OS readahead, the OS throws much of the data away immediatly. if we can identify the files that are the bottlenecks it would be very interesting to see the result of puttng them on a solid-state drive. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
On Wed, 21 Nov 2007, Ian G Batten wrote: >> however a fsync on a journal ed filesystem just means the data needs to be >> written to the journal, it doesn't mean that the journal needs to be >> flushed to >> disk. >> >> on ext3 if you have data=journal ed then your data is in the journal as well >> and >> all that the system needs to do on a fsync is to write things to the >> journal (a >> nice sequential write), > > Assuming the journal is on a distinct device and the distinct device can take > the load. It isn't on ZFS, although work is in progress. I was responding to the comments about ext3 and other journal ed filesystems as alternatives to zfs and the claim that doing a fsync on one of them required flushing the entire journal. sorry if I wasn't clear enough about this. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
On Thu, 22 Nov 2007, Gabor Gombas wrote: > On Tue, Nov 20, 2007 at 09:56:37AM -0800, David Lang wrote: > >> for cyrus you should have the same sort of requirements that you would have >> for >> a database server, including the fact that without a battery-backed disk >> cache >> (or solid state drive) to handle your updates, you end up being throttled by >> your disk rotation rate (you can only do a single fsync write per rotation, >> and >> that good only if you don't have to seek), RAID 5/6 arrays are even worse, as >> almost all systems will require a read of the entire stripe before writing a >> single block (and it's parity block) back out, and since the stripe is >> frequently larger then the OS readahead, the OS throws much of the data away >> immediatly. > > You're mixing things up. Readahead has absolutely zero influence on when > data is evicted from the cache. if the system is set to do a 1M readahead and to do that readahead it needs to read in 5M of data to verify the integrity, the system doesn't keep all 5M of data in it's cache, only the 1M that is it's readahead (or at least in some cases this is true) David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be lockingissues
On Tue, 4 Mar 2008, Ian G Batten wrote: > software RAID5 is a performance > disaster area at the best of times unless it can take advantage of > intimate knowledge of the intent log in the filesystem (RAID-Z does > this), actually, unless you have top-notch hardware raid controllers, software raid 5 may be better then hardware raid 5. many controllers only do a decent job doing raid 0 or raid 1. this is something to measure with your particular hardware. I've seen many cases where the cards do a horrible job with raid 5 compared to software. > and three-disk RAID5 assemblages are a performance disaster > area irrespective of hardware in a failure scenario. The rebuild > will involve taking 50% of the IO bandwidth of the two remaining > disks in order to saturate the new target; rebuild performance --- > contrary to intuition --- improves with larger assemblages as you can > saturate the replacement disk with less and less of the bandwidth of > the surviving spindles. this is true, up to the point where the bus gets saturated with the re-sync info, after that more disks will not improve the rebuild time. > For a terabyte, 3x500GB SATA drives in a RAID5 group will be blown > out of the water by 4x500GB SATA drives in a RAID 0+1 configuration > in terms of performance and (especially) latency, especially if it > can do the Solaris trick of not faulting an entire RAID 0 sub-group > if one spindle fails. Rebuild still isn't pretty, mind you. either of these cases will survive a single drive failure, what I would look at is either 3x1TB drives in raid 1, or 4x500G drives in raid 6 to get the ability to survive 2 drives failing. it takes long enough to rebuild an array with large drives that the chances of a second drive failing during the rebuild become noticable. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Miserable performance of cyrus-imapd 2.3.9 -- seems to be lockingissues
disks to 3. ZFS is not available everywhere, and it is not suitable for all workloads (specificly database type workloads, which is a fair approximation for cyrus) you say it's not worth reducing 4 disks to 3, but what about 6 disks to 4? (useing your example of a machine with 4 SATA drives it's the difference between useing the machine you have or buying a new one) if that's not enough, what about 8 disks to 5? (6 if you do raid 6 or want a hot-spare) what is the point that you would consider the difference valid? David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus infrastructure performance less than expected
On Tue, 29 Apr 2008, Eric Déchaux wrote: Pascal Gienger a écrit : Eric Déchaux <[EMAIL PROTECTED]> wrote: The older infrastructure can stand the 42 000 concurrent sessions, the new one can't : I was expecting each frontend to be able to handle 5 500 concurrent sessions but they are not. Around 3 000 / 3 500 concurrent sessions the frontends begin to SWAP and are not more able keep up the load. Did I undertand correctly: You are virtualizing each component and your frontends begin to swap in their virtual environment? Is there any reason why you don't assign more RAM to them? Does your frontend virtual machines run a 64 bit kernel? That's it. The frontend begin to swap, not the ESX. We have scaled the ESX RAM to avoid SWAPING on the ESX level. In fact we can't assign anymore memory as the X4600 can't handle more than 64 Gb. Another point is we have doubled total RAM compared to the actual platfrom and I don't understand why it behaves so badly. were you virtualized before? adding virtualization causes a fairly significant overhead. David Lang We abandoned all Linux for our Cyrus Servers and switched to Solaris 10 with Zoning and ZFS. We have less concurrent users than you but more storage (10 T at the moment). I would have loved to put Solaris, Zones and Massaging Server here but it was not a possibility. Custorme chose was VMware + Linux + Cyrus. Many thanks. -- Eric Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus vs Dovecot
On Wed, 13 Aug 2008, Wesley Craig wrote: > On 13 Aug 2008, at 10:31, kbajwa wrote: >> I think you are missing a point which is most important, i.e., what >> type of >> support Cyrus vs Dovecot offers. In my experience: >> >> Cyrus = 0 >> Dovecot= 100 > > As someone who answers many help requests for cyrus (and I'm very far > from the only one), I can honestly say I've never seen a requests > from you. Perhaps you've had a lot of occasion to ask for help with > Dovecot. I'm happy to hear you've gotten that help. Community is a > lot of what open source software is about. As for your experience > with the cyrus imapd community, perhaps your sample size is too small. > > Or perhaps you're thinking of paid support? Because I know very well > that you can get that for cyrus imap. can you provide links to where from? David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: suggestion need to design an email system.
On Wed, 17 Sep 2008, Wesley Craig wrote: > On 17 Sep 2008, at 11:40, Jens Hoffrichter wrote: >> Why does cyrus need it's own >> structure for the mailboxes, which is similar, but not wholly >> compatible, to maildir. Maildir and cyrus both suffer from the same >> disadvantages (huge needs in terms of inodes etc.), yet I see no >> distinctive advantage for the cyrus mailbox format to maildir. > > Performance. I could go on & on, but that's the answer, basically. actually, I suspect that Cyrus existed before Maildir and since Cyrus is a 'black box' server there's no advantage to moving to maildir. Also Cyrus has cache, index, and flags stored seperatly from the message itself, which means that when these things are changed the message file itself doesn't need to change. doign a quick google check on maildir it also appears that maildir is not as standard as people think it is, it's defined almost entirely by the implementation (DJB started it, but never worked to turn it into a standard for others to use) David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: suggestion need to design an email system.
On Wed, 17 Sep 2008, Scott M. Likens wrote: > With ZFS your leaving a "ton" of stone-age worries behind. You can go > much beyond inodes in the perks of ZFS. and gaining some new worries along the way. while some are convinced that ZFS is the best thing ever others see it as trading a set of known problems for a set of unknown problems (plus it severly limits what OS you can run, which can bring it's own set of problems along) David Lang > Vincent Fox wrote: >> Wesley Craig wrote: >> >>>> Maildir and cyrus both suffer from the same >>>> disadvantages (huge needs in terms of inodes etc.), >>>> >> With ZFS, inodes are among the many stone-age worries you leave behind. >> >> ;-) >> >> >> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html >> >> >> !DSPAM:48d1d451236501804284693! >> >> >> > > > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: suggestion need to design an email system.
On Wed, 17 Sep 2008, Scott M. Likens wrote: > Hi David, > > If you want ZFS you have several choices, OS X (Leopard), as there is a > development version that supports RAID-Z and works quite well. You > can't boot off of it, but that will change in Snow Leopard. > > Additionally you can use Opensolaris such as Nexenta, or you can use > fuse to get ZFS into Linux. fuse on linux is not a real option for most uses due to the performance hit. so you have OS X and Solaris (with opensolaris varients), that's not a very wide supproted base for running servers. > Truthfully I would rather use Nexenta to get ZFS which at least gives > you a decent working system... Unlike a standard Solaris where you are > forced to get either Sun One (Forte?) or an ancient version of gcc off > of sunfreeware and start building. :( no comment > However I do look forward to the day when Sun dual licenses ZFS as GPL > and sticks it in the Linux kernel and throws us all a wrench, for good > or bad ZFS introduces some excellent overlooked ideas that it's about > damned time someone introduced. I don't disagree that ZFS is a prod to get development on filesystems moving again or to say that they didn't introduce any good ideas, but I do disagree with people who seem to think that ZFS is perfect, and is so much better then any other filesystem that the availablity of ZFS should be the only consideration on what OS to run. David Lang > Scott > > David Lang wrote: >> On Wed, 17 Sep 2008, Scott M. Likens wrote: >> >>> With ZFS your leaving a "ton" of stone-age worries behind. You can go >>> much beyond inodes in the perks of ZFS. >> >> and gaining some new worries along the way. while some are convinced >> that ZFS is the best thing ever others see it as trading a set of >> known problems for a set of unknown problems (plus it severly limits >> what OS you can run, which can bring it's own set of problems along) >> >> David Lang >> >>> Vincent Fox wrote: >>>> Wesley Craig wrote: >>>> >>>>>> Maildir and cyrus both suffer from the same >>>>>> disadvantages (huge needs in terms of inodes etc.), >>>>>> >>>> With ZFS, inodes are among the many stone-age worries you leave behind. >>>> >>>> ;-) >>>> >>>> >>>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >>>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >>>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/ >>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki >>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html >>> > > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: suggestion need to design an email system.
On Thu, 18 Sep 2008, Adam Tauno Williams wrote: > On Wed, 2008-09-17 at 21:12 -0700, David Lang wrote: >> On Wed, 17 Sep 2008, Wesley Craig wrote: >>> On 17 Sep 2008, at 11:40, Jens Hoffrichter wrote: >>>> Why does cyrus need it's own >>>> structure for the mailboxes, which is similar, but not wholly >>>> compatible, to maildir. Maildir and cyrus both suffer from the same >>>> disadvantages (huge needs in terms of inodes etc.), yet I see no >>>> distinctive advantage for the cyrus mailbox format to maildir. >>> Performance. I could go on & on, but that's the answer, basically. >> actually, I suspect that Cyrus existed before Maildir and since Cyrus is a >> 'black box' server there's no advantage to moving to maildir. >> Also Cyrus has cache, index, and flags stored seperatly from the message >> itself, >> which means that when these things are changed the message file itself >> doesn't >> need to change. > > Yep, which is why the sealed mailstore is a good idea: meta-data. You > can keep reliable meta-data if third parties can go in and muck about. one of those "can" should be "can't" :-) David Lang >> doign a quick google check on maildir it also appears that maildir is not as >> standard as people think it is, it's defined almost entirely by the >> implementation (DJB started it, but never worked to turn it into a standard >> for >> others to use) > > > > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: choosing a file system
On Wed, 31 Dec 2008, Adam Tauno Williams wrote: > On Wed, 2008-12-31 at 11:47 +0100, LALOT Dominique wrote: >> Thanks for everybody. That was an interesting thread. Nobody seems to >> use a NetApp appliance, may be due to NFS architecture problems. > > Personally, I'd never use NFS for anything. Over the years I've had way > to many NFS related problems on other things to ever want to try it > again. NFS has some very interesting capabilities and limitations. it's really bad for multiple processes writing to the same file (the cyrus* files for example) and for atomic actions (writing the message files for example) there are ways that you can configure it that will work, but unless you already have a big NFS server you are probably much better off using a mechanism that makes the drives look more like local drives (SAN, iSCSI, etc) or try one of the cluster filesystems that has different tradeoffs than NFS does >> I believe I'll look to ext4 that seemed to be available in last >> kernel, and also to Solaris, but we are not enough to support another >> OS. > > We've used Cyrus on XFS for almost a years, no problems. > > In regards to ext3 I'd pay attention to the vintage of problem reports > and performance issues; ext3 of several years ago is not the ext3 of > today, many improvements have been made. "data=writeback" mode can help > performance quite a bit, as well as enabling "dir_index" if it isn't > already (did it ever become the default?). The periodic fsck can also > be disabled via tune2fs. I only point this out since, if you already > have any ext3 setup, trying the above are all painless and might buy > you something. it's definantly worth testing different filesystems. I last did a test about two years ago and confirmed XFS as my choice. I have one instance of cyrus still running on ext3 and I definantly see it as a user in the performance. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: choosing a file system
On Sat, 3 Jan 2009, Rob Mueller wrote: >> But the new Solid-State-Disks seem very promising. They are claimed to >> give 30x the throughput of a 15k rpm disk. If IO improves by 30 times >> that should make all these optimizations unnecessary. >> As my boss used to tell me ... Good hardware always compensates for >> not-so-good software. > > What we've found is that the meta-data (eg mailbox.db, seen db's, quota > files, cyrus.* files) use WAY more IO than the email data, but only use > 1/20th the space. > > By separating the meta data onto RAID1 10k/15k RPM drives, and the email > data onto RAID5/6 7.2k RPM drives, you can get a good balance of > space/speed. how do you move the cyrus* files onto other drives? David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Security risk of POP3 & IMAP protocols
On Fri, 13 Feb 2009, Ian Batten wrote: > On 13 Feb 09, at 0149, Joseph Brennan wrote: >> >> The protocol itself is no less secure than POP. > > Security isn't about protocols, it's about systems, and I suspect POP3 > vs IMAP is metonymic for local vs remote mail storage. > > I can see an argument that says that one problem with IMAP is that > your entire mail store, which is much more interesting to an attacker > than a message in flight or your current mail pending collection a la > POP3, is under someone else's control. So if, say, you use a whole > disk encryption product, mail delivered via traditional POP3 will be > wrapped in the arms of the encryption immediately after collection, > while mail stored on a remote server and accessed via IMAP will have > whatever security features the server has. > > If you control the IMAP server (for some suitable value of `you') then > a risk assessment is the same task in both scenarios. However, if, as > is common in many situations, the IMAP server isn't within the scope > of a risk assessment, then I can imagine that your 27001 life is a > little easier if you don't have a large pool of potentially sensitive > data under someone else's (for some value of `someone else') > control. Data at rest is a different class of problem to data in > motion, and IMAP implies a _lot_ of data at rest. > > To make this more concrete, imagine you're an HR department within a > large enterprise, handling job applications, CVs, disciplinary > processes, dismissals, etc. You need to demonstrate your compliance > with your local data protection regulations. The theft of a day's > email would be severely embarrassing, but is analogous to the theft of > a day's postal mail: a risk which most businesses would accept. It > would expose limited amounts of information about a small subset of > your employees. > > However, the theft of a year's or a decade's email would expose > substantial information about a large percentage of your employees, > and would be analogous to allowing a few filing cabinets to be stolen. > > Your email system is run by your corporation's IT function in another > jurisdiction which has laxer data protection laws --- say, an EU > company whose head office is in the USA. > > Do you (a) store all your long term records in the other jurisdiction > or (b) store them locally? > > Now I'm not defending the argument, and indeed here we have ~4TB of > email on our Cyrus servers. But I don't think the position is > entirely without merit, and having gone through the simplifying and > distorting mirror of sales droids I can see where it's come from... the flip side of the complience issue is that it's a LOT easier to control retention policies (including backups) on a central server than on everybody's individual desktops/laptops. as for the concerns about laxer data security in other juristictions, that's something that needs to be addressed when you outsource your mail (via contract with whoever you are having host your mail for you) David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: List to Spam Harvest
On Fri, 27 Feb 2009, John Thomas wrote: > I know little, so please forgive if this is wrong. > The following link seems to be crawled by Google and exposes our email > addresses to spam harvesters. I wonder if it makes sense and is > possible to not do this or obfuscate the addresses? > http://cyrusimap.web.cmu.edu/archive/mailbox.php?mailbox=archive.info-cyrus if you send mail to a public mailing list it can be harvested by spammers. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Does Cyrus benefit greatly from increased FS buffer cache?
On Thu, 16 Apr 2009, Sebastian Hagedorn wrote: --On 16. April 2009 10:58:15 +1000 Rob Mueller wrote: http://blog.fastmail.fm/2007/09/21/reiserfs-bugs-32-bit-vs-64-bit-kernel s-cache-vs-inode-memory/ Anyone have any specific thoughts? Is there any other benefit we might see from large memory allocation in 64-bit architecture? Given that I wrote that blog post, I can only tell you that in our environment, 64-bit kernels made a big difference. I wonder if ext3 behaves differently, Red Hat's 32-bit behaves differently, or if something altogether different is going on. We are currently running RHEL 3 in 32-bit mode, our servers have 16 GB, and most of it is used for caching: # free total used free sharedbuffers cached Mem: 16214344 16197612 16732 0 86944 13415172 -/+ buffers/cache:2695496 13518848 Swap: 4192944 84364184508 So it would seem that a 64-bit kernel wouldn't improve on that, right? Or is that a difference between 2.4 and 2.6? 64 bit kernels will be significantly more efficiant, and more reliable with that much memory. in addition, in 64 bit mode the system is able to use twice as many registers on the CPU, which can frequently be a significant win in and of itself (even on machines with 1G of ram) I've run both kernels on the same system and always have found that the 64 bit kernel is an advantage. I tend to do the same thing for userspace (unless I am running something that doesn't work with 64 bit userspace), but there the benifit is more hit-and-miss David Lang ATT1922831.dat Description: ATT1922831.dat Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
server-side signature verification through IMAP?
the attached messages were posted to the mulberry mailing list short version, in order to do s/mime verification the client must retreive the entire message to do the verification client-side. is there any way to do this server-side? David Lang--- Begin Message --- Hi. I've finally identified the source of a problem that has gradually been driving me nuts. Part of the finger points at IMAP, but Mulberry's behavior makes it much worse. I thought I'd at least warn others about the problem. Suppose a message is received with, say, an introductory body part containing a few kilobytes of comments or information, followed by many megabytes of "attachment" body parts. The obvious thing to do is to open (or synchronize, if working largely offline) the first body part, read it, and then download the additional body parts as needed. Asynchronous prefetching (which Mulberry doesn't do) aside, that is a model that ought to be supported by any competent IMAP client-server pair and Mulberry does it very nicely. However, if the message is signed with S/MIME, the signature is over the entire message, including all body parts. Mulberry wants to verify signatures when messages are opened. One can uncheck that preference option, but, if one does, it is relatively hard to verify selectively -- no provision for adding a "verify signature" button, no "verify sig" entry in the per-message pull-down from the TOC page, etc. There is a per-message Verify/Decrypt entry on the main "Message" pull-down, but it seems to often be grayed out even when signed messages are selected. So, suppose one keeps the "Verify signed messages on opening" preference checked. Now, if one has a signed message and tries to open that first body part, Mulberry feels obligated to download the entire, multi-megabyte message in order to verify the signature. No provisions for a "this is going to take a while, do you mean it" warning or anything else (such as the traditional "the message you are about to open is large..." warning/question) -- just innocently click on a message which is obviously "small first body part, huge attachment(s)" (and less obviously signed, little pencil icon notwithstanding) and then go sit on one's hands for a while. Mulberry then manages to add insult to injury: when one actually selects the attachment to open or download it, it generates the "message you are about to open is large" warning and then, upon getting a "yes", proceeds to download it again. I understand enough of Mulberry's internal model to know why that happens but, as the number of people who are automatically signing messages by default continues to rise, the overall picture is going to become a fairly nasty combination of misfeatures. At least a "this message is rather large, do you really want to take the time to download all of the body parts so the signature can be verified?" dialog box would seem to be in order. Obviously this will never be noticed by anyone with fast LAN connections to their IMAP servers. But for those of us who are either not as well off technologically or who travel extensively to environments with poor connectivity... john --- End Message --- --- Begin Message --- --On Tuesday, October 13, 2009 6:03 PM -0400 John C Klensin wrote: > However, if the message is signed with S/MIME, the signature is > over the entire message, including all body parts. Ouch. That's a nasty drawback. I wonder if there are IMAP extensions to do the verification server-side? --- End Message --- Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Exec'ing a script from Cyrus when imapd has a client
On Tue, 20 Oct 2009, Greg A. Woods wrote: > At Tue, 20 Oct 2009 19:36:24 +0200, Xavier Bestel > wrote: > Subject: Exec'ing a script from Cyrus when imapd has a client >> >> I have a small install with cyrus-imapd 2.3.14, which reads some of its >> mails with fetchmail. To limit the delay in mail delivery, fetchmail >> awakes each minute to get mails. >> What I would like is let fetchmail do that only when there's a client >> actually reading its mails, i.e. an MUA actually connected to imapd. > > I don't get it. Are you saying you are using fetchmail to inject > messages into a locally running Cyrus install which you then connect to > with a locally running IMAP MUA? I think what he is saying is that he does not have a MTA. he uses fetchmail to download mail from elsewhere and put it in cyrus. currently he crons fetchmail to run once a min so that when people are logged in they see new mail with low latencies. however, if nobody is logged in to Cyrus, this is a waste of time, and he would be better off running fetchmail less frequently (or not at all). so he is asking if there is a way to tell if anyone is connected to Cyrus or not, so that if not he can skip the fetchmail run. David Lang > Maybe you should just eliminate fetchmail from the picture and then see > if things make more sense. Just point your MUA at the originating IMAP > server and eliminate everything in between. > > If you're using an IMAP capable client, and you're using IMAP to read > your e-mail, then you don't need or want fetchmail -- it's just a rather > nasty old hack for the days before IMAP. Personally I think the last > usable copy of fetchmail should be archived away on an ancient 9-track > magnetic tape reel and put behind a little glass door which has "Break > glass in case of emergency" written on it. Everything is ever so much > easier and simpler if you can just eliminate all the unnecessary > complexities, such as fetchmail. > > (if you want to keep a copy of your messages on your local machine, then > you can and should probably just use your MUA to do that) > > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: cyrus replication : master to replica and replica to master
On Thu, 22 Oct 2009, David Touzeau wrote: > On Thu, Oct 22, 2009 at 12:56:03AM -0700, Jon . wrote: >> On Wed, Oct 21, 2009 at 9:20 PM, Rob Mueller wrote: >> ... >> >>> The difference between "in theory this would work" and the practice > of >>> actually doing it are huge. Basically it works only if you are 100% > sure >>> that only one side is ever being accessed at a time. eg. > IMAP/POP/LMTP/etc. > > Pretty much. With appropriate fencing, non-local bind and a service IP > address that's feasible. But Rob won't let me do it. Fair enough too, > it's pretty messy. implementing this should not be that hard allow non-local bind in /etc/sysctl heartbeat (linux-ha.org) can handle moving the service IP and fencing (up to and including turning a box off if the cluster decides that it has failed) I've been doing this (without going to the extent of turning the failed box off) on my firewalls for years. it sounds more complicated than it is. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Exec'ing a script from Cyrus when imapd has a client
On Thu, 22 Oct 2009, Greg A. Woods wrote: At Tue, 20 Oct 2009 22:54:12 +0200, Xavier Bestel wrote: Subject: Re: Exec'ing a script from Cyrus when imapd has a client Le mardi 20 octobre 2009 à 13:00 -0700, David Lang a écrit : On Tue, 20 Oct 2009, Greg A. Woods wrote: At Tue, 20 Oct 2009 19:36:24 +0200, Xavier Bestel wrote: Subject: Exec'ing a script from Cyrus when imapd has a client I have a small install with cyrus-imapd 2.3.14, which reads some of its mails with fetchmail. To limit the delay in mail delivery, fetchmail awakes each minute to get mails. What I would like is let fetchmail do that only when there's a client actually reading its mails, i.e. an MUA actually connected to imapd. I don't get it. Are you saying you are using fetchmail to inject messages into a locally running Cyrus install which you then connect to with a locally running IMAP MUA? I think what he is saying is that he does not have a MTA. he uses fetchmail to download mail from elsewhere and put it in cyrus. currently he crons fetchmail to run once a min so that when people are logged in they see new mail with low latencies. however, if nobody is logged in to Cyrus, this is a waste of time, and he would be better off running fetchmail less frequently (or not at all). so he is asking if there is a way to tell if anyone is connected to Cyrus or not, so that if not he can skip the fetchmail run. Yes, that's it, precisely. OK, that's just the kind of crazy overkill and complexity I was talking about. If you don't have an MTA (but you do have an IMAP-capable MUA), then you really do not need or want Cyrus, and you certainly do not want fetchmail either. Just use your IMAP MUA directly for goodness sake! there can be cases where you are providing mail services for several people, or have multiple machines you use yourself where having an IMAP server is worthwhile. and if you are going to setup an IMAP server, why use anything less than the best? ;-) now, it's unusual to use something like this without having a full MTA, but it's not unheard of. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Exec'ing a script from Cyrus when imapd has a client
On Fri, 23 Oct 2009, Greg A. Woods wrote: > At Thu, 22 Oct 2009 18:43:41 -0700 (PDT), David Lang > wrote: > Subject: Re: Exec'ing a script from Cyrus when imapd has a client >> >> there can be cases where you are providing mail services for several people, >> or >> have multiple machines you use yourself where having an IMAP server is >> worthwhile. > > Neither of those things make any real sense whatsoever. They certainly > don't define any clear requirements that make sense in this context. > > Every modern and useful IMAP-capable MUA can collect e-mail from any > combination of many IMAP servers anywhere and everywhere all at once. > > If "fetchmail" can fetch the mail from an IMAP server, then so can any > MUA. > > Just get rid of all the unnecessary complexity in the middle and just > use the MUA for what it's designed to be used for! as long as you are willing to limit yourself to a single MUA on a single desktop/laptop. if you want to be able to access your mail from different devices you need a mail server, not just a MUA David Lang > > >> now, it's unusual to use something like this without having a full MTA, but >> it's >> not unheard of. > > It's not unusual for people to create all kinds of crazy complicated > setups that have no real purpose, in every domain in life. > > I'm sure I make my own life more complicated than it needs to be in some ways. > > However things do not _need_ to be made more complicated than necessary, > > Here the OP's question provides a perfect clue showing that something is > far more complicated than it needs to be because we see that it will > even have to get more complex (and even less robust) before it begins to > work the way it would actually work without any of this unnecessary > complexity in the middle in the first place. > > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Exec'ing a script from Cyrus when imapd has a client
On Fri, 23 Oct 2009, Greg A. Woods wrote: > At Fri, 23 Oct 2009 13:00:34 -0700 (PDT), David Lang > wrote: > Subject: Re: Exec'ing a script from Cyrus when imapd has a client >> >> as long as you are willing to limit yourself to a single MUA on a single >> desktop/laptop. >> >> if you want to be able to access your mail from different devices you need a >> mail server, not just a MUA > > Huh? What in the heck are you talking about? > > I run multiple IMAP clients (some on the same computer, some on > different computers) all _simultaneously_, all accessing a half-dozen > different mail accounts on different servers around the Internet. > > All my MUAs access the same folders and same messages directly, and > simultaneously. > > I certainly don't need yet another IMAP server and a whole bunch of > unnecessary complexity with things like fetchmail just to do this. I possibly missed it, but I didn't see anything that said that fetchmail was grabbing things via IMAP. if the remote account is POP then doing something like this has value if you have intermittent/expensive-per-min internet connectivity doing something like this has value. I did something similar to this several years ago where a non-profit only had dial-up internet. I ran a local cyrus server and then had a process to bring up the internet connection as-needed. In that case I just polled for mail every half hour and people were willing to live with that at that time. In this case I actually used UUCP through a fixed mail server to do the routing instead of fetchmail, but the basic concept is the same. another reason to run your own server is just to be free from quotas. many ISPs have small mail quotas. David Lang > What I would really like to learn is why anyone would falsely believe > that they do somehow need their own IMAP server for this reason. There > must be some false conception or expectation permeating some parts of > the ether out there. > > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Exec'ing a script from Cyrus when imapd has a client
On Mon, 26 Oct 2009, Greg A. Woods wrote: > At Fri, 23 Oct 2009 13:37:30 -0700 (PDT), David Lang > wrote: > Subject: Re: Exec'ing a script from Cyrus when imapd has a client >> >> I possibly missed it, but I didn't see anything that said that fetchmail was >> grabbing things via IMAP. > > Yup, I think you missed it. > >> if you have intermittent/expensive-per-min internet connectivity doing >> something >> like this has value. > > Nope, not really. All modern useful IMAP clients can work offline too. > > All another IMAP server is doing is adding to the complexity _and_ > decreasing, i.e. lowering, the robustness of the overall solution. > >> another reason to run your own server is just to be free from quotas. many >> ISPs >> have small mail quotas. > > All modern useful IMAP clients can also store message locally -- moving > them from server to server, or server to local (or back), is as simple > as selecting and saving/dragging messages between folders. in my mind, having the IMAP client copy all messages to the local drive goes a long way to defeating the benifits of using IMAP in the first place. what do you consider a 'modern IMAP client' that is actually reasonably efficiant to use? there are a lot of 'IMAP clients' out there that treat IMAP as if it was POP (downloading everything and then working on it locally, taking _no_ advantage of the server capabilities) I am interested in finding such a client because at the moment I am using pine and mulberry, both of which are very good at using the server, but not exactly 'modern'. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Exec'ing a script from Cyrus when imapd has a client
On Mon, 26 Oct 2009, Xavier Bestel wrote: > On Mon, 2009-10-26 at 10:07 -0700, David Lang wrote: >> On Mon, 26 Oct 2009, Greg A. Woods wrote: >> >>> At Fri, 23 Oct 2009 13:37:30 -0700 (PDT), David Lang >>> wrote: >>> Subject: Re: Exec'ing a script from Cyrus when imapd has a client >>>> >>>> I possibly missed it, but I didn't see anything that said that fetchmail >>>> was >>>> grabbing things via IMAP. >>> >>> Yup, I think you missed it. >>> >>>> if you have intermittent/expensive-per-min internet connectivity doing >>>> something >>>> like this has value. >>> >>> Nope, not really. All modern useful IMAP clients can work offline too. >>> >>> All another IMAP server is doing is adding to the complexity _and_ >>> decreasing, i.e. lowering, the robustness of the overall solution. >>> >>>> another reason to run your own server is just to be free from quotas. many >>>> ISPs >>>> have small mail quotas. >>> >>> All modern useful IMAP clients can also store message locally -- moving >>> them from server to server, or server to local (or back), is as simple >>> as selecting and saving/dragging messages between folders. >> >> in my mind, having the IMAP client copy all messages to the local drive goes >> a >> long way to defeating the benifits of using IMAP in the first place. > > The drive is not exactly local, it's on a separate server (which does > mainly mail and file server), which is accessed remotely or not, > depending on who uses it and when. I was responding to Greg, who was saying that all modern IMAP clients will copy the mail to the local drive so that they can work offline. David Lang >> what do you consider a 'modern IMAP client' that is actually reasonably >> efficiant to use? there are a lot of 'IMAP clients' out there that treat >> IMAP as >> if it was POP (downloading everything and then working on it locally, taking >> _no_ advantage of the server capabilities) I am interested in finding such a >> client because at the moment I am using pine and mulberry, both of which are >> very good at using the server, but not exactly 'modern'. > > I admit I have yet to find the ideal IMAP client, efficiency-wise. But > that's another problem. > > Xav > > > > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: VMware for Cyrus?
On Mon, 9 Nov 2009, Sebastian Hagedorn wrote: --On 9. November 2009 08:37:46 -0300 Reinaldo de Carvalho wrote: You need analyse the I/O consumition. # iostat -d 1 I trust real world experiences more than benchmarks ... either people on this list have successfully run Cyrus under ESX or they haven't. I don't want to be the first to try it. iostat is not a benchmark, it's a tool to measure your I/O patterns (number of I/O's, size of transfers, etc) while I understand your desire to have real-world experiance, you do need to know your usage patterns to size the system accordingly. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: VMware for Cyrus?
On Mon, 9 Nov 2009, Sebastian Hagedorn wrote: --On 9. November 2009 14:10:54 +0100 Simon Matter wrote: While virtualization has advantages it has also disadvantages. One thing is that it introduces an additional layer of complexity into the game. It's my impression that in many areas virtualization gets introduced not because of technical reasons but because of political pressure. In our case I wouldn't necessarily call it political pressure ... it's more like organizational pressure. We have fewer personnel resources than we used to, and have to run more systems with them! and every time you introduce virtualization you introduce an additional system that you need to run. remember that you need to admin all the virtual machines, just like you would if they were on their own physical boxes, plus you now need to admin the host OS. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Exec'ing a script from Cyrus when imapd has a client
On Thu, 12 Nov 2009, Greg A. Woods wrote: > At Sun, 8 Nov 2009 06:54:30 +1100, Bron Gondwana wrote: > Subject: Re: Exec'ing a script from Cyrus when imapd has a client >> >> On Sat, Nov 07, 2009 at 11:08:31AM -0500, Greg A. Woods wrote: >>> >>> Just get a forwarding alias installed on the remote mail server and then >>> you'll be using the MTA to move your mail in a robust, secure, and >>> fail-safe manner to the IMAP server where you desire it to be finally >>> delivered. >> >> Maybe unreliable network connectivity? > > That's _exactly_ where you want to use SMTP or some other store and > forward mechanism to create a robust and reliable mail transport link! > > Use SMTP to breech the unreliable link! It's safe, proven, and designed > for that very task! no, SMTP only works if you have network connectivity that is up most of the time. it will handle short outages, but it will not handle the case where your network connectivity is off most of the time >> A dynamic IP where they don't >> want a stale DynDNS pointer to cause someone else to get the mail? > > Well, amateurs can and will run whatever hacks they want, and they're > not usually interested in doing the kinds of things necessary for > production systems in the first place either. > > Further, if anyone is stupid enough to try to use dynamic IP addresses > where static IP addresses are REQUIRED for proper functionality and > robust operations then they get every problem they deserve and I have no > interest whatsoever in catering to any of their hacks and abominations. > > Use proper client/server protocols for dynamic IP clients! SMTP is not a proper protocol for a dynamic IP environement. >> Pull vs Push in the abstract is an age old question that never has only >> one answer, much as you are trying to paint it that way. > > Well, in Internet e-mail delivery there has always been one and only one > answer to the push vs. pull philosophy. I'm only talking about e-mail > here. not true, in the beginning UUCP was the primary mechanism for transporting e-mail. it was designed for the (then current) environement where connectivity was very intermittent and expensive to leave idle (which happens to match the use case here as well, but using UUCP takes far more cooperation on the part of the Internet server, thus the fetchmail approach) > Fight the way e-mail has always worked and you have to fight the whole > infrastructure and use fringe tools with known risks and problems. > > If you want your e-mail to work reliably then you have to work with the > existing infrastructure and with the tried and tested tools that > designed and implemented to work that way. > > Note how even in SMTP the proposed mechanisms for pull-like > functionality have been lost, broken, and forgotten forever, and even > there, like in UUCP, it's still fundamentally store-and-FORWARD even if > the client makes the call. Nobody has _ever_ made "pull" work for > e-mail in any significant widely accepted and implemented way. UUCP that's acttivated when the client connects and tickles the server to let it know that it's connected is effectivly a pull mechanism. >>> The rest of this is kinda just BS about how to use a proper IMAP client. >> >> Er, you know a perfect IMAP client? I've never been able to find a good >> one, which is why I use offlineimap to local Maildirs and mutt to talk >> to them. > > I didn't say "perfect" -- I said "proper". :-) > > Mutt is not a proper IMAP client so far as I can tell, for example. > > Pine, Emacs Wanderlust, Thunderbird, Apple Mail, etc. are all "proper" > IMAP clients in most respects, I think. Pick your poison. :-) Thunderbird? my understanding (from watching people use it) is that it wants to pull a copy of all your mail to the local box before processing it. how is this a proper IMAP client? David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Exec'ing a script from Cyrus when imapd has a client
On Thu, 12 Nov 2009, Gabor Gombas wrote: > On Thu, Nov 12, 2009 at 11:55:25AM -0800, David Lang wrote: > >>> Use SMTP to breech the unreliable link! It's safe, proven, and designed >>> for that very task! >> >> no, SMTP only works if you have network connectivity that is up most of the >> time. it will handle short outages, but it will not handle the case where >> your >> network connectivity is off most of the time > > ETRN can solve this, you just need a relay that understands ETRN and is > willing to hold your e-mails while you're off-line. ODMR is better for > dynamic addresses (you don't need dynamic DNS) but it seems to be less > supported. I have not used either yet, so YMMV. the big advantage of the fetchmail approach that the initial poster was asking about is that it does not require special configuration of the upstream mail server. I definantly agree that it is not the best possible approach. but as far as I can tell, every approach that's better requires cooperation from a mail server that does have full-time Internet connectivity. unfortunantly that's not always available (at least, not available at an affordable price in time and money) David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Exec'ing a script from Cyrus when imapd has a client
On Thu, 12 Nov 2009, Greg A. Woods wrote: > At Thu, 12 Nov 2009 21:21:24 +0100, Xavier Bestel > wrote: > Subject: Re: Exec'ing a script from Cyrus when imapd has a client >> >> Do you gain anything if Cyrus doesn't >> fulfill the needs of some users ? > > Did I ever say Cyrus should or should not meet the needs of some users? > > I think you've got something backwards here. > > Why should any one "product" meet the needs of all users? That's the > "if all you've got is a hammer..." analogy. Cyrus IMAP will _NEVER_ > meet the needs of all users, and that's fundamentally what I've been > trying to say from the start. If you don't need it then don't try to > wedge it into your implementation! > > We all gain if we can avoid the "all you've got is a hammer" trap, and > indeed we all gain if we can help each other avoid that trap too. > > If I'm not too mistaken it seems most everyone who wanted to use an IMAP > server as a client-level tool (by employing the likes of fetchmail) were > clearly falling for the "I have this Cyrus IMAP Hammer and I want to use > it to manage all of my e-mail even though I'm not running a server" trap. > Certainly that seemed to me to be the OP's situation. > > The real fear I have is that as a result others will believe that this > kind of use is condoned and approved, or worse that these folks were > actually setting up such things for other groups of users. In both > cases we end up with naive users who will not understand the issues > involved. Issues such as mangled headers, unreliable delivery, loss of > e-mail, and so on. who are you to officially condone or approve any particular use? who on this list (or any list) has the authority to do so? it's fine to suggest other solutions, but if people then say that those solutions cannot be used in these cases (and especially when they give the reasons) continuing to argue that it's wrong and evil to use the tool that way is not productive. you may not choose that solution, but that may be because you have resources available to you that make a different solution better. > Sure, it's all fine and dandy for someone to want to learn about a > product like Cyrus IMAP (or even fetchmail) by installing and using it > in some form where they can personally make use of it. > > However if the goal is just to make something work in the real world for > end users then we really need to go back to the fundamental end-user > requirements and figure out how best to meet them without creating > hidden problems along the way simply because we've got this hammer in > our hands and we're dying to bang away on something. If the user wants > some screws installed then we'd be doing them a huge favour if we would > go and find the proper screwdriver to do the job for them! even if the OP was using UUCP for the mail transport, the question he asked (how to have a job run frequently when a user is logged in, and not run if there are no users logged in) is still a very useful question to get an answer to. you have focused on the fact that he wants to use fetchmail as the transport between the full-time internet and his intermittently connected network and are telling everyone that he absolutly, under no conditions should try to do what he's attempting. that's where we have severe disagreements. most of the rest of us see situations where the basic approach he's doing could be the best possible approach. we may quibble over exact details of some of the things (use fetchmail vs UUCP vs other), but that doesn't make the basic approach invalid. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Exec'ing a script from Cyrus when imapd has a client
On Thu, 12 Nov 2009, Greg A. Woods wrote: > At Thu, 12 Nov 2009 11:55:25 -0800 (PST), David Lang > wrote: > Subject: Re: Exec'ing a script from Cyrus when imapd has a client >> >> no, SMTP only works if you have network connectivity that is up most of the >> time. it will handle short outages, but it will not handle the case where >> your >> network connectivity is off most of the time > > If your link is off most of the time then use UUCP -- that's what it was > designed for. Tunnel it through SSL if you're using public IP > addressing and routing for the same link. you can only do this if the servers you are connecting to support UUCP, most do not. but even if you do this, it's valid to want UUCP to check frequently when imapd has a client and infrequently (if at all) when it doesn't. > Or just use IMAP from a client MUA -- your link will be up when you want > to read mail and all your mail will be available at the IMAP server at > that time. over a slow, high latency, error-prone link IMAP is very painful to use. > All these excuses for doing strange things with things like fetchmail > are really really lame and stretching the imagination beyond belief! >>> Use proper client/server protocols for dynamic IP clients! >> >> SMTP is not a proper protocol for a dynamic IP environement. > > Indeed, SMTP is not a client/server style protocol in the sense I meant. > > IMAP would be a proper client/server style protocol in the sense I meant. so you are claiming that his business need is invalid. you are not in a position to declare this. >> Thunderbird? my understanding (from watching people use it) is that it wants >> to >> pull a copy of all your mail to the local box before processing it. how is >> this >> a proper IMAP client? > > How is it not a proper IMAP client? Like I said, pick your poison. > Some MUAs will want to copy everything over at once for one kind of > performance profile, some will request only headers (enough to form the > summary index) for another kind of performance profile. Thunderbird > fully understands the concept of multiple folders on arbitrary servers, > and it more or less speaks true IMAPv4, giving the user an interface to > do most everything that IMAPv4 will easily allow. It has additional > features that make it possible to work offline. That sounds like a > proper IMAP client to me. that 'more or less speaks true IMAPv4' makes me say it's not a proper IMAP client. I also consider any client that uses IMAP to pull the data to the local system and does everything else there to be a POP client that happens to use IMAP to fetch it's messages. A proper IMAP client would fetch only the portions of a message that the user needs, and would use the capabilities of the server (or at least the basic IMAPv4 capabilities, they may not use all of the enhancements) rather than duplicating the functionality on local copies. this doesn't prevent the client from offering a disconnected mode of operation, but if disconnected mode is not in use, the server should be used. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Future Ideas wiki page
one thing that I saw mentioned elsewhere as a limitation of IMAP (and therefor I don't know if there is a way to address it reasonably) is the lack of a fuzzy search capability. the IMAP search is a exact match search, it would be useful to have the hooks to be able to use a search-engine like search capibility as well (not just exact matches, but matches with only some of the search terms, matches with plural versions of the search terms, etc) As I understand it this would require a slight variation of the search request to indicate that you want the fuzzy match, and a variation of the search response to be able to indicate the quality of each match returned. David Lang On Thu, 7 Jan 2010, Bron Gondwana wrote: > Date: Thu, 7 Jan 2010 20:01:52 -0800 > From: Bron Gondwana > To: cyrus-de...@lists.andrew.cmu.edu, cyrus-proj...@lists.andrew.cmu.edu > Cc: info-cyrus@lists.andrew.cmu.edu > Subject: Future Ideas wiki page > > Hi All, > > I've set up a new wiki page here: > > http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/FutureIdeas > > Linked from the roadmap: > > http://cyrusimap.web.cmu.edu/twiki/bin/view/Cyrus/RoadMap > > I've updated the Roadmap with the items I have ready right > now for 2.4, and put everything else into the bright future > under 2.5 for now, subject to "actually gets finished and > stable", obviously. > > Most of these ideas have been fleshed out in some detail > already in my postings to the devel mailing list. Some are > still a little bit raw (like having the mailbox path on disk > depend on the uniqueid rather than the actual mailbox name. > I'll expand on this another time... and it's not yet mentioned > on the wiki, but it makes a lot of things nicer!) > > Anyway, I'd love feedback on any or all of it, and if there > are other things that you feel are really important for the > future viability of Cyrus I'd love to hear about them as well. > I haven't yet had a chance to look at the QRESYNC stuff that > Ken's already done for 2.4, and we might wind up releasing > a 2.4 without a lot of these changes just because there's a > lot of work in there! > > That said, I'm pushing myself pretty aggressively to have that > list finished by April of THIS YEAR. Particularly the low > bandwidth replication, which depends on all pretty much all > of the others! > > Regards, > > Bron. > > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Future Ideas wiki page
On Fri, 8 Jan 2010, Bron Gondwana wrote: > On Fri, 08 Jan 2010 09:56 -0800, "David Lang" > wrote: >> one thing that I saw mentioned elsewhere as a limitation of IMAP (and >> therefor I >> don't know if there is a way to address it reasonably) is the lack of a >> fuzzy >> search capability. > > Without a specification document, it's hard to add anything that you expect > clients to actually use. true, but without a sample implementation it's unlikely to become a standard, so the discussion needs to stop somewhere. >> the IMAP search is a exact match search, it would be useful to have the >> hooks to >> be able to use a search-engine like search capibility as well (not just >> exact >> matches, but matches with only some of the search terms, matches with >> plural >> versions of the search terms, etc) > > Yes, that would be lovely to have. You'd probably run a separate > search-engine > process and have the IMAP server just send out a request and map the document > IDs back to folder/uid on response. sounds right. >> As I understand it this would require a slight variation of the search >> request >> to indicate that you want the fuzzy match, and a variation of the search >> response to be able to indicate the quality of each match returned. > > It would require a brand new spec for the search result - an ordered list of > UIDs wouldn't cut it any more! > > While we're at it, I'm much more interested in cross-folder searching with > sort > order that doesn't require folder as the first item, but that's significantly > more > complex! If we have to define a new search response, it may be reasonable to handle both. Does anyone who has been part of the IMAP standards work want to think/talk about what would work well and fit the flavor of IMAP? On the same subject, what other wishes do people have related to search (if there are enough things people are interested in, possibly someone will get motivated enough to write the code to make it work) > Thankfully, this is all pretty orthagonal to everything that I'm doing, so > it's not > a consideration I need to give much thought to at the moment. Someone else > who considers it worth putting effort in to could do it pretty independently. agreed. > The charset changes would allow an initial pre-processing pass to spit out the > "document" as UTF-8 rather than its original MIME encoding for processing by > the search engine, but that's the only interaction it would have. If the > search > engine supports a chunked input, it would probably be worth embedding that > target into the lib/charset.c as a character filter sink, and chaining the > documents > into it rather than building an entire buffer at once. There's already code > that > does that just using a standard buffer and sending it to the squatter callback > whenever it reaches a fixed size, then resetting it. Easy enough to do. If that interface is enough more efficiant, it may be worth making that a requirement and let the external tool deal with splitting it back up if needed. So, if I am understanding you correctly, the hooks into Cyrus to support something like this are fairly easy to do, the hard thing would be the IMAP command and response. David Lang > Regards, > > Bron. > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Future Ideas wiki page
On Sat, 9 Jan 2010, Bron Gondwana wrote: > On Sat, 09 Jan 2010 15:10 +0100, "Alexey Melnikov" > wrote: >> Bron Gondwana wrote: >>> While we're at it, I'm much more interested in cross-folder searching with >>> sort >>> order that doesn't require folder as the first item, but that's >>> significantly more >>> complex! >>> >>> >> <http://tools.ietf.org/html/draft-ietf-morg-multimailbox-search-01>? >> If you have some additional use cases in mind, please let me know. > > "with sort order that doesn't require folder as the first item". I guess you > could return > multiple ESEARCH responses with the same folder mentioned to get the > ordering... > > It's still more folder-centric than otherwise, and it's going to make > following threads > across folders (say INBOX and a couple of time based archive folders) tricky! thinking about this a bit more, it sounds almost like what you are wanting is a third mode of addressing messages. currently we can do message # in this folder messageUID in this folder and something like folderUID:messageUID would open up what you are looking for (probably plus more) Would it be possible to take a character that can't appear in a message# or UID position in the existing protocol and define it as a delimiter for this? (I used ':' in my example above as I believe that '-' is used to indicate a range of messages) If it is, then this would 'just' be a new addressing option like UID currently is, and like UID, clients would opt-in to this new mode. (it would still need a RFC for the new mode, but does this sound like a solution to what you are looking for? David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Future Ideas wiki page
On Wed, 13 Jan 2010, Wil Cooley wrote: And anyway, would it be faster to open and list 1,000 files in 23 directories than to open one directory and list 23,000 files? Would that be overshadowed by the cost of opening all 23,000 files (which I presume it would need to if it were resorting to listing them). there are two scenerios that are important here 1. open a directory and list 23,000 files for this case, just about all filesystems will be faster with one directory as they can just do a sequential walk through the directory. 2. open file 22,987 for this case you will see a drastic difference between different filesystems (and potentially even the same filesystem with different options). For example, this will be very slow on Ext2, or on Ext3 without hashdir enabled. It's supposed to be much faster on Ext3 with hashdir, but I've had mixed results when I've tried it. On the other hand this is a very fast operation on XFS. David Lang signature.asc Description: OpenPGP digital signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Future Ideas wiki page
On Wed, 13 Jan 2010, Bron Gondwana wrote: > On Wed, 13 Jan 2010 16:26 -0800, "David Lang" > wrote: >> On Sat, 9 Jan 2010, Bron Gondwana wrote: >> >>> On Sat, 09 Jan 2010 15:10 +0100, "Alexey Melnikov" >>> wrote: >>>> Bron Gondwana wrote: >>>>> While we're at it, I'm much more interested in cross-folder searching >>>>> with sort >>>>> order that doesn't require folder as the first item, but that's >>>>> significantly more >>>>> complex! >>>>> >>>>> >>>> <http://tools.ietf.org/html/draft-ietf-morg-multimailbox-search-01>? >>>> If you have some additional use cases in mind, please let me know. >>> >>> "with sort order that doesn't require folder as the first item". I guess >>> you could return >>> multiple ESEARCH responses with the same folder mentioned to get the >>> ordering... >>> >>> It's still more folder-centric than otherwise, and it's going to make >>> following threads >>> across folders (say INBOX and a couple of time based archive folders) >>> tricky! >> >> thinking about this a bit more, it sounds almost like what you are >> wanting is a >> third mode of addressing messages. >> >> currently we can do >> >> message # in this folder >> >> messageUID in this folder >> >> and something like folderUID:messageUID would open up what you are >> looking for >> (probably plus more) >> >> Would it be possible to take a character that can't appear in a message# >> or UID >> position in the existing protocol and define it as a delimiter for this? >> (I used >> ':' in my example above as I believe that '-' is used to indicate a range >> of >> messages) >> >> If it is, then this would 'just' be a new addressing option like UID >> currently >> is, and like UID, clients would opt-in to this new mode. (it would still >> need a >> RFC for the new mode, but does this sound like a solution to what you are >> looking for? > > Absolutely - two issues. > > 1: how to you give folders UIDs? I thought that there was mention in your list of addressing folders by UID for replication purposes. > 2: how do "ranges" work? a range within a folder works as-is, a range with different folders is an error > The first one is the killer, because you'd have to be able to map back to > select > by UID as well. SELECT 195455, or something. > > Actually, the whole "SELECTED" idea would probably be discarded, instead just > addressing everything by UID and never actually selecting folders. yes, for some things this is extremely useful for others it's just added complication. > By which time, why not just define a brand new protocol not called IMAP which > includes the good bits of what IMAP currently does, and discards anything that > doesn't fit the multi-folder worldview. So long as you made the storage and > meta-data requirements compatible with already existing Cyrus and other IMAP > servers, you could just write a whole new daemon that talked your new protocol > and be happy with that. you could, but just like not every client uses UIDs and still use message numbers, this is being done in a backwards compatible manner so that the client can use either one, and a server can support both. David Lang > Bron ( yes, I have been tempted to write something that talks sync_client > protocol, >why do you ask ;) > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Future Ideas wiki page
On Wed, 13 Jan 2010, Bron Gondwana wrote: > On Wed, 13 Jan 2010 17:40 -0800, "David Lang" > wrote: >>> Absolutely - two issues. >>> >>> 1: how to you give folders UIDs? >> >> I thought that there was mention in your list of addressing folders by >> UID for >> replication purposes. > > UniqueID - it's an internal 16 hex digit field. Tacking one of those on every > result item would be expensive for bandwidth. If you have a lot of of messages being identified it will be noticable, but I'll point out that the user dowloading one word doc attachement is several thousand messages worth of IDs. Another option would be to use the folder name (but you would have to worry about escaping some characters, etc) >>> The first one is the killer, because you'd have to be able to map back to >>> select >>> by UID as well. SELECT 195455, or something. >>> >>> Actually, the whole "SELECTED" idea would probably be discarded, instead >>> just >>> addressing everything by UID and never actually selecting folders. >> >> yes, for some things this is extremely useful for others it's just added >> complication. > > There's not much that it's useful for with cross folder work! > > Internally our web interface actually already does exactly what you describe > so it > can support cross folder searching. Every UnqMsg is :, where > FolderId is a reference to a database field which contains a mapping to the > underlying folder. We also store a bunch of metadata about how to present > that > particular folder to the user (number of messages to display, type of preview, > default sort, etc) if the only thing that can really use this is cross folder searching, then only implement this as output from the server in search results (and possibly sorting commands), allow it for input (message specification) for other commands. Worst case you could require the client to select the folder. One other thing that something like this would be useful for is for things that want to watch for messages from any of several folders. Today they need to either open a seperate connection for each folder they want to watch, or cycle through the folders polling them. I suspect that if this addressing mechanism was implemented, it would be about as easy to add it everywhere that UID is an option as it would be to only add it to some commands. >>> By which time, why not just define a brand new protocol not called IMAP >>> which >>> includes the good bits of what IMAP currently does, and discards anything >>> that >>> doesn't fit the multi-folder worldview. So long as you made the storage and >>> meta-data requirements compatible with already existing Cyrus and other IMAP >>> servers, you could just write a whole new daemon that talked your new >>> protocol >>> and be happy with that. >> >> you could, but just like not every client uses UIDs and still use message >> numbers, this is being done in a backwards compatible manner so that the >> client >> can use either one, and a server can support both. > > The problem is you keep paying a higher and higher complexity cost at the > server end. > Once you start talking not having a folder selected, that cost (and associated > locking issues) is going to blow your implementation complexity way up. > Cross folder is > far enough away from the initial design of IMAP that the impedance mismatch > is grating > pretty badly! I'm missing the locking issues here. even with single folder searching you can have a message disappear immediatly after the search results have been provided. Folders don't get locked when they are selected. What is it that needs to be locked for this sort of addressing? I agree with the climbing complexity, but is there really so much stuff that _everyone_ agrees should be dumped with IMAPv4 that it's worth the break? If so it may be time for IMAPv5, but have we really moved that far? Most of the time you really do want to act on one folder at a time, and as such selecting that folder makes sense. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: visibility of Mailbox-folders
On Tue, 19 Jan 2010, Dr. Harry Knitter wrote: >> >> That's interesting. Since cyradm is an imap client, why does it see the >> mailboxes and your other clients don't? Can it be that your clients are >> configured to use another folder separator or something? I really don't >> understand this. >> >> Simon >> > > I don' t understand it either. We have tried it with different Client programs > (Thunderbird, Outlook, KMail unfortunately has problems with large amounts of > imap folders). The result was allways the same. > Other folders which show the same results in cyradm show their subfolders and > files without any problems. > > So what can I do to solve this problem? Silly question (since I haven't been following this thread), have you checked the subscribed status of the folders? David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus crashed on redundant platform - need better availability?
On Fri, 10 Sep 2004, Michael Loftis wrote: Date: Fri, 10 Sep 2004 13:15:05 -0600 From: Michael Loftis <[EMAIL PROTECTED]> To: Paul Dekkers <[EMAIL PROTECTED]>, [EMAIL PROTECTED] Subject: Re: Cyrus crashed on redundant platform - need better availability? The theory only translates if you're using a JOURNALED file system. Linux ext3, reiserfs AIX JFS, Sun/others veritas are all examples of this. AFAIK FreeBSD hasn't any journalling file systems, i could be wrong though since I haven't really looked for one (my freebsd boxes just run...and run...and run...) That said, the machine shouldn't' have crashed in the first place, but you are running 5.x which is clearly labeled as *NOT* production (4.10 for that)... All of my produciton boxen are 4.x based (of the FreeBSD herd) However even a Journaled filesystem won't protect you completely from corruption. even the filesystems you list can loose data when there is a crash and if one system goes haywire and starts scribbling on the shared disk it will trash any filesystem. David Lang --On Friday, September 10, 2004 13:24 +0200 Paul Dekkers <[EMAIL PROTECTED]> wrote: Hi, We're implementing a new mailplatform running on two dell 2650-servers (2 xeon cpu's with each 3 Ghz, HTT and 3Gb of memory) and with a disk array of 4 Tb connected with a adaptec 39160 scsi controller for storage. We installed FreeBSD 5.2.1 on it, and - of course - cyrus 2.2.8 (from the ports) as IMAP server. Our MTA is postfix. There are two machines for redundancy. If one fails, the other one should take over: mount the disks from the array, and move on. Unfortunally, the primary server crashed twice already. The first time it did while synchronising two IMAP-spools from the old server to the new one. There was not much data on it back then. The second time was worse, around 10Gb of mail was stored on the disks. We discovered that the fsck took about 30 minutes, so although we have two machines for redundancy it takes still quite some time before the mail is available again. (And we still have about 90 Gb of mail to migrate, so when all users are migrated it takes much longer.) I mounted the filesystems synchronous now: although it slows down the system I hope it speeds up the fsck a bit when there is another crash. The second crash was while removing a lot of mailboxes (dm) while some of them where removed the same time using a webmail app (squirrelmail). I'm not sure why the box crashed; there was nothing in the logs, there was nothing on the screen when we came there, it just booted up again. Of course I'm interested if anyone has any thoughts on this. Although many on the list claim that this (having 2 boxes with 1 disk-array) is a nice way for redundancy I'm in doubt now if this is true. It still takes 30 mins before everything is back again! It seems to me that if there was a "live" version of cyrus available with a synchronised mail-spool, that there was no outage noticeable for users (except in losing a connection maybe). Am I right? Maybe it's time to continue on the "High availability ... again"-discussion we had a while ago. If the cyrus developers are able to implement this with some funding there are still some questions left for me: how much time would it take before a "stable" solution is ready? How many funding is expected? I still have to talk to management about this, but I would really support this development and I'm certainly willing to convince some managers. Regards, Paul --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- Undocumented Features quote of the moment... "It's not the one bullet with your name on it that you have to worry about; it's the twenty thousand-odd rounds labeled `occupant.'" --Murphy's Laws of Combat --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus crashed on redundant platform - need better availability?
also take a look at the heartbeat package at linux-ha.org This works on linux, *BSD, and solaris (there were people working on a AIX port, but they apparently dropped it shortly before finishing) David Lang On Wed, 15 Sep 2004, Jure [UTF-8] PeÄ~Mar wrote: Date: Wed, 15 Sep 2004 17:07:20 +0200 From: "Jure [UTF-8] PeÄ~Mar" <[EMAIL PROTECTED]> To: Paul Dekkers <[EMAIL PROTECTED]> Cc: [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: Re: Cyrus crashed on redundant platform - need better availability? On Wed, 15 Sep 2004 13:38:43 +0200 Paul Dekkers <[EMAIL PROTECTED]> wrote: But I suppose RH's cluster manager takes care of mounting the partitions and checking them if there are any errors. Not really, at least not by itself. See http://people.redhat.com/jrfuller/cms/ for detailed documentation of what is included with RH AS 2.1 (it's some $500 extra for AS 3). I had to write some pretty paranoid scripts that take care of assembling software raids, checking the fs and mountig it while taking care about the other machine to prevent problems. Of course all this would be much easier with some kind of clustered fs, but clustered fs brings a new problem: locking. Almost all i've seen so far have an external 'locking manager' on a separate box, which brings ethernet latency into every lock operation, which i'm sure is very noticable in the lock-heavy usage patterns as mail is. But this is just my feeling, i haven't yet benchmarked any :) Do you think using RH's cluster software is a valuable consideration for this kind of clustering setup? Using FreeBSD there are not that many clustering solutions for now, and if it's advisable to at least consider using RH here (although I have no experience with RH) we can certainly look at it. (Any idea how fast RH would "recover services"?) This RH cluster software is nothing fancy; i'm sure equivalents exists for BSDs. See documentation link above. Actually it is just Kimberlite (http://oss.missioncriticallinux.com/projects/kimberlite/), sold with RedHat support. "Speed" of recovery is almost completely out of the cluster control. The only thing that matters for the cluster is what your cyrus init script returns when called with 'status' parameter. Everything else is up to your init scripts. Of course, if one box dies completely, the other takes over in the configurable time. -- Jure Peÿÿar --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus crashed on redundant platform - need better availability?
how much are you asking for? David Lang On Wed, 15 Sep 2004, Ken Murchison wrote: Date: Wed, 15 Sep 2004 11:44:45 -0400 From: Ken Murchison <[EMAIL PROTECTED]> To: Paul Dekkers <[EMAIL PROTECTED]> Cc: David Carter <[EMAIL PROTECTED]>, [EMAIL PROTECTED] Subject: Re: Cyrus crashed on redundant platform - need better availability? Paul Dekkers wrote: David Carter wrote: On Wed, 15 Sep 2004, Paul Dekkers wrote: On the other hand, if there is a application level redundancy on its way, it doesn't really matter on what platform the machine runs, so it would still make me happier and even with FreeBSD. And I would rather put my money there. Even if it means we'll have to wait for some months, I wouldn't hold out hope of anything being available in "some months". I wrote my replication code two years ago, and submitted it to Rob and Ken about this time last year. Neither I or they have put any significant work into the code since then. As I indicated in my previous message, we all have other priorities right now. I can imagine, but I hoped that priorities would change a bit with the amount of users that repeatedly <http://www.interglot.com/toclipboard.php?b=1&d=2&t=herhaaldelijk&s=herhaaldelijk&w=repeatedly>showed This link appears dead. All I get is "To clipboard". interest in this feature and with the money we are willing to put in :-| I'm willing to work on it if there is money available. You are the only one that has says that you would commit money. Where are the rest of the folks? Based on the number of people that stepped up to pay for virtdomains support (zero), I'm guessing there are fewer out there willing to spend money than you think. But I could be wrong. Other than the altnamespace project ($5000) that I did for a (unnamed) company in Texas, Jeremy Howard at Fastmail is the only one who has consistently paid for features. I'll let him disclose what he has spent, if he chooses to, but its safe to say that its been more than just pizza and beer. I'd have to look at David's patch again and discuss things with CMU to get a good time estimate, but I'm guessing that a project like this would cost a few thousand dollars. -- Kenneth Murchison Oceana Matrix Ltd. Software Engineer 21 Princeton Place 716-662-8973 x26 Orchard Park, NY 14127 --PGP Public Key--http://www.oceana.com/~ken/ksm.pgp --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Funding Cyrus High Availability
On Thu, 16 Sep 2004, Ken Murchison wrote: Question: Are people looking at this as both redundancy and performance, or just redundance? for performance we already have murder, what we currently lack is redundancy. once we have redundancy then the next enhancement is going to be to teach murder about it so that it can failover to the backup box(s) as needed, but for now simply having the full data at the backup location would be so far ahead of where we are now that the need to reconfigure murder for a failover is realitivly trivial by comparison. David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Funding Cyrus High Availability
On Fri, 17 Sep 2004, Paul Dekkers wrote: Date: Fri, 17 Sep 2004 08:25:26 +0200 From: Paul Dekkers <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: Funding Cyrus High Availability Hi, Eric S. Pulley wrote: Question: Are people looking at this as both redundancy and performance, or just redundance? Cyrus performs pretty well already. Background redundancy would be awesome. Especially if we had control over when the syncing process occurred either via time interval or date/time. I would say not at an interval but as soon as there is an action performed on one mailbox, the other one would be pushed to do something. I believe that is called rolling replication. I would not be really happy with a interval synchronisation. It would make it harder to use both platforms at the same time, and that is what I want as well. So there is a little-bit of load-balancing involved, but more and more _availability_. Being able to use both platforms at the same time maybe implies that there is either no master/slave role or that this is auto-elected between the two and that this role is floating... right, but there are already tools freely available on most platforms to do the election and changing of the role (by switching between config files and restarting the master) what is currently lacking is any ability to do the master/slave role. once we have that it's just a little scripting to tie just about any failover software in to make it automatic. one thing we need to watch out for here is that we don't set an impossible/unreasonable goal. don't try to solve every problem and add every availablity feater you can imagine all at once. instead let's look at the building blocks that are needed and identify what's currently not available. currently we have murder which will spread the load across multiple machines. currently we have many tools available to detect a server failure and run local scripts to reconfigure machines (HACMP on AIX, hearbeat for Linux, *BSD, Solaris, etc) what we currently do not have is any ability to have one mailstore updated to match changes in another one. once we have that ability there are many things that can be built by glueing togeather existing code. once we have a bit of experiance with people actually useing these features it will then be obvious which features need better integration with Cyrus and which make sense to remain seperate. I also would not be really satisfied with interval synchronisation as the only choice. I think we need something where the primary mailstore pushes a record of it's changes to the secondary mailstore This can then be tweaked in several directions. 1. locking can be added so that the primary doesn't complete it's command until the secondary says it has a permanent record of the change (two-phase commit or a reasonable facimily of such) 2. batch up the changes until they hit some threshold (size or time or combination) and then send a batch of changes all at once 3. recongnise it's own changes to gain the ability to push updates in both directions at the same time (true two-phase commit with bi-directional replication, some horribile performance pathalogical cases, but attractive in some cases) or other varients but these all share the same common need the ability for the master to output all it's changes and the ability for a slave to read such changes and update itself to match the nice thing is that with IMAP much of the data needed is already output, you could do a first approximation of this with a client that opened a seperate connection to every folder on the primary server and just sat watching for server messages and whenever it saw an update send the matching command to the slave (fetching the full data as needed to get all the info). this obviously won't scale to any reasonalbe size, but this means that most of what's needed is already identified so the core could be just a common output of the exisitng messages with a little more data (mailbox and folder in most cases, message contents in a few) let's get these small, but critical pieces done and then we can grow and experiment from there. David Lang Paul --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Funding Cyrus High Availability
On Fri, 17 Sep 2004, Ken Murchison wrote: David Lang wrote: On Thu, 16 Sep 2004, Ken Murchison wrote: Question: Are people looking at this as both redundancy and performance, or just redundance? for performance we already have murder, what we currently lack is redundancy. once we have redundancy then the next enhancement is going to be to teach murder about it so that it can failover to the backup box(s) as needed, but for now simply having the full data at the backup location would be so far ahead of where we are now that the need to reconfigure murder for a failover is realitivly trivial by comparison. Actually what I was really asking, is are people looking for an active-active config and an active-passive config? I think that everyone would love to have the active-active option, the problem I have with this is that the active-passive config will solve many peoples problems and I believe that is will be far simpler to do so I don't want the ideal goal of active-active to end up side tracking the huge progress that would be achieved by active-passive. active-active also requires significantly different choices if the nodes are seperated by significant distances. I'd hate to end up with an active-active solution that works only with the machines all local and still have no solution to the disaster recovery senerio. David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: Funding Cyrus High Availability
On Fri, 17 Sep 2004 [EMAIL PROTECTED] wrote: My biggest question here is, simply, why recreate what's already out there? There are a number of projects (LVM, PVFS) which do this kind of replication/distribution/virtulization for filesystems. There are a number of databases which have active/active clustering (mysql, DB2, Oracle, et al) and master/slave. Personally, I would LOVE to see a full RDBMS-backed system. You define your database(s) in the config file ... and that is all. All configuration options are stored on the central RDBMS. All mailboxes are stored there. You can then rely 100% on the RDBMS systems for clustering/failover/scalability/backing up ... all datastorage domain problems which they have already addressed/solved. The other advantages would be very nice integration with other applications which can query against databases. (ex: postfix directly supports mysql lookups.) But then, I can't afford to really help with this myself so take my thoughts with a big "hope" pill. =D Mike, one of the problems with this is that different databases have different interfaces and capabilities. if you design it to work on Oracle then if you try to make it work on MySQL there are going to be quite a few things you need to change. if you start on MySQL and then port to Oracle then you either ignore a large chunk of Oracle functionality that you could use or you end up having to re-write a bunch of stuff to take advantage of it. I also would love this option (I would choose postgres as the back-end) but this is significantly more complicated then a master->slave replication modification to Cyrus. As such it would cost more to get written and you would have fewer people willing to pay for any particular version. another issue in all this is the maintainance of the resulting code. If this code can be used in many different situations then more people will use it (probably including CMU) and it will be maintained as a side effect of any other changes. however if it's tailored towards a very narrow situation then only the people who have that particular problem will use it and it's likly to have issues with new changes. David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: Funding Cyrus High Availability
On Fri, 17 Sep 2004 [EMAIL PROTECTED] wrote: From: David Lang [mailto:[EMAIL PROTECTED] Mike, one of the problems with this is that different databases have different interfaces and capabilities. if you design it to work on Oracle then if you try to make it work on MySQL there are going to be quite a few things you need to change. --snip another issue in all this is the maintainance of the resulting code. If this code can be used in many different situations then more people will use it (probably including CMU) and it will be maintained as a side effect of any other changes. however if it's tailored towards a very narrow situation then only the people who have that particular problem will use it and it's likly to have issues with new changes. I'd actually figured something like ODBC would be used, with prepared statements. /shrug. Abstract the whole interface issue. unfortunantly there are a few problems with this to start with ODBC is not readily available on all platforms. secondly ODBC can't cover up the fact that different database engines have vastly differeing capabilities. if you don't use any of these capabilities then you don't run into this pitfall, but if you want to you will. I really wish that ODBC did live up to it's hype, but in practice only the most trivial database users can transparently switch from database to database by changing the ODBC config David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Funding Cyrus High Availability
software Con: increased complexity runs the risk of breaking some deployment senerios in an attempt to simplify others 5. Active/Active designate one of the boxes as primary and identify all items in the datastore that absolutly must not be subject to race conditions between the two boxes (message UUID for example). In addition to implementing the replication needed for #1 modify all functions that need to update these critical pieces of data to update them on the master and let the master update the other box. Pro: best use of available hardware as the load is split almost evenly between the boxes. best availability becouse if there is a failure half of the clients won't see it at all Con: significantly more complex then the other options. behavior during a failure is less obvious split-brain recovery is not straightforward and if automatic failover is active the sysadmin will have no option to have things degraded slightly while a problem is fixed depending on the implementation this may be very sensitive to network latency between the machines and could be very suitable for working with machines in the same datacenter, but worthless for machines thousands of miles apart. 6. active/active/active/... Take #5 and extend the idea to more then a pair of boxes. this makes the updates more complex to propogate (they now need to be sent to every other machine in the cluster) Pro: better load balancing then #5 allows for the ability to have a HA pair in a primary location and a backup in a remote location (i.e. your main HQ has two boxes, but your disaster recovery center has one as well) Con: the complexity goes up significantly when you shift from 2 to n boxes in a cluster. the bandwidth required for updates increases by a factor of roughly n! significantly more split-brain senerios become possible and need to be accounted for. - while #6 is the ideal option to have it can get very complex personally I would like to see #1 (with a sample daemon or two to provide basic functionality and leave the doors open for more creative uses) followed by #3 while people try and figure out all the problems with #5 and #6 there are a lot of senerios that are possible with #1 or #3 that are not possible with #5 and very little of the work needed to release #1 and #3 as supported options is not work that needs to be done towards #5/6 anyway (the pieces need to be identified in the code and hooks put in place in the code at those locations. the details of the hooks will differ slightly David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Funding Cyrus High Availability
On Sun, 19 Sep 2004, David Carter wrote: On Sun, 19 Sep 2004, David Lang wrote: 5. Active/Active designate one of the boxes as primary and identify all items in the datastore that absolutly must not be subject to race conditions between the two boxes (message UUID for example). In addition to implementing the replication needed for #1 modify all functions that need to update these critical pieces of data to update them on the master and let the master update the other box. We may be talking at cross purposes (and its entirely likely that I've got the wrong end of the stick!), but I consider active-active to be the case where there is no primary: users can make changes to either system, and if the two systems lose touch with each other they have to resolve their differences when contact is reestablished. UUIDs aren't a problem (each machine in a cluster owns its own fraction of the address space). Message UIDs are a big problem. I guess in the case of conflict, you could bump the UIDvalidity value on a mailbox and reassign UIDs for all the messages, using timestamps determine the eventual ordering of messages. Now that I think about it, maybe that's not a totally absurd idea. It would involve a lot of work though. the problem is that when they are both up you have to have one of them allocate the message UID's or you have to change the UIDVALIDITY for every new message that arrives. here is the problem. you have a new message created on both servers at the same time. how do you allocate the UID without any possibility of stepping on each other? the only way to do this is to have some sort of locking so that only one machine at a time can allocate UID's. you can shuffle this responsibility back and forth between machines, but there's a significant amount of overhead in doing this so the useual answer is just to have one machine issue the numbers and the other ask the first for a number when it needs it. changing UIDVALIDITY while recovering from a split-brain is probably going to be needed. but as you say it's a lot of work (which is why I'm advocating the simpler options get released first :-) Pro: best use of available hardware as the load is split almost evenly between the boxes. best availability becouse if there is a failure half of the clients won't see it at all Actually this is what I do right now by having two live mailstores. Half the mailboxes on each system are active, the remainder are passive. right, but what this would allow is sharing the load on individual mailboxes useually this won't matter, but I could see it for shared mailboxes David Lang -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Funding Cyrus High Availability
please don't misunderstand my posts. it's not that I don't think that active/active/active is possible, it's just that I think it's far more complicated. assiming that the simplest method would cost ~$3000 to code I would make a wild guess that the ballpark figures would be 1. active/passive without automatic failover $3k 2. active/passive with automatic failover (limited to two nodes or withing a murder cluster) $4k 3. active/passive with updates pushed to the master $5k 4. #3 with auto failover (failover not limited to two nodes or a single murder cluster) $7k 5. active/active (limited to a single geographic location) $10k 6. active/active/active (no limits) $30k in addition to automaticly re-merge things after a split-brin has happened would probably be another $5k now this doesn't mean that all ofs must be done in this funded project. I believe that people would end up going from #1 or #3 to #2 or #4 by individuals coding the required pieces and sharing them (#4 has as much of a jump over #3 becouse of all the different senrios that are involved, each one is individually simple) however #5 and #6 are significantly more difficult and I would not expent them to just happen (they are also much more intrusinve to the code so there is some possibility of them not getting merged into the core code quickly) David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare P.S. #1-4 all could qualify as the first way, #5 and #6 are both complicated enough to start with that it is really hard to keep them out of the second way --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Funding Cyrus High Availability
On Mon, 20 Sep 2004, David Carter wrote: On Sun, 19 Sep 2004, David Lang wrote: assiming that the simplest method would cost ~$3000 to code I would make a wild guess that the ballpark figures would be 1. active/passive without automatic failover $3k 2. active/passive with automatic failover (limited to two nodes or withing a murder cluster) $4k 3. active/passive with updates pushed to the master $5k 4. #3 with auto failover (failover not limited to two nodes or a single murder cluster) $7k 5. active/active (limited to a single geographic location) $10k 6. active/active/active (no limits) $30k in addition to automaticly re-merge things after a split-brin has happened would probably be another $5k I think that you are missing a zero (or at least a fairly substantial multipler!) from 5. 1 -> 4 can be done without substantial changes to the Cyrus core code, and Ken would be able to use my code as a reference implementation, even if he wanted to recode everything from scratch. 5 and 6 would require a much more substantial redesign and I suspect quite a lot of trial and error as this is unexplored territory for IMAP servers. Thanks, this is exactly the type of feedback that I was hopeing to get. so you are saying that #5 is more like $50k-100k and #6 goes up from there Ok folks, how much are you really willing to pay for this and since the amount of work involved translates fairly directly into both cost and time how long are you willing to go with nothing? David Lang -- David Carter Email: [EMAIL PROTECTED] University Computing Service,Phone: (01223) 334502 New Museums Site, Pembroke Street, Fax: (01223) 334679 Cambridge UK. CB2 3QH. --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Funding Cyrus High Availability
On Mon, 20 Sep 2004, Paul Dekkers wrote: David Carter wrote: 5. Active/Active designate one of the boxes as primary and identify all items in the datastore that absolutly must not be subject to race conditions between the two boxes (message UUID for example). In addition to implementing the replication needed for #1 modify all functions that need to update these critical pieces of data to update them on the master and let the master update the other box. We may be talking at cross purposes (and its entirely likely that I've got the wrong end of the stick!), but I consider active-active to be the case where there is no primary: users can make changes to either system, and if the two systems lose touch with each other they have to resolve their differences when contact is reestablished. I'd go for #5 as well: Since this is a setup where there is no primary at all, I suppose this is quite some different design then the #1-4 solutions. And because of that, I would think that it's rather useless to have these steps done in order to get #5 right, but I might as well be wrong. actually I think most of the work nessasary for #1 is also needed for #5-6. for #1 you need to have the ability for a system report all it's changes to a daemon and the ability for a system to read in changes and implement them. #5 needs the same abilities plus the ability to resolve conflicts. the HA steps of #2 and #4 don't gain that much, but they can also be done external to cyrus so it's not a problem to skip them. #3 involves changes to the update code to have cyrus take special actions with soem types of updates. there would need to be changes in the same area for #5, but they would be different. David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: setrlimit: Unable to set file descriptors limit to -1: Operationnot permitted
On Mon, 18 Oct 2004, Luca Olivetti wrote: That's not actually a problem (under linux it's not possible to use -1 and so it retries with the maximum value) and it's not the cause of the "signaled to death by 11". Among other causes it's possible that cyrus is using one version of berkeley db while openldap is using another. what is the maximum value for file decripters? David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Anyone with experience using imapsync
--On Monday, August 21, 2006 10:29:26 PM -0700 Rob Tanner <[EMAIL PROTECTED]> wrote: David, I tried various combinations based on your suggestions. The --prefix2 "INBOX." was the most interesting. Instead of userxyz, I ended up with INBOXxyz. Just a variation on the same problem. One thing I did forget to mention. The saslauthd program is pointing to the shadow file at the moment, and none of the folks (ultimately I need to move about 4,500 accounts) are in the shadow file. I have to eventually switch the pointer to LDAP but that requires some other prep that I'm not yet ready for. But could that possibly explain my problem? I think that the fact that it's dropping the . when it is trying to access the destination folder is the key. when you add the --prefix INBOX. you may have to quote it to have it keep the . in the name. when I get home tonight I'll check the command line I've been useing at home. David Lang -- Rob On 08/21/2006 05:21 PM, David Lang wrote: I'm working with it to copy some things currently, I'm doing it a user at a time, and found that I needed to set --prefix2 "INBOX." (note the . ) to get things to copy properly. it looks like you may need to set --sep or --sep2 to "." as well. also, take a look at imapcopy, if you are just doing a one-way move it may be easier to setup (imapsync can keep the two in sync while you are testing) David Lang On Mon, 21 Aug 2006, Rob Tanner wrote: Hi, I'm trying to migrate mail from one IMAP server to another using the perl program imapsync. Both the source and destination servers are Cyrus IMAP4 v2.2.3 servers. I have added a second partition to the destination server and made it the default by configuring imapd.conf as follows: partition-default: /var/spool/imap partition-belgarath: /var/spool/belgarath defaultpartition: belgarath With this setup, I can use cyradm and by hand correctly adds users to the new partition. When I use imapsync to copy users over. Instead of folders such as user.xyz and user.xyx.sent and user.xyz.drafts, I get userxyz and userxyzsent and userxyzdrafts -- all as separate and entirely independent folders and not even a hierarchy. Here's a script I used just to test and move over one user: # ! /bin/bash ./imapsync \ --host1 belgarath.linfield.edu --user1 cyrus --passfile1 /home/rtanner/imapsync/cyrus.pwd \ --host2 polgara.linfield.edu --user2 cyrus --passfile2 /home/rtanner/imapsync/cyrus.pwd \ --syncinternaldates \ --subscribe \ --include "^user\.aabryan.*$" Any idea what I'm doing wrong? Thanks, Rob Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: IMAP sync tool (rsync for IMAP)
On Thu, 21 Dec 2006, Florin Andrei wrote: I'm currently using two IMAP accounts, one on Cyrus-2.2 the other on Cyrus-2.3. The one on Cyrus-2.3 will get decomissioned so I need to transfer all my email, preserving the folders/subfolders tree, under a specific folder (oldmail/foo/bar) on the 2.2 server. I need to do the bulk of the transfer sometime soon, then sync up again a few times after that, until the day the account on the 2.3 server gets nuked. Essentially, I need a tool that I can point at servers A and B and tell it "get all the email from my account on server A to a specific folder on my account on server B, preserving the subfolders hierarchy". The tool needs to be smart enough to repeat the operation later on but then it must only transfer the new messages. The tool may run on one or the other IMAP servers, or even on a 3rd machine, since it should be network-based. Pretty much all systems are Linux 'round here, some Windows stragglers too. Sort of like rsync for IMAP, if that makes sense. So far, the only tool I've found is imapsync: http://freshmeat.net/projects/imapsync/ Anyone tried it with Cyrus? Good/bad experiences? I just used it to move from 2.1 to 2.3, there were a handful of messages it didn't like (~30 out of a few hundred thousand messages) but it appears to have worked well enough to fix the last few messages manually. most of the errors were cases of invalid headers that 2.3 wouldn't accept, but 2.1 obviously did. David Lang Are there any other tools that work better with Cyrus? Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Patches used at FastMail.FM
On Tue, 9 Jan 2007, Ken Murchison wrote: Disable "anyone" ACL the usual reason for allowing the "anyone" ACL is to allow for + addressing to work. is there another way to do this? in most cases I think that a global 'allow + addressing' config option is really more appropriate then having to configure things on a per-folder basis, possibly with a 'no, don't allow + addressing to this folder' override. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Patches used at FastMail.FM
On Wed, 10 Jan 2007, Rob Mueller wrote: the usual reason for allowing the "anyone" ACL is to allow for + addressing to work. is there another way to do this? The admin user can still set the anyone acl, it's just non-admin users can't change/set it. The way we do this to allow + addressing is when we create the users top level folder, we set the "anyone p" acl on it, and any new folders created after that by the user automatically inherit it. but this is in conflict with the the idea that in a large installation of people who don't know each other the 'anyone' permission doesn't make sense. what is really desired for + addressing is to say that messages that arrive via the lmtp interface are allowed to write to all folders (not just the inbox folders) without allowing other users on the system to write arbatrary data to other people's folders via the IMAP interface. at least if it's arriving via the lmtp interface you have reason to believe that it's been (somewhat) validated by your MTA. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Patches used at FastMail.FM
On Wed, 10 Jan 2007, Rob Mueller wrote: but this is in conflict with the the idea that in a large installation of people who don't know each other the 'anyone' permission doesn't make sense. what is really desired for + addressing is to say that messages that arrive via the lmtp interface are allowed to write to all folders (not just the inbox folders) without allowing other users on the system to write arbatrary data to other people's folders via the IMAP interface. at least if it's arriving via the lmtp interface you have reason to believe that it's been (somewhat) validated by your MTA. That's really what the "p" permission is all about: p - post (send mail to submission address for mailbox, not enforced by IMAP4 itself) So setting "anyone p" means that email via LMTP can be put into any persons folder by the delivery agent, but that folder isn't visible or accessible via any IMAP commands. At least that how I believe it works, and what we've observed. Maybe Ken can clarify? Ok, I thought that 'post' pre-dated lmtp and was the IMAP function to write a message into the folder. i.e. a program like imapsync would need the 'p' permission to write the messages, (but would need other permissions to check for messages, set flags, etc) I'll play around with things a bit while waiting for clarification. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Clustering and replication
On Mon, 29 Jan 2007, Tom Samplonius wrote: - "Bron Gondwana" <[EMAIL PROTECTED]> wrote: On Fri, Jan 26, 2007 at 12:20:15PM -0800, Tom Samplonius wrote: * the system monitoring scripts do a 'du -s' on the sync directory every 2 minutes and store the value in a database so our status commands can see if any store is behind (the trigger for noticing is 10kb, that's a couple of minutes worth of log during the U.S. day). This also emails us if it gets above 100kb (approx 20 mins behind) And what do you do if it gets behind? I have three Cyrus groups right now, that are never going to catch up. They log about 20KB in 20 minutes, so the update rate is not that high. The machines are dedicated, and the replicas aren't doing anything. tcpdump confirms that there is traffic to the replica, but the entire sync_client is so opaque it is hard to see what it is doing. So sync_client can't keep up at all, and since it also quits from time to time, it gets even worse. I'm planning to hack the log, and add some logging to sync_client, particularly to find the number of records per second it is able to process. And then maybe someway to find why it quits all the time. Either that, or my only alternative is to switch to using DRBD to sync the filesystem to a standby server. * a "monitorsync" process that runs from cron every 10 minutes and reads the contents of the sync directory, comparing any log-(\d+) file's PID with running processes to ensure it's actually being run and tries to sync_client -r -f the file if there isn't one. It also checks that there is a running sync_client -r process (no -f) for the store. Wow, a lot of protection to protect against sync_client just exiting. sync_client isn't very big, so it shouldn't be that hard to find the different places that it exits, and fix them? * a weekly "checkreplication" script which logs in as each user to both the master and replica via IMAP and does a bunch of lists, examines, even fetches and compares the output to ensure they are identical. Between all that, I'm pretty comfortable that replication is correct and we'll be told if it isn't. It's certainly helped us find our share of issues with the replication system! Well, I know our replicas are out of sync, so we just don't use them. I just hope the master's don't fail. Each pair has about 30,000 accounts, and about 300GB of online mail. Tom, in your situation you may want to seriously look at disabling fsync. doing so could let your replicas keep up. it's definantly not ideal, but if I was forced to choose between 1. single box with fsync and no replica or 2. master without fsync and replicas without fsync, but up to date I would choose 2, as it won't loose any data due to a master failing, no matter what happens on the master, and I'm only vunerable to something that would take down both the primary and it's replica at the same time (don't have them both on the same UPS!) David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: load balancing at fastmail.fm
On Mon, 12 Feb 2007, urgrue wrote: If it's using block level replication, how does it offer instant recovery on filesystem corruption? Does it track every block written to disk, and can thus roll back to effectively what was on disk at a particular instant in time, so you then just remount the filesystem and the replay of the journal should restore to a good state? Yes. I may be wrong but to my understanding at least NetApp has this capability. No, NetApp takes snapshots of the filesystems on a schedule (hourly, daily, weekly, etc), and you can read files off of those snapshots. you cannot getany more granular then that. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Cyrus with a NFS storage. random DBERROR
On Sat, 9 Jun 2007, Rob Mueller wrote: > So - added is a new option "uuidmode" in imapd.conf. Set it to md5 and you > will get UUIDs of the form: 02(first 11 bytes of the MD5 value for the > message) which takes up the same space, but allows pretty good integrity > checking. > > Is it safe? - we calulated that with one billion messages you have a one in > 1 billion chance of a birthday collision (two random messages with the same > UUID). They then have to get in the same MAILBOXES collection to sync_client > to affect each other anyway. The namespace available for generated UUIDs is > much smaller than this, since they have no collision risk - but if you had > that many delivering you would hit the limits and start getting blank UUIDs > anyway. does the IMAP spec specify how large a UUID can be? David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Global mailbox
On Mon, 30 Jul 2007, Boris Andratzek wrote:
> Lorenzo Milesi wrote:
>> Michael Menge ha scritto:
>>> Hi,
>>>
>>> as far as i know the seen status is managed for each user, but the
>>> other flags such as deleted, spam, or other custom flags are managed
>>> per email and are GLOBAL.
>>
>> Hi
>> Thanks for your reply.
>> Isn't it possible to make also the seen flag global?
>>
>
> Hej Lorenzo,
>
>
> I started a thread with quite the same in here ('mark shared folder's
> mails as read') and in debian.user.de and had to consider that this is
> almost impossible. I also thought about server-side filtering with sieve
> and this might be an option for you but it'll be a lot of work
what you could do as a work-around is make a subfolder 'read' and configure
your
clients to move messages that have been read into that folder.
David Lang
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Virtual Domains
I had all sorts of problems getting this to work.
I have my firewall running sendmail sending the messages to an internal server
via lmtp, and then authenticating against postgres. the biggest problems were
getting the lmtp connection to include the domain of the destination and
makeing
the authentication pass through the domain the user typed in.
David Lang
my cyrus.conf is
asgard dlang # cat /etc/cyrus.conf
# $Header: /var/cvsroot/gentoo-x86/net-mail/cyrus-imapd/files/cyrus.conf,v 1.4
2004/07/18 04:02:23 dragonheart Exp $
# Standard standalone server configuration.
START {
# Do not delete this entry!
recover cmd="ctl_cyrusdb -r"
# This is only necessary if using idled for IMAP IDLE.
idled cmd="idled"
}
# UNIX sockets start with a slash and are put into /var/imap/socket.
SERVICES {
# Add or remove based on preferences.
imap cmd="imapd" listen="imap2" prefork=0
pop3 cmd="pop3d" listen="pop-3" prefork=0
# Don't forget to generate the needed keys for SSL or TLS
# (see doc/html/install-configure.html).
imaps cmd="imapd -s" listen="imaps" prefork=0
pop3s cmd="pop3d -s" listen="pop3s" prefork=0
sieve cmd="timsieved" listen="sieve" prefork=0
# at least one LMTP is required for delivery
lmtp cmd="lmtpd -a" listen="lmtp" prefork=0
lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0
# this is only necessary if using notifications
#notify cmd="notifyd" listen="/var/imap/socket/notify" proto="udp"
prefork=1
}
EVENTS {
# This is required.
checkpointcmd="ctl_cyrusdb -c" period=30
# This is only necessary if using duplicate delivery suppression.
delprune cmd="ctl_deliver -E 3" period=1440
# This is only necessary if caching TLS sessions.
tlsprune cmd="tls_prune" period=1440
}
my imapd.conf
asgard dlang # cat /etc/imapd.conf
# $Header: /var/cvsroot/gentoo-x86/net-mail/cyrus-imapd/files/imapd.conf,v 1.5
2004/08/27 06:02:45 langthang Exp $
# Don't forget to use chattr +S (if you are using ext[23])
# when you change these directories (read the docs).
configdirectory:/var/imap
partition-default: /movies/imap
sievedir: /var/imap/sieve
virtdomains:yes
#defaultdomain lang.hm
#tls_ca_path:/etc/ssl/certs
#tls_cert_file: /etc/ssl/cyrus/server.crt
#tls_key_file: /etc/ssl/cyrus/server.key
# Don't use an everyday user as admin.
admins: cyrus
hashimapspool: yes
allowanonymouslogin:no
allowplaintext: yes
# Allow renaming of top-level mailboxes.
#allowusermoves: yes
# Use this if sieve-scripts could be in ~user/.sieve.
#sieveusehomedir: yes
# Use saslauthd if you want to use pam for imap.
# But be warned: login with DIGEST-MD5 or CRAM-MD5
# is not possible using pam.
#sasl_pwcheck_method: saslauthd
## This is a recommended authentication method if you
## emerge cyrus-sasl with 'postgres' or 'mysql'
## To use with mysql database uncomment those lines below.
sasl_pwcheck_method: auxprop
sasl_auxprop_plugin: sql
## possible values for sasl_auxprop_plugin 'mysql', 'pgsql', 'sqlite'.
sasl_sql_engine: pgsql
## all possible values.
sasl_mech_list: LOGIN PLAIN CRAM-MD5 DIGEST-MD5
## or limit to CRAM-MD5 only
#sasl_mech_list: CRAM-MD5
## change below to suit your setup.
sasl_sql_user: mailuser
sasl_sql_passwd: password
sasl_sql_database: maildb
sasl_sql_hostnames: localhost
sasl_sql_select: SELECT clear FROM users WHERE email = '[EMAIL PROTECTED]'
my sendmail.mc
bifrost:/etc/mail# cat sendmail.mc
define(`_USE_ETC_MAIL_')dnl
include(`/usr/share/sendmail/cf/m4/cf.m4')dnl
VERSIONID(`DI Basebuild 3.1 07-20-05')
OSTYPE(`debian')dnl
DOMAIN(`debian-mta')dnl
dnl # Items controlled by /etc/mail/sendmail.conf - DO NOT TOUCH HERE
undefine(`confHOST_STATUS_DIRECTORY')dnl#DAEMON_HOSTSTATS
dnl # Items controlled by /etc/mail/sendmail.conf - DO NOT TOUCH HERE
FEATURE(`virtusertable',`hash /etc/mail/virtusertable')
VIRTUSER_DOMAIN_FILE(`/etc/mail/virtdomaintable')
FEATURE(`mailertable',`hash /etc/mail/mailertable')
FEATURE(`use_cw_file')
FEATURE(`preserve_local_plus_detail')
FEATURE(always_add_domain)
FEATURE(nouucp,`reject')
define(`confLOCAL_MAILER',`cyrusv2')
define(`CYRUSV2_MAILER_ARGS',`TCP asgard lmtp')
dnl MAILER(`smtp')
MAILER(`cyrusv2')
MAILER(`smtp')
MAILER_DEFINITIONS
Mlmtp, P=[IPC], F=lsDFMnqA@/:|SmXz, E=\r\n,
S=EnvFromL, R=EnvToL/H
Re: need help recovering from disk crash
On Tue, 25 Sep 2007, Scott M. Likens wrote:
> Hi,
>
> If you have a dump of the mailbox's (ctl_mboxlist) then you can restore
> those, personally I back those up weekly as well as /var/spool/imap
I don't think I have that.
> If you don't, re-add the users, then do reconstruct -r -f user.username
> (obviously replace username with the username in question) and it will
> reconstruct the mailbox and find all the folders for you and add them to
> the mailboxes db.
>
> Then do that on all your users, and you should be good.
Ok, that's not bad.
> As far as 2.3.9 vs. 2.2... if you're only dealing with the message
> spool... it should be easy to massage it together. if you're trying to
> bring back duplicate, sieve, seen, etc... it might be a hassle.
it would be really nice to retain the seen state, I don't care about duplicates
and don't have sieve in use. what else would I loose?
> Rather vague email I just wrote... but you seemed to have the basics...
> if not reply all.
thanks.
David Lang
> Scott
>
> [EMAIL PROTECTED] wrote:
>> I lost my OS drive on my home server, the mail partition was on a raid array
>> and survived, I have some of the rest of the config info, but it looks like I
>> lost the configdir contents (the directories are still there, but the files
>> are
>> missing) I may be able to recover some stuff from lost+found if I can get
>> hints
>> on what to search for.
>>
>> now, to make things more interesting, the old system was running gentoo and
>> has
>> cyrus 2.3.recent on it
>>
>> the new system is ubuntu with 2.2.something on it (I couldn't get a recent
>> gentoo installer to run reliably on this hardware)
>>
>> can I make this work or should I compile 2.3.9?
>>
>> with reconstruct -m now working how can I recover the mailbox?
>>
>> the good news is that I only have 3 users on the system (with about 3G of
>> mail
>> in several hundred folders betwen us) so manual fixes may be practical
>>
>> the config files were saved, and are:
>>
>> #cat imapd.conf |grep -v "^#" |grep "^[a-z]"
>> configdirectory:/var/imap
>> partition-default: /movies/imap
>> sievedir: /var/imap/sieve
>> virtdomains:yes
>> admins: cyrus
>> hashimapspool: yes
>> allowanonymouslogin:no
>> allowplaintext: yes
>> sasl_pwcheck_method: auxprop
>> sasl_auxprop_plugin: sasldb
>> sasl_mech_list: PLAIN
>>
>>
>> # cat cyrus.conf |grep -v "^ *#" |grep "[a-z]"
>>recover cmd="ctl_cyrusdb -r"
>>idled cmd="idled"
>>imap cmd="imapd" listen="imap2" prefork=0
>>pop3 cmd="pop3d" listen="pop-3" prefork=0
>>imaps cmd="imapd -s" listen="imaps" prefork=0
>>pop3s cmd="pop3d -s" listen="pop3s" prefork=0
>>sieve cmd="timsieved" listen="sieve" prefork=0
>>lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0
>>checkpointcmd="ctl_cyrusdb -c" period=30
>>delprune cmd="ctl_deliver -E 3" period=1440
>>tlsprune cmd="tls_prune" period=1440
>> [EMAIL PROTECTED]:/etc# cat cyrus.conf |grep -v "^ *#" |grep "[a-z{}]"
>> START {
>>recover cmd="ctl_cyrusdb -r"
>>idled cmd="idled"
>> }
>> SERVICES {
>>imap cmd="imapd" listen="imap2" prefork=0
>>pop3 cmd="pop3d" listen="pop-3" prefork=0
>>imaps cmd="imapd -s" listen="imaps" prefork=0
>>pop3s cmd="pop3d -s" listen="pop3s" prefork=0
>>sieve cmd="timsieved" listen="sieve" prefork=0
>>lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0
>> }
>> EVENTS {
>>checkpointcmd="ctl_cyrusdb -c" period=30
>>delprune cmd="ctl_deliver -E 3" period=1440
>>tlsprune cmd="tls_prune" period=1440
>> }
>>
>> David Lang
>>
>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>>
>>
>> !DSPAM:46f8b94b179948275421122!
>>
>>
>>
>
>
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: need help recovering from disk crash
On Tue, 25 Sep 2007, David Lang wrote:
> On Tue, 25 Sep 2007, Scott M. Likens wrote:
>
>> Hi,
>>
>> If you have a dump of the mailbox's (ctl_mboxlist) then you can restore
>> those, personally I back those up weekly as well as /var/spool/imap
>
> I don't think I have that.
>
>> If you don't, re-add the users, then do reconstruct -r -f user.username
>> (obviously replace username with the username in question) and it will
>> reconstruct the mailbox and find all the folders for you and add them to
>> the mailboxes db.
>>
>> Then do that on all your users, and you should be good.
>
> Ok, that's not bad.
I'm running into a problem with this, I can do this for one domain, but not for
a second.
David Lang
>> As far as 2.3.9 vs. 2.2... if you're only dealing with the message
>> spool... it should be easy to massage it together. if you're trying to
>> bring back duplicate, sieve, seen, etc... it might be a hassle.
>
> it would be really nice to retain the seen state, I don't care about
> duplicates
> and don't have sieve in use. what else would I loose?
>
>> Rather vague email I just wrote... but you seemed to have the basics...
>> if not reply all.
>
> thanks.
>
> David Lang
>
>> Scott
>>
>> [EMAIL PROTECTED] wrote:
>>> I lost my OS drive on my home server, the mail partition was on a raid array
>>> and survived, I have some of the rest of the config info, but it looks like
>>> I
>>> lost the configdir contents (the directories are still there, but the files
>>> are
>>> missing) I may be able to recover some stuff from lost+found if I can get
>>> hints
>>> on what to search for.
>>>
>>> now, to make things more interesting, the old system was running gentoo and
>>> has
>>> cyrus 2.3.recent on it
>>>
>>> the new system is ubuntu with 2.2.something on it (I couldn't get a recent
>>> gentoo installer to run reliably on this hardware)
>>>
>>> can I make this work or should I compile 2.3.9?
>>>
>>> with reconstruct -m now working how can I recover the mailbox?
>>>
>>> the good news is that I only have 3 users on the system (with about 3G of
>>> mail
>>> in several hundred folders betwen us) so manual fixes may be practical
>>>
>>> the config files were saved, and are:
>>>
>>> #cat imapd.conf |grep -v "^#" |grep "^[a-z]"
>>> configdirectory:/var/imap
>>> partition-default: /movies/imap
>>> sievedir: /var/imap/sieve
>>> virtdomains:yes
>>> admins: cyrus
>>> hashimapspool: yes
>>> allowanonymouslogin:no
>>> allowplaintext: yes
>>> sasl_pwcheck_method: auxprop
>>> sasl_auxprop_plugin: sasldb
>>> sasl_mech_list: PLAIN
>>>
>>>
>>> # cat cyrus.conf |grep -v "^ *#" |grep "[a-z]"
>>>recover cmd="ctl_cyrusdb -r"
>>>idled cmd="idled"
>>>imap cmd="imapd" listen="imap2" prefork=0
>>>pop3 cmd="pop3d" listen="pop-3" prefork=0
>>>imaps cmd="imapd -s" listen="imaps" prefork=0
>>>pop3s cmd="pop3d -s" listen="pop3s" prefork=0
>>>sieve cmd="timsieved" listen="sieve" prefork=0
>>>lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0
>>>checkpointcmd="ctl_cyrusdb -c" period=30
>>> delprune cmd="ctl_deliver -E 3" period=1440
>>>tlsprune cmd="tls_prune" period=1440
>>> [EMAIL PROTECTED]:/etc# cat cyrus.conf |grep -v "^ *#" |grep "[a-z{}]"
>>> START {
>>>recover cmd="ctl_cyrusdb -r"
>>>idled cmd="idled"
>>> }
>>> SERVICES {
>>>imap cmd="imapd" listen="imap2" prefork=0
>>>pop3 cmd="pop3d" listen="pop-3" prefork=0
>>>imaps cmd="imapd -s" listen="imaps" prefork=0
>>>pop3s cmd="pop3d -s" listen="pop3s" prefork=0
>>>sieve cmd="timsieved" listen="sieve" prefork=0
>>>lmtpunix cmd="lmtpd" listen="/var/imap/socket/lmtp" prefork=0
>>> }
>>> EVENTS {
>>>checkpointcmd="ctl_cyrusdb -c" period=30
>>>delprune cmd="ctl_deliver -E 3" period=1440
>>>tlsprune cmd="tls_prune" period=1440
>>> }
>>>
>>> David Lang
>>>
>>> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
>>> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
>>> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>>>
>>>
>>> !DSPAM:46f8b94b179948275421122!
>>>
>>>
>>>
>>
>>
>
> Cyrus Home Page: http://cyrusimap.web.cmu.edu/
> Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
>
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: need help recovering from disk crash
On Wed, 26 Sep 2007, Rudy Gevaert wrote: > David Lang wrote: >> On Tue, 25 Sep 2007, David Lang wrote: >> >>> On Tue, 25 Sep 2007, Scott M. Likens wrote: >>> >>>> Hi, >>>> >>>> If you have a dump of the mailbox's (ctl_mboxlist) then you can restore >>>> those, personally I back those up weekly as well as /var/spool/imap >>> I don't think I have that. >>> >>>> If you don't, re-add the users, then do reconstruct -r -f user.username >>>> (obviously replace username with the username in question) and it will >>>> reconstruct the mailbox and find all the folders for you and add them to >>>> the mailboxes db. >>>> >>>> Then do that on all your users, and you should be good. >>> Ok, that's not bad. >> >> I'm running into a problem with this, I can do this for one domain, but not >> for a second. > > I don't know if 2.2 supports virtual domains. You'd better check that. I > think there is a bug in reconstruct with virtual domains. I have to run: > > reconstruct -r -f user/[EMAIL PROTECTED] > and > reconstruct -r -f user/login/[EMAIL PROTECTED] > > (I'm also using unix hierarchy seperator, but you aren't I think.) I tracked this down last night. Since I was also fighting SASL (my old SASL had been connected to a postgres database, the new one wasn't compiled to support that, so I was having to learn a different way to set passwords) I had gotten [EMAIL PROTECTED] to work and be able to login, and apparently if you login with one virtual domain you can't do anything to any other virtual domain, even if you are listed as the admin user. I setup a password for the user cyrus and set it to be the admin user and was able to work with all the different domains. I did run into one other bug that I couldn't figure out. no user was able to modify their INBOX (delete items, etc). I could copy messages to folders, but not delete anything. to get back online (and since this is just my home machine) I ended up setting anyone all acl's on every inbox. it sounds like I really need to upgrade to 2.3.x, I downloaded it, but haven't compiled it yet. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: LARGE single-system Cyrus installs?
On Tue, 9 Oct 2007, Andrew Morgan wrote: > On Sat, 6 Oct 2007, Rob Mueller wrote: > >> As it turns out, the memory leaks weren't critical, because the the pages do >> seem to be reclaimed when needed, though it was annoying not knowing exactly >> how much memory was really free/used. The biggest problem was that with >> cyrus you have millions of small files, and with a 32bit linux kernel the >> inode cache has to be in low memory, severely limiting how many files the OS >> will cache. >> >> See this blog post for the gory details, and why a 64-bit kernel was a nice >> win for us. >> >> http://blog.fastmail.fm/2007/09/21/reiserfs-bugs-32-bit-vs-64-bit-kernels-cache-vs-inode-memory/ > > Yesterday I checked my own Cyrus servers to see if I was running out of > lowmem, and it sure looked like it. Lowmem had only a couple MB free, and > I had 2GB of free memory that was not being used for cache. > > I checked again today and everything seems to be fine - 150MB of lowmem > free and almost no free memory (3GB cached)! Grrr. > > Anyways, I was looking into building a 64-bit kernel. I'm running Debian > Sarge (I know, old) on a Dell 2850 with Intel Xeon (Nocona) CPUs and 4GB > RAM. My kernel version is 2.6.14.5, built from kernel.org sources. It > has "High Memory Support (64GB)" selected. > > When I run menuconfig, I'm not seeing any obvious place to switch from > 32-bit to 64-bit. Could you elaborate a bit about how you switched to a > 64-bit kernel? Also, are you running a full 64-bit distro, or just a > 64-bit kernel? you need a full 64 bit toolchain to compile a 64 bit kernel, the easy way to do this is to compile the kernel on a 64 bit distro. if you have the toolchain you can add arch=x86_64 to your make command if you are not converting everything over to 64 bit remember to enable 32 bit userspace (cyrus won't take advantage of all the ram, but the kernel will, so it's definantly still a win) with some older versions of the iptables binaries you can run into trouble with a 64 bit kernel and 32 bit userspace. unless you take the time to make sure that you aren't running versions that have this problem don't execute any iptables commands when running in mixed mode. David Lang Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: UC Davis Cyrus Incident September 2007
Omen Wild (University of California Davis) The root problem seems to be an interaction between Solaris' concept of global memory consistency and the fact that Cyrus spawns many processes that all memory map (mmap) the same file. Whenever any process updates any part of a memory mapped file, Solaris freezes all of the processes that have that file mmaped, updates their memory tables, and then re-schedules the processes to run. When we have problems we see the load average go extremely high and no useful work gets done by Cyrus. Logins get processed by saslauthd, but listing an inbox either takes a long time or completely times out. Apparently AIX also runs into this issue. I talked to one email administrator that had this exact issue under AIX. That admin talked to the kernel engineers at IBM who explained that this is a feature, not a bug. They eventually switched to Linux which solved their issues, although they did move to more Linux boxes with fewer users per box. Oh man... Horrible memories just flood right back... Wow. I was reading your e-mail and thinking to myself that this sounded like the same problem we had. Then I got to the above section and *bam*, there it was... We had significant problems with our e-mail last year (this year was a perfect start!) a week before students came back. We didn't resolve the problems until the end of September and we were dismayed at our final solution. We run Tru64 5.1b on a 4 member cluster. Tru64's kernel suffers from the same exact issue as described above. We have regularly 12,000 cyrus procs running at any one time during the day, and that cluster also receives on average 300k-500k e-mails each day (that is after spam/virus work). What was finally identified was that the number of "processes" that were mapped to that single physical "executable" (/usr/cyrus/imapd) was causing a lot of lock contention in the kernel. The executable would have a link list of all the processes running off of it in kernel memory. When one of the processes would go away, the kernel would start at the beginning of the list and search for the process in order to clean up its resources. During that time, the kernel would lock everything and execution would essentially stop for everything (basically, the whole system appeared to simply freeze on us). The kernel would reach a time threshold and stop in order to let other things happen (unfreeze). This time was very short, but if we had a lot of processes going away in a very short period of time, we would noticeably see the freeze, since the kernel was going into this lock-down mode a lot in a very short period of time. That is a simplified view of what really happened. could someone whip up a small test that could be used to check different operating systems (and filesystems) for this concurrancy problem? it doesn't even need to use any cyrus code, (in fact it would probably be better if it didn't) it sounds like there are a couple different aspects to check 1. large number of copies of a single program running, find the impact of starting and stopping a process 1a. single process that forks lots of copies 1b. master process that execs lots of copies 2. large number of processes mmapping a single file. 2a. impact to add or remove a process from this group 2b. impact on modifying this file personally I expect 1b and 1a to be significantly different on different OSs. some OSs will gain huge memory savings in 1a due to copy-on-write savings (and to partially account for this it may be worth making the program allocate a chink of ram and write to it after the fork), while on other OSs the overhead of multiple mappings of a page will dominate. David Lang --On Tuesday, October 16, 2007 3:39 PM -0700 Vincent Fox <[EMAIL PROTECTED]> wrote: Omen Wild (University of California Davis) The root problem seems to be an interaction between Solaris' concept of global memory consistency and the fact that Cyrus spawns many processes that all memory map (mmap) the same file. Whenever any process updates any part of a memory mapped file, Solaris freezes all of the processes that have that file mmaped, updates their memory tables, and then re-schedules the processes to run. When we have problems we see the load average go extremely high and no useful work gets done by Cyrus. Logins get processed by saslauthd, but listing an inbox either takes a long time or completely times out. Apparently AIX also runs into this issue. I talked to one email administrator that had this exact issue under AIX. That admin talked to the kernel engineers at IBM who explained that this is a feature, not a bug. They eventually switched to Linux which solved their issues, although they did move to more Linux boxes with fewer users per box. Oh man... Horrible memories just flood right back... Wow. I was reading your e-mail and thinking
Re: looking for Cyrus mail format documentation
you stated that you want to have the outside box act as a secondary MX for the inside one, if you do this and accept the extra bandwidth used then you could still do this and have the mail only delivered to the inside box and then replicated out to the outside one. this doesn't solve the problem of changing flags, but does solve the problem of getting the messages in correctly. for the flags the real question is do you HAVE to allow them to be updated when the primary can't be reached? or can your users tolorate being able to see their mail, but not have the flags change if you have a connection problem? (or possibly allow some flags to be changed and queued up, seen flags can be reconsiled by changing both sides to the the or of the two when they reconnect, deletes can be queued and processed later, etc) there are some useage patterns here that can narrow the scope of the limitation more then the generic database two-way-sync problem David Lang On Tue, 4 Feb 2003, Phil Howard wrote: > Date: Tue, 4 Feb 2003 03:19:12 -0600 > From: Phil Howard <[EMAIL PROTECTED]> > To: Rob Siemborski <[EMAIL PROTECTED]> > Cc: [EMAIL PROTECTED] > Subject: Re: looking for Cyrus mail format documentation > > On Tue, Feb 04, 2003 at 02:16:36AM -0500, Rob Siemborski wrote: > > | On Tue, 4 Feb 2003, Phil Howard wrote: > | > | > Does the RFC say that the IMAP UIDs have to be the file name? > | > | No, of course not. > | > | > Do the IMAP UIDs have to be the same between different sessions? > | > | They cannot change without also chanigng the UIDVALIDITY of the mailbox, > | which is an expensive operation for disconnected clients (it forces them > | to resync) > | > | So yes, every time you need to resync, you can increment the uidvalidity, > | but your disconnected users are going to hate you for it, and this isn't a > | tremendously good solution for the real world (where temporary outages > | between distant nodes is the norm). > > So the message with UID 123 during one session has to still have UID 123 > during the next session. That indeed will break the ability to have > unique remote syncronization. > > What's curious to me is how, with a Maildir format, that IMAP could be > implemented to retain that state without either storing some extra data > or updating the files in place. I had thought that real unique message > IDs were the same as in RFC 822. I didn't read RFC 2060 because I had > been talked out of implementing my own IMAP daemon. But I guess I > should have read it, anyway, to understand its limitations. Probably > better do that soon before I design something else that can't work :-) > > -- > - > | Phil Howard - KA9WGN | Dallas | http://linuxhomepage.com/ | > | [EMAIL PROTECTED] | Texas, USA | http://ka9wgn.ham.org/| > - >
Re: Cyrus process model...
as someone attempting to get apache 2 running (reliably) in a high volume environment I can say the idea is interesting, but I definantly wouldn't rush into useing it. if you have some time and want to get a start on something that may (or may not) be worth doing in the long run you can start on it, but don't stop maintaining the current version, the apache core code may not be the right thing in the long run. David Lang On Wed, 26 Feb 2003, Rob Mueller wrote: > Date: Wed, 26 Feb 2003 16:45:00 +1100 > From: Rob Mueller <[EMAIL PROTECTED]> > To: Lawrence Greenfield <[EMAIL PROTECTED]>, > Rob Siemborski <[EMAIL PROTECTED]> > Cc: Ken Murchison <[EMAIL PROTECTED]>, > info-cyrus <[EMAIL PROTECTED]> > Subject: Cyrus process model... > > [ Continued from an off mailing list conversation about killing cyrus lmtpd > processes when they go haywire, and cyrus process accounting ] > > > > Surely this is a relatively well solved problem? Just about every unix > > > system uses this master/forked child approach? How does apache do it? > > > Net::Server::PreFork? I can't imagine that there aren't cookbook > solutions > > > to this issue since it's what unix has been doing for 30 years? Or is > there > > > something I'm missing here? > > > There are many different possibilities. Most other systems limit the > > number of clients instead of forking new processes on demand without a > > set limit. Apache also doesn't have differentiated children or > > substantial shared state. (All children are members of the same > > service or you don't particularly care how many additional unused children > > you have...) > > I was under the impression that Apache 2 was planning on making it's > forking/threading model much more generic, and supporting a general > 'services' model, including a library to abstract the underlying OS? Hmmm, > looking into that, it appears that it's mostly done already. > > http://apr.apache.org/ > http://apr.apache.org/apr2_0intro/apr2_0intro.htm > > And more: > > Contains following functionality > -Reading and writing of files > -Character set conversion > -Network communications using sockets > -Time management used for Internet type conversions > -String management like C++ including natural order management > -UNIX Password management routines > -Table management routines > -UUID Internet generation > -Filename canonicalization > -Random data generation > -Global lock management > -Threads and process management > -Dynamic library loading routines > -Memory mapped and shared memory > > - > > http://www.arctic.org/~dean/apache/2.0/process-model.html > > I think the above is general enough to implement the interesting process > models, and to implement optimizations that are available only in some of > the multi-threaded models. Note that nothing above is specific to HTTP, and > I believe that we should strive to keep the abstraction so that the same > libraries can be used to implement other types of servers (i.e. FTP, > streaming video/audio, corba). > > - > > Would the cyrus team think it worthwhile to consider refactoring to use the > new Apache 2 APR modules? I know off hand that it would be a lot of work, > but it could be a gradual re-factoring process, and the idea of actually > reusing code between projects would be *really* nice. > > Joel Spolsky is a big proponent of refactoring over time to improve software > and you can read some of his thoughts here. > > http://www.joelonsoftware.com/articles/fog69.html > http://www.joelonsoftware.com/news/fog000328.html > > Ooops, I'm feeling a rant come along... > > *** RANT MODE *** > > I know this is a little off topic, but the source for cyrus is really > showing it's age a bit. I know that happens with all software, you start > with certain assumptions, and the more you go on, the more the original > assumptions get blown away, so you hack this in here, and there, and then > every now and then, you go on a big cleanup spree! The problem I feel is > that the cleanup hasn't been big enough or often enough. > > Also, over time programming habits change. Many old C idioms are pretty much > dead. Most of the C string handling methods are now annoying, or downright > dangerous. There are several dozen replacement libraries, including the APR > one above, and good ones like > http://www.annexia.org/freeware/c2lib/index.msp. This library also > implements automatically resizing arrays and memory pools, a common way to > avoid all subtle leaks introduced by malloc() and the like, and to avoid th
Re: Cyrus process model...
I will also add that on current *nix systems the advantages of threads over processes is a lot less then it used to be. In my case we are running apache2 on AIX and found no noticable difference between the two (so we are useing processes for the stability reasons you note below) David Lang On Thu, 27 Feb 2003, Rob Mueller wrote: > > It is always a big pain to update code that was never written to be > > threaded, to be thread-safe. Apache2 has a problems with just about every > > third party module supported under Apache 1.3. I imagine that Cyrus would > > have all sorts of thread issues. There is no magic solution for that. > > I'm not convinced that it's necessary to make it thread safe. In many > situations I think threads are a step backwards. While it always feels a bit > odd to think of it as a positive, the multiple process model introduces an > inherent stability, even for non-optimal (buggy) code that can crash a > process. In that case, only one connection/instance is lost, and no-one else > is affected. In multithreaded code, one bad crashed thread *can* take out > the entire process and all connections. Of course, if your code has to share > a lot of information between each 'instance', then threads are very useful. > > In the case of cyrus, I think you can quite happily stick with the > multi-process model, I wasn't advocating moving to a threaded model. The > discussion started due to an issue with killing child processes. Apparently > there are currently race conditions in 'master' that means that a killed > child may not be correctly recognised by the 'master' process as a dead > child. I commented that I thought a master/forked child idiom had been used > in unix for 30 years, and shouldn't there be cookbook solutions for most of > these issues? Which started me looking for libraries that might have already > done this... > > > Besides, if anyone really wants to take Cyrus to the next generation, > > create a new NG branch in CVS (on your own CVS server if necessary), and > > start "refactoring" away. (Of course, "refactoring" has to be the most > > overused term in software development at the moment, and is touted as a > > solution for everything from bad design, to poor management). > > The thing is, in my experience, refactoring actually works, regardless of > it's buzz word of the week or not. Better yet, *continuous* refactoring > seems to work the best! Hmmm, not that I find that easy to define. I guess > it's being aware as you work on a project, which parts are clearly beginning > to feel 'wrong' (hmmm, more subjective thoughts...), and devoting some time > to actually fixing up those problem areas. This is generally a lot easier if > you're good at creating interfaces and sticking to them. Of course, being > forced to work around an interface is one of the clear signs of something > being 'wrong'. > > Rob >
Re: 1. 2., etc., files in user. directory
in netscape I think the option is called something like 'compact folder' David Lang On Sun, 24 Dec 2000, William K. Hardeman wrote: > Date: Sun, 24 Dec 2000 02:31:21 -0500 > From: William K. Hardeman <[EMAIL PROTECTED]> > To: Brian Capouch <[EMAIL PROTECTED]>, [EMAIL PROTECTED] > Subject: Re: 1. 2., etc., files in user. directory > > Brian, > > Cool. Someone else that's using a Slackware system. Mine is > Slackware-current. :) > > It sounds to me as if Netscape isn't doing an expunge. From my > understanding of the IMAP protocol and how Cyrus implements it, no messages > are actually deleted until an expunge operation is requested by the client. > What happens is that the Delete flag is set, only. I think that both > Netscape and Pine aren't seeing them because the Delete flag has been set > in some prior session, but I could well be wrong. (If I am, I hope someone > will correct me.) > > I'm not sure what might be happening here, but one suggestion is to try > using a client that allows Deleted messages to be explicitely Expunged, and > see what happens to your mail store on disk, then. One such client is > Mulberry, which has Windows, Mac and Linux versions available. I don't > really know of any others, although I'm sure there are some. > > If the explicitely called Expunge works, then I'd say it's a client issue. > If the Expunge does not work, and the messages you expect to have been > deleted are still on your drive, then there might be some mis-configuration > or bug somewhere in your Cyrus install. I've not had any problems with mine > that I've noticed, so far. > > Hope this helps, > Will > William Hardeman > [EMAIL PROTECTED] > > --On Saturday, December 23, 2000 21:09:33 -0500 Brian Capouch > <[EMAIL PROTECTED]> wrote: > > > I just finished installing cyrus 2.0.9 on a Slackware Linux system. I > > had the same problems reported earlier today getting cyradm to install > > properly; I finally resorted to going to another machine running an > > earlier beta (2.0.7) and then manually copying files out of the ~/perl > > directory into the various perl5 system locations. Now it works like a > > charm. > > > > System, if it's germane to anyone, is Linux Slackware 7.0 running > > 2.4.0-test12. > > > > Now my question: I am the only user on this box for the time being. I > > want to make sure things are stable before moving any of my other users > > over. > > > > I seem to be able to receive messages just fine, and sieve is running > > (out of my home directory). However, I notice that the various messages > > I have received, then moved to the Trash and then later "emptied" out > > via Netscape, are still there in the cyrus user directory where they > > were initially delivered. Stated different, even though my IMAP client > > can't see them, *I* can if I manually go into the cyrus directory space > > and "ls" > > > > Is this to be expected? They don't show up when I look at my IMAP mail > > from either Netscape or Pine, and I'm hoping perhaps that they get > > purged by some other entity. > > > > But I don't think 2.0.7 did that. Can anyone tell me for sure? > > > > Thanks. > > > > B. > > > > > >
Re: Session-usage of Outlook Express
I think this is just outlook showing it's POP3 roots. there is a bunch of mail software out there that was written for POP3 that has been 'upgraded' to IMAP but still acts as if it was talking to a POP3 server (including in many cases keeping a local copy of the mail messages) David Lang On Thu, 22 Mar 2001, Anton Roeckseisen wrote: > Date: Thu, 22 Mar 2001 11:01:21 +0100 > From: Anton Roeckseisen <[EMAIL PROTECTED]> > To: [EMAIL PROTECTED] > Subject: Session-usage of Outlook Express > > Hi, > > I'm trying the cyrus-server and found that outlook express is logging in and > out to check for new mail availlable on the imap-server every few minutes > instead of keeping the session open for several hours and checking in the open > sesson. > > I'm not used to M$... so does anyone know if I can configure outlook express > to keep the session open? And further and even more important: is this kind of > behaviour the expected behaviour of imap-clients? Or is outlook express just > an exception? > > Thanks, > Anton > >
Re: best filesystem for imap server
I've done some testing and seen a HUGE speedup when switching from EXT2/3 to XFS. unfortunantly I haven't had a chance to do the same comparison with Reiserfs (I need to, but haven't had time) I was even able to see a dramatic difference with a single user accessing a fairly large mailbox (thousands of messages in the inbox) David Lang On Wed, 1 Dec 2004, John Madden wrote: Date: Wed, 1 Dec 2004 13:12:57 -0500 (EST) From: John Madden <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: best filesystem for imap server I dont want to start a religious battle, but could I have some opinions on filesystems for a 100ish user imap server? I have 2x 250G western digital disks to use. I think the performance of those disks (and the RAID you put on them) will be much more significant that the filesystem you use, considering the size of your user population. And given that factor, I'd say that even ext3 won't give you any problems performance-wise. Still, reiserfs, IMO, would be preferable for mail files. John -- John Madden UNIX Systems Engineer Ivy Tech State College [EMAIL PROTECTED] --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: best filesystem for imap server
On Wed, 1 Dec 2004, Jim Miller wrote: I feel that XFS is a bad choice since it is not a 'truly' journaled file system. If you have a power failure/system crash/lockup, etc., etc. You could very easily end up with a corrupt file system -- XFS doesn't write out to the disks immediately (caching unwritten data to memory). EXT3 is journaled but very slow. ReiserFS is a better choice for a journaled file system and if you can hold off until all the bugs are worked out, Reiser4FS would be the best choice (IMHO). note that most journaling filesystems journal the metadata, not the file data (and ext3 does this as well by default, but it has a mode to enable journaling everything) and actually ext3 had the option to journal everything becouse in the initial implementation the peopel writing the code couldn't seperate the two types of data so to simplify things they journaled everything. the reason that not everything is journaled is a simple performance issue. having to write the data to the journal, read it from the journal and write it to the final location, then update the journal requires a LOT more IO bandwidth then if you just do this for the metadata. personally I have trouble trusting reiserfs ever since it was revealed that one reason that it was doing so well on benchmarks is that it delaye up to 30 seconds before writing anything to disk so in many cases the benchmark was completed before any disk activity took place. This has been changed, but it leaves a bad taste behind. also note that if you are useing IDE drives you have no way of really knowing when the data has hit the platter (as opposed to just being in the buffer of the drive) as many of the drives will lie to you and tell you the write is complete once it hits the buffers. David Lang Jim --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: best filesystem for imap server
On Thu, 2 Dec 2004, Jules Agee wrote: Date: Thu, 02 Dec 2004 10:11:21 -0800 From: Jules Agee <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Subject: Re: best filesystem for imap server David Lang wrote: also note that if you are useing IDE drives you have no way of really knowing when the data has hit the platter (as opposed to just being in the buffer of the drive) as many of the drives will lie to you and tell you the write is complete once it hits the buffers. I think they use capacitors that will hold enough charge to allow flushing the buffers to disk when there's a power loss. they used to, but nowdays when the bugger is 8M (or larger), potentially with many seeks they don't have any capacitors large enough to hold that much power (disassemble a failed drive sometime and try to find any significant capacitors in it) David Lang -- Jules Agee System Administrator Pacific Coast Feather Co. [EMAIL PROTECTED] x284 --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: best filesystem for imap server
On Thu, 2 Dec 2004, John Madden wrote: Date: Thu, 2 Dec 2004 14:53:07 -0500 (EST) From: John Madden <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: best filesystem for imap server I think they use capacitors that will hold enough charge to allow flushing the buffers to disk when there's a power loss. And another set of caps to keep the spindles spinning so that data can be written? I'm not yet willing to buy the bridge you're selling. :) 10 or so years ago when the drives had significantly more rotating mass and significantly lower data density there were (high-end SCSI) drives that could use their rotational energy to power their electronics to write the data and adjust the dataclock as the spindle slowed, but I don't think any drive does this anymore. David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Can I use Cyrus IMAP w/Outlook?
Paul, I recently took a look at useing thunderbird 1.0 with IMAP and found that it was storing a lot of info locally, is it really that good an IMAP client? David Lang On Wed, 12 Jan 2005, Paul Dekkers wrote: Date: Wed, 12 Jan 2005 10:17:10 +0100 From: Paul Dekkers <[EMAIL PROTECTED]> To: Thomas Kessler <[EMAIL PROTECTED]> Cc: info-cyrus@lists.andrew.cmu.edu Subject: Re: Can I use Cyrus IMAP w/Outlook? Thomas Kessler wrote: Can I use Cyrus IMAP w/Outlook? Is this even possible? I setup Cyrus assuming that I could, but I havent been able to find any saying that is works, just that Insight Server application that works with Cyrus IMAP. If I cant use Cyrus with Outlook, can someone recommend an IMAP server that will? Of course you can use it. It's not the best IMAP client in the world (I would rather use (and support) Mozilla/Thunderbird ;-) or, ok, maybe even rather Outlook Express then Outlook because of its IMAP support), but it surely works. Just set up an IMAP account with the correct account info & credentials. Paul P.S. If authentication fails against the Cyrus server you should probably tune your sasl_mech_list to something like "PLAIN LOGIN" depending on your backend ofcourse. --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Once again: action hooks
Take a look at Popfile (on sourceforge) it will train when you move a message into a folder and retrain when you move a message to another folder. the current version is single user (you would need to run one copy per user), but the development version is adding multi-user capability) David Lang On Tue, 25 Jan 2005, Denis Jedig wrote: Date: Tue, 25 Jan 2005 12:19:27 +0100 From: Denis Jedig <[EMAIL PROTECTED]> To: info-cyrus@lists.andrew.cmu.edu Subject: Once again: action hooks Greetings, list. I already crawled the web, checked the docs and posted to the local newsgroups, but I am still lacking a solution for my problem: I want to train bayesian filters via putting messages into and removing them out of IMAP mailboxes [*] I considered several ways to achieve this: a) start the learning process for everything in an IMAP mailbox periodically (cron) b) use notifications for starting the learning process However, the problem for both approaches is: there is no clean way to "unlearn" a message if it is moved out (not deleted) of a mailbox. So I would like to know, if there is some way to implement notifications on message move actions. Thank you in advance. [1] this seems the most convinient way to do so for the users. Forwarding messages to a specific e-mail address does not work really well - we would need two e-mail addresses per user, per bayes bucket, one for "learn", one for "unlearn" and the users do not understand this concept. Denis Jedig --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Large email account
On Mon, 20 Dec 2004, Prasanna Buddhika wrote: Date: Mon, 20 Dec 2004 16:14:17 -0800 (PST) From: Prasanna Buddhika <[EMAIL PROTECTED]> To: info-cyrus@lists.andrew.cmu.edu Subject: Large email account Hi, I have win2000 server running exchange 2000 server. There are 2 large email accounts (70 emails in a single account) which we don't want to delete. But I want to move these 2 accounts to Linux machine with cyrus-imap server. Migration went very well. Only problem is the speed. When the 2 mail accounts on the exchange server and when I use exchange method to access the account via Microsoft outlook client it won't take even 1 minute to show all the massages. But if I use cyrus-imap to retriew (sync) the email it takes 30 minutes. But after sync it's fast. Only problem is when ever I logged off the system and log back in to the system again I have to wait 30 minutes to access the mail. Can anyone have a good suggestion to speedup this apart from deleting emails.:) Thank you, Prasan What filesystem did you use on the linux box? the ext2/ext3 filesystems slow down significantly when you have large files or directories with large numbers of files in them, and you are definantly far beyone the point where they will slow down (around 10,000 messages they start to slow down and ~15,000-20,000 they become very bad) if you want to use ext2/ext3 you will need to reasearch and implement the htree filesystem extention and see how well it helps you. Otherwise try converting to XFS or Reiserfs (my personal prefrence is XFS, but test for yourself) one way to tell that it's a server-side problem is to login to the server, cd to the directory for the mailbox and do a ls, you will be amazed at how long it will take to respond. once you get the server working well then the next step is to get a better client then outlook as the other posts have pointed out, but if you were to switch to another client without fixing the server you would find that every action you take is very slow (now moving from outlook you may not notice quite how slow, as outlook is one of the slower clients to start with) David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
RE: Database backend?
I would be interested in this, thanks. David Lang On Wed, 11 May 2005, ¿øÅÂȯ wrote: Date: Wed, 11 May 2005 23:29:55 +0900 From: ¿øÅÂȯ <[EMAIL PROTECTED]> To: Markus Heller <[EMAIL PROTECTED]> Cc: info-cyrus@lists.andrew.cmu.edu Subject: RE: Database backend? Hi, I had an experience to implement mysql backend to store mailbox paths. The mysql backend was intended for special purpose. But it would help you to start to make your DBMS backend implementation. Let me know if you want it. ;-) Tawan Won _ From: [EMAIL PROTECTED] ÀÌ(°¡) ´ÙÀ½ »ç¶÷ ´ë½Å º¸³¿ Markus Heller Sent: 2005-05-11 (¼ö) ¿ÀÈÄ 8:01 To: info-cyrus@lists.andrew.cmu.edu Subject: Database backend? Dear List, has anybody tried to store the contents of the imap repository in a database like postgres? Is there an interface for postgres or mysql? Thanks in advance, Markus --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus <http://asg.web.cmu.edu/cyrus> Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu <http://cyruswiki.andrew.cmu.edu> List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html <http://asg.web.cmu.edu/cyrus/mailing-list.html> -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Imap timeout with 27k messages...
the problem is reated to the speed of accessing evrything on the filesystem. there's no easy fix for this (other then possibly extending the timeouts) to 'fix' this you could create a new folder and manually copy a bunch of the messages to that new folder then run reconstruct on both the old and new folders. beyond that do some testing with huge message folders on different filesystems. you may find that other filesystems handle huge folders better then what you're useing. David Lang On Mon, 8 Aug 2005, Jared Watkins wrote: Date: Mon, 08 Aug 2005 13:26:31 -0400 From: Jared Watkins <[EMAIL PROTECTED]> Reply-To: Jared Watkins <[EMAIL PROTECTED]> To: info-cyrus@lists.andrew.cmu.edu Subject: Imap timeout with 27k messages... Hello all.. I have a situation here where an 'exempt' user has accumulated nearly 27k messages and 1.5G of mail in their sent items folder and now any attempt to access this folder has imap timeout problems and stuck processes. The cyradm utility is also not able to work with the folder... attempts to rename.. or reconstruct it results in a stuck process. This under RHEL3 with reiserfs and cyrus 2.2.3.. I've reviewed the changelog and I don't see anything obviously related to this up to 2.2.12. Any ideas on what might be causing this... or what I can do to fix it... short of deleting the folder? Thanks, Jared --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: [POLL] pop3d/nntpd and IMAP flags
On Tue, 25 Oct 2005, Henrique de Moraes Holschuh wrote: On Tue, 25 Oct 2005, Ken Murchison wrote: It would be fairly straightforward to have an option that updated \Seen state whenever a POP3 client issues a RETR command or an NNTP client issues a BODY or ARTICLE command. I vote for this change, and if it is optional, I'd vote for it to be the default behaviour. I obviously haven't used pop and imap on the same folder much, this is actually the behavior I would have expected (it's the same message, however you read it, the fact that you have read it means it's seen.) My question is what do people think of the interaction between pop3d/nntpd and the \Deleted flag? Should these daemons ignore articles that have this flag set? Should a POP3 DELE command or a NNTP cancel message just set the \Deleted flag instead of expunging the message? I vote for two possible behaviours, selectable via imapd.conf: 1. pop3/nntp DELE/cancel sets \Delete flag. pop3 QUIT causes expunge 2. what we have now (this would be the default). #1 sounds like the proper flag, there's not much that's more annoying then having fetchmail crach in the middle of a run and result in loosing messages as a result. and expunging after each message can kill your server (in fact, if there was a lazy expunge this would be the perfect situation to use it) Should any setting which enables pop3d/nntpd to use IMAP flags be global or per-mailbox? I'd be happy enough with it being global. If it is made per-mailbox, IMHO it would be good to have it work as follows: a "global" option that applies to every mailbox, and a per-mailbox annotation that overrides the global option for this mailbox subtree. I agree with those who say it should be both, with the specific overriding the general. David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: improving concurrency/performance
On Mon, 7 Nov 2005, Jure Pe?ar wrote: On Sun, 6 Nov 2005 14:20:03 -0800 (PST) Andrew Morgan <[EMAIL PROTECTED]> wrote: mkfs -t ext3 -j -m 1 -O dir_index /dev/sdb1 tune2fs -c 0 -i 0 /dev/sdb1 What about 1k blocks? I think they'd be more useful than 4k on mail spools ... I was recently doing some testing of lots of small files on the various filesystems, and I ran into a huge difference (8x) depending on what allocator was used for ext*. the default allocator changed between ext2 and ext3 (you can override it as a mount option) and when reading 1M files (10 dirs of 10 dirs of 10 dirs of 1000 1K files) the time to read them went from ~5 min with the old allocator useed in ext2 to 40 min for the one that's the default for ext3. David Lang -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: improving concurrency/performance
On Mon, 7 Nov 2005, Sergio Devojno Bruder wrote: David Lang wrote: (..) I was recently doing some testing of lots of small files on the various filesystems, and I ran into a huge difference (8x) depending on what allocator was used for ext*. the default allocator changed between ext2 and ext3 (you can override it as a mount option) and when reading 1M files (10 dirs of 10 dirs of 10 dirs of 1000 1K files) the time to read them went from ~5 min with the old allocator useed in ext2 to 40 min for the one that's the default for ext3. David Lang (!!) Interesting. You said mount options? man mount man page only show me data=journal, data=ordered, data=writeback, etcetera. How can I change that? I found more things listed under /usr/src/linux/Documentation/filesystems there are ext2.txt and ext3.txt files that list all the options available. note that with my test all the files were created in order, it may be that if the files are created in a random order things would be different, so further testing is warrented. I was doing tests to find how long it took to tar/untar these files (with the tarball on a different drive). here are the notes I made at the time. this was either 2.6.8.1 or 2.6.13.4 (I upgraded about that time, but I'm not sure what the exact timeing was note that on my cyrus server I actually use XFS with very large folders (20,000 mails in one folder) and it seems lightning fast. I haven't reconciled that observed bahavior with the tests listed below the fact that on ext* filesystems I had tests range from 5 min to 80 min is somewhat scary. I did make sure to clear memory (by reading a file larger then available ram and doing a sync) between tests David Lang on ext2 reading the tarball takes 53 seconds, createing the tar takes 10m, untarring it takes 4 min, copying it between drives on different controllers takes 62 seconds. XFS looks bad for small files (13 min to untar, 9:41 to tar), but good for large files (47 sec to read) reiserfs: reading the tar 43 sec 4:50 to tar 2:06 to untar (it was designed for tiny files and it appears to do that well) a couple tests I ran on reiserfs that I hadn't thought to run on the others, untaring on top of an existing directory took 7m, ls -lR took 2:40, ls -flR (unsorted) took 2:40, find . -print took 21sec, rm -r took 3m jfs: 57 sec to read, untar 15:30, no other tests run ext3: untar 3:30, read 64sec, tar 5:46, untarring on top of an existing directory 5:20, ls -lR 53 sec, ls -flR 47 sec, find . -print 7 sec enabling dir_hash changed the read (36 sec) ls -flr (57 sec), ls -lR 61 sec, find (25 sec), tar 81m!!! turning off dir_hash and removing the journal (effectivly turning it into ext2 again) and mounting noatime the tar goes to 34 min mounting with oldalloc,noatime untar is 4:45, tar is 5:51. -- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. -- C.A.R. Hoare Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
