Cyrus IMAP Murder and (SUN)Solaris Cluster
Hi there, We are running a cyrus IMAP murder for thousands of users with around 3TB of mail. We have a number of backends that each have attached storage for the mail store. We are currently using solaris 10 to run the murder on. We want to try and improve the resilience of our setup and are looking at a number of options to provide High Availability: 1. Using the latest version of cyrus to implement backend replication so that we have stand by spares for the murder if one of the backends go down. 2. Use Sun cluster to provide automatic failover for the backends somehow or another. 3. Some other solution we haven't thought of yet If anyone out there has though this kind of thing through or has experience of implementing Cyrus IMAP in a SUN Cluster environment, we'd love to hear from you, regards, Gavin. -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
upgrading a 2.2.12 murder to 2.3.14
We are planning towards upgrading our existing murder. The murder has four front ends, three backends and separate mupdate and lmtp servers. We want to move from version 2.2.12 to 2.3.14 so that we can make use of delayed expunge an possible replication. We have several thousand users currently having 4 TB of mail. Any comments on the following would be welcome: 1. We plan to gradually migrate users from the existing backend machines to new backend servers running 2.3.14 that have been integrated into our murder. We plan to do this using xfer. Although this is very time consuming we are under the impression that cyrus recommends using imap itself to do migrations rather than trying underlying filesystem copies of some kind. 2. We should end up then with our existing murder but with three backends running 2.3.14. We then plan to upgrade the other machines in the murder to 2.3.14 in the following order: frontends then lmtp and finally the mupdate server. Does this make sense? 3. As part of our preparation for this work we have been experimenting with cyrus replication. The replication protocol seems pretty solid, however we have some concerns about how to make use of it in our upgrade. We are considering having a replicant machine for each of the new backends. But this makes the migration even slower. In our tests, if we migrate users via xfer to a machine that is doing rolling replication, the replication takes around three times as long to complete as the xfer. Does anyone have any experience of migrating to a replicating environment? many thanks, Gavin Gray -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: upgrading a 2.2.12 murder to 2.3.14
Quoting Dave McMurtrie : >> 3. As part of our preparation for this work we have been experimenting >> with cyrus replication. The replication protocol seems pretty solid, >> however we have some concerns about how to make use of it in our >> upgrade. We are considering having a replicant machine for each of the >> new backends. But this makes the migration even slower. In our tests, >> if we migrate users via xfer to a machine that is doing rolling >> replication, the replication takes around three times as long to >> complete as the xfer. Does anyone have any experience of migrating to >> a replicating environment? > > Perhaps consider treating the migration and replication as two separate > things. It's not that you have to for technical reasons, but it will > probably make your life less complicated. Nothing stops you from > enabling replication once you're all done with the upgrade. > Hi, thanks for getting back to me... Assuming we do do the migration first, how would you suggest we subsequently enable replication? Can we just start the sync_client doing rolling replication, or should we do an initial replication of all users by running sync_client manually with a list of users? -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Synchronisation two cyrus-imapd servers
Hi there,
I also have this problem. I can't get the sync_server to run
under 2.3.15. Here is a snippet fom our logs:
Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 143423 local6.error]
DBERROR: reading /var/imap/db/skipstamp, assuming the worst: No such
file or directory
Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 518349 local6.debug] executed
Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 301543 local6.info]
skiplist: checkpointed /var/imap/mailboxes.db (0 records, 144 bytes)
in 0 seconds
Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 301543 local6.info]
skiplist: checkpointed /var/imap/annotations.db (0 records, 144 bytes)
in 0 seconds
Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 921384 local6.debug]
accepted connection
Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 177842 local6.debug]
cmdloop(): startup
Oct 8 12:42:36 mailbe9r last message repeated 1 time
Oct 8 12:42:38 mailbe9r master[25640]: [ID 970914 local6.error]
process 25647 exited, signaled to death by 11
Oct 8 12:42:38 mailbe9r master[25640]: [ID 621917 local6.debug]
service syncserver pid 25647 in BUSY state: terminated abnormally
I've had a go at trying to debug the problem, but had no success.
Does anyone have any ideas?
regards,
Gavin Gray
Quoting Alexander Demin :
> Hello.
>
> I have problem with synchronisation two cyrus-imapd servers.
>
> *** Start "Replica" host configuration ***
> OS: FreeBSD 7.2-STABLE i386
> cyrus-imapd-2.3.15 WITH_BDB=true WITH_REPLICATION=true
> cyrus-sasl-2.1.23 WITH_AUTHDAEMOND=true WITH_LOGIN=true WITH_PLAIN=true
> WITH_CRAM=true WITH_DIGEST=true
> cyrus-sasl-saslauthd-2.1.23
> All soft installed from ports.
>
> Cyrus configuration:
> /usr/local/etc/cyrus.conf
> START {
> recover cmd="ctl_cyrusdb -r"
> }
>
> SERVICES {
> imapcmd="imapd" listen="imap" prefork=0
> imaps cmd="imapd -s" listen="imaps" prefork=0
> pop3cmd="pop3d" listen="pop3" prefork=0
> pop3s cmd="pop3d -s" listen="pop3s" prefork=0
> sieve cmd="timsieved" listen="sieve" prefork=0
> lmtpunixcmd="lmtpd" listen="/data/imap/socket/lmtp" prefork=0
> smmap cmd="smmapd" listen="/data/imap/socket/smmap" prefork=1
> syncserver cmd="sync_server" listen="csync" prefork=1
> }
>
> EVENTS {
> checkpoint cmd="ctl_cyrusdb -c" period=30
> delprunecmd="cyr_expire -E 3" at=0400
> tlsprunecmd="tls_prune" at=0400
> }
>
> /usr/local/etc/imapd.conf
> configdirectory: /backup/imap
> partition-default: /backup/spool/imap
> unixhierarchysep: no
> altnamespace: yes
> allowanonymouslogin: no
> allowplaintext: yes
> imapidresponse: yes
> admins: cyrus
> munge8bit: 0
> rfc2046_strict: 0
> sievedir: /backup/imap/sieve
> sendmail: /usr/sbin/sendmail
> postmaster: postmaster
> annotation_db: skiplist
> duplicate_db: berkeley-nosync
> mboxlist_db: skiplist
> ptscache_db: berkeley
> seenstate_db: skiplist
> subscription_db: flat
> sasl_pwcheck_method: auxprop
> sasl_auxprop_plugin: sasldb
> sasl_log_level: 7
> sasl_mech_list: plain cram-md5 digest-md5 login
> lmtpsocket: /backup/imap/socket/lmtp
> virtdomains: userid
> lmtp_downcase_rcpt: 1
>
> #
> # EOF
>
> /etc/services
> csync 2005/tcp
>
> /etc/rc.conf (show only cyrus/sasl params)
> cyrus_imapd_enable="YES"
> saslauthd_enable="YES"
> saslauthd_flags="-a sasldb"
> *** End "Replica" host configuration ***
>
> *** Start "Master" host configuration ***
> OS: FreeBSD 7.2-STABLE amd64
> cyrus-imapd-2.3.15 WITH_BDB=true WITH_REPLICATION=true
> cyrus-sasl-2.1.23 WITH_AUTHDAEMOND=true WITH_LOGIN=true WITH_PLAIN=true
> WITH_CRAM=true WITH_DIGEST=true
> cyrus-sasl-saslauthd-2.1.23
> All soft installed from ports.
>
> Cyrus configuration:
> /usr/local/etc/cyrus.conf
> START {
> recover cmd="ctl_cyrusdb -r"
> }
>
> SERVICES {
> imapcmd="imapd" listen="imap" prefork=0
> imaps cmd="imapd -s" listen="imaps" prefork=0
> pop3cmd="pop3d" listen="pop3" prefork=0
> pop3s cmd="pop3d -s" listen="pop3s" prefork=0
> sieve cmd="timsieved" listen="sieve" prefork=0
> lmtpunixcmd="lmtpd" listen="/data/imap/socket/
Re: Synchronisation two cyrus-imapd servers
Hi there,
Yes you're right. If I add a defaultpartition and
corresponding partition- to imapd.conf all is well,
many thanks,
Gavin.
Quoting steff...@gmx.de:
> Hi,
>
> you are probably missing 'defaultpartition' in imapd.conf:
> https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3170
>
> Regards,
> Stephan
>
> Original-Nachricht
>> Datum: Thu, 08 Oct 2009 14:04:57 +0100
>> Von: Gavin Gray
>> An: info-cyrus@lists.andrew.cmu.edu
>> Betreff: Re: Synchronisation two cyrus-imapd servers
>
>> Hi there,
>> I also have this problem. I can't get the sync_server to run
>> under 2.3.15. Here is a snippet fom our logs:
>>
>> Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 143423 local6.error]
>> DBERROR: reading /var/imap/db/skipstamp, assuming the worst: No such
>> file or directory
>> Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 518349 local6.debug]
>> executed
>> Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 301543 local6.info]
>> skiplist: checkpointed /var/imap/mailboxes.db (0 records, 144 bytes)
>> in 0 seconds
>> Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 301543 local6.info]
>> skiplist: checkpointed /var/imap/annotations.db (0 records, 144 bytes)
>> in 0 seconds
>> Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 921384 local6.debug]
>> accepted connection
>> Oct 8 12:42:36 mailbe9r syncserver[25647]: [ID 177842 local6.debug]
>> cmdloop(): startup
>> Oct 8 12:42:36 mailbe9r last message repeated 1 time
>> Oct 8 12:42:38 mailbe9r master[25640]: [ID 970914 local6.error]
>> process 25647 exited, signaled to death by 11
>> Oct 8 12:42:38 mailbe9r master[25640]: [ID 621917 local6.debug]
>> service syncserver pid 25647 in BUSY state: terminated abnormally
>>
>> I've had a go at trying to debug the problem, but had no success.
>>
>> Does anyone have any ideas?
>>
>> regards,
>>
>> Gavin Gray
>>
>>
>> Quoting Alexander Demin :
>>
>> > Hello.
>> >
>> > I have problem with synchronisation two cyrus-imapd servers.
>> >
>> > *** Start "Replica" host configuration ***
>> > OS: FreeBSD 7.2-STABLE i386
>> > cyrus-imapd-2.3.15 WITH_BDB=true WITH_REPLICATION=true
>> > cyrus-sasl-2.1.23 WITH_AUTHDAEMOND=true WITH_LOGIN=true WITH_PLAIN=true
>> > WITH_CRAM=true WITH_DIGEST=true
>> > cyrus-sasl-saslauthd-2.1.23
>> > All soft installed from ports.
>> >
>> > Cyrus configuration:
>> > /usr/local/etc/cyrus.conf
>> > START {
>> >recover cmd="ctl_cyrusdb -r"
>> > }
>> >
>> > SERVICES {
>> >imapcmd="imapd" listen="imap" prefork=0
>> >imaps cmd="imapd -s" listen="imaps" prefork=0
>> >pop3cmd="pop3d" listen="pop3" prefork=0
>> >pop3s cmd="pop3d -s" listen="pop3s" prefork=0
>> >sieve cmd="timsieved" listen="sieve" prefork=0
>> >lmtpunixcmd="lmtpd" listen="/data/imap/socket/lmtp" prefork=0
>> >smmap cmd="smmapd" listen="/data/imap/socket/smmap" prefork=1
>> >syncserver cmd="sync_server" listen="csync" prefork=1
>> > }
>> >
>> > EVENTS {
>> >checkpoint cmd="ctl_cyrusdb -c" period=30
>> >delprunecmd="cyr_expire -E 3" at=0400
>> >tlsprunecmd="tls_prune" at=0400
>> > }
>> >
>> > /usr/local/etc/imapd.conf
>> > configdirectory: /backup/imap
>> > partition-default: /backup/spool/imap
>> > unixhierarchysep: no
>> > altnamespace: yes
>> > allowanonymouslogin: no
>> > allowplaintext: yes
>> > imapidresponse: yes
>> > admins: cyrus
>> > munge8bit: 0
>> > rfc2046_strict: 0
>> > sievedir: /backup/imap/sieve
>> > sendmail: /usr/sbin/sendmail
>> > postmaster: postmaster
>> > annotation_db: skiplist
>> > duplicate_db: berkeley-nosync
>> > mboxlist_db: skiplist
>> > ptscache_db: berkeley
>> > seenstate_db: skiplist
>> > subscription_db: flat
>> > sasl_pwcheck_method: auxprop
>> > sasl_auxprop_plugin: sasldb
>> > sasl_log_level: 7
>> > sasl_mech_list: plain cram-md5 digest-md5 login
>> > lmtpsocket: /backup/imap/soc
intermittent problems with kick_mupdate
Hi there, We are busy migrating lots of users to new backends in our cyrus murder using xfer. The existing backends (and the rest of the murder) are cyrus 2.2 where as the new backend machines are cyrus 2.3.15. Mostly everything works just fine. However every now and them our new backends fail to connect to our mupdate server at the start of an xfer. We see the new backend log into the mupdate server, but then a subsequent call to kick_mupdate fails and consequently so does the xfer:- The error on the originating backend for the failed xfer is: The remote Server(s) denied the operation The error on the new backend receiving the error is: kick_mupdate: can't connect to target: No such file or directory the underlying OS of the new backend is openSolaris any thoughts? -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: intermittent problems with kick_mupdate
Forgot to mention the more specific errors we have seen on the backends we're moving users from: imap[29289]: [ID 886451 local6.error] Could not trigger remote push to mupdate serverduring move of user... and imap[22505]: [ID 772019 local6.error] Could not set remote acl on user Quoting Gavin Gray : > Hi there, > > We are busy migrating lots of users to new backends in our cyrus > murder using xfer. > > The existing backends (and the rest of the murder) are cyrus 2.2 where > as the new backend machines are cyrus 2.3.15. > > Mostly everything works just fine. However every now and them our new > backends fail to connect to our mupdate server at the start of an > xfer. We see the new backend log into the mupdate server, but then a > subsequent call to kick_mupdate fails and consequently so does the > xfer:- > > The error on the originating backend for the failed xfer is: > The remote Server(s) denied the operation > > The error on the new backend receiving the error is: > kick_mupdate: can't connect to target: No such file or directory > > the underlying OS of the new backend is openSolaris > > any thoughts? > > > -- > Gavin Gray > Edinburgh University Information Services > Rm 2013 JCMB > Kings Buildings > Edinburgh > EH9 3JZ > UK > tel +44 (0)131 650 5987 > email gavin.g...@ed.ac.uk > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > > Cyrus Home Page: http://cyrusimap.web.cmu.edu/ > Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki > List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html > > -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: intermittent problems with kick_mupdate
Yes, all that you're saying makes sense in terms of what we're experiencing, Quoting Wesley Craig : > On 03 Jun 2010, at 09:24, Gavin Gray wrote: >> imap[29289]: [ID 886451 local6.error] Could not trigger remote push to >> mupdate serverduring move of user... > > During the xfer, the local backend sets some information in the mupdate > master WRT the new mailbox location. However, this information may be > a bit of a guess. The MUPDATEPUSH instructs the remote backend to set > whatever the correct information is in the mupdate master. Stupidly, > the above log doesn't report what the problem was, and neither does the > remote backend. That should be fixed (can you open a bug report?). > However, the error is not fatal. This error does not interrupt xfers and cause them to fail. >> imap[22505]: [ID 772019 local6.error] Could not set remote acl on user > > This error is fatal. In fact, you ought to not execute the following > MUPDATEPUSH, because not being able to set the ACL is not permissible. > Perhaps you're seeing this problem: > > https://bugzilla.andrew.cmu.edu/show_bug.cgi?id=3218 > > Of course, the logging again fails to tell us *why* we aren't able to > set the remote ACL (another good opportunity to report a bug). This error does cause xfers to halt and fail. Although we can easily restart them and they pick up from where they left off and complete successfully. >> The error on the new backend receiving the error is: >> kick_mupdate: can't connect to target: No such file or directory > > > Is this a unified murder? Skimming imapd.c, I see a mix of calls to > kick_mupdate(), some protected by checks for the type of murder, some > not. Perhaps that's the problem. In any case, kick_mupdate() is void, > so errors relating to it are probably cascades from some other failed > step in the process. It's not a unified murder, we have a number of frontends and a dedicated mupdate master. As for kick_mupdate we see lots of those errors say over the course of the night when the xfers are taking place ,but perhaps only one instance of a xfer failing, which is matched by exactly one of these entries in our logs: Jun 3 01:00:23 backend machine imap[2554]: [ID 130975 local6.error] connect(mupdate master machine) failed: Connection refused Jun 3 01:00:23 backend machine imap[2554]: [ID 320383 local6.error] mupdate_connect failed: unknown error The logs then go on to say some operation couldn't happen because of not being able to connect to the mupdate server, however the error reported by the failed xfer process is always the "Could not set remote acl on user" one. I don't really understand what kick_mupdate does or how it does it from having a quick look at the code so I feel at a bit of a loss there. Is their any possibility that with lots of concurrent xfers going on we could be hitting some limit on how many connections the mupdate master process will accept? I'll look at the bug report you mentioned and think about submitting one regarding the error logging, many thanks, Gavin Gray -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
imapd dumping core due to SEGV
Hi there, We have three backend servers in our cyrus murder all like this: name : Cyrus IMAPD version: v2.3.15 2009/09/09 12:35:48 vendor : Project Cyrus support-url: http://cyrusimap.web.cmu.edu os : SunOS os-version : 5.11 environment: Built w/Cyrus SASL 2.1.23 Running w/Cyrus SASL 2.1.23 Built w/Berkeley DB 4.7.25: (May 15, 2008) Running w/Berkeley DB 4.7.25: (May 15, 2008) Built w/OpenSSL 0.9.8a 11 Oct 2005 (+ security fixes for: CVE-2006-2937 CVE-2006-2940 CVE-2006-3738 CVE-2006-4339 CVE-2006-4343 CVE-2007-3108 CVE-2007-4995 CVE-2007-5135 CVE-2008-5077 CVE-2009-0590) Running w/OpenSSL 0.9.8a 11 Oct 2005 (+ security fixes for: CVE-2006-2937 CVE-2006-2940 CVE-2006-3738 CVE-2006-4339 CVE-2006-4343 CVE-2007-3108 CVE-2007-4995 CVE-2007-5135 CVE-2008-5077 CVE-2009-0590) Built w/zlib 1.2.3 Running w/zlib 1.2.3 CMU Sieve 2.3 NET-SNMP mmap = shared lock = fcntl nonblock = fcntl idle = poll Two of them have had imapd processes crash and leave core dumps in the past couple of days. Looking at the core dumps with dbx we see . t...@1 (l...@1) program terminated by signal SEGV (no mapping at the fault address) 0xfe5848d3: strncmp+0x0033: movb 0x0002(%esi),%al (dbx) where current thread: t...@1 =>[1] strncmp(0x8098d2b, 0xfbed7fe6, 0x81824e0, 0x20), at 0xfe5848d3 [2] message_pendingboundary(0xfbed7fe4, 0x8183750, 0x8040f84, 0x819364d), at 0x8098d2b [3] message_parse_content(0x8040fd0, 0x0, 0x8196698, 0x8040f80), at 0x8098b4f [4] message_parse_body(0x8040fd0, 0x0, 0x8196698, 0x81422d7, 0x8040f80, 0x140, 0x8040cc0, 0x8040f80), at 0x80969df [5] message_parse_multipart(0x8040fd0, 0x0, 0x8196550, 0x8040f80), at 0x8098967 [6] message_parse_body(0x8040fd0, 0x0, 0x8196550, 0x81422d7, 0x8040f80, 0xa0, 0x8040ea0, 0x8040f80), at 0x809690f [7] message_parse_multipart(0x8040fd0, 0x0, 0x8182438, 0x8040f80), at 0x8098967 [8] message_parse_body(0x8040fd0, 0x0, 0x8182438, 0x81422d7, 0x8040f80, 0x0, 0x8040ff8), at 0x809690f [9] message_parse_mapped(0xfbed, 0x8000, 0x8182438, 0xfe5be856, 0x8041050, 0x1), at 0x809648a [10] message_parse_file(0x8174b68, 0x0, 0x0, 0x80441e0, 0x81ed030, 0x30), at 0x80962c1 [11] append_fromstage(0x8044200, 0x80441e0, 0x818c880, 0x485baf50, 0x8189dc0, 0x1), at 0x8086364 [12] cmd_append(0x817f6c0, 0x817f7a0, 0x0, 0x0), at 0x8068d7c [13] cmdloop(0xfee20118, 0xfe4e2a00, 0x7ab8a40), at 0x8063852 [14] service_main(0x1, 0x8175218, 0x8047e3c, 0xf, 0xfeffdbb0, 0x8047878), at 0x8062e9e [15] main(0x1, 0x8047e34, 0x8047e3c), at 0x8061fde and t...@1 (l...@1) program terminated by signal SEGV (no mapping at the fault address) 0xfe5babb3: _smalloc+0x00c3:movl 0x0008(%ecx),%edx (dbx) where current thread: t...@1 =>[1] _smalloc(0x10, 0xfe6a, 0x8044b88, 0xfe5bac43), at 0xfe5babb3 [2] _malloc_unlocked(0x10), at 0xfe5bae3e [3] malloc(0x10, 0x8044c60, 0xfe6a1de8, 0x8180070), at 0xfe5bac0d [4] xmalloc(0x10, 0x8044bf0, 0xfe621e65, 0xfe60c37d, 0x65746164, 0x81a8800), at 0x80bd8b1 [5] appendstrlist_withdata(0x8044c64, 0x81a5e70, 0x0, 0x0), at 0x809d1c4 [6] appendstrlist(0x8044c64, 0x81a5e70, 0x816566c, 0x0), at 0x809d23e [7] cmd_fetch(0x817f688, 0x817f768, 0x0, 0x8063451), at 0x806a2fa [8] cmdloop(0xfe8e0118, 0xfe4f2a00, 0x7ab8a40), at 0x8064177 [9] service_main(0x1, 0x8175218, 0x8047e3c, 0xf, 0xfeffdbb0, 0x8047878), at 0x8062e9e [10] main(0x1, 0x8047e34, 0x8047e3c), at 0x8061fde Is there any known issues with 2.3.15? Also when this happens the cyrus master process kills all other active imapd processes and restarts, is there a reason for this? regards, Gavin Gray -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: imapd dumping core due to SEGV
Sorry for the delay getting back about this, I meant to let people know that the reason for this: >> Also when this happens the cyrus master process kills all other active >> imapd processes and restarts, is there a reason for this? > > I've never heard of master doing that in response to ANY child > behavior. Does master log anything? was the way we had setup SMF on Solaris to control Cyrus IMAP. One needs to make sure SMF is setup to ignore core dumps and child processes signaling death otherwise SMF will restart the entire service. On Mon, 12 Jul 2010 18:36:59 +0100, Wesley Craig wrote: > On 05 Jul 2010, at 10:56, Gavin Gray wrote: >> Two of them have had imapd processes crash and leave core dumps in >> the past couple of days. Looking at the core dumps with dbx we see > > I'm not aware of bug fixes in those code paths. Given how little those > two code paths have in common, I'd suspect memory corruption. > >> Also when this happens the cyrus master process kills all other active >> imapd processes and restarts, is there a reason for this? > > I've never heard of master doing that in response to ANY child > behavior. Does master log anything? > > :wes > > -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Replication sync-server and Delayed Delete
Hi there, We have a cyrus murder using replication and we have a few questions about the behaviour we are seeing on our system. 1. cyr_expire on the master doesn't cause any replication to happen. Is that 'correct'? In other words if we want to delete folders from the DELETED heirarchy on the replicant then we need to also run cyr_expire on the replicant? 2. We're also a little unclear about replication vis a vis the delayed expunge and the unexpunge facility. Could you explain what ought to happen in terms of replication when email is expunged and then possibly unexpunged if anything? 3. We are seeing a strange anomaly on the replication of deleting a folder. e.g a user deletes a folder the folder goes into the DELETED heirarchy of the partition the user's mailbox is on the folder is also deleted from the replicant as we would expect however the folder on the replicant goes into the DELETED heirarchy on a different partition(the default partition as specified in cyrus.conf). Is this normal? many thanks, Gavin Gray name : Cyrus IMAPD version: v2.3.15 2009/09/09 12:35:48 vendor : Project Cyrus support-url: http://cyrusimap.web.cmu.edu os : SunOS os-version : 5.11 environment: Built w/Cyrus SASL 2.1.23 Running w/Cyrus SASL 2.1.23 Built w/Berkeley DB 4.7.25: (May 15, 2008) Running w/Berkeley DB 4.7.25: (May 15, 2008) Built w/OpenSSL 0.9.8a 11 Oct 2005 (+ security fixes for: CVE-2006-2937 CVE-2006-2940 CVE-2006-3738 CVE-2006-4339 CVE-2006-4343 CVE-2007-3108 CVE-2007-4995 CVE-2007-5135 CVE-2008-5077 CVE-2009-0590) Running w/OpenSSL 0.9.8a 11 Oct 2005 (+ security fixes for: CVE-2006-2937 CVE-2006-2940 CVE-2006-3738 CVE-2006-4339 CVE-2006-4343 CVE-2007-3108 CVE-2007-4995 CVE-2007-5135 CVE-2008-5077 CVE-2009-0590) Built w/zlib 1.2.3 Running w/zlib 1.2.3 CMU Sieve 2.3 NET-SNMP mmap = shared lock = fcntl nonblock = fcntl idle = poll -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Replication sync-server and Delayed Delete
Hi there, So is the side effect of deleted folders ending up on the default partition when delayed delete is switched on on a replicant machine a known issue for sync_server? A knock on effect of this seems to be that cyr_expire on the replicant doesn't find these DELETED folders when it runs to do a purge. Could you suggest a safe alternative to cyr_expire in order to purge these misplace deleted folders? regards, Gavin Gray On Wed, 15 Sep 2010, Bron Gondwana wrote: > On Wed, Sep 15, 2010 at 12:29:18PM +0100, Gavin Gray wrote: >> Hi there, >> >> We have a cyrus murder using replication and we have a few questions >> about the behaviour we are seeing on our system. >> >> 1. cyr_expire on the master doesn't cause any replication to happen. >> Is that 'correct'? In other words if we want to delete folders from >> the DELETED heirarchy on the replicant then we need to also run >> cyr_expire on the replicant? > > Yeah, pretty much. > >> 2. We're also a little unclear about replication vis a vis the >> delayed expunge and the unexpunge facility. Could you explain what >> ought to happen in terms of replication when email is expunged and >> then possibly unexpunged if anything? > > It's a bit messy. Unexpunge is a sin against IMAP by the way, and > has been replaced with "generate new UID and promote" in 2.4. In > which case it's just like a new append wit the same flags, and > replicates like an append :) > > 2.3 replication ignores expunges - it's as if they don't exist! When > the mailbox syncs, it nukes the records that aren't "alive" on the > master from the replica. If you re-inject them with unexpunge, it > should find them and sync_combine_commit() the result. I don't know > if unexpunge inserts replication events though - somewhat doubt it. > >> 3. We are seeing a strange anomaly on the replication of deleting a folder. >>e.g a user deletes a folder >>the folder goes into the DELETED heirarchy of the partition >> the user's mailbox is on >>the folder is also deleted from the replicant as we would expect >>however the folder on the replicant goes into the DELETED >> heirarchy on a different partition(the default partition as >> specified in cyrus.conf). Is this normal? > > Replication and partitions is broken in some ways in 2.3. It should > be better in 2.4 I believe, though I haven't tested it. I'm going to > be releasing an alpha super-soon (it's been pushed to git.cyrusimap.org > now!) > > Bron. > > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Replication sync-server and Delayed Delete
Actually, I was talking rubbish cyr_expire does find the deleted folders older than x days, I just got my dates wrong when looking into it. apologies, Gavin. On Thu, 16 Sep 2010, gavin.g...@ed.ac.uk wrote: > Hi there, > So is the side effect of deleted folders ending up on the default > partition when delayed delete is switched on on a replicant machine a known > issue for sync_server? A knock on effect of this seems to be that cyr_expire > on the replicant doesn't find these DELETED folders when it runs to do a > purge. > > Could you suggest a safe alternative to cyr_expire in order to purge these > misplace deleted folders? > > regards, > > Gavin Gray > > > On Wed, 15 Sep 2010, Bron Gondwana wrote: > >> On Wed, Sep 15, 2010 at 12:29:18PM +0100, Gavin Gray wrote: >>> Hi there, >>> >>> We have a cyrus murder using replication and we have a few questions >>> about the behaviour we are seeing on our system. >>> >>> 1. cyr_expire on the master doesn't cause any replication to happen. >>> Is that 'correct'? In other words if we want to delete folders from >>> the DELETED heirarchy on the replicant then we need to also run >>> cyr_expire on the replicant? >> >> Yeah, pretty much. >> >>> 2. We're also a little unclear about replication vis a vis the >>> delayed expunge and the unexpunge facility. Could you explain what >>> ought to happen in terms of replication when email is expunged and >>> then possibly unexpunged if anything? >> >> It's a bit messy. Unexpunge is a sin against IMAP by the way, and >> has been replaced with "generate new UID and promote" in 2.4. In >> which case it's just like a new append wit the same flags, and >> replicates like an append :) >> >> 2.3 replication ignores expunges - it's as if they don't exist! When >> the mailbox syncs, it nukes the records that aren't "alive" on the >> master from the replica. If you re-inject them with unexpunge, it >> should find them and sync_combine_commit() the result. I don't know >> if unexpunge inserts replication events though - somewhat doubt it. >> >>> 3. We are seeing a strange anomaly on the replication of deleting a >>> folder. >>>e.g a user deletes a folder >>>the folder goes into the DELETED heirarchy of the partition >>> the user's mailbox is on >>>the folder is also deleted from the replicant as we would expect >>>however the folder on the replicant goes into the DELETED >>> heirarchy on a different partition(the default partition as >>> specified in cyrus.conf). Is this normal? >> >> Replication and partitions is broken in some ways in 2.3. It should >> be better in 2.4 I believe, though I haven't tested it. I'm going to >> be releasing an alpha super-soon (it's been pushed to git.cyrusimap.org >> now!) >> >> Bron. >> >> > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
lmtpd processes 'hanging'
Hi there We are runnning cyrus 2.3.15 in a murder set up. We have recently started having intermittent problems with mail delivery to the backends. Every so often the lmtpd processes on each of the backends stop doing anything and a mail queue begins to build up on out lmtp server. After around five minutes or so everything bursts back into life with the lmtpd processes on the backends and the queue on the lmtp server drains. We can't figure out why this is happening. If anyone has any ideas they would be greatly appreciated During working hours we usually have around 3000 imapd processes per backend and normally deliver a few hundred messages per minute. We have three identical backend servers. One of them has become buisier in terms of active users ie larger mail store, larger deliverdb. We use berkley db for the cyrus databases. regards, Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
problem with xfer from 2.3.15 to 2.4.17
Hi there, We are have been running a cyrus imap 2.3.15 murder for some time, we have intoduced some new cyrus 2.4.17 backends into that murder and would like to migrate users to these new machines, however xfer's are failing for this reason: Mar 24 12:16:22 backend.server imap[2370]: [ID 160988 local6.error] possible MITM attack:list of available SASL mechanisms changed Mar 24 12:16:22 backend.server imap[2370]: [ID 872191 local6.error] Could not move mailbox: user.imaptest, Initial backend connect failed we believe this is to to do with the list of SASL auth mechanisms returned in the capability string from a server before and after a login, but we can't see what is causing the mismatch or how to resolve. Any ideas gratefully appreciated. The strange thing is GSSAPI logins work fine between the servers in question. here are some details of the 2.3.15 and 2.4.17 backends in question: name : Cyrus IMAPD version: v2.4.17 d1df8aff 2012-12-01 vendor : Project Cyrus support-url: http://www.cyrusimap.org os : SunOS os-version : 5.11 environment: Built w/Cyrus SASL 2.1.23 Running w/Cyrus SASL 2.1.23 Built w/Berkeley DB 4.7.25: (May 15, 2008) Running w/Berkeley DB 4.7.25: (May 15, 2008) Built w/OpenSSL 1.0.0e 6 Sep 2011 Running w/OpenSSL 1.0.0e 6 Sep 2011 Built w/zlib 1.2.3 Running w/zlib 1.2.3 CMU Sieve 2.4 NET-SNMP mmap = shared lock = fcntl nonblock = fcntl idle = idled C: C01 CAPABILITY S: * CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ SORT=DISPLAY THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE LIST-EXTENDED WITHIN QRESYNC SCAN XLIST URLAUTH URLAUTH=BINARY X-NETSCAPE MUPDATE=mupdate://mupdate-ucs.www-dyn.ed.ac.uk/ LOGINDISABLED AUTH=GSSAPI COMPRESS=DEFLATE IDLE and name : Cyrus IMAPD version: v2.3.15 2009/09/09 12:35:48 vendor : Project Cyrus support-url: http://cyrusimap.web.cmu.edu os : SunOS os-version : 5.11 environment: Built w/Cyrus SASL 2.1.15 Running w/Cyrus SASL 2.1.23 Built w/Berkeley DB 4.7.25: (May 15, 2008) Running w/Berkeley DB 4.7.25: (May 15, 2008) Built w/OpenSSL 1.0.0e 6 Sep 2011 Running w/OpenSSL 1.0.0e 6 Sep 2011 Built w/zlib 1.2.3 Running w/zlib 1.2.3 CMU Sieve 2.3 mmap = shared lock = fcntl nonblock = fcntl idle = idled C: C01 CAPABILITY S: * CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID MUPDATE=mupdate://mupdate-ucs.www-dyn.ed.ac.uk/ LOGINDISABLED AUTH=GSSAPI COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE CATENATE CONDSTORE SCAN IDLE LISTEXT LIST-SUBSCRIBED X-NETSCAPE URLAUTH Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
xfer problems between 2.3.15 and 2.4.17
I have been testing xfer of accounts within a cyrus murder from 2.3.15 backends to new 2.4.17. backends. all the email and folders seem to migrate perfectly and the xfer'd accounts can send and receive email. However when reading email with an IMAP client I am having strange issues setting flags within folders on messages. In particular setting and unsetting deletion flags is very erratic. In some folders it doesn't work at all, on others I can set the deletion flag but can't unset it. All of the backends have delayed expunge switched on. debug output from the alpine IMAP client seems to suggest the server is doing what it's told: IMAP DEBUG 12:04:30 5/7: 0169 STORE 93 +Flags (\DELETED) IMAP DEBUG 12:04:30 5/7: 0169 OK Completed IMAP DEBUG 12:04:31 5/7: 016a STORE 94 +Flags (\DELETED) IMAP DEBUG 12:04:31 5/7: 016a OK Completed but then something seems to immediately remove the flag, because on issuing an expunge the client finds nothing to expunge. nothing of note seems to be logged on the backend, even logging in debug mode. the other baffling thing is that in some folders within the same users account, this whole process works perfectly. does anyone have any ideas what could be causing this and if there might be a solution? many thanks, Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
Any help with this would be much appreciated. We keep coming across folders that once they have been migrated seem to have corrupt cyrus.index files. The only way to fix them is to remove the files and do a reconstruct. This is not a workable solution from our users point of view as is sets all the messages back to flaggged as new etc. We have tried various tests, but we can't discover the cause of the corruption of the cyrus.index files. regards, Gavin Gray On Wed, 7 May 2014, gavin.g...@ed.ac.uk wrote: > I have been testing xfer of accounts within a cyrus murder from 2.3.15 > backends to new 2.4.17. backends. > > all the email and folders seem to migrate perfectly and the xfer'd > accounts can send and receive email. However when reading email with an > IMAP client I am having strange issues setting flags within folders on > messages. In particular setting and unsetting deletion flags is very > erratic. In some folders it doesn't work at all, on others I can set the > deletion flag but can't unset it. All of the backends have delayed > expunge switched on. > > debug output from the alpine IMAP client seems to suggest the server is > doing what it's told: > > IMAP DEBUG 12:04:30 5/7: 0169 STORE 93 +Flags (\DELETED) > IMAP DEBUG 12:04:30 5/7: 0169 OK Completed > IMAP DEBUG 12:04:31 5/7: 016a STORE 94 +Flags (\DELETED) > IMAP DEBUG 12:04:31 5/7: 016a OK Completed > > but then something seems to immediately remove the flag, because on > issuing an expunge the client finds nothing to expunge. > > nothing of note seems to be logged on the backend, even logging in debug > mode. > > the other baffling thing is that in some folders within the same users > account, this whole process works perfectly. > > does anyone have any ideas what could be causing this and if there might > be a solution? > > many thanks, > > Gavin Gray > Edinburgh University Information Services > Rm 2013 JCMB > Kings Buildings > Edinburgh > EH9 3JZ > UK > tel +44 (0)131 650 5987 > email gavin.g...@ed.ac.uk > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
Thanks Ken, We have tried the -G and -R options to no avail. I've begun to experiment with your seen database idea. It would be nice to discover the reason for the corruption. We still don't even know whether the problem was there before the xfer or if it's a result of the xfer process. Obviously there are some major changes from cyrus 2.3.15 to 2.4.17, but then mostly folders are migrated and 'upgraded' correctly. cheers, Gavin Gray On Thu, 15 May 2014, Ken Murchison wrote: > Hi Gavin, > > Have you tried running reconstruct with the -G and/or -R options to see if > they fix the corruption without having to remove cyrus.index? > > Another option is to place a copy of the user's .seen file from the 2.3 > machine on the 2.4 machine prior to removing cyrus.index and reconstructing. > I *think* the server will then upgrade the seen state from .seen to > cyrus.index. > > > > On 05/15/2014 11:53 AM, gavin.g...@ed.ac.uk wrote: >> Any help with this would be much appreciated. >> >> We keep coming across folders that once they have been migrated seem to >> have corrupt cyrus.index files. The only way to fix them is to remove the >> files and do a reconstruct. This is not a workable solution from our users >> point of view as is sets all the messages back to flaggged as new etc. >> >> We have tried various tests, but we can't discover the cause of the >> corruption of the cyrus.index files. >> >> regards, >> >> Gavin Gray >> >> >> On Wed, 7 May 2014, gavin.g...@ed.ac.uk wrote: >> >> > I have been testing xfer of accounts within a cyrus murder from 2.3.15 >> > backends to new 2.4.17. backends. >> > >> > all the email and folders seem to migrate perfectly and the xfer'd >> > accounts can send and receive email. However when reading email with an >> > IMAP client I am having strange issues setting flags within folders on >> > messages. In particular setting and unsetting deletion flags is very >> > erratic. In some folders it doesn't work at all, on others I can set the >> > deletion flag but can't unset it. All of the backends have delayed >> > expunge switched on. >> > >> > debug output from the alpine IMAP client seems to suggest the server is >> > doing what it's told: >> > >> > IMAP DEBUG 12:04:30 5/7: 0169 STORE 93 +Flags (\DELETED) >> > IMAP DEBUG 12:04:30 5/7: 0169 OK Completed >> > IMAP DEBUG 12:04:31 5/7: 016a STORE 94 +Flags (\DELETED) >> > IMAP DEBUG 12:04:31 5/7: 016a OK Completed >> > >> > but then something seems to immediately remove the flag, because on >> > issuing an expunge the client finds nothing to expunge. >> > >> > nothing of note seems to be logged on the backend, even logging in debug >> > mode. >> > >> > the other baffling thing is that in some folders within the same users >> > account, this whole process works perfectly. >> > >> > does anyone have any ideas what could be causing this and if there might >> > be a solution? >> > >> > many thanks, >> > >> > Gavin Gray >> > Edinburgh University Information Services >> > Rm 2013 JCMB >> > Kings Buildings >> > Edinburgh >> > EH9 3JZ >> > UK >> > tel +44 (0)131 650 5987 >> > email gavin.g...@ed.ac.uk >> > >> > -- >> > The University of Edinburgh is a charitable body, registered in >> > Scotland, with registration number SC005336. >> > >> > Cyrus Home Page: http://www.cyrusimap.org/ >> > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> > To Unsubscribe: >> > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus >> > >> > >> Gavin Gray >> Edinburgh University Information Services >> Rm 2013 JCMB >> Kings Buildings >> Edinburgh >> EH9 3JZ >> UK >> tel +44 (0)131 650 5987 >> email gavin.g...@ed.ac.uk >> >> -- >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ >> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > -- > Kenneth Murchison > Principal Systems Software Engineer > Carnegie Mellon University > > > Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
So I'm going to attach the cyrus metadata files for a folder that behaved the way I am describing having been xfer'd between 2.3.15 and 2.4.17. these are the files from the source server ie the 2.3.15 backend. If anyone with far deeper knowledge of cyrus than we do, could have a look to see if they can identify a problem and possibly find a solution then we would be very grateful. Gavin Gray On Thu, 15 May 2014, gavin.g...@ed.ac.uk wrote: Any help with this would be much appreciated. We keep coming across folders that once they have been migrated seem to have corrupt cyrus.index files. The only way to fix them is to remove the files and do a reconstruct. This is not a workable solution from our users point of view as is sets all the messages back to flaggged as new etc. We have tried various tests, but we can't discover the cause of the corruption of the cyrus.index files. regards, Gavin Gray On Wed, 7 May 2014, gavin.g...@ed.ac.uk wrote: I have been testing xfer of accounts within a cyrus murder from 2.3.15 backends to new 2.4.17. backends. all the email and folders seem to migrate perfectly and the xfer'd accounts can send and receive email. However when reading email with an IMAP client I am having strange issues setting flags within folders on messages. In particular setting and unsetting deletion flags is very erratic. In some folders it doesn't work at all, on others I can set the deletion flag but can't unset it. All of the backends have delayed expunge switched on. debug output from the alpine IMAP client seems to suggest the server is doing what it's told: IMAP DEBUG 12:04:30 5/7: 0169 STORE 93 +Flags (\DELETED) IMAP DEBUG 12:04:30 5/7: 0169 OK Completed IMAP DEBUG 12:04:31 5/7: 016a STORE 94 +Flags (\DELETED) IMAP DEBUG 12:04:31 5/7: 016a OK Completed but then something seems to immediately remove the flag, because on issuing an expunge the client finds nothing to expunge. nothing of note seems to be logged on the backend, even logging in debug mode. the other baffling thing is that in some folders within the same users account, this whole process works perfectly. does anyone have any ideas what could be causing this and if there might be a solution? many thanks, Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. cyrus.index Description: Binary data �� Cyrus mailbox header "The best thing about this system was that it had lots of goals." --Jim Morris on Andrew user.jaw76a0f44a45710262 $NotJunk JunkRecorded $Junk NonJunk jaw lrswipkxtecda cyrus.cache Description: Binary data cyrus.expunge Description: Binary data Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
xfer problems between 2.3.15 and 2.4.17
As you may be aware we are attempting this and have run into various problems. Currently we have a mixed murder of 2.3.15 backends and 2.4.17 backends. We are now fairly confident that we can xfer accounts succesfully between these backends. The problems we had appear to have been with a very small number of accounts on our older backends that had corrupt cyrus.index files. However we are now having trouble configuring frontends that will work with this mixed murder environment while we xfer our users accross. If we use our existing 2.3.15 frontends then users have who have been migrated lose the ability to see other accounts in the "Other Users" name space. On the other hand if we introduce 2.4.17 frontends then we see strange behaviour around folder creation. Clients can create the folders but autosubscription fails with the client being told the new folder doesn't exist. If one waits a minute or two one can manually subscribe to the folder. So far we have not upgraded the mupdate master. Is this a mistake? In terms of the frontend config we have added suppress_capabilities: ESEARCH QRESYNC WITHIN XLIST LIST-EXTENDED to the 2.4.17 frontends, otherwise the config is identical to our 2.3.15 frontends. Is there any other config changes we should be aware of? any ideas greatly appreciated... Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
All we see is the request from the client to subscrobe to the newly created folder failing for the reason that, the mailbox does not exist. we get this from logging the client's imap session. thanks, Gavin Quoting Dave McMurtrie on Thu, 5 Jun 2014 16:18:16 +: > On Thu, 2014-06-05 at 16:15 +0100, gavin.g...@ed.ac.uk wrote: >> As you may be aware we are attempting this and have run into various >> problems. >> >> Currently we have a mixed murder of 2.3.15 backends and 2.4.17 backends. >> We are now fairly confident that we can xfer accounts succesfully between >> these backends. The problems we had appear to have been with a very small >> number of accounts on our older backends that had corrupt cyrus.index >> files. >> >> However we are now having trouble configuring frontends that will work >> with this mixed murder environment while we xfer our users accross. >> >> If we use our existing 2.3.15 frontends then users have who have been >> migrated lose the ability to see other accounts in the "Other Users" name >> space. >> >> On the other hand if we introduce 2.4.17 frontends then we see strange >> behaviour around folder creation. Clients can create the folders but >> autosubscription fails with the client being told the new folder doesn't >> exist. If one waits a minute or two one can manually subscribe to the >> folder. >> >> So far we have not upgraded the mupdate master. Is this a mistake? >> >> In terms of the frontend config we have added >> >> suppress_capabilities: ESEARCH QRESYNC WITHIN XLIST LIST-EXTENDED >> >> to the 2.4.17 frontends, otherwise the config is identical to our 2.3.15 >> frontends. Is there any other config changes we should be aware of? >> >> any ideas greatly appreciated... > > What you're doing should work just fine. It's exactly what we did to > migrate our murder environment from 2.3.x to 2.4.x. I think I posted to > the info-cyrus list about how we upgraded, but maybe I didn't. I got > all the 2.4 backend servers built and ready to go, then I upgraded all > the frontends to 2.4, then I used XFER to move all the mail from 2.3 to > 2.4 servers. I don't recall exactly when I upgraded our mupdate server, > but I don't think it matters. I don't believe anything changed in > mupdate protocol or in the mailbox.db format between 2.3 and 2.4. > > Have you tried grabbing telemetry on the 2.4 server when the > subscription fails? Is there anything being logged? > > Thanks! > > Dave > -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
yes it looks like for some reason the mupdate procedure that happens within the murder upon folder creation is getting held up for some reason with regard to the 2.4 frontend. but we don't know why or how to correct it. thanks for getting back to us, Quoting Andrew Morgan on Thu, 5 Jun 2014 11:02:11 -0700 (PDT): > On Thu, 5 Jun 2014, gavin.g...@ed.ac.uk wrote: > >> As you may be aware we are attempting this and have run into various >> problems. >> >> Currently we have a mixed murder of 2.3.15 backends and 2.4.17 backends. >> We are now fairly confident that we can xfer accounts succesfully between >> these backends. The problems we had appear to have been with a very small >> number of accounts on our older backends that had corrupt cyrus.index >> files. >> >> However we are now having trouble configuring frontends that will work >> with this mixed murder environment while we xfer our users accross. >> >> If we use our existing 2.3.15 frontends then users have who have been >> migrated lose the ability to see other accounts in the "Other Users" name >> space. >> >> On the other hand if we introduce 2.4.17 frontends then we see strange >> behaviour around folder creation. Clients can create the folders but >> autosubscription fails with the client being told the new folder doesn't >> exist. If one waits a minute or two one can manually subscribe to the >> folder. > > This is tickling my memory, but I can't recall exactly what it was. > I remember running into a problem like this as well. Something about > the frontend's mailbox database not being updated in a timely > fashion... > >> So far we have not upgraded the mupdate master. Is this a mistake? >> >> In terms of the frontend config we have added >> >> suppress_capabilities: ESEARCH QRESYNC WITHIN XLIST LIST-EXTENDED >> >> to the 2.4.17 frontends, otherwise the config is identical to our 2.3.15 >> frontends. Is there any other config changes we should be aware of? > > I used the following when I upgraded from 2.3 to 2.4: > > suppress_capabilities: ESEARCH LIST-EXTENDED QRESYNC WITHIN XLIST > ENABLE SORT=DISPLAY > > There was a thread I started back in October 2011 with the subject > "2.3 to 2.4 Murder upgrade" where I ran through the upgrade and the > workarounds I had to make. > > Andy > > -- Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
Actually we get very little logged on our mupdate master, just logins, checkpointing notices etc, nothing about the work done to maintain folder state. It's one of the reasons we feel so in the dark with this issue. On Sat, 7 Jun 2014, Wesley Craig wrote: > On 07 Jun 2014, at 14:49, Gavin Gray wrote: >> yes it looks like for some reason the mupdate procedure that happens within >> the murder upon folder creation is getting held up for some reason with >> regard to the 2.4 frontend. but we don't know why or how to correct it. > > You should have logs from the mupdate master indicating when the various > folder state changes happen, when the various frontends receive that > information, etc. > > :wes > > Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
yes we have allowallsubscribe: yes in our frontend imapd.conf thanks On Tue, 10 Jun 2014, Michael Menge wrote: Hi, did you use allowallsubscribe in your imapd.conf? copied from imapd.conf manpage allowallsubscribe: 0 Allow subscription to nonexistent mailboxes. This option is typically used on backend servers in a Murder so that users can subscribe to mailboxes that don't reside on their "home" server. This option can also be used as a workaround for IMAP clients which don't play well with nonexistent or unselectable mailboxes (e.g., Microsoft Outlook). Regards Michael Menge Quoting gavin.g...@ed.ac.uk: Actually we get very little logged on our mupdate master, just logins, checkpointing notices etc, nothing about the work done to maintain folder state. It's one of the reasons we feel so in the dark with this issue. On Sat, 7 Jun 2014, Wesley Craig wrote: > On 07 Jun 2014, at 14:49, Gavin Gray wrote: > > yes it looks like for some reason the mupdate procedure that happens > > within the murder upon folder creation is getting held up for some > > reason with regard to the 2.4 frontend. but we don't know why or how to > > correct it. > > You should have logs from the mupdate master indicating when the various > folder state changes happen, when the various frontends receive that > information, etc. > > : wes > > Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
sorry I should've been more explicit, we also have the same thing in our backend's imapd.conf On Tue, 10 Jun 2014, Michael Menge wrote: Hi, Quoting gavin.g...@ed.ac.uk: yes we have allowallsubscribe: yes in our frontend imapd.conf the man page mentions backend servers thanks On Tue, 10 Jun 2014, Michael Menge wrote: > Hi, > > did you use allowallsubscribe in your imapd.conf? > > copied from imapd.conf manpage > > allowallsubscribe: 0 > Allow subscription to nonexistent mailboxes. This option is > typically > used on backend servers in a Murder so that users can subscribe to > mailboxes that don't reside on their "home" server. This option can > also be used as a workaround for IMAP clients which don't play > well with nonexistent or unselectable mailboxes (e.g., Microsoft > Outlook). > > Regards > > Michael Menge > > > Quoting gavin.g...@ed.ac.uk: > > > Actually we get very little logged on our mupdate master, just logins, > > checkpointing notices etc, nothing about the work done to maintain > > folder state. It's one of the reasons we feel so in the dark with this > > issue. > > > > On Sat, 7 Jun 2014, Wesley Craig wrote: > > > > > On 07 Jun 2014, at 14:49, Gavin Gray wrote: > > > > yes it looks like for some reason the mupdate procedure that > > > > happens > > within the murder upon folder creation is getting held > > > > up for some > > reason with regard to the 2.4 frontend. but we > > > > don't know why or how to > > correct it. > > > > You should have logs from the mupdate master indicating when the > > > > various > folder state changes happen, when the various frontends > > > > receive that > information, etc. > > > > : wes > > > > Gavin Gray > > Edinburgh University Information Services > > Rm 2013 JCMB > > Kings Buildings > > Edinburgh > > EH9 3JZ > > UK > > tel +44 (0)131 650 5987 > > email gavin.g...@ed.ac.uk > > > > -- > > The University of Edinburgh is a charitable body, registered in > > Scotland, with registration number SC005336. > > > > Cyrus Home Page: http://www.cyrusimap.org/ > > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > > To Unsubscribe: > > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > > > > > > M.MengeTel.: (49) 7071/29-70316 > Universität Tübingen Fax.: (49) 7071/29-5912 > Zentrum für Datenverarbeitung mail: > michael.me...@zdv.uni-tuebingen.de > Wächterstraße 76 > 72074 Tübingen > Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
I've switched to debug logging now on our mupdate master, thanks On Mon, 9 Jun 2014, Wesley Craig wrote: > On 09 Jun 2014, at 07:14, gavin.g...@ed.ac.uk wrote: >> Actually we get very little logged on our mupdate master, just logins, >> checkpointing notices etc, nothing about the work done to maintain folder >> state. It's one of the reasons we feel so in the dark with this issue. > > Set logging on the mupdate master to debug, it shares lots of details. In > particular, you want to see cmd_set and cmd_find. The cmd_set is from the > backend making changes to the mupdate master. The cmd_find is *either* a > frontend looking for particular mailboxes *or* the normal streaming of state > change events from the mupdate master to the frontends. Obviously, there can > be timing issues, but the typical paradigm implemented on the frontends is > that when an expected mailbox isn't there, the frontend will specifically ask > the mupdate master about it using a kick_mupdate(). > > :wes > Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
Hi Wes, It looks like the whole mupdate thing is working perfectly well. If I create a folder while connected to my 2.4.17 frontend then the logs show the backend issuing the cmd_set and then a bunch of cmd_find going out including to the frontend in question. Furthermore the new folder really is there in the mailboxes.db on the frontend. So in a way that's reassuring, but then why is the frontend telling email clients that the folder doesn't exist when a request to subscribe to it comes in? We aren't seeing any kick_mupdate getting logged. The only thing is we see two cmd_sets is that normal? : Jun 10 11:07:29 mupd1 mupdate[31917]: cmd_set(fd:5, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_set(fd:5, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:29, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:29, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:24, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:24, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:25, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:25, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:10, user.gaving.gavarella) Jun 10 11:07:30 mupd1 mupdate[31917]: cmd_find(fd:10, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:11, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:11, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:36, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:36, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:26, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:38, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:38, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:14, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:22, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:22, user.gaving.gavarella) Jun 10 11:07:31 mupd1 mupdate[31917]: cmd_find(fd:26, user.gaving.gavarella) ... fd 38, and 26 is the 2.4.17 frontend in question. Thanks for helping out with this. Gavin On Mon, 9 Jun 2014, Wesley Craig wrote: > On 09 Jun 2014, at 07:14, gavin.g...@ed.ac.uk wrote: >> Actually we get very little logged on our mupdate master, just logins, >> checkpointing notices etc, nothing about the work done to maintain folder >> state. It's one of the reasons we feel so in the dark with this issue. > > Set logging on the mupdate master to debug, it shares lots of details. In > particular, you want to see cmd_set and cmd_find. The cmd_set is from the > backend making changes to the mupdate master. The cmd_find is *either* a > frontend looking for particular mailboxes *or* the normal streaming of state > change events from the mupdate master to the frontends. Obviously, there can > be timing issues, but the typical paradigm implemented on the frontends is > that when an expected mailbox isn't there, the frontend will specifically ask > the mupdate master about it using a kick_mupdate(). > > :wes > Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: xfer problems between 2.3.15 and 2.4.17
this fix, detailed below, has indeed solved our problem with folder creation on our new 2.4.17 frontends. many thanks On Tue, 10 Jun 2014, Andrew Morgan wrote: > On Tue, 10 Jun 2014, gavin.g...@ed.ac.uk wrote: > >> Hi Wes, >> >> It looks like the whole mupdate thing is working perfectly well. >> If I create a folder while connected to my 2.4.17 frontend then the logs >> show the backend issuing the cmd_set and then a bunch of cmd_find going >> out including to the frontend in question. Furthermore the new folder >> really is there in the mailboxes.db on the frontend. So in a way that's >> reassuring, but then why is the frontend telling email clients that the >> folder doesn't exist when a request to subscribe to it comes in? We aren't >> seeing any kick_mupdate getting logged. > > I'm pretty sure your problem is that you have the proxyservers variable set > in imapd.conf on your frontend. See this message from the archives: > > > http://asg.andrew.cmu.edu/archive/message.php?mailbox=archive.info-cyrus&searchterm=Murder%20mailbox%20create%20race%20condition&msg=54193 > > I ran into this same problem, which was introduced by changes in v2.4.13. The > imapd.conf manpage says: > > proxyservers: >A list of users and groups that are allowed to proxy for other >users, separated by >spaces. Any user listed in this will be allowed to login for any >other user: use with >caution. In a standard murder this option should ONLY be set on >backends. DO NOT SET > on frontends or things won't work properly. > > Let us know if removing proxyservers from your frontends fixes the problem! > > Andy > > > Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
possible problems with deduplication database
We run our cyrus 2.4 murder with deduplication switched on. Generally speaking this works well, but every so often user's complain about receiving duplicate messages. We've been taking a look at our delivery logs and our delivery db and it does seem to sometimes behave oddly e.g a series of messages gets delivered with the same Message-ID and if dump deliver.db we can see this sort of thing: id: to: user. at: 1411970492 uid: 24022 id: to: user. at: 1411974127 uid: 24023 id: to: user. at: 1411977693 uid: 24026 id: to: user. at: 1411988514 uid: 24031 id: to: user. at: 1411992092 uid: 24033 id: to: user. at: 1412059168 uid: 24042 id: to: user. at: 1412060492 uid: 24043 id: to: user. at: 1412070549 uid: 24051 id: to: user. at: 1412073074 uid: 24053 id: to: user. at: 1412075354 uid: 24055 these are quite isolated incidents, but surely this shouldn't happen at all? I suppose I'm asking if there are any known reasons why the deduplication process might behave in this way? We run cyr_expire -E 1 every night Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
deduplication issue
We run our cyrus 2.4 murder with deduplication switched on. Generally speaking this works well, but every so often user's complain about receiving duplicate messages. We've been taking a look at our delivery logs and our delivery db and it does seem to sometimes behave oddly e.g a series of messages gets delivered with the same Message-ID and if dump deliver.db we can see this sort of thing: id: to: user. at: 1411970492 uid: 24022 id: to: user. at: 1411974127 uid: 24023 id: to: user. at: 1411977693 uid: 24026 id: to: user. at: 1411988514 uid: 24031 id: to: user. at: 1411992092 uid: 24033 id: to: user. at: 1412059168 uid: 24042 id: to: user. at: 1412060492 uid: 24043 id: to: user. at: 1412070549 uid: 24051 id: to: user. at: 1412073074 uid: 24053 id: to: user. at: 1412075354 uid: 24055 these are quite isolated incidents, but surely this shouldn't happen at all? I suppose I'm asking if there are any known reasons why the deduplication process might behave in this way? We run cyr_expire -E 1 every night Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: delete and expunge delayed and delprune on replica?
Hello there, We run a murder with several backends each of which replicates to a slave. Our understanding is that the if one is using delayed deletion and expunge then any cyr_expire jobs must be run on both the master and the replicant as the cyr_expire process is local to a backend regardless of whether that backend is a master or a replicant. In our case we run delprune cmd="cyr_expire -D 30 -E 1 -X 4" at=0400 on every backend However I think there is a lot of confusion over the details of all this since 2.4.x e.g. We don't really understand how the configuration option expunge_days: 7 affects cyr_expire I'd also like to know what the details of the difference between running cyr_expire with and without the -a option actually are. If anyone can helpt to clarify all this then i@m dure many would appreciate it, thanks, Gavin Gray On Thu, 2 Oct 2014, Marcus Schopen wrote: > Hi, > > in a master/slave setup I've activated delete_mode and expunge_mode on > master and salve side. > > imapd.conf: > delete_mode: delayed > expunge_mode: delayed > > cyrus.conf: > delprune cmd="/usr/sbin/cyrus expire -E 1 -X 7 -D 7" at=0501 > > Does is make sense to set delete and expunge mode to delayed and run > delpune as an event on slave side too or should this only configured on > master side and delete/expunge delayed and delprune configuration on > master will also effect the replica? > > Ciao! > > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > Gavin Gray Edinburgh University Information Services Rm 2013 JCMB Kings Buildings Edinburgh EH9 3JZ UK tel +44 (0)131 650 5987 email gavin.g...@ed.ac.uk -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
