imap flags lost after upgrade (2.0.15 to 2.2.6)
Hello everybody, The university I work for is upgrading its cyrus 2.0.15 (+patches) imap-server to a cyrus 2.2.6 murder. The mailboxes on the old server are being copied over, databases converted, sieve scripts compiled etc. One problem though, all IMAP flags seems to be gone after running 'reconstruct' on a moved mailbox. It seems that the SYSFLAGS USERFLAGS bitmasks are all set to . I am running: 'reconstruct -r -f user.MAILBOX' Running reconstruct with the '-x' flag would recreate all found mailboxes as new, but without '-x' reconstruct should try to preserve the data in the files cyrus.*. This does not seem to work properly, at least not between versions 2.0.15 and 2.2.6. Has anyone else run into this problem? Is there some "upgrade-tool" that I should be using, but haven't found? /skitta -- Tomas 'Skitta' Lindroos. Åbo Akademi University Computing Centre --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
sieve configuration (turning features on/off)
Hello everybody, I am configuring a cyrus 2.2.6 murder, and I am going to use sieve for mail filtering. Is there any good way of configuring the 'capabilities' for sieve to allow or disallow certain features. I'd like to disable at least "reject" and "vacation" (users are not allowed to forward their mail, at least not via the imap-server, and vacation will be done elsewhere, if allowed). Can this be done in some configuration file (imapd.conf?), or by editing some file in the sieve library source? The documentation seems to be quite sparse on this topic, but I think I've seen a post to this list mentioning that some sieve features might be configurable in imapd.conf. /skitta -- Tomas 'Skitta' Lindroos. Åbo Akademi University Computing Centre --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: imap flags lost after upgrade (2.0.15 to 2.2.6) [problem solved]
Hello again. Sorry for replying to my own post. It seems that cyrus cannot upgrade the cyrus.index files in mailboxes between versions 2.0.x and 2.2.x. I had to write a program that 'upgrades' this database, which is then run on all mailboxes before running 'reconstruct' on the new server. Now all flags and other data is transferred correctly. (The records stored in the file has changed format and length, so reconstruct gets an error message from the library function that reads index-files. After that, it rebuilds the index-file from scratch, with most values set to zero. This program upgrades the index file from "minor version" 3 to 6.) /skitta Tomas Lindroos wrote: Hello everybody, The university I work for is upgrading its cyrus 2.0.15 (+patches) imap-server to a cyrus 2.2.6 murder. The mailboxes on the old server are being copied over, databases converted, sieve scripts compiled etc. One problem though, all IMAP flags seems to be gone after running 'reconstruct' on a moved mailbox. It seems that the SYSFLAGS USERFLAGS bitmasks are all set to . I am running: 'reconstruct -r -f user.MAILBOX' Running reconstruct with the '-x' flag would recreate all found mailboxes as new, but without '-x' reconstruct should try to preserve the data in the files cyrus.*. This does not seem to work properly, at least not between versions 2.0.15 and 2.2.6. Has anyone else run into this problem? Is there some "upgrade-tool" that I should be using, but haven't found? /skitta --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: imap flags lost after upgrade (2.0.15 to 2.2.6) [problem solved]
Ken Murchison wrote: Tomas Lindroos DC wrote: I had to write a program that 'upgrades' this database, which is then run on all mailboxes before running 'reconstruct' on the new server. Now all flags and other data is transferred correctly. Exactly what does your script have to change? I'd like to fix the mailbox code. I took the header source code from mailbox.h in version 2.0.15 and 2.2.6 and looked at the differences, and checked with the documentation to see what the records in cyrus.index should look like. Having the structs for index_header and index_record for both versions, I read records from the old cyrus.index and constructed a new one. If everything seemed allright, i made a rename(oldfile, backupfile) and rename(newfile, oldfile). So I didn't do as much changes as I created a new database from scratch... I do a couple of assumptions while converting the files, which may or may not be sane. Do any of these look too dangerous to be used? index_record->content_lines = 0; /* wrong, but I don't parse all the messages to look for line counts right now */ index_record->cache_version = 0; /* I hope this contributes to getting the cache file rebuild from scratch */ index_header->generation_no = 0; /* as above, hopefully a cache gen mismatch. (this could be read from the cache-file, of course) */ Does this seem ok? Also, no flags are shown to the imap client if the counters deleted, answered and flagged in cyrus.index are set to 0, so while converting the index_records I count these flags and set the values in the header. Anyway, when I run reconstruct, mailbox_open_index() fails (line 515 in reconstruct.c), whereafter mailbox_read_index_record() does something that makes the check "if (old_index.uid == uid[msg])" false. Which in the end means that a lot of values for the current message are set to zero. I'm sorry, but I don't have the time to delve into this any further right now. Ken, I could supply you one of my example-mailboxes that fails to convert, if that would help you get on with debugging this problem. /skitta -- Tomas 'Skitta' Lindroos. Planerare vid Datacentralen, UNIX-sektionen <[EMAIL PROTECTED]>, tel.nr. (215)4469 --- Cyrus Home Page: http://asg.web.cmu.edu/cyrus Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Murder + virtal hosting with ipaliasing problem
Hello everybody!
I am trying to build a cyrus murder with virtual hosting enabled. In the
future we will probably have two or three frontends on round robin dns, so
I have imapd:s running on an IP-aliased interface. This works fine, let's
call it foo.abo.fi.
Now, on the same frontend I need another set of imapd:s which run on yet
another IP-alised interface, which has a DNS-entry on another domain,
say bar.baz.org. The reasons for doing this is:
- they want "their" imap-server on their own domain
- they want to log in as "username", not "[EMAIL PROTECTED]"
- they want to authenticate to their active directory (through
saslauthd-krb5, all other virtual domains use mysql auxprop)
Now, if I create a mailbox "[EMAIL PROTECTED]" and log in as
[EMAIL PROTECTED] everything works as it should:
> telnet foo.abo.fi imap
* OK foo.abo.fi Cyrus IMAP4 Murder v2.2.12-AAU-RPM-2.2.12-3.RHEL4.1.aau server
ready
1 LOGIN [EMAIL PROTECTED]
1 OK User logged in
2 LIST "" *
* LIST (\Noinferiors) "." "INBOX"
2 OK Completed
3 SELECT INBOX
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]
* 3 EXISTS
* 1 RECENT
* OK [UNSEEN 3]
* OK [UIDVALIDITY 1160720446]
* OK [UIDNEXT 4]
3 OK [READ-WRITE] Completed
4 LOGOUT
* BYE LOGOUT received
4 OK Completed
But if I try to use the server on bar.baz.org, the LIST command fails to
retrieve any mailboxes:
> telnet bar.baz.org imap
* OK foo.abo.fi Cyrus IMAP4 Murder v2.2.12-AAU-RPM-2.2.12-3.RHEL4.1.aau server
ready
1 LOGIN [EMAIL PROTECTED]
1 OK User logged in
2 LIST "" *
2 OK Completed
3 SELECT INBOX
3 NO Mailbox does not exist
4 LOGOUT
* BYE LOGOUT received
4 OK Completed
The same thing happens when I omit the @baz.org part:
> telnet bar.baz.org imap
* OK foo.abo.fi Cyrus IMAP4 Murder v2.2.12-AAU-RPM-2.2.12-3.RHEL4.1.aau server
ready
1 LOGIN testuser
1 OK User logged in
2 LIST "" *
2 OK Completed
3 SELECT INBOX
3 NO Mailbox does not exist
4 LOGOUT
* BYE LOGOUT received
4 OK Completed
If I dump the contents of mailboxes.db I get the correct results on all
machines in the murder.
Ok, then some snippets from the config files:
bar.baz.org-imapd.conf:
configdirectory: /var/imap
sievedir: /var/sieve
defaultpartition: virtual
partition-virtual: /var/spool/virtual
hashimapspool: false
unixhierarchysep: no
altnamespace: yes
sharedprefix: shared
singleinstancestore: yes
duplicatesuppression: yes
allowusermoves: yes
servername: bar.baz.org
virtdomains: on
loginrealms: baz.org
defaultdomain: baz.org
sasl_pwcheck_method: saslauthd
(... + other SASL, lmtp, tls and mupdate stuff)
The working server foo.abo.fi has the same config, except for:
servername: foo.abo.fi
loginrealms: xxx yyy zzz abo.fi
defaultdomain: abo.fi
sasl_pwcheck_method: auxprop
I get the same result when using "virtdomains userid". I tried to mix the
forms but that didn't work at all (but there could have been other reasons
as well.)
Any ideas? It seems like the fact that the virtual server is on another
network (in DNS, not physically) somehow screws up the mailbox lookups
from mailboxes.db. (I've run several instances of imapd:s on other servers
with different configs, though not with virtual domains on, and I think
other people use it as well.) This should be possible to do, right? There
are no complaints in the logs.
cyrus.conf:
SERVICES {
mupdate cmd="/usr/lib64/cyrus-imapd/mupdate" listen=3905 prefork=1
imap cmd="/usr/lib64/cyrus-imapd/proxyd -C /etc/imapd-foo.conf"
listen="foo.abo.fi:imap" prefork=0
imaps cmd="/usr/lib64/cyrus-imapd/proxyd -s -C /etc/imapd-foo.conf"
listen="foo.abo.fi:imaps" prefork=0
himap cmd="/usr/lib64/cyrus-imapd/proxyd -C /etc/imapd-baz.org.conf"
listen="bar.baz.org:imap" prefork=0
himaps cmd="/usr/lib64/cyrus-imapd/proxyd -s -C /etc/imapd-baz.org.conf"
listen="bar.baz.org:imaps" prefork=0
}
I use the RHEL4-supplied x86_64 packages, though recompiled to disable the
autocreate patches.
/skitta
--
Tomas 'Skitta' Lindroos.
UNIX systemadministratör, Åbo Akademi, Datacentralen
[EMAIL PROTECTED], 02-215-4469
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: Spam and sieve vacation
On Mon, 27 Aug 2007, Janne Peltonen wrote: 5/week? Whee. We might achieve something remotely approaching that with personalized bayesian filtering (a multi-discipline, internationaly connected university receives quite a lot of ham that looks very much like spam, so we are a bit paranoid about false positives). Due to spam levels we have been disallowing "vacation" the last 4-5 years. This summer we made it possible for our users to enable an autoresponder. We use greylisting and spamassassin with bayesian filtering, and we are quite low on spam ending up in users inboxes, and also on false positives. We don't provide vacation via sieve though, we considered it not being the IMAP server that should send the answers. Instead we use a home-grown python script on a smtp server. Since may 73 people have actually used this service. During the last 12 weeks 17129 messages were not answered, of which 3002 had a spam level of 3 or more and 1103 messages would have been answered if they had come from inside addresses (our users can choose to answer only inside our domain). 2570 auto-replies were sent. Looking at the logs it seems that perhaps 25% of these replies should not have been sent and most of these can actually be caught with some more filters in the python script. That makes 200 backscatter messages a week for us this summer - probably worth it, and the filters can be improved. /skitta checking out rfc3834 right now - we seem to be fairly compliant :) -- Tomas 'Skitta' Lindroos. UNIX systems administrator, Åbo Akademi University, Computing Centre [EMAIL PROTECTED], +358-40-5177716 Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
