Incorrect quota usage after XFER between servers
Hi, We run a traditional murder with Cyrus 2.3.12p2 - 2 frontend, 2 backend, and a mupdate server. Whenever we migrate mailboxes all the mailboxes for the domain are moved at the same time, and there's a high chance that the quota usage will get messed up and mailboxes will have more than 100% of their quota used. Running the quota -f command will fix the usage on the new server but why is it going wrong in the first place? The migration is performed using some Perl similar to the following: $imapAdmin = Cyrus::IMAP::Admin->new( 'mailstore1' ); $imapAdmin->authenticate( User => "adminuser", Password => "adminpassword" ); $result = $imapAdmin->xfer( 'user/usern...@example.com', 'mailstore2' ); Luckily we don't have to migrate mailboxes very often, but it's annoying when we do as we have to disable the lmtp_over_quota_perm_failure option in imapd.conf otherwise we'll end up bouncing mail between the migration and quota repair finishing. In the changelog for 2.3 http://www.cyrusimap.org/docs/cyrus-imapd/2.3.16/changes.php I can see that 2.3.14 says # Fixed incorrect quota calculations on sync_server when replicating unexpunged messages (thanks David Carter) Is this fix likely to apply to mailboxes moved between servers with the XFER command as well? If so, I'll see if we can get 2.3.16 installed. Regards, Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Does anyone allow unlimited or extremely large quotas?
On 16 Nov 2010, at 13:38, Adam Tauno Williams wrote: > > Our largest quota's a 4GB; without any issues. > > I think the issue you will encounter first is clients will start to > fall > down when folders exceed a 'reasonable' number of messages. Common > IMAP > clients I've seen start to exhibit severe performance issues beyond a > few hundred thousand messages. Is that with the server and client on the same LAN or with the client on a low speed WAN connection? We find that 50,000 messages in a folder is more than enough to make Thunderbird/Outlook unresponsive for minutes at a time when connecting to a remote server. Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: XFER problems with 2.4.6
On 15 Feb 2011, at 13:42, karave...@mail.bg wrote: > > Feb 11 14:41:10 stor5 cyrus/imap[11569]: LOSTQUOTA: unable to record > quota file .xx!user.000 > Feb 11 13:23:47 stor5 cyrus/imap[11445]: Deleted mailbox ^xx! > user^000^pay This looks similar to when I tried to migrate a mailbox from 2.4.6 back to 2.3.12, there were some weird log entries that had mismatched '.' vs '^' substitutions. unixhierarchysep is set to on for both servers. In your example, it says .xx!user.000 for the quota, but then deleting ^xx!user^000 - this is the same thing that I had. After the XFER failed, dumping the mailboxes.db on the mupdate server revealed that there were duplicated mailboxes - one in each format. The mailbox in question was xx...@.co.uk (masked of course) cyrus@mupdate $ /usr/lib/cyrus-imapd/cyr_dbtool /var/lib/imap/ mailboxes.db skiplist show ^co^uk!user^xx^yy ^co^uk!user^xx^yy^Sent ^co^uk!user^xx^yy^Trash .co.uk!user.xx^yy .co.uk!user.xx^yy.Sent .co.uk!user.xx^yy.Trash The two different formats were associated with different servers, the first set was shown as being on the 2.3.12 server where it had failed to move it, tried to back out the move, and failed with that too. The second (correctly formatted) ones were shown as being on the 2.4.6 server. The mailbox contents were actually still on the 2.4.6 server so it was a matter of removing the broken entries using cyr_dbtool. This was 2.4.6 -> 2.3.12 so maybe there's some discrepancy in what 2.4.6 sends vs what 2.3 and 2.4 expect? Simon -- Simon Amor Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Move of mailboxes
On 5 May 2011, at 09:15, Frank Elsner wrote: > we have 2 backends in a murder environment (running cyrus 2.3.11) which works. > > And now for the problem: > > We want to replace both backends by new hardware. Therefore the mailboxes > from both must be moved to the replacement machines. > > Is there a recommended procedure to do the move? Any pointers (even to > pitfalls) are welcome. You should upgrade Cyrus on the new machines because there's a quota bug < 2.3.14 (I think) where the quota on the new machine counts expunged messages or something similar so you end up needing to run quota -f to fix it. For a live migration, where downtime per mailbox is minimal - add the new hardware into the murder as machines 3 and 4. Use XFER to move the mailboxes to the new machines. XFER from 2.3.12 to 2.4.8 worked fine here. The problem with this method is you have to issue an XFER command for each mailbox separately - fine if you have a separate database with mailboxes in it, but not so helpful if you only have the cyradm "listmailbox" output. Advantages are that you can transfer a test mailbox over first to check that the new server works ok (authentication etc) and you can do the migration over the course of a few days rather than having to do them all at once. If you can afford to take the servers offline for a few hours to half a day or more depending on spool size, you can simply copy the data over. This has the advantage that you can name the new server exactly the same as the old server. Stop cyrus (and exim/sendmail/postfix) on both old and new servers, use rsync to copy the spool and conf directories over to the new servers and then start Cyrus. Pitfalls here include things like mismatched BDB versions (convert mailboxes.db and others to skiplist format first). I've successfully migrated 2.2.x (CentOS4) to 2.4.8 using this method. Simon -- Simon Amor si...@leaky.org http://www.leaky.org/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: mailboxes.db discrepancies between mailbox and mupdate servers
On 9 Jul 2013, at 22:20, Shawn Winnington-Ball wrote:
>
>> You can force a backend to push all of its mailboxes to the mupdate master
>> by running "ctl_mboxlist -m" on the backend. If you're not 100% sure
>> whether you want to push every mailbox before you know what state things
>> are in, you can individually push mailboxes, again using mupdate protocol.
>> Log in to the backend and run
>>
>> $ mupdatetest your.mupdate.server.com.
>> 1 MUPDATEPUSH user.foo
>
> However, I tried running this command and got
>
> B01 MUPDATEPUSH user.foo
> B01 BAD "Unrecognized command"
>
> I can't find MUPDATEPUSH in RFC 3656 either.
>
According to my notes, MUPDATEPUSH is used through an admin user's IMAP
connection (via imtest) not through mupdatetest.
If the mailbox doesn't actually exist, you might need to use LOCALCREATE via
IMAP to the backend prior to MUPDATEPUSH.
LC1 LOCALCREATE user.foo
ACL0 SETACL user.foo foo lrswipkxtea
MP1 MUPDATEPUSH user.foo
I use virtual domains so the syntax of the SETACL command may be slightly off
('foo' is the authenticated user as far as I can remember).
If you've already told the mupdate server about the mailbox, you probably don't
need the last 2 commands, just the localcreate should be enough to let them
connect to the mailbox.
Simon
Disclaimer: I'm not a cyrus expert - I just break it a lot :)
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re:
Hi Stefan, On 24 Jul 2013, at 22:13, "Stefan Schlörholz" wrote: > Hello Simon, > >> Did you try running "reconstruct -r -f ..."? > > I did try to run "reconstruct -r user.paul". The -f switch is not > known/accepted by my cyradm. Use the binary not the cyradm command. Depending on the system, it could be somewhere like /usr/lib/cyrus-imapd/reconstruct It's in the same place as imapd, lmtpd, and cyrus-master live (at least on CentOS) so you might be able to locate it using something like this command: ps ax | grep cyrus-master 32268 ?Ss 0:07 /usr/lib/cyrus-imapd/cyrus-master -d The -f argument to reconstruct according to the man page: Examine the filesystem underneath mailbox, adding all directories with a cyrus.header found there as new mailboxes. Useful for restoring mailboxes from backups. Regards, Simon (a different one) -- Simon Amor si...@leaky.org Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: undelete a single message, delete_mode + expunge_mode activated
On 10 Mar 2014, at 20:31, Marcus Schopen wrote:
>
> When deleting a single message from Inbox or a subfolder these message
> are not moved into a DELETED structure and keep staying at their
> originally place on filesystem. Those messages are definitely delete. I
> can't see them in my imap client anymore (tested with evolution and
> Thunderbird). How do I undelete these single messages?
Undeleting messages in a mailbox
The following examples assume you have an installation of cyrus where there are
binaries in /usr/lib/cyrus-imapd/ - if not, adjust path to suit.
List messages available to unexpunge:
# su cyrus -c "/usr/lib/cyrus-imapd/unexpunge -l user/si...@leaky.org"
Each message will give you something like the following:
UID: 11422
Size: 7786
Sent: Mon Mar 10 12:00:00 2014
Recv: Mon Mar 10 16:06:32 2014
Expg: Mon Mar 10 16:53:55 2014
From: john wade
To :
Cc :
Bcc :
Subj: {44}
re: sieveshell aborts with "expected rparen"
To unexpunge a single message:
# su cyrus -c "/usr/lib/cyrus-imapd/unexpunge -udv user/si...@leaky.org 11422"
restoring expunged messages in mailbox 'user/si...@leaky.org'
Unexpunged user/si...@leaky.org: 11422 => 11438
restored 1 expunged messages
To unexpunge all the messages and mark them as undeleted as well:
# su cyrus -c "/usr/lib/cyrus-imapd/unexpunge -adv user/si...@leaky.org"
NOTE: This isn't recursive
It will only restore the inbox. To find other folders run
# su cyrus -c "/usr/lib/cyrus-imapd/ctl_mboxlist -d" | grep leaky.org
leaky.org!user.simon0 default si...@leaky.org lrswipkxtecda
leaky.org!user.simon.Lists 0 default si...@leaky.org lrswipkxtecda
leaky.org!user.simon.Lists.cyrus0 default si...@leaky.org
lrswipkxtecda
leaky.org!user.simon.Deleted Messages 0 default si...@leaky.org
lrswipkxtecda
Run the unexpunge command for every folder that needs to have mail undeleted.
For folder names that have spaces ' ', the spaces need to be escaped with a
backslash
su cyrus -c "/usr/lib/cyrus-imapd/unexpunge -adv user/simon/Deleted\
messa...@leaky.org"
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
