Dovecot gzip u.files on sdboxes
Hi, I was wondering if it's possible to lighten the process of converting sdbox mailboxes to zlib by, instead of using dsync (that brings much more io, copying from alt - and copying of unneeded data as we have many mails already compressed from activating zlib a couble of years ago) just compressing each of the u files. By compression of the u files I mean opening the uncompressed u file for reading, a u.temp file for writting and: 1. copy the dbox message header from u to u.tmp 2. compress the plain message and write it to u.tmp 3. fix the message_size_hex on the u.tmp dbox message header to the new, compressed size 4. copy the dbox metadata from u to u.tmp with Zphysical_size added 5. copy the u stat times to u.tmp 6. replace u with u.tmp Is the message_size_hex stored anywhere else (indexes) that may cause this to not work? Thank you.
Re: Dovecot gzip u.files on sdboxes
> On 01/02/2023 14:07 EET Jose Celestino wrote: > > > Hi, > > I was wondering if it's possible to lighten the process of converting > sdbox mailboxes to zlib by, instead of using dsync (that brings much more io, > copying from alt - and copying of unneeded data as we have many mails already > compressed from activating zlib a couble of years ago) just compressing each > of the u files. > > By compression of the u files I mean opening the uncompressed u file for > reading, a u.temp file for writting and: > > 1. copy the dbox message header from u to u.tmp > > 2. compress the plain message and write it to u.tmp > > 3. fix the message_size_hex on the u.tmp dbox message header to the new, >compressed size > > 4. copy the dbox metadata from u to u.tmp with Zphysical_size added > > 5. copy the u stat times to u.tmp > > 6. replace u with u.tmp > > Is the message_size_hex stored anywhere else (indexes) that may cause this to > not work? > > Thank you. The header around the u file is about the physical contents only, you can rewrite that header as long as you preserve any other fields. You can use `doveadm dump u.1` to see how it works. Aki
Corrupted sizes in cache once again
Hi,
I run a fairly large Dovecot Installation (around 100k mailboxes) on
several servers.
gzip compression is on.
Every once in a while I get the dreaded "cache corruption" messages in
the log:
Error: Corrupted record in index cache file
/[redacted]/Maildir/dovecot.index.cache: UID 3868: Broken physical size
in mailbox INBOX:
read(zlib(/[redacted]/Maildir/cur/1674129792.M797543P21755.node2,S=8099,W=8276:2,))
failed: Cached message size smaller than expected (2877 < 8099,
box=INBOX, UID=3868)
Error: Corrupted record in index cache file
/[redacted]/Maildir/dovecot.index.cache: UID 3875: Broken physical size
in mailbox INBOX:
read(zlib(/[redacted]/Maildir/cur/1674212201.M985809P29112.node2,S=13907,W=14121:2,))
failed: Cached message size smaller than expected (5533 < 8192,
box=INBOX, UID=3875)
The first entry shows 2877 (size on disk) vs. 8099 (real size unzipped,
also in the filename: S=8099).
The second entry shows 5533 (size on disk) vs. 8192 - this is not
correct in any way. Size on disk is 13907 as noted in the filename.
Both mails were delivered trough LMTP and retrieved by the POP3 service.
Anyone with an idea what might be happening here? I've read all
available info in the doc and in the previous discussions / bug reports,
but nothing seems to match my case. And where does that 8192 come from -
it looks suspicious?
Version is 2.3.7.2 (Ubuntu 20.04)
doveconf -n excerpt:
protocol lmtp {
mail_plugins = " old_stats mail_log notify zlib sieve quota"
}
protocol imap {
mail_max_userip_connections = 50
mail_plugins = " old_stats mail_log notify zlib imap_old_stats quota
imap_quota imap_sieve"
}
protocol pop3 {
mail_max_userip_connections = 50
mail_plugins = " old_stats mail_log notify zlib"
}
Tim
Re: Corrupted sizes in cache once again
I would like to add some more observations:
I hav another mailbox with one mail's size being saved wrong in the
cache file:
doveadm dump:
RECORD: seq=40, uid=1202, flags=0x00
- ext 5 cache : 8504 (3821)
- ext 6 vsize : 145625 (d9380200)
: vsize = 5166845802713
: highest_uid = 0
: message_count = 8716
- cache offset=8504 size=212, prev_offset = 0
- hdr.From: 42: From: [redacted]
- hdr.Message-ID: 45: Message-ID: [redacted]
- hdr.Subject: 44: Subject: [redacted]
- flags: (2000)
RECORD: seq=41, uid=1203, flags=0x00
- ext 5 cache : 8716 (0c22)
- ext 6 vsize : 105141 (b59a0100)
: vsize = 105141
: highest_uid = 0
: message_count = 0
- cache offset=8716 size=280, prev_offset = 0
- hdr.From: 104: From: [redacted]
- hdr.Message-ID: 108: Message-ID: [redacted]
- hdr.Subject: 105: Subject: [redacted]
- flags: (2000)
The message with seq=40 is the mail dovecot complains about:
Error: Corrupted record in index cache file
/[redacted]/Maildir/dovecot.index.cache: UID 1202: Broken physical size
in mailbox INBOX:
read(zlib(/[redacted]/Maildir/new/1675176154.M829333P14495.node2,S=232816,W=236045))
failed: Cached message size smaller than expected (145082 < 146788,
box=INBOX, UID=1202)
Size on disk is 145082
Size after decompression is 232816 (as expected)
Message with seq=41 is correct with both vsize in cache and decompressed
size being equal.
Before the POP3 access that produced the error both mails were only ever
touched by the dovecot lmtp delivery process.
Am 01.02.23 um 14:13 schrieb Tim Evers:
Hi,
I run a fairly large Dovecot Installation (around 100k mailboxes) on
several servers.
gzip compression is on.
Every once in a while I get the dreaded "cache corruption" messages in
the log:
Error: Corrupted record in index cache file
/[redacted]/Maildir/dovecot.index.cache: UID 3868: Broken physical
size in mailbox INBOX:
read(zlib(/[redacted]/Maildir/cur/1674129792.M797543P21755.node2,S=8099,W=8276:2,))
failed: Cached message size smaller than expected (2877 < 8099,
box=INBOX, UID=3868)
Error: Corrupted record in index cache file
/[redacted]/Maildir/dovecot.index.cache: UID 3875: Broken physical
size in mailbox INBOX:
read(zlib(/[redacted]/Maildir/cur/1674212201.M985809P29112.node2,S=13907,W=14121:2,))
failed: Cached message size smaller than expected (5533 < 8192,
box=INBOX, UID=3875)
The first entry shows 2877 (size on disk) vs. 8099 (real size
unzipped, also in the filename: S=8099).
The second entry shows 5533 (size on disk) vs. 8192 - this is not
correct in any way. Size on disk is 13907 as noted in the filename.
Both mails were delivered trough LMTP and retrieved by the POP3 service.
Anyone with an idea what might be happening here? I've read all
available info in the doc and in the previous discussions / bug
reports, but nothing seems to match my case. And where does that 8192
come from - it looks suspicious?
Version is 2.3.7.2 (Ubuntu 20.04)
doveconf -n excerpt:
protocol lmtp {
mail_plugins = " old_stats mail_log notify zlib sieve quota"
}
protocol imap {
mail_max_userip_connections = 50
mail_plugins = " old_stats mail_log notify zlib imap_old_stats quota
imap_quota imap_sieve"
}
protocol pop3 {
mail_max_userip_connections = 50
mail_plugins = " old_stats mail_log notify zlib"
}
Tim
Re: IMAP tuning for Outlook 365
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi, On Tue, 2023-01-31 at 11:14 -0500, Paul Kudla wrote: > > Ok sorry to NOT have an answer you are looking for but I don't want > anyone wasting their time on what is clearly a microsoft issue. > > Outlook 365 / Outlook since 2010 simply does not support imap. I can second this. I personally don't use Windows, or MacOS, but have had to support it on desktops for other people. I found that from time to time Outlook would not exit properly (hence people not seeing new incoming email) requiring either a restart of the machine, or using the system utils to manually kill the background processes that exit cleanly. I do think that Outlook is not the best email client in any given environment. My solution was to find another email client - I don't like thunderbird, there is Canary, but if you want the closest drop-in replacement for Outlook on a Windows desktop there is "eM Client" ... there is a port of Evolution (my preferred client) for Windows, but I'm not so sure about the sources I have found for it - the MacOS ports of Evolution are reputable, and I would recommend that. - -- Nikolai Lusan -BEGIN PGP SIGNATURE- iQIyBAEBCgAdFiEEVfd4GW6z4nsBxdLo4ZaDRV2VL6QFAmPbHz4ACgkQ4ZaDRV2V L6QYjw/4lr1lAKB4NIlQED14vIq5p83qmnYG32Vw0jjwFYJ1pwpb4E8kz4VMrQBM //IrFB9DM/zzJwulAt7yUnFLzduWKC/G7lZlSpBiQfMlD8Iu8f2kzaAMA7WdG4IF Amobnh0jFoDooKQ1Fo6ExzyMV6iefsTwjN5snc9SDVO5/R+uXhvLttCnT0faWoRp tUz43yuzfozdUe+WNQ47S6zNbyOyJrBLFQr9zRyDheCiWynJSSvZa1CF8S66Ho29 4fndeJRDTQcrbzRzMBjcNR6VtX+yQ9GwhZXd7fO+pAw/n+jiRnSVO2hlepWbZYEo 3JgzOrF6JsTd19eoM0rAcrtVfILHPGSo8sSpb+utqIhQVgJyN62NgZp1gM7XvWbP ueZWStqnuIo2P3qA6TwkkEP4ckt0Jm3pndxXhKv9xLOWWpXadKx1rcTaGBcxKKJr PahfAgu3YaViIAzGGr7oGTQTHJq7JS7zkm7VAtWbVoAjgbJXZvrpoezOTnyG7XAb 5EQIf0Qkywm8qvlr5HKVmYAxOPFM1+3EJq2Gd2PMmcC2mk3gG5HN6L62MU3xjxWO f8dSVjj9XXM93lz1i807T3ydAsTsHDwmzI4GJncf8X/icqiN5uUJnXG9VqvT3T0n EdtopYbU6yXa5zJj/v4Bu+GjnaN/Vrjz9bO2CXKZyOcKcdmelg== =j+SC -END PGP SIGNATURE-
