HIGHESTMODSEQ not increasing
Hi, I have a problem with my Dovecot IMAP Server on Debian Linux. I migrated it from FreeBSD to Debian, but since then the HIGHESTMODSEQ value for INBOX is not increasing, when I receive new e-mail. This causes my Groupware not to report any new e-mail via ActiveSync. Here is the protocol transscript: [I] <~>-> openssl s_client -crlf -connect mail.forberger-online.de:993 CONNECTED(0003) [...] --- read R BLOCK * OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN AUTH=LOGIN] Dovecot (Debian) ready. A LOGIN REDACTED REDACTED A OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE] Logged in A SELECT INBOX (CONDSTORE) * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent Junk NonJunk $label4 $label1 $label2 $label3 $label5 Old $Forwarded NOTJUNK $has_cal) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent Junk NonJunk $label4 $label1 $label2 $label3 $label5 Old $Forwarded NOTJUNK $has_cal \*)] Flags permitted. * 7286 EXISTS * 0 RECENT * OK [UIDVALIDITY 1291740963] UIDs valid * OK [UIDNEXT 158448] Predicted next UID * OK [HIGHESTMODSEQ 190086] Highest A OK [READ-WRITE] Select completed (0.001 + 0.000 secs). A SELECT INBOX (CONDSTORE) * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent Junk NonJunk $label4 $label1 $label2 $label3 $label5 Old $Forwarded NOTJUNK $has_cal) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent Junk NonJunk $label4 $label1 $label2 $label3 $label5 Old $Forwarded NOTJUNK $has_cal \*)] Flags permitted. * 7288 EXISTS * 0 RECENT * OK [UIDVALIDITY 1291740963] UIDs valid * OK [UIDNEXT 158450] Predicted next UID * OK [HIGHESTMODSEQ 190086] Highest A OK [READ-WRITE] Select completed (0.002 + 0.000 + 0.001 secs). The Groupware's IMAP client library is reporting, that MODSEQ is not increasing. It stays at 190086, nevertheless I receive new e-mail. I have tried the commands doveadm index and doveadm force-resync on the INBOX, without any change. My filesystem where the mailboxes reside is on ext4. Can you help me what is wrong? Best regards, Ronny Forberger -- Ronny Forberger ronnyforber...@ronnyforberger.de
How to configure Dovecot with Keycloak correctly
Hello I’m trying to configure Dovecot with Keycloak I’m using the proxy with
password grant method.
Here’s my dovecot-oauth2.plain.conf.ext
grant_url =
https://server-1.gary-springfield-mo.net/realms/sso/protocol/openid-connect/token
client_id = dovecot
client_secret = WtZ5uZsBi2QPfh1EWuMWjUVGNNT4HEGR
introspection_url =
https://server-1.gary-springfield-mo.net/realms/sso/protocol/openid-connect/token/introspect
introspection_mode = post
use_grant_password = yes
debug = yes
username_attribute = username
pass_attrs = host=server-1:8181 proxy=y proxy_mech=xoauth2
pass=%{oauth2:access_token}
And my dovecot-oauth2.token.conf.ext
grant_url =
https://server-1.gary-springfield-mo.net/realms/sso/protocol/openid-connect/token
client_id = dovecot
client_secret = WtZ5uZsBi2QPfh1EWuMWjUVGNNT4HEGR
tokeninfo_url =
https://server-1.gary-springfield-mo.net/realms/sso/protocol/openid-connect/token
introspection_url =
https://server-1.gary-springfield-mo.net/realms/sso/protocol/openid-connect/token/introspect
introspection_mode = post
use_grant_password = no
debug = yes
username_attribute = username
pass_attrs = pass=%{oauth2:access_token}
Do I also need to configure the dovecot-oauth2.conf.ext
Sent from my iPhone
Backups and disaster recovery
Hello all! After reading some of the past threads on backups, I was wondering if I could get a sanity check... I run a Maildir configuration for a small (10 mailboxes) mail server. Using "doveadm backup", for each mailbox I do: - Weekly full backups and then copy the files to a network-based filesystem.- Daily incremental backups and then copy the files to a network-based filesystem. My recovery assumption is that in case of hardware failure, I would re-set up the mail service and for each mailbox I will recover at the right directory the latest full mailbox backup and on top of it, each incremental backup to the latest day. Does this sound like a sound strategy? Also, I was thinking of setting up a second dovecot server on another server and replicating my primary on an hourly basis to decrease recovery time. But I looked at mbsync and it seems to require mailbox login/password for each mailbox (which I don't have). Is there an alternative? Thank you!
Re: Backups and disaster recovery
On 12/3/22 16:02, GDS wrote: After reading some of the past threads on backups, I was wondering if I could get a sanity check... I run a Maildir configuration for a small (10 mailboxes) mail server. Using "/doveadm backup"/, for each mailbox I do: - Weekly full backups and then copy the files to a network-based filesystem. - Daily incremental backups and then copy the files to a network-based filesystem. My recovery assumption is that in case of hardware failure, I would re-set up the mail service and for each mailbox I will recover at the right directory the latest full mailbox backup and on top of it, each incremental backup to the latest day. Does this sound like a sound strategy? Also, I was thinking of setting up a second dovecot server on another server and replicating my primary on an hourly basis to decrease recovery time. But I looked at mbsync and it seems to require mailbox login/password for each mailbox (which I don't have). Is there an alternative? At 07:00 every day, my mailserver rsyncs its entire root filesystem to another server with a large btrfs filesystem. All the mail is on that filesystem. I am using Maildir, not sure how to figure out whether that is Maildir++ or not: 00 7 * * * rsync -axH --delete --delete-excluded --exclude=.git / server.domain.tld:/storage0/bilbofull/ The target server has a snapshot maintenance script I wrote that runs every night, as well as weekly and monthly: 45 23 * * * /usr/local/sbin/snapshot-maintenance storage0 daily 50 23 1 * * /usr/local/sbin/snapshot-maintenance storage0 monthly 55 23 * * 7 /usr/local/sbin/snapshot-maintenance storage0 weekly The script keeps 7 daily snaps, 4 weekly snaps, and 6 monthly snaps. I deleted all the snapshots recently because I was doing significant maintenance on the filesystem and wanted to actually recover the disk space. This shows the snaps that currently exist: elyograg@smeagol:/usr/local/sbin$ sudo btrfs subvolume list /storage0 ID 68160 gen 711353 top level 5 path .snapshot/2022.11.01.storage0.monthly ID 68163 gen 718255 top level 5 path .snapshot/2022.11.06.storage0.weekly ID 68170 gen 729261 top level 5 path .snapshot/2022.11.13.storage0.weekly ID 68177 gen 746501 top level 5 path .snapshot/2022.11.20.storage0.weekly ID 68181 gen 756805 top level 5 path .snapshot/2022.11.24.storage0.daily ID 68182 gen 759352 top level 5 path .snapshot/2022.11.25.storage0.daily ID 68183 gen 761844 top level 5 path .snapshot/2022.11.26.storage0.daily ID 68184 gen 779474 top level 5 path qemu ID 68185 gen 764384 top level 5 path .snapshot/2022.11.27.storage0.weekly ID 68186 gen 766942 top level 5 path .snapshot/2022.11.28.storage0.daily ID 68187 gen 769388 top level 5 path .snapshot/2022.11.29.storage0.daily ID 68188 gen 771998 top level 5 path .snapshot/2022.11.30.storage0.daily ID 68189 gen 774588 top level 5 path .snapshot/2022.12.01.storage0.monthly ID 68190 gen 777168 top level 5 path .snapshot/2022.12.02.storage0.daily The end result of this is that I have backups of all the email at many different points in time. I back up a lot of other things into the btrfs filesystem as well. elyograg@smeagol:~$ df -h /storage0 Filesystem Size Used Avail Use% Mounted on /dev/sdb1 22T 3.5T 19T 17% /storage0 elyograg@smeagol:~$ sudo du -hs /storage0/bilbofull/var/vmail 6.5G /storage0/bilbofull/var/vmail I also have 10 mailboxes across several domains. There are more than 10 defined, but only 10 of them actually have email. The users are in a postfixadmin database. You should be able to use rsync to copy Maildirs from one server to another. That is IMHO one of the advantages to Maildir ... each change to the mailbox occurs with a single message file, so it is unlikely the mailbox will be corrupted if it changes during the copy. Thanks, Shawn
Re: Backups and disaster recovery
On Sat, Dec 03, 2022 at 11:02:54PM +, GDS wrote: > Hello all! After reading some of the past threads on backups, I was > wondering if I could get a sanity check... I run a Maildir > configuration for a small (10 mailboxes) mail server. Using "doveadm > backup", for each mailbox I do: > > - Weekly full backups and then copy the files to a network-based > filesystem. > > - Daily incremental backups and then copy the files to a > network-based filesystem. > > My recovery assumption is that in case of hardware failure, I would > re-set up the mail service and for each mailbox I will recover at the > right directory the latest full mailbox backup and on top of it, each > incremental backup to the latest day. Does this sound like a sound > strategy? Assumptions are dangerous things. Have you tested your assumptions - i.e. simulated recovering from a hardware failure - in order to be sure your backups and procedures are adequate? Are your server, and your network-based filesystem protected against bit-rot? > Also, I was thinking of setting up a second dovecot server on another > server and replicating my primary on an hourly basis to decrease > recovery time. But I looked at mbsync and it seems to require mailbox > login/password for each mailbox (which I don't have). Is there an > alternative? You might want to consider using ZFS. https://en.wikipedia.org/wiki/ZFS https://openzfs.org Jim Salter has written some fairly accessible tutorials. For instance: https://arstechnica.com/information-technology/2014/02/ars-walkthrough-using-the-zfs-next-gen-filesystem-on-linux/3/ https://arstechnica.com/information-technology/2015/12/rsync-net-zfs-replication-to-the-cloud-is-finally-here-and-its-fast/ https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/ https://jrs-s.net/category/open-source/zfs/ Sam
Re: HIGHESTMODSEQ not increasing
It works now again. It was supposingly an error with the maildir. I created the maildir around year 2002 and copied it several times between mail servers and OSes. I created an new maildir and copied back all messages. Best regards, Ronny Forberger Am 03.12.2022 um 10:31 schrieb Ronny Forberger: Hi, I have a problem with my Dovecot IMAP Server on Debian Linux. I migrated it from FreeBSD to Debian, but since then the HIGHESTMODSEQ value for INBOX is not increasing, when I receive new e-mail. This causes my Groupware not to report any new e-mail via ActiveSync. Here is the protocol transscript: [I] <~>-> openssl s_client -crlf -connect mail.forberger-online.de:993 CONNECTED(0003) [...] --- read R BLOCK * OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN AUTH=LOGIN] Dovecot (Debian) ready. A LOGIN REDACTED REDACTED A OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE] Logged in A SELECT INBOX (CONDSTORE) * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent Junk NonJunk $label4 $label1 $label2 $label3 $label5 Old $Forwarded NOTJUNK $has_cal) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent Junk NonJunk $label4 $label1 $label2 $label3 $label5 Old $Forwarded NOTJUNK $has_cal \*)] Flags permitted. * 7286 EXISTS * 0 RECENT * OK [UIDVALIDITY 1291740963] UIDs valid * OK [UIDNEXT 158448] Predicted next UID * OK [HIGHESTMODSEQ 190086] Highest A OK [READ-WRITE] Select completed (0.001 + 0.000 secs). A SELECT INBOX (CONDSTORE) * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent Junk NonJunk $label4 $label1 $label2 $label3 $label5 Old $Forwarded NOTJUNK $has_cal) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent Junk NonJunk $label4 $label1 $label2 $label3 $label5 Old $Forwarded NOTJUNK $has_cal \*)] Flags permitted. * 7288 EXISTS * 0 RECENT * OK [UIDVALIDITY 1291740963] UIDs valid * OK [UIDNEXT 158450] Predicted next UID * OK [HIGHESTMODSEQ 190086] Highest A OK [READ-WRITE] Select completed (0.002 + 0.000 + 0.001 secs). The Groupware's IMAP client library is reporting, that MODSEQ is not increasing. It stays at 190086, nevertheless I receive new e-mail. I have tried the commands doveadm index and doveadm force-resync on the INBOX, without any change. My filesystem where the mailboxes reside is on ext4. Can you help me what is wrong? Best regards, Ronny Forberger -- Ronny Forberger ronnyforber...@ronnyforberger.de
Backups
I started to investigate using doveadm backup to backup my mail system. I have a small number of users and the mail store is not large. It uses maildir format. I setup a test system that is not connected to the internet and started up dovecot. I used the following command to backup one user: doveadm backup -u ben remote:test ben is the user is in the mail store. Test is the actual server name. That worked just fine. The maildir was copied completely (as best as I can tell with ls). Then I tried the second user: doveadm backup -u jean remote:test This gives 2 error messages: doveadm(jean)[]: Error: Mailbox INBOX: Failed to get attribute vendor/vendor.dovecot/pvt/server/sieve/files/.dovecot: Mailbox attributes not enabled doveadm(jean)[]<0IwxIlI0jGMgUwAAZU03Dg>: Error: Remote command returned error 65: ssh test doveadm dsync-server -ujean -U In addition, the maildir directories are created, but there are no emails in any of them (e.g., cur). What is the problem with the 2nd and why does it behave differently from the first? -- Doug
