Better not post your email password on a public mailing list, was: Re: no full syncs after upgrading to dovecot 2.3.18

[Daniel Lange]

Am 26.04.22 um 11:36 schrieb Paul Kudla (SCOM.CA Internet Services Inc.):

#imapc_host = mail.scom.ca
#imapc_password = Pk554669
#imapc_user = p...@scom.ca


I suggest to change that password immediately.

$ openssl s_client -crlf -connect mail.scom.ca:993
CONNECTED(0003)
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ 
AUTH=PLAIN AUTH=LOGIN] SCOM.CA Internet Services Inc. - Dovecot ready
A login p...@scom.ca Pk554669
A OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT 
SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND 
URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED 
I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH 
LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE 
SAVEDATE LITERAL+ NOTIFY SPECIAL-USE] Logged in
A status INBOX (messages)
* STATUS INBOX (MESSAGES 344)
A OK Status completed (0.002 + 0.000 + 0.001 secs).
^C

Kind regards,
Daniel

Re: no full syncs after upgrading to dovecot 2.3.18

[Aki Tuomi]

Hi!

This is probably going to get fixed in 2.3.19, this looks like an issue we are 
already fixing.

Aki

> On 26/04/2022 16:38 Paul Kudla (SCOM.CA Internet Services Inc.) 
>  wrote:
> 
>  
> Agreed there seems to be no way of posting these kinds of issues to see 
> if they are even being addressed or even known about moving forward on 
> new updates
> 
> i read somewhere there is a new branch soming out but nothing as of yet?
> 
> 2.4 maybe 
> 5.0 
> 
> my previous replication issues (back in feb) went unanswered.
> 
> not faulting anyone, but the developers do seem to be disconnected from 
> issues as of late? or concentrating on other issues.
> 
> I have no problem with support contracts for day to day maintence 
> however as a programmer myself they usually dont work as the other end 
> relies on the latest source code anyways. Thus can not help.
> 
> I am trying to take a part the replicator c programming based on 2.3.18 
> as most of it does work to some extent.
> 
> tcps just does not work (ie 600 seconds default in the c programming)
> 
> My thoughts are tcp works ok but fails when the replicator through 
> dsync-client.c when asked to return the folder list?
> 
> 
> replicator-brain.c seems to control the overall process and timing.
> 
> replicator-queue.c seems to handle the que file that does seem to carry 
> acurate info.
> 
> 
> things in the source code are documented enough to figure this out but i 
> am still going through all the related .h files documentation wise which 
> are all over the place.
> 
> there is no clear documentation on the .h lib files so i have to walk 
> through the tree one at a time finding relative code.
> 
> since the dsync from doveadm does see to work ok i have to assume the 
> dsync-client used to compile the replicator is at fault somehow or a 
> call from it upstream?
> 
> Thanks for your input on the other issues noted below, i will keep that 
> in mind when disassembling the source code.
> 
> No sense in fixing one thing and leaving something else behind, probably 
> all related anyways.
> 
> i have two test servers avaliable so i can play with all this offline to 
> reproduce the issues
> 
> Unfortunately I have to make a living first, this will be addressed when 
> possible as i dont like systems that are live running this way and 
> currently only have 5 accounts with this issue (mine included)
> 
> 
> 
> 
> Happy Tuesday !!!
> Thanks - paul
> 
> Paul Kudla
> 
> 
> Scom.ca Internet Services 
> 004-1009 Byron Street South
> Whitby, Ontario - Canada
> L1N 4S3
> 
> Toronto 416.642.7266
> Main 1.866.411.7266
> Fax 1.888.892.7266
> 
> On 4/26/2022 9:03 AM, Reuben Farrelly wrote:
> > 
> > I ran into this back in February and documented a reproducible test case 
> > (and sent it to this list).  In short - I was able to reproduce this by 
> > having a valid and consistent mailbox on the source/local, creating a 
> > very standard empty Maildir/(new|cur|tmp) folder on the remote replica, 
> > and then initiating the replicate from the source. This consistently 
> > caused dsync to fail replication with the error "dovecot.index reset, 
> > view is now inconsistent" and sync aborted, leaving the replica mailbox 
> > in a screwed up inconsistent state. Client connections on the source 
> > replica were also dropped when this error occurred.  You can see the 
> > error by enabling debug level logging if you initiate dsync manually on 
> > a test mailbox.
> > 
> > The only workaround I found was to remove the remote Maildir and let 
> > Dovecot create the whole thing from scratch.  Dovecot did not like any 
> > existing folders on the destination replica even if they were the same 
> > names as the source and completely empty.  I was able to reproduce this 
> > the bare minimum of folders - just an INBOX!
> > 
> > I have no idea if any of the developers saw my post or if the bug has 
> > been fixed for the next release.  But it seemed to be quite a common 
> > problem over time (saw a few posts from people going back a long way 
> > with the same problem) and it is seriously disruptive to clients.  The 
> > error message is not helpful in tracking down the problem either.
> > 
> > Secondly, I also have had an ongoing and longstanding problem using 
> > tcps: for replication.  For some reason using tcps: (with no other 
> > changes at all to the config) results in a lot of timeout messages 
> > "Error: dsync I/O has stalled, no activity for 600 seconds".  This goes 
> > away if I revert back to tcp: instead of tcps - with tcp: I very rarely 
> > get timeouts.  No idea why, guess this is a bug of some sort also.
> > 
> > It's disappointing that there appears to be no way to have these sorts 
> > or problems addressed like there once was.  I am not using Dovecot for 
> > commercial purposes so paying a fortune for a support contract for a 
> > high end installation just isn't going to happen, and this list seems to 
> > be quite ordinary fo

Can I set a different certificate per listen port?

[Kees van Vloten]

Hi all,

I am trying to setup dovecot to listen to imaps on the local network and 
through haproxy from the internet.


service imap-login {
  inet_listener imaps {
    port = 993
    ssl = yes
  }
  inet_listener imaps_haproxy {
    haproxy = yes
    port = 10993
    ssl = yes
  }
}

Obviously the dns-name on the internet connection (10993) is different 
than on the lan (993).


In the docs 
(https://doc.dovecot.org/configuration_manual/dovecot_ssl_configuration/) 
I found multiple options, but unfortunately none of those have the 
option to distinguish per listen port.


Is there a way to setup two different certificates for the two listeners?

- Kees

Sv: Better not post your email password on a public mailing list, was: Re: no full syncs after upgra

[Sebastian Nielsen]

Even more stupid that the IMAP port is available to the public. Should have 
been firewalled to authorized IPs only, then it wouldn't have mattered that the 
password have leaked.

-Ursprungligt meddelande-
Från: dovecot-boun...@dovecot.org  För Daniel Lange
Skickat: den 27 april 2022 14:59
Till: Paul Kudla (SCOM.CA Internet Services Inc.) 
Kopia: dovecot@dovecot.org
Ämne: Better not post your email password on a public mailing list, was: Re: no 
full syncs after upgrading

Am 26.04.22 um 11:36 schrieb Paul Kudla (SCOM.CA Internet Services Inc.):
> #imapc_host = mail.scom.ca
> #imapc_password = Pk554669
> #imapc_user = p...@scom.ca

I suggest to change that password immediately.

$ openssl s_client -crlf -connect mail.scom.ca:993
CONNECTED(0003)
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ 
AUTH=PLAIN AUTH=LOGIN] SCOM.CA Internet Services Inc. - Dovecot ready A login 
p...@scom.ca Pk554669 A OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID 
ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS 
THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN 
NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT 
SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY 
PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE] Logged 
in A status INBOX (messages)
* STATUS INBOX (MESSAGES 344)
A OK Status completed (0.002 + 0.000 + 0.001 secs).
^C

Kind regards,
Daniel

Recovering deleted mailbox

[Sean McBride]

Hi all,

I have a user (coworker) that accidentally deleted a mailbox and all its 
sub-mailboxes.

I use Maildir format storage.  I have backups.

Is it enough to put the mailbox folder back where it was?  I'm talking about 
the folder that contains 'cur', 'new', 'tmp', 'dovecot-uidlist', etc.   Or 
would this desynchronize or otherwise confuse dovecot?  Or is it preferable to 
use some doveadm command?  Or...?

Thanks,

Sean

Re: Recovering deleted mailbox

[Ted Hatfield]

I've done so with no issues as long as permissions and ownership are 
accurate.


Ted Hatfield

On Wed, 27 Apr 2022, Sean McBride wrote:


Hi all,

I have a user (coworker) that accidentally deleted a mailbox and all its 
sub-mailboxes.

I use Maildir format storage.  I have backups.

Is it enough to put the mailbox folder back where it was?  I'm talking 
about the folder that contains 'cur', 'new', 'tmp', 'dovecot-uidlist', 
etc.   Or would this desynchronize or otherwise confuse dovecot?  Or is 
it preferable to use some doveadm command?  Or...?


Thanks,

Sean

Re: Recovering deleted mailbox

[Shawn Heisey]

On 4/27/22 16:18, Sean McBride wrote:

I have a user (coworker) that accidentally deleted a mailbox and all its 
sub-mailboxes.

I use Maildir format storage.  I have backups.

Is it enough to put the mailbox folder back where it was?  I'm talking about 
the folder that contains 'cur', 'new', 'tmp', 'dovecot-uidlist', etc.   Or 
would this desynchronize or otherwise confuse dovecot?  Or is it preferable to 
use some doveadm command?  Or...?



Disclaimer:  I am not affiliated with the project, and I am definitely 
not an expert.  I've been running dovecot for my personal mail server 
for a long time, thankfully with very few incidents.  I have done some 
manual surgery on my maildir mailbox and seen how it reacts.  Dovecot is 
very resilient.


What you describe should be sufficient.  It's how I would proceed with a 
restore.  In most cases I would copy the backup on top of any existing 
structure, rather than doing a wholesale replace, because any new mail 
received should have different filenames than what is in the backup.


If it were me, I would probably delete all the files that have a 
filename starting with "dovecot" in that user's mailbox, and restart 
dovecot, letting dovecot rebuild those files when the user connects.  I 
don't really have any experience with how things operate over POP3, I've 
always used IMAP with dovecot.


I'm interested to know whether the real experts here have different 
advice than this, in case I ever find myself in that situation.  There 
might be some doveadm commands that accomplish the dovecot* file 
rebuilding in a cleaner way.


Thanks,
Shawn

Re: Recovering deleted mailbox

[Aki Tuomi]

> On 28/04/2022 01:57 Shawn Heisey  wrote:
> 
>  
> On 4/27/22 16:18, Sean McBride wrote:
> > I have a user (coworker) that accidentally deleted a mailbox and all its 
> > sub-mailboxes.
> >
> > I use Maildir format storage.  I have backups.
> >
> > Is it enough to put the mailbox folder back where it was?  I'm talking 
> > about the folder that contains 'cur', 'new', 'tmp', 'dovecot-uidlist', etc. 
> >   Or would this desynchronize or otherwise confuse dovecot?  Or is it 
> > preferable to use some doveadm command?  Or...?
> 
> 
> Disclaimer:  I am not affiliated with the project, and I am definitely 
> not an expert.  I've been running dovecot for my personal mail server 
> for a long time, thankfully with very few incidents.  I have done some 
> manual surgery on my maildir mailbox and seen how it reacts.  Dovecot is 
> very resilient.
> 
> What you describe should be sufficient.  It's how I would proceed with a 
> restore.  In most cases I would copy the backup on top of any existing 
> structure, rather than doing a wholesale replace, because any new mail 
> received should have different filenames than what is in the backup.
> 
> If it were me, I would probably delete all the files that have a 
> filename starting with "dovecot" in that user's mailbox, and restart 
> dovecot, letting dovecot rebuild those files when the user connects.  I 
> don't really have any experience with how things operate over POP3, I've 
> always used IMAP with dovecot.
> 
> I'm interested to know whether the real experts here have different 
> advice than this, in case I ever find myself in that situation.  There 
> might be some doveadm commands that accomplish the dovecot* file 
> rebuilding in a cleaner way.
> 
> Thanks,
> Shawn

There is no reason to delete the dovecot files after recovery. You can run 
`doveadm force-resync` to ensure everything is synced. Removing the files just 
cause more problems than benefit usually.

Aki

Re: Can I set a different certificate per listen port?

[Aki Tuomi]

> On 27/04/2022 22:14 Kees van Vloten  wrote:
> 
>  
> Hi all,
> 
> I am trying to setup dovecot to listen to imaps on the local network and 
> through haproxy from the internet.
> 
> service imap-login {
>    inet_listener imaps {
>      port = 993
>      ssl = yes
>    }
>    inet_listener imaps_haproxy {
>      haproxy = yes
>      port = 10993
>      ssl = yes
>    }
> }
> 
> Obviously the dns-name on the internet connection (10993) is different 
> than on the lan (993).
> 
> In the docs 
> (https://doc.dovecot.org/configuration_manual/dovecot_ssl_configuration/) 
> I found multiple options, but unfortunately none of those have the 
> option to distinguish per listen port.
> 
> Is there a way to setup two different certificates for the two listeners?
> 
> - Kees

Hi!

Currently port is not supported. What we usually recommend here is that you use 
haproxy to distribute connections to different local IP addresses and use

local 127.0.0.5/32 {
  ssl_cert=

Re: Recovering deleted mailbox

[Shawn Heisey]

On 4/27/2022 11:27 PM, Aki Tuomi wrote:

There is no reason to delete the dovecot files after recovery. You can run 
`doveadm force-resync` to ensure everything is synced. Removing the files just 
cause more problems than benefit usually.


Thanks for that information!  Very helpful for future fiddling.

Does that resync command also maybe force a full FTS reindex?  I'm using 
fts_solr.  The way that I currently manage a full reindex is with the 
following shell script:


---
#!/bin/sh
DOVECOT_SOLR_URL_BASE="http://localhost:8983/solr/dovecot";
MAIL_ROOT=/var/vmail

# DO NOT MODIFY BELOW
# WITHOUT GOOD REASON
#
DEL_ALL_QUERY_XML="*:*"
service dovecot stop
curl \
  "${DOVECOT_SOLR_URL_BASE}/update?commit=true&optimize=true" \
  -H "Content-Type: text/xml" \
  --data-binary "${DEL_ALL_QUERY_XML}"
find ${MAIL_ROOT} -name "dovecot.*index*" -print0 | xargs -0 rm -f
service dovecot start
doveadm index -A -q '*'
---

-s

Re: Recovering deleted mailbox

[Aki Tuomi]

> On 28/04/2022 08:33 Shawn Heisey  wrote:
> 
>  
> On 4/27/2022 11:27 PM, Aki Tuomi wrote:
> > There is no reason to delete the dovecot files after recovery. You can run 
> > `doveadm force-resync` to ensure everything is synced. Removing the files 
> > just cause more problems than benefit usually.
> 
> Thanks for that information!  Very helpful for future fiddling.
> 
> Does that resync command also maybe force a full FTS reindex?  I'm using 
> fts_solr.  The way that I currently manage a full reindex is with the 
> following shell script:
> 
> ---
> #!/bin/sh
> DOVECOT_SOLR_URL_BASE="http://localhost:8983/solr/dovecot";
> MAIL_ROOT=/var/vmail
> 
> # DO NOT MODIFY BELOW
> # WITHOUT GOOD REASON
> #
> DEL_ALL_QUERY_XML="*:*"
> service dovecot stop
> curl \
>    "${DOVECOT_SOLR_URL_BASE}/update?commit=true&optimize=true" \
>    -H "Content-Type: text/xml" \
>    --data-binary "${DEL_ALL_QUERY_XML}"
> find ${MAIL_ROOT} -name "dovecot.*index*" -print0 | xargs -0 rm -f
> service dovecot start
> doveadm index -A -q '*'
> ---
> 
> -s

# drop fts data
doveadm fts rescan -u user
# rebuild index
doveadm index -u user "*"

Aki

Re: Recovering deleted mailbox

[Shawn Heisey]

On 4/27/2022 11:48 PM, Aki Tuomi wrote:

# drop fts data
doveadm fts rescan -u user
# rebuild index
doveadm index -u user "*"


I do full reindexes a lot more often than a typical user would. When I 
do a reindex, I want it to happen for all users, not one at a time.  Do 
you have a sequence of commands to accomplish that?


I'm part of the Solr project.  I do reindexes for performance testing 
when I think of something to try, or I want to upgrade Solr.  And also 
when I think of a change that I want to make to the index config for 
dovecot.  A full reindex for my install takes less than 10 minutes as I 
only have about 160K messages total. Not something I would recommend 
doing frequently when there are millions of messages.


Thanks,
Shawn

Re: Recovering deleted mailbox

[Aki Tuomi]

> On 28/04/2022 08:57 Shawn Heisey  wrote:
> 
>  
> On 4/27/2022 11:48 PM, Aki Tuomi wrote:
> > # drop fts data
> > doveadm fts rescan -u user
> > # rebuild index
> > doveadm index -u user "*"
> 
> I do full reindexes a lot more often than a typical user would. When I 
> do a reindex, I want it to happen for all users, not one at a time.  Do 
> you have a sequence of commands to accomplish that?
> 
> I'm part of the Solr project.  I do reindexes for performance testing 
> when I think of something to try, or I want to upgrade Solr.  And also 
> when I think of a change that I want to make to the index config for 
> dovecot.  A full reindex for my install takes less than 10 minutes as I 
> only have about 160K messages total. Not something I would recommend 
> doing frequently when there are millions of messages.
> 
> Thanks,
> Shawn

Then I would recommend using curl to drop all indexes from solr, and then run 
`doveadm index -A -q '*'` as you are doing already.

Aki

Re: Recovering deleted mailbox

[Narcis Garcia]

__
I'm using this dedicated address because personal addresses aren't 
masked enough at this mail public archive. Public archive administrator 
should fix this against automated addresses collectors.

El 28/4/22 a les 7:48, Aki Tuomi ha escrit:



On 28/04/2022 08:33 Shawn Heisey  wrote:

  
On 4/27/2022 11:27 PM, Aki Tuomi wrote:

There is no reason to delete the dovecot files after recovery. You can run 
`doveadm force-resync` to ensure everything is synced. Removing the files just 
cause more problems than benefit usually.


Thanks for that information!  Very helpful for future fiddling.

Does that resync command also maybe force a full FTS reindex?  I'm using
fts_solr.  The way that I currently manage a full reindex is with the
following shell script:

---
#!/bin/sh
DOVECOT_SOLR_URL_BASE="http://localhost:8983/solr/dovecot";
MAIL_ROOT=/var/vmail

# DO NOT MODIFY BELOW
# WITHOUT GOOD REASON
#
DEL_ALL_QUERY_XML="*:*"
service dovecot stop
curl \
    "${DOVECOT_SOLR_URL_BASE}/update?commit=true&optimize=true" \
    -H "Content-Type: text/xml" \
    --data-binary "${DEL_ALL_QUERY_XML}"
find ${MAIL_ROOT} -name "dovecot.*index*" -print0 | xargs -0 rm -f
service dovecot start
doveadm index -A -q '*'
---

-s


# drop fts data
doveadm fts rescan -u user
# rebuild index
doveadm index -u user "*"

Aki


I took note of this:
# Repair
doveadm -v force-resync -u $User '*'
# Drop fts data
doveadm -v fts rescan -u $User
# Rebuild index
doveadm -v index -u $User '*'

But I'm not clear about $User it it's for my system "vmail" account that 
run Dovecot services, or it refers to mail account.

In first case, how to specify mail account or maildir tree to act to?

Thank you.

Re: Recovering deleted mailbox

[Aki Tuomi]

> On 28/04/2022 09:56 Narcis Garcia  wrote:
> 
>  
> __
> I'm using this dedicated address because personal addresses aren't 
> masked enough at this mail public archive. Public archive administrator 
> should fix this against automated addresses collectors.
> El 28/4/22 a les 7:48, Aki Tuomi ha escrit:
> > 
> >> On 28/04/2022 08:33 Shawn Heisey  wrote:
> >>
> >>   
> >> On 4/27/2022 11:27 PM, Aki Tuomi wrote:
> >>> There is no reason to delete the dovecot files after recovery. You can 
> >>> run `doveadm force-resync` to ensure everything is synced. Removing the 
> >>> files just cause more problems than benefit usually.
> >>
> >> Thanks for that information!  Very helpful for future fiddling.
> >>
> >> Does that resync command also maybe force a full FTS reindex?  I'm using
> >> fts_solr.  The way that I currently manage a full reindex is with the
> >> following shell script:
> >>
> >> ---
> >> #!/bin/sh
> >> DOVECOT_SOLR_URL_BASE="http://localhost:8983/solr/dovecot";
> >> MAIL_ROOT=/var/vmail
> >>
> >> # DO NOT MODIFY BELOW
> >> # WITHOUT GOOD REASON
> >> #
> >> DEL_ALL_QUERY_XML="*:*"
> >> service dovecot stop
> >> curl \
> >>     "${DOVECOT_SOLR_URL_BASE}/update?commit=true&optimize=true" \
> >>     -H "Content-Type: text/xml" \
> >>     --data-binary "${DEL_ALL_QUERY_XML}"
> >> find ${MAIL_ROOT} -name "dovecot.*index*" -print0 | xargs -0 rm -f
> >> service dovecot start
> >> doveadm index -A -q '*'
> >> ---
> >>
> >> -s
> > 
> > # drop fts data
> > doveadm fts rescan -u user
> > # rebuild index
> > doveadm index -u user "*"
> > 
> > Aki
> 
> I took note of this:
> # Repair
> doveadm -v force-resync -u $User '*'
> # Drop fts data
> doveadm -v fts rescan -u $User
> # Rebuild index
> doveadm -v index -u $User '*'
> 
> But I'm not clear about $User it it's for my system "vmail" account that 
> run Dovecot services, or it refers to mail account.
> In first case, how to specify mail account or maildir tree to act to?
> 
> Thank you.

It refers to user's mail account, not system vmail account.

Aki