Re: smtp_address_cmp does not ignore case of localpart

[John Fawcett via dovecot]

On 11/02/2019 22:43, Andreas Jobs via dovecot wrote:
> Hi,
>
> in src/lib-smtp/smtp-address.c line 759 the localpart is compared with
> null_strcmp, which does not ignore the case. While this is the correct
> behavior (according to the RFC) it causes problems in real life.
>
> In my case the vacation notices are not sent, if someone writes to
> first.l...@ruhr-uni-bochum.de because :addresses contains (only)
> first.l...@ruhr-uni-bochum.de.
>
>
>
> Regards,
> Andreas
>
In practice if you convert the username part to lower case in the MTA
then you will accept mail with mixed case but only have to deal with
lowercase in Dovecot.

John

Re: smtp_address_cmp does not ignore case of localpart

[John Fawcett via dovecot]

On 12/02/2019 21:41, Andreas Jobs via dovecot wrote:
> On Tue, Feb 12, 2019 at 08:12:43AM +0100, John Fawcett via dovecot wrote:
>> On 11/02/2019 22:43, Andreas Jobs via dovecot wrote:
>> In practice if you convert the username part to lower case in the MTA
>> then you will accept mail with mixed case but only have to deal with
>> lowercase in Dovecot.
>
> Yes, but the vacation action (also) compares :addresses against the
> header to|cc|sender... And changing the headers is a bad idea.
>
> Maybe my subject line was a bad choice :-)
>
>
> --Andreas
>
You're right, ignore my suggestion. Only the domain is treated case
insensitive and localpart is compared case sensitive.

The change in itself looks trivial, but it would affect more than
vacation (smtp_address_equals is used in a few places in dovecot and
pigeonhole)

John

Re: Upgrading to 2.3

[John Fawcett via dovecot]

On 09/03/2019 14:10, Christian Schmidt via dovecot wrote:
> Hi,
>
> @lbutlr via dovecot, 08.03.19:
>> On 8 Mar 2019, at 05:54, Aki Tuomi via dovecot 
>> wrote:
>>> https://wiki.dovecot.org/Upgrading
>>
>> Duh. I wasn't looking for a URL that was specific.
>
> https://wiki2.dovecot.org/Upgrading/2.3 ;-)
>
> Kind Regards
> Christian
>
Noticed a small typo in the wiki for upgrading to 2.3

recipient_delimiters

should read

recipient_delimiter

John

Re: SMTPUTF8 support

[John Fawcett via dovecot]

On 06/04/2019 10:07, André Rodier via dovecot wrote:
> On 05/04/2019 08:57, David Bürgin via dovecot wrote:
>> André, are you quite sure you have it working?
>>
>> In this thread someone from Open-Xchange stated that no, Dovecot doesn’t
>> have SMTPUTF8 support implemented, and the same response was given by
>> another Dovecot developer last September (it ‘is being considered’ was
>> the answer then, see
>> https://dovecot.org/pipermail/dovecot/2018-September/112887.html).
>>
>> I am using LMTP to deliver mail to Dovecot from Postfix, and delivery
>> fails with Postfix reporting: ‘SMTPUTF8 is required, but was not offered
>> by host mail.my.org[private/dovecot-lmtp]’. I doubt that this can work
>> without a change in Dovecot LMTP.
>>
>> Anyway, I am also interested in SMTPUTF8 support, so +1 from me!
>>
>>
>
> Hello David et al,
>
> I have it working with my set up, and I confirm it is not easy.
>
> If I limit the stack to OpenLDAP, Postfix and Dovecot to their
> simplest configuration, it is relatively easy, even with slightly
> outdated software in Debian Stretch. 

I think you are confusing acceptance of UTF8 in the localpart of the
email address (which if I remember correctly worked in the 2.2 branch)
with SMTPUTF8 support. As mentioned in the thread above Dovecot does not
as yet support SMTPUTF8. Support for SMTPUTF8 would mean among other
things changing the source code of Dovecot so that it announces the
SMTPUTF8 key word to the upstream MTA.

John

Re: SMTPUTF8 support

[John Fawcett via dovecot]

On 06/04/2019 11:34, André Rodier via dovecot wrote:
> On Sat, 2019-04-06 at 11:19 +0200, John Fawcett via dovecot wrote:
>> On 06/04/2019 10:07, André Rodier via dovecot wrote:
>>> On 05/04/2019 08:57, David Bürgin via dovecot wrote:
>>>> André, are you quite sure you have it working?
>>>>
>>>> In this thread someone from Open-Xchange stated that no, Dovecot doesn’t
>>>> have SMTPUTF8 support implemented, and the same response was given by
>>>> another Dovecot developer last September (it ‘is being considered’ was
>>>> the answer then, see
>>>> https://dovecot.org/pipermail/dovecot/2018-September/112887.html).
>>>>
>>>> I am using LMTP to deliver mail to Dovecot from Postfix, and delivery
>>>> fails with Postfix reporting: ‘SMTPUTF8 is required, but was not offered
>>>> by host mail.my.org[private/dovecot-lmtp]’. I doubt that this can work
>>>> without a change in Dovecot LMTP.
>>>>
>>>> Anyway, I am also interested in SMTPUTF8 support, so +1 from me!
>>>>
>>>>
>>> Hello David et al,
>>>
>>> I have it working with my set up, and I confirm it is not easy.
>>>
>>> If I limit the stack to OpenLDAP, Postfix and Dovecot to their
>>> simplest configuration, it is relatively easy, even with slightly
>>> outdated software in Debian Stretch. 
>> I think you are confusing acceptance of UTF8 in the localpart of the
>> email address (which if I remember correctly worked in the 2.2 branch)
>> with SMTPUTF8 support. As mentioned in the thread above Dovecot does not
>> as yet support SMTPUTF8. Support for SMTPUTF8 would mean among other
>> things changing the source code of Dovecot so that it announces the
>> SMTPUTF8 key word to the upstream MTA.
>>
>> John
> Hello John,
>
> You are probably right.
>
> I am not well aware of the low level details and the terminology.
>
> I confess I am mostly interested in the proper reception of the emails
> with accentuated user names, which is already sometimes difficult to
> achieve.
>
> Is there any link where I can learn the communication rules between
> Dovecot and Postfix and the SMTPUTF8 standard ?
>
> Kind regards,
> André
>
André

Daniel Lange has posted some links to this topic from Postfix and
earlier discussions on the mailing list.

Postfix has this SMTPUTF8 documentation:

http://www.postfix.org/SMTPUTF8_README.html

On that page you can also find links to the relevant standards including
SMTPUTF8 standard.

John

Re: dovecot[30008]: auth-worker: Fatal: master: service(auth-worker): child 31074 killed with signal 11

[John Fawcett via dovecot]

On 08/04/2019 16:22, Herbert via dovecot wrote:
> Hi
>
> Since a recent DNF update, I see many log file entries like the following:
>
> Apr 08 16:16:36 udoo.symmetrix.ch audit[31074]: ANOM_ABEND
> auid=4294967295 uid=0 gid=0 ses=4294967295
> subj=system_u:system_r:dovecot_auth_t:s0 pid=31074 comm="auth"
> exe="/usr/libexec/dovecot/auth" sig=11 res=1
> Apr 08 16:16:36 udoo.symmetrix.ch kernel: traps: auth[31074] general
> protection fault ip:7ff45b5f98a7 sp:7fff0df96830 error:0 in
> libmariadb.so.3[7ff45b5d7000+46000]
> Apr 08 16:16:36 udoo.symmetrix.ch dovecot[30008]: auth-worker: Fatal:
> master: service(auth-worker): child 31074 killed with signal 11 (core
> not dumped - https://dovecot.org/bugreport.html#coredumps)
> Apr 08 16:16:38 udoo.symmetrix.ch abrt-dump-journal-oops[1139]:
> abrt-dump-journal-oops: Found oopses: 1
> Apr 08 16:16:38 udoo.symmetrix.ch abrt-dump-journal-oops[1139]:
> abrt-dump-journal-oops: Creating problem directories
> Apr 08 16:16:39 udoo.symmetrix.ch abrt-dump-journal-oops[1139]:
> Reported 1 kernel oopses to Abrt
> Apr 08 16:16:39 udoo.symmetrix.ch abrt-server[31104]: Can't find a
> meaningful backtrace for hashing in '.'
> Apr 08 16:16:39 udoo.symmetrix.ch abrt-server[31104]: Option
> 'DropNotReportableOopses' is not configured
> Apr 08 16:16:39 udoo.symmetrix.ch abrt-server[31104]: Preserving oops
> '.' because DropNotReportableOopses is 'no'
> Apr 08 16:16:39 udoo.symmetrix.ch dbus-daemon[1022]: [system]
> Activating service name='org.freedesktop.problems' requested by
> ':1.779' (uid=0 pid=31122 comm="/usr/bin/python3
> /usr/bin/abrt-action-notify -d /v"
> label="system_u:system_r:abrt_t:s0-s0:c0.c1023") (using servicehelper)
> Apr 08 16:16:40 udoo.symmetrix.ch dbus-daemon[1022]: [system]
> Successfully activated service 'org.freedesktop.problems'
> Apr 08 16:16:40 udoo.symmetrix.ch abrt-notification[31129]: System
> encountered a non-fatal error in ??()
>
>
> # uname -a
> Linux udoo.symmetrix.ch 5.0.5-100.fc28.x86_64 #1 SMP Wed Mar 27
> 22:16:29 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>
> # dovecot --version
> 2.3.5 (513208660)
>
> # mysql --version
> mysql  Ver 15.1 Distrib 10.2.22-MariaDB, for Linux (x86_64) using
> readline 5.1
>
> Unfortunately I didn't manage to get a core dump, sorry.
>
> Kind regards,
> Herbert
>
>
Herbert

it's just a guess without a backtrace, but it looks similar to an
already known issue of a double mysql_close. The code has not yet in a
release but it's in the master-2.3 branch. If you are building from
source you might be able to patch it yourself while waiting for the next
2.3.x release.

https://github.com/dovecot/core/commit/8d78721bbf86657966750da3bcfc7fe9a92c7417

John

Re: Where to report (potential) Dovecot bugs

[John Fawcett via dovecot]

On 09/04/2019 22:03, Mark Moseley via dovecot wrote:
> I'm curious if this is still the right place to report potential bugs
> with Dovecot. 
>
> Is there a Dovecot bug tracker somewhere?

https://www.dovecot.org/bugreport.html

Re: failed: read(/var/run/dovecot/dns-client)

[John Fawcett via dovecot]

On 11/04/2019 00:18, Laura Smith via dovecot wrote:
> ‐‐‐ Original Message ‐‐‐
> On Wednesday, April 10, 2019 10:24 PM, Aki Tuomi  
> wrote:
>
>>> On 10 April 2019 23:56 Laura Smith via dovecot < dovecot@dovecot.org> wrote:
>>>
>>> ‐‐‐ Original Message ‐‐‐
>>> On Wednesday, April 10, 2019 9:14 PM, Aki Tuomi < 
>>> aki.tu...@open-xchange.com> wrote:
>>>
> On 10 April 2019 23:13 Laura Smith via dovecot dovecot@dovecot.org wrote:
> Sent with ProtonMail Secure Email.
> ‐‐‐ Original Message ‐‐‐
> On Wednesday, April 10, 2019 8:20 PM, Aki Tuomi 
> aki.tu...@open-xchange.com wrote:
>>> On 10 April 2019 22:13 Laura Smith via dovecot dovecot@dovecot.org 
>>> wrote:
>>> On Wednesday, April 10, 2019 7:57 PM, Aki Tuomi 
>>> aki.tu...@open-xchange.com wrote:
> On 10 April 2019 21:26 Laura Smith via dovecot dovecot@dovecot.org 
> wrote:
> ==
> dsync( foo...@example.com): Error: imapc(foobar.example.com:993): 
> dns_lookup(foobar.example.com) failed: 
> read(/var/run/dovecot/dns-client) failed: read(size=512) failed: 
> Connection reset by peer
 This is dovecot's internal dns-client, and something goes wrong when 
 talking to the service.
> dsync( foo...@example.com): Error: Failed to initialize user: imapc: 
> Login to foobar.example.com failed: Disconnected from server
 This is btw dsync service, not imap service.
> ===
> Initially I thought "oh no, not another AppArmor block".
> But then surely the second message would not appear if the DNS lookup 
> was not successful ?
> Also "dig foobar.example.com" works fine.
> How should I be troubleshooting this ? And if it is still likely to 
> be AppArmor, what is calling it ? "doveadm" itself or something else 
> ? What does "/var/run/dovecot/dns-client" do and why doesn't dovecot 
> use standard OS calls like everyone else ?
 Because the "standard OS call" is blocking and we would prefer it to 
 not block everything else.
> So many questions !
 Aki
>>> Thanks for your reply, but both those message are generated from a 
>>> simple :
>>> doveadm -v -o mail_fsync=never backup -R -u foo...@example.com imapc:
>>> So I don't know what you mean about dsync service failing ? Surely the 
>>> DNS lookup succeeded if the 'dsync service' failed due to remote 
>>> disconnect ?
>>> I'm still none the wiser as to where to start looking for 
>>> troubleshoting ?
>> Did you check dovecot logs? Maybe there is something useful?
>> Aki
> Only the same old cryptic message about dns-client ?
> master: Fatal: execv(/usr/lib/dovecot/dns-client) failed: Permission 
> denied
 Something prevents executing the dns-client binary.
> master: Error: service(dns_client): command startup failed, throttling 
> for 16 secs
> dns_client: Fatal: master: service(dns_client): child 14293 returned 
> error 84 (exec() failed)
 Aki
>>> Yes but is it being called by doveadm directly or by some other dovecot 
>>> program ? If I'm going to have to go down the AppArmor route, then I would 
>>> prefer if you told me what was calling it instead of me having to 
>>> un-necessarily spend time doing straces !
>>>
>>> Also, should I be able to call dns-client directly myself ? (or is there a 
>>> way to do so to enable testing ?
>> It is started by dovecot's master process when you connect to dns-client 
>> unix socket. You can try
>>
>> socat stdio unix-connect:/var/run/dovecot/dns-client
>>
>> I thought apparmor tells when something is blocked into kernel log? have you 
>> checked dmesg?
>>
>> Apologies for your frustration. 
>> ---
> Yeah nothing in dmesg.  I'm still hunting around to find some log somewhere 
> but so far silence.
>
> "socat stdio unix-connect:/var/run/dovecot/dns-client" runs but returns 
> nothing. Is that expected ?
>
> When you say "dovecot's master process", so  doveadm sync talks to the master 
> process ?  So in terms of apparmor I would therefore be looking at 
> /usr/sbin/dovecot ?  If that's the case, the relevant apparmor permisssions 
> are already provided :
>   /{,var/}run/dovecot/ rw,
>   /{,var/}run/dovecot/** rw,

Laura

Do the above apparmor settings give permission to dovecot to execute
/usr/lib/dovecot/dns-client, assuming that the user under which dovecot
is running already has file system permissions to do that?

John

Re: failed: read(/var/run/dovecot/dns-client)

[John Fawcett via dovecot]

On 11/04/2019 00:51, Laura Smith via dovecot wrote:
> ‐‐‐ Original Message ‐‐‐
> On Wednesday, April 10, 2019 11:48 PM, John Fawcett via dovecot 
>  wrote:
>
>> On 11/04/2019 00:18, Laura Smith via dovecot wrote:
>>
>>> ‐‐‐ Original Message ‐‐‐
>>> On Wednesday, April 10, 2019 10:24 PM, Aki Tuomi aki.tu...@open-xchange.com 
>>> wrote:
>>>
>>>>> On 10 April 2019 23:56 Laura Smith via dovecot < dovecot@dovecot.org> 
>>>>> wrote:
>>>>> ‐‐‐ Original Message ‐‐‐
>>>>> On Wednesday, April 10, 2019 9:14 PM, Aki Tuomi < 
>>>>> aki.tu...@open-xchange.com> wrote:
>>>>>
>>>>>>> On 10 April 2019 23:13 Laura Smith via dovecot dovecot@dovecot.org 
>>>>>>> wrote:
>>>>>>> Sent with ProtonMail Secure Email.
>>>>>>> ‐‐‐ Original Message ‐‐‐
>>>>>>> On Wednesday, April 10, 2019 8:20 PM, Aki Tuomi 
>>>>>>> aki.tu...@open-xchange.com wrote:
>>>>>>>
>>>>>>>>> On 10 April 2019 22:13 Laura Smith via dovecot dovecot@dovecot.org 
>>>>>>>>> wrote:
>>>>>>>>> On Wednesday, April 10, 2019 7:57 PM, Aki Tuomi 
>>>>>>>>> aki.tu...@open-xchange.com wrote:
>>>>>>>>>
>>>>>>>>>>> On 10 April 2019 21:26 Laura Smith via dovecot dovecot@dovecot.org 
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> ==
>>>>>>>>>>>
>>>>>>>>>>> dsync( foo...@example.com): Error: imapc(foobar.example.com:993): 
>>>>>>>>>>> dns_lookup(foobar.example.com) failed: 
>>>>>>>>>>> read(/var/run/dovecot/dns-client) failed: read(size=512) failed: 
>>>>>>>>>>> Connection reset by peer
>>>>>>>>>>> This is dovecot's internal dns-client, and something goes wrong 
>>>>>>>>>>> when talking to the service.
>>>>>>>>>>> dsync( foo...@example.com): Error: Failed to initialize user: 
>>>>>>>>>>> imapc: Login to foobar.example.com failed: Disconnected from server
>>>>>>>>>>> This is btw dsync service, not imap service.
>>>>>>>>>>>
>>>>>>>>>>> ===
>>>>>>>>>>>
>>>>>>>>>>> Initially I thought "oh no, not another AppArmor block".
>>>>>>>>>>> But then surely the second message would not appear if the DNS 
>>>>>>>>>>> lookup was not successful ?
>>>>>>>>>>> Also "dig foobar.example.com" works fine.
>>>>>>>>>>> How should I be troubleshooting this ? And if it is still likely to 
>>>>>>>>>>> be AppArmor, what is calling it ? "doveadm" itself or something 
>>>>>>>>>>> else ? What does "/var/run/dovecot/dns-client" do and why doesn't 
>>>>>>>>>>> dovecot use standard OS calls like everyone else ?
>>>>>>>>>>> Because the "standard OS call" is blocking and we would prefer it 
>>>>>>>>>>> to not block everything else.
>>>>>>>>>>> So many questions !
>>>>>>>>>>> Aki
>>>>>>>>>>> Thanks for your reply, but both those message are generated from a 
>>>>>>>>>>> simple :
>>>>>>>>>>> doveadm -v -o mail_fsync=never backup -R -u foo...@example.com 
>>>>>>>>>>> imapc:
>>>>>>>>>>> So I don't know what you mean about dsync service failing ? Surely 
>>>>&

Re: failed: read(/var/run/dovecot/dns-client)

[John Fawcett via dovecot]

On 11/04/2019 10:02, Laura Smith via dovecot wrote:
> ‐‐‐ Original Message ‐‐‐
> On Thursday, April 11, 2019 12:55 AM, John Fawcett via dovecot 
>  wrote:
>
>> On 11/04/2019 00:51, Laura Smith via dovecot wrote:
>>
>>> ‐‐‐ Original Message ‐‐‐
>>> On Wednesday, April 10, 2019 11:48 PM, John Fawcett via dovecot 
>>> dovecot@dovecot.org wrote:
>>>
>>>> On 11/04/2019 00:18, Laura Smith via dovecot wrote:
>>>>
>>>>> ‐‐‐ Original Message ‐‐‐
>>>>> On Wednesday, April 10, 2019 10:24 PM, Aki Tuomi 
>>>>> aki.tu...@open-xchange.com wrote:
>>>>>
>>>>>>> On 10 April 2019 23:56 Laura Smith via dovecot < dovecot@dovecot.org> 
>>>>>>> wrote:
>>>>>>> ‐‐‐ Original Message ‐‐‐
>>>>>>> On Wednesday, April 10, 2019 9:14 PM, Aki Tuomi < 
>>>>>>> aki.tu...@open-xchange.com> wrote:
>>>>>>>
>>>>>>>>> On 10 April 2019 23:13 Laura Smith via dovecot dovecot@dovecot.org 
>>>>>>>>> wrote:
>>>>>>>>> Sent with ProtonMail Secure Email.
>>>>>>>>> ‐‐‐ Original Message ‐‐‐
>>>>>>>>> On Wednesday, April 10, 2019 8:20 PM, Aki Tuomi 
>>>>>>>>> aki.tu...@open-xchange.com wrote:
>>>>>>>>>
>>>>>>>>>>> On 10 April 2019 22:13 Laura Smith via dovecot dovecot@dovecot.org 
>>>>>>>>>>> wrote:
>>>>>>>>>>> On Wednesday, April 10, 2019 7:57 PM, Aki Tuomi 
>>>>>>>>>>> aki.tu...@open-xchange.com wrote:
>>>>>>>>>>>
>>>>>>>>>>>>> On 10 April 2019 21:26 Laura Smith via dovecot 
>>>>>>>>>>>>> dovecot@dovecot.org wrote:
>>>>>>>>>>>>> ==
>>>>>>>>>>>>> dsync( foo...@example.com): Error: imapc(foobar.example.com:993): 
>>>>>>>>>>>>> dns_lookup(foobar.example.com) failed: 
>>>>>>>>>>>>> read(/var/run/dovecot/dns-client) failed: read(size=512) failed: 
>>>>>>>>>>>>> Connection reset by peer
>>>>>>>>>>>>> This is dovecot's internal dns-client, and something goes wrong 
>>>>>>>>>>>>> when talking to the service.
>>>>>>>>>>>>> dsync( foo...@example.com): Error: Failed to initialize user: 
>>>>>>>>>>>>> imapc: Login to foobar.example.com failed: Disconnected from 
>>>>>>>>>>>>> server
>>>>>>>>>>>>> This is btw dsync service, not imap service.
>>>>>>>>>>>>> ===
>>>>>>>>>>>>> Initially I thought "oh no, not another AppArmor block".
>>>>>>>>>>>>> But then surely the second message would not appear if the DNS 
>>>>>>>>>>>>> lookup was not successful ?
>>>>>>>>>>>>> Also "dig foobar.example.com" works fine.
>>>>>>>>>>>>> How should I be troubleshooting this ? And if it is still likely 
>>>>>>>>>>>>> to be AppArmor, what is calling it ? "doveadm" itself or 
>>>>>>>>>>>>> something else ? What does "/var/run/dovecot/dns-client" do and 
>>>>>>>>>>>>> why doesn't dovecot use standard OS calls like everyone else ?
>>>>>>>>>>>>> Because the "standard OS call" is blocking and we would prefer it 
>>>>>>>>>>>>> to not block everything else.
>>>>>>>>>>>>> So many questions !
>&

Re: failed: read(/var/run/dovecot/dns-client)

[John Fawcett via dovecot]

On 11/04/2019 22:09, Laura Smith via dovecot wrote:
> ‐‐‐ Original Message ‐‐‐
> On Thursday, April 11, 2019 9:01 PM, John Fawcett via dovecot 
>  wrote:
>
>> On 11/04/2019 10:02, Laura Smith via dovecot wrote:
>>
>>> ‐‐‐ Original Message ‐‐‐
>>> On Thursday, April 11, 2019 12:55 AM, John Fawcett via dovecot 
>>> dovecot@dovecot.org wrote:
>>>
>>>> On 11/04/2019 00:51, Laura Smith via dovecot wrote:
>>>>
>>>>> ‐‐‐ Original Message ‐‐‐
>>>>> On Wednesday, April 10, 2019 11:48 PM, John Fawcett via dovecot 
>>>>> dovecot@dovecot.org wrote:
>>>>>
>>>>>> On 11/04/2019 00:18, Laura Smith via dovecot wrote:
>>>>>>
>>>>>>> ‐‐‐ Original Message ‐‐‐
>>>>>>> On Wednesday, April 10, 2019 10:24 PM, Aki Tuomi 
>>>>>>> aki.tu...@open-xchange.com wrote:
>>>>>>>
>>>>>>>>> On 10 April 2019 23:56 Laura Smith via dovecot < dovecot@dovecot.org> 
>>>>>>>>> wrote:
>>>>>>>>> ‐‐‐ Original Message ‐‐‐
>>>>>>>>> On Wednesday, April 10, 2019 9:14 PM, Aki Tuomi < 
>>>>>>>>> aki.tu...@open-xchange.com> wrote:
>>>>>>>>>
>>>>>>>>>>> On 10 April 2019 23:13 Laura Smith via dovecot dovecot@dovecot.org 
>>>>>>>>>>> wrote:
>>>>>>>>>>> Sent with ProtonMail Secure Email.
>>>>>>>>>>> ‐‐‐ Original Message ‐‐‐
>>>>>>>>>>> On Wednesday, April 10, 2019 8:20 PM, Aki Tuomi 
>>>>>>>>>>> aki.tu...@open-xchange.com wrote:
>>>>>>>>>>>
>>>>>>>>>>>>> On 10 April 2019 22:13 Laura Smith via dovecot 
>>>>>>>>>>>>> dovecot@dovecot.org wrote:
>>>>>>>>>>>>> On Wednesday, April 10, 2019 7:57 PM, Aki Tuomi 
>>>>>>>>>>>>> aki.tu...@open-xchange.com wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 10 April 2019 21:26 Laura Smith via dovecot 
>>>>>>>>>>>>>>> dovecot@dovecot.org wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ==
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> dsync( foo...@example.com): Error: 
>>>>>>>>>>>>>>> imapc(foobar.example.com:993): dns_lookup(foobar.example.com) 
>>>>>>>>>>>>>>> failed: read(/var/run/dovecot/dns-client) failed: 
>>>>>>>>>>>>>>> read(size=512) failed: Connection reset by peer
>>>>>>>>>>>>>>> This is dovecot's internal dns-client, and something goes wrong 
>>>>>>>>>>>>>>> when talking to the service.
>>>>>>>>>>>>>>> dsync( foo...@example.com): Error: Failed to initialize user: 
>>>>>>>>>>>>>>> imapc: Login to foobar.example.com failed: Disconnected from 
>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>> This is btw dsync service, not imap service.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ===
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Initially I thought "oh no, not another AppArmor block".
>>>>>>>>>>>>>>> But then surely the second message would not appear if the DNS 
>>>>>>>>>>>>>>> lookup was not successful ?
>>>>>

Re: Secure Client-Initiated Renegotiation

[John Fawcett via dovecot]

On 11/04/2019 23:28, sergio via dovecot wrote:
> Hello.
>
> I've just tested my system that runs dovecot 2.3.4.1 on debian buster
> with testssl.sh (https://testssl.sh/) and is says:
>
> Secure Renegotiation (CVE-2009-3555)    not vulnerable (OK)
> Secure Client-Initiated Renegotiation   VULNERABLE (NOT ok), potential
> DoS threat
>
> Is this a configuration or a compilation issue and how to solve it?
>
This should be interpreted as meaning that client initiated
renegotiation is enabled. The tool does not test whether the software
mitigates the dos threat by limiting the number of negotiations to a
configurable limit.

Having said that, I think that mitigating this in dovecot would require
a code change. What I'm not sure about is whether the best route would
be to turn off client side renegotiation or only limit it. A previous
version of openssl turned it off but then it was re-introduced. That
would require further investigation to understand the best solution.

John

Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 12/04/2019 12:09, Peter Mogensen via dovecot wrote:
> Looking further at tcpdumps of the Dovecot->Solr traffic and Solr
> metrics it doesn't seem like there's anything suspicious apart from the
> TCP windows running full and Dovecot backing of ... until it times out
> and close the connection.
>
> From my understanding of how Dovecot operates towards Solr it will flush
> ~1000 documents towards Solr in /update request until it has traversed
> the mailbox (let's say 20.000 mails), doing softCommits after each.
>
> But is it really reasonable for Dovecot to expect that no request will
> take more than 60s to process by Solr?
> It doesn't seem like my Solr can handle that, although it does process
> documents and it does reasonably fast clear pending documents after
> Dovecot closes the connection.
>
> On the surface it looks like Dovecot is too impatient.
>
> /Peter

If this value was made configurable people could set it to what they
want. However the underlying problem is likely on solr configuration.

Is this a problem only on initial indexing or an ongoing problem after
initial indexing?

The parameters that the solr plugin are using are designed to make
documents visible to searches quickly.

If there is an issue on initial indexing, where you are not really
concerned about qucik visibility but just getting things into the index
efficiently, a better approach would be for dovecot plugin not to send
any commit or softCommit (or waitSearcher either) and that should speed
things up. You'd need to configure solr with a long autoSoftCommit
maxTime and a reasonable autoCommit maxTime, which you could then
reconfigure when the load was done.

If you're using dovecot built from source code it should be possible to
test that by some minor modification of the code in  fts-backend-solr.c.

John

[PATCH] Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 13/04/2019 17:16, Shawn Heisey via dovecot wrote:
> On 4/13/2019 4:29 AM, John Fawcett via dovecot wrote:
>> If this value was made configurable people could set it to what they
>> want. However the underlying problem is likely on solr configuration.
>
> The Jetty that is included in Solr has its idle timeout set to 50
> seconds.  But in practice, I have not seen this timeout trigger ...
> and if the OP is seeing a 60 second timeout, then the 50 second idle
> timeout in Jetty must not be occurring.
>
> There may be a socket timeout configured on inter-server requests --
> distributed queries or the load balancing that SolrCloud does.  I can
> never remember whether this is the case by default.  I think it is.
>
>> If there is an issue on initial indexing, where you are not really
>> concerned about qucik visibility but just getting things into the index
>> efficiently, a better approach would be for dovecot plugin not to send
>> any commit or softCommit (or waitSearcher either) and that should speed
>> things up. You'd need to configure solr with a long autoSoftCommit
>> maxTime and a reasonable autoCommit maxTime, which you could then
>> reconfigure when the load was done.
>
> Solr ships with autoCommit set to 15 seconds and openSearcher set to
> false on the autoCommit.  The autoSoftCommit setting is not enabled by
> default, but depending on how the index was created, Solr might try to
> set autoSoftCommit to 3 seconds ... which is WAY too short.
>
> I will usually increase the autoCommit time to 60 seconds, just to
> reduce the amount of work that Solr is doing.  The autoSoftCommit
> time, if it is used, should be set to a reasonably long value ...
> values between two and five minutes would be good.  Attempting to use
> a very short autoSoftCommit time will usually lead to problems.
>
> This thread says that dovecot is sending explicit commits.  One thing
> that might be happening to exceed 60 seconds is an extremely long
> commit, which is usually caused by excessive cache autowarming, but
> might be related to insufficient memory.  The max heap setting on an
> out-of-the-box Solr install (5.0 and later) is 512MB.  That's VERY
> small, and it doesn't take much index data before a much larger heap
> is required.
>
> Thanks,
> Shawn

I looked into the code (version 2.3.5.1): the fts-solr plugin is not
sending softCommit every 1000 emails. Emails from a single folder are
batched in up to a maximum of 1000 emails per request, but the
softCommit gets sent once per mailbox folder at the end of all the
requests for that folder.

I immagine that one of the reasons dovecot sends softCommits is because
without autoindex active and even if mailboxes are periodically indexed
from cron, the last emails received with be indexed at the moment of the
search.  So while sending softCommit has the advantage of including
recent mails in searches, it means that softCommits are being done upon
user request. Frequency depends on user activity.

Going back to the original problem: seems the first advice to Peter is
to look into solr configuration as others have said.

>From dovecot point of view I can see the following as potentially useful
features:

1) a configurable batch size would enable to tune the number of emails
per request and help stay under the 60 seconds hard coded http request
timeout. A configurable http timeout would be less useful, since this
will potentially run into other timeouts on solr side.

2) abilty to turn off softCommits so as to have a more predictable
softCommit workload. In that case autoSoftCommit should be configured in
solr. In order to minimize risk of recent emails not appearing in search
results, periodic indexing could be set up by cron.

I've attached a patch, any comments are welcome (especially about
getting settings from the backend context).

Example config

plugin {
  fts = solr
  fts_solr =
url=https://user:passw...@solr.example.com:443/solr/dovecot/
batch_size=500 no_soft_commit
}

John

--- src/plugins/fts-solr/fts-solr-plugin.h.orig 2019-04-14 15:12:07.694289402 
+0200
+++ src/plugins/fts-solr/fts-solr-plugin.h  2019-04-14 14:04:17.213939414 
+0200
@@ -12,8 +12,10 @@
 
 struct fts_solr_settings {
const char *url, *default_ns_prefix;
+   unsigned int batch_size;
bool use_libfts;
bool debug;
+   bool no_soft_commit;
 };
 
 struct fts_solr_user {
--- src/plugins/fts-solr/fts-solr-plugin.c.orig 2019-04-14 11:41:03.591782439 
+0200
+++ src/plugins/fts-solr/fts-solr-plugin.c  2019-04-14 14:37:46.059433864 
+0200
@@ -10,6 +10,8 @@
 #include "fts-solr-plugin.h"
 
 
+#define DEFAULT_SOLR_BATCH_SIZE 1000
+
 const char *fts_solr_plugin_version = DOVECOT_ABI_VERSION;
 struct http_client *solr_http_client = NULL;
 
@@ -37,6 +39,10 @@
} else if (

Re: [PATCH] Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 14/04/2019 16:04, Aki Tuomi via dovecot wrote:
>
>> On 14 April 2019 16:59 John Fawcett via dovecot < dovecot@dovecot.org
>> <mailto:dovecot@dovecot.org>> wrote:
>>
>>
>> On 13/04/2019 17:16, Shawn Heisey via dovecot wrote:
>>> On 4/13/2019 4:29 AM, John Fawcett via dovecot wrote:
>>>> If this value was made configurable people could set it to what they
>>>> want. However the underlying problem is likely on solr configuration.
>>> The Jetty that is included in Solr has its idle timeout set to 50
>>> seconds.  But in practice, I have not seen this timeout trigger ...
>>> and if the OP is seeing a 60 second timeout, then the 50 second idle
>>> timeout in Jetty must not be occurring.
>>> There may be a socket timeout configured on inter-server requests --
>>> distributed queries or the load balancing that SolrCloud does.  I can
>>> never remember whether this is the case by default.  I think it is.
>> >> If there is an issue on initial indexing, where you are not really
>> >> concerned about qucik visibility but just getting things into the
>> index
>> >> efficiently, a better approach would be for dovecot plugin not to
>> send
>> >> any commit or softCommit (or waitSearcher either) and that should
>> speed
>> >> things up. You'd need to configure solr with a long autoSoftCommit
>> >> maxTime and a reasonable autoCommit maxTime, which you could then
>> >> reconfigure when the load was done.
>> >
>>> Solr ships with autoCommit set to 15 seconds and openSearcher set to
>>> false on the autoCommit.  The autoSoftCommit setting is not enabled by
>>> default, but depending on how the index was created, Solr might try to
>>> set autoSoftCommit to 3 seconds ... which is WAY too short.
>>> I will usually increase the autoCommit time to 60 seconds, just to
>>> reduce the amount of work that Solr is doing.  The autoSoftCommit
>>> time, if it is used, should be set to a reasonably long value ...
>>> values between two and five minutes would be good.  Attempting to use
>>> a very short autoSoftCommit time will usually lead to problems.
>>> This thread says that dovecot is sending explicit commits.  One thing
>>> that might be happening to exceed 60 seconds is an extremely long
>>> commit, which is usually caused by excessive cache autowarming, but
>>> might be related to insufficient memory.  The max heap setting on an
>>> out-of-the-box Solr install (5.0 and later) is 512MB.  That's VERY
>>> small, and it doesn't take much index data before a much larger heap
>>> is required.
>>> Thanks,
>>> Shawn
>> I looked into the code (version 2.3.5.1): the fts-solr plugin is not
>> sending softCommit every 1000 emails. Emails from a single folder are
>> batched in up to a maximum of 1000 emails per request, but the
>> softCommit gets sent once per mailbox folder at the end of all the
>> requests for that folder.
>>
>> I immagine that one of the reasons dovecot sends softCommits is because
>> without autoindex active and even if mailboxes are periodically indexed
>> from cron, the last emails received with be indexed at the moment of the
>> search.  So while sending softCommit has the advantage of including
>> recent mails in searches, it means that softCommits are being done upon
>> user request. Frequency depends on user activity.
>>
>> Going back to the original problem: seems the first advice to Peter is
>> to look into solr configuration as others have said.
>>
>> From dovecot point of view I can see the following as potentially useful
>> features:
>>
>> 1) a configurable batch size would enable to tune the number of emails
>> per request and help stay under the 60 seconds hard coded http request
>> timeout. A configurable http timeout would be less useful, since this
>> will potentially run into other timeouts on solr side.
>>
>> 2) abilty to turn off softCommits so as to have a more predictable
>> softCommit workload. In that case autoSoftCommit should be configured in
>> solr. In order to minimize risk of recent emails not appearing in search
>> results, periodic indexing could be set up by cron.
>>
>> I've attached a patch, any comments are welcome (especially about
>> getting settings from the backend context).
>>
>> Example config
>>
>> plugin {
>>   fts = solr
>>   fts_solr =
>> url= https://user:passw...@solr.example.com:443/solr/dovecot/
>> batch_size=500 no_soft_commit
>> }
>>
>> John
>
> Can you please open a pull request to https://github.com/dovecot/core ?
> ---
> Aki Tuomi

Done, thanks for considering it.

John

Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 14/04/2019 17:16, Peter Mogensen via dovecot wrote:
> sorry... I got distracted half way and forgot to put a meaningfull
> subject so the archive could figure out the thread. - resending.
>
> On 4/14/19 4:04 PM, dovecot-requ...@dovecot.org wrote:
>
>>> Solr ships with autoCommit set to 15 seconds and openSearcher set to
>>> false on the autoCommit.? The autoSoftCommit setting is not enabled by
>>> default, but depending on how the index was created, Solr might try to
>>> set autoSoftCommit to 3 seconds ... which is WAY too short.
> I just run with the default. 15s autoCommit and no autoSoftCommit
>
>>> This thread says that dovecot is sending explicit commits.? 
> I see explicit /update req. with softCommit and waitSearcer=true in a
> tcpdump.
>
>>> One thing
>>> that might be happening to exceed 60 seconds is an extremely long
>>> commit, which is usually caused by excessive cache autowarming, but
>>> might be related to insufficient memory.? The max heap setting on an
>>> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY
>>> small, and it doesn't take much index data before a much larger heap
>>> is required.
> I run with
>
> SOLR_JAVA_MEM="-Xmx8g -Xms2g"
>
>> I looked into the code (version 2.3.5.1):
> This is 2.2.35. I haven't checked the source difference to 2.3.x I must
> admit.
>
>> I immagine that one of the reasons dovecot sends softCommits is because
>> without autoindex active and even if mailboxes are periodically indexed
>> from cron, the last emails received with be indexed at the moment of the
>> search.? 
> I expect that dovecot has to because of it's default behavior by only
> bringing the index up-to-date just before search. So it has towait for
> the index result to be available if there's been any new mails indexed.
>
>> 1) a configurable batch size would enable to tune the number of emails
>> per request and help stay under the 60 seconds hard coded http request
>> timeout. A configurable http timeout would be less useful, since this
>> will potentially run into other timeouts on solr side.
> Being able to configure it is great.
> But I don't think it solves much. I recompiled with 100 as batch size
> and it still ended in timeouts.
> Then I recompiled with 10min timeout and now I see all the batches
> completing and their processesing time is mostly between 1 and 2 minutes
> (so all would have failed).
>
> To me it looks like Solr just takes too long time to index. This is no
> small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
> and for this test it's not doing anything else, so I'm a bit surprised
> that even with only a few users this takes so long time.
>
> /Peter
>
>
Peter

I suppose you could go with a batch size of 50. If it's linear, you
could still keep under the default 60 seconds http request time :-)

I'm now testing with solr settings autoCommit 15 seconds, autoSoftCommit
60 seconds and sending no softCommits from dovecot and 500 batch size.

I've set up

    /usr/local/bin/doveadm index -A "*"

in crontab every 5 minutes so indexes will stay mostly up to date to
minimize amount of mail not already visible in the index when searches
are done.

The solr server is a small test virtual machine with 0.2 (shared) vCPU
and 0.6MB of memory and non SSD storage. It can index around 2000 emails
per minute when there is no other activity. Average email size is about
45Kb. I'm not indexing attachments.

John

Re: Solr connection timeout hardwired to 60s

[John Fawcett via dovecot]

On 14/04/2019 17:55, John Fawcett via dovecot wrote:
> The solr server is a small test virtual machine with 0.2 (shared) vCPU
> and 0.6MB of memory and non SSD storage. It can index around 2000 emails
> per minute when there is no other activity. Average email size is about
> 45Kb. I'm not indexing attachments.
>
> John
>
I more than double the performance by using a 0.5 shared vCPU and SSD
storage. Approx 4500 emails per minute.

John

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 01:15, Larry Rosenman via dovecot wrote:
> Given all the discussion on FTS/Solr, etc, I have a question:
>
> I have autoindex set, and searching in roundcube most of the time does
> NOT find things,
> HOWEVER if I do:
> doveadm fts rescan
> doveadm index
>
> I can find things in the mailboxes.
>
> WHY?
>
> (doveconf -n attached).
>
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
> 
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

have you been able to check your solr logs to see if the queries that
find nothing actually arrive at solr? What is logged for those queries?

Some clients (don't know if roundcube is among those) don't send the
query in some circumstances.

Also when email arrives in dovecot is the automatic indexing working: do
you see solr logging for adding those messages.

John

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 01:39, Larry Rosenman via dovecot wrote:
>
> full solr.log at:
> https://www.lerctr.org/~ler/solr.log
>
> The search DOES make it to SOLR:
> ⌂77% [l...@thebighonker.lerctr.org:~] 130 $ grep sysutils
> /var/log/solr/solr.log
> 2019-04-14 18:31:34.749 INFO  (qtp349420578-7538) [   x:dovecot]
> o.a.s.c.S.Request [dovecot]  webapp=/solr path=/select
> params={q={!lucene+q.op%3DAND}(hdr:sysutils\/+OR+body:sysutils\/)&fl=uid,score&sort=uid+asc&fq=%2Bbox:8097632f69627b5b5895bbe98eac+%2Buser:ler&rows=22644&wt=xml}
> hits=0 status=0 QTime=460
>
> Pick showing subjects:
> https://www.lerctr.org/~ler/sysutils_mail.png
>
> What else?
>
> I'm happy to provide access.
>
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
> 
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

so the search is returning no hits as you said. But can you show that
there is data in the index that should match?

doveadm search -u u...@example.com mailbox inbox body "sysutils/"

Can you do a controlled test and send yourself a test message with that
string and show the solr log where it is being inserted into the index
and then search for it with doveadm (just to rule out roundcube for the
moment) and show solr log for that search?

John

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:
> Note the hits after the fts rescan/index.
>
> Get Outlook for Android 
>
> 
> *From:* Aki Tuomi 
> *Sent:* Monday, April 15, 2019 12:55:07 AM
> *To:* Larry Rosenman; John Fawcett
> *Cc:* Dovecot Mailing List
> *Subject:* Re: SOLR/Index?
>  
>
>
> On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
>> ⌂72% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox
>> lists/freebsd/ports-commiters  body 'sysutils'
>> [l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
>> [l...@thebighonker.lerctr.org:~] $ doveadm index -q
>> lists/freebsd/ports-commiters
>> ⌂64% [l...@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog
>> Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected
>> (auth failed, 1 attempts in 2 secs): user=, method=PLAIN,
>> rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed,
>> session=
>> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login:
>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS,
>> session=
>> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged
>> out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>> Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP
>> address 23.100.68.192
>> Apr 14 19:30:55 thebighonker exim[14846]:
>> H=(DaVinci-MWare.prophet21lab.com
>> ) [23.100.68.192]:52130
>> I=[192.147.25.65]:25 sender verify defer for > >: host lookup did not complete
>> Apr 14 19:30:55 thebighonker exim[14846]:
>> H=(DaVinci-MWare.prophet21lab.com
>> ) [23.100.68.192]:52130
>> I=[192.147.25.65]:25 F=mailto:i...@duke.org>>
>> temporarily rejected RCPT mailto:jpo...@why.net>>:
>> Could not complete sender verify
>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login:
>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS,
>> session=
>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged
>> out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>> Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP
>> address 80.253.235.35
>> Apr 14 19:31:19 thebighonker dovecot[2507]:
>> indexer-worker(ler/14919): Indexed 1578 messages in
>> lists/freebsd/ports-commiters (UIDs 21067..22644)
>> ^C
>> [l...@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox
>> lists/freebsd/ports-commiters  body 'sysutils/'
>
>
> Just minor nit, but you are searching for 'sysutils' first, then
> 'sysutils/'. FTS does not do substring searches by default.
>
> Aki
>
>>
>> -- 
>> Larry Rosenman                     http://www.lerctr.org/~ler
>> 
>> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
>> 
>> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Larry

just to be sure: are you running a standard unmodified 2.3.5.1 version
which you built from source code?

I can see that first you search for sysutils, then do a rescan and
reindex (which is shown in the log) and then you are able to find sysutils/.

It is better when doing these tests to search for the same string before
and after, just to eliminate too many different factors in the test.

Nevertheless I did not see your logging for what happens when you
receive a test message containing sysutils/. Dovecot should be outputing
info about autoindexing given your setup. Does it do that or does it
give some other message? Can you show those logs?

John

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 10:31, Larry Rosenman via dovecot wrote:
> It always shows the autoindex. And yes built from sources.  I'm the
> FreeBSD port maintainer for mail/docecot.  This has been happening for
> several releases.
>
> Get Outlook for Android <https://aka.ms/ghei36>
>
> 
> *From:* dovecot  on behalf of John
> Fawcett via dovecot 
> *Sent:* Monday, April 15, 2019 2:06:55 AM
> *To:* dovecot@dovecot.org
> *Subject:* Re: SOLR/Index?
>  
> On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:
>> Note the hits after the fts rescan/index.
>>
>> Get Outlook for Android <https://aka.ms/ghei36>
>>
>> 
>> *From:* Aki Tuomi 
>> *Sent:* Monday, April 15, 2019 12:55:07 AM
>> *To:* Larry Rosenman; John Fawcett
>> *Cc:* Dovecot Mailing List
>> *Subject:* Re: SOLR/Index?
>>  
>>
>>
>> On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
>>> ⌂72% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox
>>> lists/freebsd/ports-commiters  body 'sysutils'
>>> [l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
>>> [l...@thebighonker.lerctr.org:~] $ doveadm index -q
>>> lists/freebsd/ports-commiters
>>> ⌂64% [l...@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog
>>> Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login: Disconnected
>>> (auth failed, 1 attempts in 2 secs): user=, method=PLAIN,
>>> rip=180.180.217.124, lip=192.147.25.65, TLS: Connection closed,
>>> session=
>>> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login:
>>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS,
>>> session=
>>> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged
>>> out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>>> Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP
>>> address 23.100.68.192
>>> Apr 14 19:30:55 thebighonker exim[14846]:
>>> H=(DaVinci-MWare.prophet21lab.com
>>> <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130
>>> I=[192.147.25.65]:25 sender verify defer for >> <mailto:i...@duke.org>>: host lookup did not complete
>>> Apr 14 19:30:55 thebighonker exim[14846]:
>>> H=(DaVinci-MWare.prophet21lab.com
>>> <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130
>>> I=[192.147.25.65]:25 F=mailto:i...@duke.org>>
>>> temporarily rejected RCPT mailto:jpo...@why.net>>:
>>> Could not complete sender verify
>>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login:
>>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS,
>>> session=
>>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged
>>> out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>>> Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP
>>> address 80.253.235.35
>>> Apr 14 19:31:19 thebighonker dovecot[2507]:
>>> indexer-worker(ler/14919): Indexed 1578 messages in
>>> lists/freebsd/ports-commiters (UIDs 21067..22644)
>>> ^C
>>> [l...@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox
>>> lists/freebsd/ports-commiters  body 'sysutils/'
>>
>>
>> Just minor nit, but you are searching for 'sysutils' first, then
>> 'sysutils/'. FTS does not do substring searches by default.
>>
>> Aki
>>
>>>
>>> -- 
>>> Larry Rosenman                     http://www.lerctr.org/~ler
>>> <http://www.lerctr.org/~ler>
>>> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
>>> <mailto:larry...@gmail.com>
>>> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
>
> Larry
>
> just to be sure: are you running a standard unmodified 2.3.5.1 version
> which you built from source code?
>
> I can see that first you search for sysutils, then do a rescan and
> reindex (which is shown in the log) and then you are able to find
> sysutils/.
>
> It is better when doing these tests to search for the same string
> before and after, just to eliminate too many different factors in the
> test.
>
> Nevertheless I did not see your logging for what happens when you
> receive a test message containing sysutils/. Dovecot should be
> outputing info about autoindexing given your setup. Does it do that or
> does it give some other message? Can you show those logs?
>
> John
>
Larry

Did you notice any difference between the logging for auto indexing and
the logging for indexing that you triggered manually? Would you mind
posting the auto indexing logging for a message to that same user (ler)?

best regards

John

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 10:59, Larry Rosenman via dovecot wrote:
> I'll run a full test when I'm back in front of areal computer vs. My
> phone.(in a few hours)
>
> Get Outlook for Android <https://aka.ms/ghei36>
>
> ----
> *From:* dovecot  on behalf of John
> Fawcett via dovecot 
> *Sent:* Monday, April 15, 2019 3:57:08 AM
> *To:* Dovecot Mailing List
> *Subject:* Re: SOLR/Index?
>  
> On 15/04/2019 10:31, Larry Rosenman via dovecot wrote:
>> It always shows the autoindex. And yes built from sources.  I'm the
>> FreeBSD port maintainer for mail/docecot.  This has been happening
>> for several releases.
>>
>> Get Outlook for Android <https://aka.ms/ghei36>
>>
>> ----------------
>> *From:* dovecot  on behalf of John
>> Fawcett via dovecot 
>> *Sent:* Monday, April 15, 2019 2:06:55 AM
>> *To:* dovecot@dovecot.org
>> *Subject:* Re: SOLR/Index?
>>  
>> On 15/04/2019 08:09, Larry Rosenman via dovecot wrote:
>>> Note the hits after the fts rescan/index.
>>>
>>> Get Outlook for Android <https://aka.ms/ghei36>
>>>
>>> 
>>> *From:* Aki Tuomi 
>>> *Sent:* Monday, April 15, 2019 12:55:07 AM
>>> *To:* Larry Rosenman; John Fawcett
>>> *Cc:* Dovecot Mailing List
>>> *Subject:* Re: SOLR/Index?
>>>  
>>>
>>>
>>> On 15.4.2019 3.33, Larry Rosenman via dovecot wrote:
>>>> ⌂72% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox
>>>> lists/freebsd/ports-commiters  body 'sysutils'
>>>> [l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
>>>> [l...@thebighonker.lerctr.org:~] $ doveadm index -q
>>>> lists/freebsd/ports-commiters
>>>> ⌂64% [l...@thebighonker.lerctr.org:~] $ tail -f /var/log/maillog
>>>> Apr 14 19:30:27 thebighonker dovecot[2507]: imap-login:
>>>> Disconnected (auth failed, 1 attempts in 2 secs): user=,
>>>> method=PLAIN, rip=180.180.217.124, lip=192.147.25.65, TLS:
>>>> Connection closed, session=
>>>> Apr 14 19:30:28 thebighonker dovecot[2507]: imap-login: Login:
>>>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>>>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14813, TLS,
>>>> session=
>>>> Apr 14 19:30:30 thebighonker dovecot[2507]: imap(ler/14813): Logged
>>>> out in=12412 out=66691 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>>>> Apr 14 19:30:54 thebighonker exim[14846]: no host name found for IP
>>>> address 23.100.68.192
>>>> Apr 14 19:30:55 thebighonker exim[14846]:
>>>> H=(DaVinci-MWare.prophet21lab.com
>>>> <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130
>>>> I=[192.147.25.65]:25 sender verify defer for >>> <mailto:i...@duke.org>>: host lookup did not complete
>>>> Apr 14 19:30:55 thebighonker exim[14846]:
>>>> H=(DaVinci-MWare.prophet21lab.com
>>>> <http://DaVinci-MWare.prophet21lab.com>) [23.100.68.192]:52130
>>>> I=[192.147.25.65]:25 F=mailto:i...@duke.org>>
>>>> temporarily rejected RCPT mailto:jpo...@why.net>>:
>>>> Could not complete sender verify
>>>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap-login: Login:
>>>> user=, method=PLAIN, rip=2001:470:1f0f:3ad:bb:dcff:fe50:d900,
>>>> lip=2001:470:1f0f:3ad:bb:dcff:fe50:d900, mpid=14910, TLS,
>>>> session=
>>>> Apr 14 19:31:04 thebighonker dovecot[2507]: imap(ctr/14910): Logged
>>>> out in=169 out=1711 fhc=0 fhb=0 fbc=0 fbb=0 del=0 exp=0 trash=0
>>>> Apr 14 19:31:16 thebighonker exim[14911]: no host name found for IP
>>>> address 80.253.235.35
>>>> Apr 14 19:31:19 thebighonker dovecot[2507]:
>>>> indexer-worker(ler/14919): Indexed 1578 messages in
>>>> lists/freebsd/ports-commiters (UIDs 21067..22644)
>>>> ^C
>>>> [l...@thebighonker.lerctr.org:~] 130 $ doveadm search mailbox
>>>> lists/freebsd/ports-commiters  body 'sysutils/'
>>>
>>>
>>> Just minor nit, but you are searching for 'sysutils' first, then
>>> 'sysutils/'. FTS does not do substring searches by default.
>>>
>>> Aki
>>>
>>>>
>>>> -- 
>>>> Larry Rosenman                     http://www.lerctr.org/~ler
>>>

Re: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 11:38, Larry Rosenman via dovecot wrote:
> ⌂63% [l...@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX
> ⌂67% [l...@thebighonker.lerctr.org:~] 1 $ mail -s "test fts1970"
> l...@lerctr.org 
> test fts1970
>
> test fts1970
> .
> EOT
> [l...@thebighonker.lerctr.org:~] $ mailq
> [l...@thebighonker.lerctr.org:~] $ grep fts1970 mail/INBOX
> Subject: test fts1970
> test fts1970
> test fts1970
>
>
> Apr 15 04:29:03 thebighonker exim[49528]: 1hFxvD-000Csq-P6 <=
> l...@lerctr.org  U=ler P=local S=388
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Connect from
> local
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(l...@lerctr.org/49364
> ): save: box=INBOX, uid=175402,
> msgid= >, size=640,
> vsize=660, from=Larry Rosenman  >, subject=test fts1970, flags=()
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(l...@lerctr.org/49364
> ): sieve:
> msgid= >: stored mail into
> mailbox 'INBOX' (subject=test fts1970 from=l...@lerctr.org
>  size=660)
> Apr 15 04:29:03 thebighonker dovecot[2507]: lmtp(49364): Disconnect
> from local: Client has quit the connection (state=READY)
> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 => ler
> mailto:l...@lerctr.org>> R=localuser T=dovecot_lmtp
> S=404 C="250 2.0.0 mailto:l...@lerctr.org>>
> 6ACWMN9OtFzUwAAAu+mOrA Saved" QT=0s DT=0s
> Apr 15 04:29:03 thebighonker exim[49535]: 1hFxvD-000Csq-P6 Completed QT=0s
> Apr 15 04:29:03 thebighonker dovecot[2507]:
> indexer-worker(l...@lerctr.org/49366 ):
> Indexed 1 messages in INBOX (UIDs 175402..175402)
>
>
> ⌂81% [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX 
> body 'fts1970'
> ⌂83% [l...@thebighonker.lerctr.org:~] $
>
>
> ⌂65% [l...@thebighonker.lerctr.org:~] 75 $ doveadm search -u
> l...@lerctr.org   mailbox INBOX body 'fts1970'
> a53a143be44bda5bd483bbe98eac 175402
> [l...@thebighonker.lerctr.org:~] $ doveadm index -q INBOX
> [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX  body
> 'fts1970'
> [l...@thebighonker.lerctr.org:~] $ doveadm fts rescan
> [l...@thebighonker.lerctr.org:~] $ doveadm index -q INBOX
> [l...@thebighonker.lerctr.org:~] $ doveadm search mailbox INBOX  body
> 'fts1970'
> a53a143be44bda5bd483bbe98eac 175402
> [l...@thebighonker.lerctr.org:~] $ doveadm search -u l...@lerctr.org
>   mailbox INBOX body 'fts1970'
> a53a143be44bda5bd483bbe98eac 175402
> [l...@thebighonker.lerctr.org:~] $
>
> So, yes, your hypothesis is correct.
>
> Question: How can I make it consistent?  
>
> I have a script that runs on the first of the month that does
> archiving, and I have similar issues in that namespace:
> ⌂67% [l...@thebighonker.lerctr.org:~] $ cat bin/archive-mail
> #!/bin/sh
> PATH=$PATH:/usr/local/bin
> #Expects to be run after midnight on the first of the month
> #  to archive all the previous months mail
> #Date Run:
> TODAY=`date "+%Y-%m-%d"`
> #last month in /MM
> YEAR_LAST_MONTH=`date -v-1d "+%Y/%m"`
> #1st of last month as 01-Mon-
> FIRST_LAST_MONTH=`date -v-1d "+01-%b-%Y"`
> echo 'TODAY=' ${TODAY}
> echo 'YEAR_LAST_MONTH=' ${YEAR_LAST_MONTH}
> echo 'FIRST_LAST_MONTH=' ${FIRST_LAST_MONTH}
> # get a list of all the mailboxes with at least one real message
> doveadm -f tab mailbox status vsize \* 2>/dev/null |
>         sed -e 1d | sort -k 1,1 |
>         awk  'BEGIN {FS="\t"} {if ($2 > 0)  print $1}' |
> while read i
> do
>    echo `date` start ${i}
>    doveadm mailbox create "ARCHIVE/${YEAR_LAST_MONTH}/${i}"
>    doveadm -f tab mailbox status messages "${i}"
>    doveadm move "ARCHIVE/${YEAR_LAST_MONTH}/${i}" mailbox \
>             "${i}" BEFORE ${TODAY} SINCE ${FIRST_LAST_MONTH}
>    doveadm -f tab mailbox status messages "${i}"
>    echo `date` done  ${i}
> done
> ⌂64% [l...@thebighonker.lerctr.org:~] $
>
>
> The Exim config can be provided as well if necessary.
>
> ler & l...@lerctr.org  *ARE THE SAME MAILBOX*
>
At the moment it looks as though you have two sets of emails indexed in
solr. One is indexed under username (the one you are running mannually
and apparently the one used by roundcube too, but that's to be verified)
and another set being indexed by autoindex = yes option using the full
email address. Once you've got it working as you require, then you may
want to clean out solr and reindex with just one of them just to reduce
volumes.

Your setup seems to have a mix of users from mysql and from /etc/passwd.
Not sure if your mysql users are all mapped to real users or they have
their own mailboxes with domain included. Your solution will depend on
what you really need and if the setup is working correctly you may not
want to tweak it too much or other things may star

Re: Fwd: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 12:59, Larry Rosenman via dovecot wrote:
> forgot to reply all.
>
> -- Forwarded message -
> From: *Larry Rosenman* mailto:larry...@gmail.com>>
> Date: Mon, Apr 15, 2019 at 5:58 AM
> Subject: Re: SOLR/Index?
> To: John Fawcett mailto:j...@voipsupport.it>>
>
>
> the local users (myself, my wife, a friend) can authenticate EITHER as
>  or @lerctr.org .
>
> switching to all virtual users is NOT going to happen. 
>
> If I login to roundcube with @lerctr.org  it
> finds the autoindexed mail.
>
> So, if I make everyone always authenticate as @lerctr.org
>  we should be fine.
>
> and change my script to do doveadm -u @ instead of
> depending on the local user running the cron job. 
>
> question: Is there some way to have dovecot change what it sees to be
> @lerctr.org  when they login as ?
>
>
Dovecot is very configurable, but it can also take some time, effort and
testing to get the configuration you want. Personally I don't mix user
types since it takes out an element of complexity.

For your case you might find it useful to look into auth_default_realm

that can specify a domain name when one is not supplied.

https://wiki2.dovecot.org/DomainLost

Other things that might be useful: there is a method for returning a
"user" field from the userdb query or passdb query which will change the
username. Or there is another setting that can overriding values of
fields returned by the userdb.

https://wiki.dovecot.org/UserDatabase

Hope it helps!

John

Re: Fwd: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 13:43, Larry Rosenman via dovecot wrote:
> Again, this doesn't help with doveadm running as the local user, and
> also doesn't help with
> the PAM authentication.
>
> passdb {
>   driver = pam
>   #[session=yes] [setcred=yes] [failure_show_msg=yes] [max_requests=20]
>   #[cache_key=] []
>   args = failure_show_msg=yes session=yes max_requests=20
>   skip = authenticated
> }
>
> How can I default the domain for PAM authentication?
> (I've set auth_default_realm and it doesn't help in this case).

Larry

I guess I don't understand enough about your setup or what is not now
working.

My understanding was that everything is now working when logging in as
user@domain, but that you would like to login as user and have dovecot
treat that as though you had logged in as user@domain, but at this point
I admit I may have misinterpreted your emails.

What's also not clear for me is the purpose in your setup of the three
passdb methods (sql, static, and pam) and two userdb methods (sql and
passwd-file). That's why I've pointed you to the docs and I'm hestitant
to give specific advice that may leave you worse off. Others on the list
may have more insights.

John

Re: Fwd: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 16:05, Larry Rosenman via dovecot wrote:
> I have normal system users, some and some virtual domains (2 different
> flavors).
>
> I have this job that runs from cron on my own userid to archive mail
> I want Dovecot to use my system password for mail authentication.
> I, therefore, use PAM to authenticate system users
> Everything works great, modulo getting the auto-index to be visible
> using RoundCube (which if I log into roundcube using @
> it does). 
> by default if I'm logged in as my normal OS user, default doveadm
> commands (as issued from the shell or from my archive script) use the
> user I'm logged in as *WITHOUT A DOMAIN*.
>
> I want to default PAM auth'd users to append @lerctr.org
>  (DOMAIN) to the OS user.
>
when they log in to roundcube as the OS user, right?

John

Re: Fwd: SOLR/Index?

[John Fawcett via dovecot]

On 15/04/2019 16:34, Larry Rosenman via dovecot wrote:
> yes, either with or without the domain.
>
>
> On Mon, Apr 15, 2019 at 9:33 AM John Fawcett via dovecot
> mailto:dovecot@dovecot.org>> wrote:
>
> On 15/04/2019 16:05, Larry Rosenman via dovecot wrote:
>> I have normal system users, some and some virtual domains (2
>> different flavors).
>>
>> I have this job that runs from cron on my own userid to archive mail
>> I want Dovecot to use my system password for mail authentication.
>> I, therefore, use PAM to authenticate system users
>> Everything works great, modulo getting the auto-index to be
>> visible using RoundCube (which if I log into roundcube using
>> @ it does). 
>> by default if I'm logged in as my normal OS user, default doveadm
>> commands (as issued from the shell or from my archive script) use
>> the user I'm logged in as *WITHOUT A DOMAIN*.
>>
>> I want to default PAM auth'd users to append @lerctr.org
>> <http://lerctr.org> (DOMAIN) to the OS user.
>>
> when they log in to roundcube as the OS user, right?
>
> John
>
>
>
> -- 
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 (c)     E-Mail: larry...@gmail.com
> <mailto:larry...@gmail.com>
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

So if all works fine when you specify user@domain in roundcube, but you
want to be able to specify only username in roundcube and have roundcube
login as user@domain, then one solution would be to configure roundcube
to add a default domain to bare usernames. As far as I remember (it's a
while since I used roundcube) it is one of the configuration options.

John

Re: Auto rebuilding of Solr indexes on settings change?

[John Fawcett via dovecot]

On 25/04/2019 13:32, Peter Mogensen via dovecot wrote:
> Hi,
>
> Looking at the source, it doesn't seem like fts-solr checks for settings
> changes using fts_index_have_compatible_settings() like fts-lucene does.
>
> Is there any special reason for why fts-solr shouldn't also rebuild
> indexes if settings has changed?
>
> /Peter

Peter

I guess that there is no setting for the fts-solr plugin that would be
require rebuilding to take place. If you change any fts-solr settings
(except for the url) I would expect the result in solr, providing there
were no errors, will be the same with or without a rebuild.

For fts-lucene there are settings which would change what is stored in
lucene. The relevant settings which are taken into account for lucene
index rebuild are in  fts_lucene_settings_checksum(): default_language,
whitespace_chars, normalize and no_snowball.

John

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 19/05/2019 12:52, mabi via dovecot wrote:
> Hello,
>
> I am trying to setup the last_login plugin on Dovecot 2.3 in order to log a 
> user's last IMAP login into a separate PostgreSQL table called last_login. I 
> am actually following that guide: 
> https://docs.iredmail.org/track.user.last.login.html with the exception that 
> I use PostgreSQL instead of MySQL.
>
> So far it works at least for the very first login of a user but it looks like 
> Dict always want to add a new record into my last_login table instead of 
> updating the existing record for a specific user. Here is the error message 
> from Dovecot:
>
> May 19 12:45:14-mbox1 dovecot: 
> imap(u...@domain.tld)<51915>: Error: last_login_dict: 
> Failed to write value: dict-server returned failure: sql dict: commit failed: 
> ERROR:  duplicate key value violates unique constraint "last_logins_pkey"
>
> It looks like Dict with PostgreSQL does not support UPDATEing an already 
> existing record in the table. Or am I missing something? or it does not work 
> with PostgreSQL? Which does not make much sense to me...
>
> Regards,
> Mabi
>
I guess it is not supported for PostgreSQL or Sqlite, since they don't
have "ON DUPLICATE KEY UPDATE" statement which is what is being used in
MySQL.

You could verify the query being used by turning on query logging in
PostgresSQL.

John

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 19/05/2019 15:42, mabi via dovecot wrote:
> ‐‐‐ Original Message ‐‐‐
> On Sunday, May 19, 2019 2:58 PM, John Fawcett via dovecot 
>  wrote:
>
>> I guess it is not supported for PostgreSQL or Sqlite, since they don't
>> have "ON DUPLICATE KEY UPDATE" statement which is what is being used in
>> MySQL.
> That's it, PostgreSQL does not have "ON DUPLICATE KEY" but instead uses "ON 
> CONFLICT" as documented here:
>
> https://www.postgresql.org/docs/10/sql-insert.html#SQL-ON-CONFLICT
>
>> You could verify the query being used by turning on query logging in
>> PostgresSQL.
> I enabled query logging as suggested and found out that Dovecot dict is not 
> using the "ON CONFLICT" feature of INSERT with PostgreSQL, as you can see 
> from the query below:
>
> INSERT INTO last_logins (last_login,username,domain) VALUES 
> (1558273000,'u...@domain.tld','domain.tld')
>
> For me this makes Dovecot's dictionary feature useless with PostgreSQL. 
> Should I open a bug for that? or is it more of a "feature request"?
>
> Regards,
> Mabi

You may be able to find a workaround, by redefining the table without a
primary key and then select the max(last_login) from the table, with
some periodic job that clears out the old entries.

Or you could make a workaround with the rules syntax from PostgresSql.

https://www.postgresql.org/docs/9.2/sql-createrule.html

Though probably the best thing is a patch to dovecot in order to support
last_login for PostgresSql.

John

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 19/05/2019 16:25, John Fawcett via dovecot wrote:
> INSERT INTO last_logins (last_login,username,domain) VALUES 
> (1558273000,'u...@domain.tld','domain.tld')

I don't have PostgresSql, would you be able to verify if this syntax would work:

INSERT INTO last_logins (last_login,username,domain) VALUES 
(1558273000,'u...@domain.tld','domain.tld') ONCONFLICT(username) UPDATE SET 
last_login=1558273000,domain='u...@domain.tld'

It's important to check that this updates only the single row for that user and 
it puts the right data in that row. If it doesn't work can you give the correct 
syntax?

John 

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 19/05/2019 17:08, mabi via dovecot wrote:
> ‐‐‐ Original Message ‐‐‐
> On Sunday, May 19, 2019 4:44 PM, John Fawcett via dovecot
>  wrote:
>
>> I don't have PostgresSql, would you be able to verify if this syntax
>> would work: INSERT INTO last_logins (last_login,username,domain)
>> VALUES (1558273000,'u...@domain.tld
>> <mailto:u...@domain.tld>','domain.tld') ON CONFLICT(username) UPDATE
>> SET last_login=1558273000,domain='u...@domain.tld
>> <mailto:u...@domain.tld>' It's important to check that this updates
>> only the single row for that user and it puts the right data in that
>> row. If it doesn't work can you give the correct syntax?
>
> So you nearly yes ;-) The only parameter missing was "DO" keyword
> before the "UPDATE". So the correct query would be:
>
> INSERT INTO last_logins (last_login,username,domain) VALUES
> (1558273000,'u...@domain.tld <mailto:u...@domain.tld>','domain.tld')
> ON CONFLICT (username) DO UPDATE SET
> last_login=1558273000,domain='domain.tld';
>
> I also adapted the domain='domain.tld' at the end of the query, you
> had domain='u...@domain.tld <mailto:u...@domain.tld>' but this is just
> a "content" detail which does not matter.
>
> Hope that helps. Let me know if I can do any further testing.
>
Attached is a tentative patch. I've verified no regression for mysql.
There should be no regression for sqlite as the code path is identical.

Are you able to test for pgsql? As mentioned by Akie it will break for
PostgresSql < 9.5 but probably it was not working anyway due to
duplicate keys. Whether this is a wider problem depends on whether the
insert code is being used for other purposes too.

If you or someone can verify it works on PostgresSql >= 9.5, then the
next step will be to make it conditional on the version.

John

John

--- dict-sql-private.h.orig 2019-05-19 19:00:12.395887496 +0200
+++ dict-sql-private.h  2019-05-19 19:04:00.147601310 +0200
@@ -13,6 +13,7 @@
HASH_TABLE(const char *, struct sql_prepared_statement *) 
prep_stmt_hash;
 
bool has_on_duplicate_key:1;
+   bool has_on_conflict_do_update:1;
 };
 
 #endif
--- dict-sql.c.orig 2019-05-19 18:58:02.435194691 +0200
+++ dict-sql.c  2019-05-19 19:17:52.613253822 +0200
@@ -105,8 +105,10 @@
i_zero(&sql_set);
sql_set.driver = driver->name;
sql_set.connect_string = dict->set->connect;
-   /* currently pgsql and sqlite don't support "ON DUPLICATE KEY" */
+   /* pgsql and sqlite don't support "ON DUPLICATE KEY" */
+   /* mysql and sqlite don't support "ON CONFLICT DO UPDATE" */
dict->has_on_duplicate_key = strcmp(driver->name, "mysql") == 0;
+   dict->has_on_conflict_do_update = strcmp(driver->name, "pgsql") == 0;
 
if (sql_db_cache_new(dict_sql_db_cache, &sql_set, &dict->db, error_r) < 
0) {
pool_unref(&pool);
@@ -1108,12 +1110,15 @@
 
str_append_str(prefix, suffix);
str_append_c(prefix, ')');
-   if (!dict->has_on_duplicate_key) {
+   if (dict->has_on_duplicate_key ) {
+   str_append(prefix, " ON DUPLICATE KEY UPDATE ");
+   } else if(dict->has_on_conflict_do_update)  {
+   str_printfa(prefix, " ON CONFLICT (%s) DO UPDATE SET 
",fields[0].map->username_field);
+   } else {
*stmt_r = sql_dict_transaction_stmt_init(ctx, str_c(prefix), 
¶ms);
return 0;
}
 
-   str_append(prefix, " ON DUPLICATE KEY UPDATE ");
for (i = 0; i < field_count; i++) {
const char *first_value_field =
t_strcut(fields[i].map->value_field, ',');

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 19/05/2019 20:31, mabi via dovecot wrote:
>
> ‐‐‐ Original Message ‐‐‐
> On Sunday, May 19, 2019 7:36 PM, John Fawcett via dovecot
>  wrote:
>>
>> Attached is a tentative patch. I've verified no regression for mysql.
>> There should be no regression for sqlite as the code path is identical.
>>
>> Are you able to test for pgsql? As mentioned by Akie it will break
>> for PostgresSql < 9.5 but probably it was not working anyway due to
>> duplicate keys. Whether this is a wider problem depends on whether
>> the insert code is being used for other purposes too.
>>
>> If you or someone can verify it works on PostgresSql >= 9.5, then the
>> next step will be to make it conditional on the version.
>>
> Thank you very much John for your patch, that's fantastic. I am on
> OpenBSD 6.5 and will recompile dovecot from the ports by adding your
> patch to it, I hope that works and will let you know if I managed. If
> I understand correctly the relevant binary file I need to replace is
> the following right:
>
> /usr/local/lib/dovecot/dict/libdriver_pgsql.so
>
> or are there any others I also need to replace in order to test? I am
> planning to test live by just replacing the relevant file(s) so that I
> hopefully don't need to re-install the whole dovecot package.

I'm not sure how the source compilation works on OpenBSD, when I do it
on linux and run "make install" it installs all relevant binaries/libraries.

I saw one issue with the fix though, it does not correctly pull out the
username field. I'm wondering if the query can be rewritten not to
mention the name of the field that fails the constraint

John

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 19/05/2019 22:37, John Fawcett via dovecot wrote:
> On 19/05/2019 20:31, mabi via dovecot wrote:
>>
>> ‐‐‐ Original Message ‐‐‐
>> On Sunday, May 19, 2019 7:36 PM, John Fawcett via dovecot
>>  wrote:
>>>
>>> Attached is a tentative patch. I've verified no regression for
>>> mysql. There should be no regression for sqlite as the code path is
>>> identical.
>>>
>>> Are you able to test for pgsql? As mentioned by Akie it will break
>>> for PostgresSql < 9.5 but probably it was not working anyway due to
>>> duplicate keys. Whether this is a wider problem depends on whether
>>> the insert code is being used for other purposes too.
>>>
>>> If you or someone can verify it works on PostgresSql >= 9.5, then
>>> the next step will be to make it conditional on the version.
>>>
>> Thank you very much John for your patch, that's fantastic. I am on
>> OpenBSD 6.5 and will recompile dovecot from the ports by adding your
>> patch to it, I hope that works and will let you know if I managed. If
>> I understand correctly the relevant binary file I need to replace is
>> the following right:
>>
>> /usr/local/lib/dovecot/dict/libdriver_pgsql.so
>>
>> or are there any others I also need to replace in order to test? I am
>> planning to test live by just replacing the relevant file(s) so that
>> I hopefully don't need to re-install the whole dovecot package.
>
> I'm not sure how the source compilation works on OpenBSD, when I do it
> on linux and run "make install" it installs all relevant
> binaries/libraries.
>
> I saw one issue with the fix though, it does not correctly pull out
> the username field. I'm wondering if the query can be rewritten not to
> mention the name of the field that fails the constraint
>
> John
>
so basically if this works just as well:

INSERT INTO last_logins (last_login,username,domain) VALUES
(1558273000,'u...@domain.tld <mailto:u...@domain.tld>','domain.tld') ON
CONFLICT DO UPDATE SET last_login=1558273000,domain='domain.tld';

then the fix can be altered to attached file which is more similar to
the MYSQL syntax and does not require extra logic to get the username field.

John



--- dict-sql-private.h.orig 2019-05-19 19:00:12.395887496 +0200
+++ dict-sql-private.h  2019-05-19 19:04:00.147601310 +0200
@@ -13,6 +13,7 @@
HASH_TABLE(const char *, struct sql_prepared_statement *) 
prep_stmt_hash;
 
bool has_on_duplicate_key:1;
+   bool has_on_conflict_do_update:1;
 };
 
 #endif
--- dict-sql.c.orig 2019-05-19 18:58:02.435194691 +0200
+++ dict-sql.c  2019-05-19 19:17:52.613253822 +0200
@@ -105,8 +105,10 @@
i_zero(&sql_set);
sql_set.driver = driver->name;
sql_set.connect_string = dict->set->connect;
-   /* currently pgsql and sqlite don't support "ON DUPLICATE KEY" */
+   /* pgsql and sqlite don't support "ON DUPLICATE KEY" */
+   /* mysql and sqlite don't support "ON CONFLICT DO UPDATE" */
dict->has_on_duplicate_key = strcmp(driver->name, "mysql") == 0;
+   dict->has_on_conflict_do_update = strcmp(driver->name, "pgsql") == 0;
 
if (sql_db_cache_new(dict_sql_db_cache, &sql_set, &dict->db, error_r) < 
0) {
pool_unref(&pool);
@@ -1108,12 +1110,15 @@
 
str_append_str(prefix, suffix);
str_append_c(prefix, ')');
-   if (!dict->has_on_duplicate_key) {
+   if (dict->has_on_duplicate_key ) {
+   str_append(prefix, " ON DUPLICATE KEY UPDATE ");
+   } else if(dict->has_on_conflict_do_update)  {
+   str_append(prefix, " ON CONFLICT DO UPDATE SET ");
+   } else {
*stmt_r = sql_dict_transaction_stmt_init(ctx, str_c(prefix), 
¶ms);
return 0;
}
 
-   str_append(prefix, " ON DUPLICATE KEY UPDATE ");
for (i = 0; i < field_count; i++) {
const char *first_value_field =
t_strcut(fields[i].map->value_field, ',');

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 19/05/2019 22:45, John Fawcett via dovecot wrote:
> On 19/05/2019 22:37, John Fawcett via dovecot wrote:
>> On 19/05/2019 20:31, mabi via dovecot wrote:
>>>
>>> ‐‐‐ Original Message ‐‐‐
>>> On Sunday, May 19, 2019 7:36 PM, John Fawcett via dovecot
>>>  wrote:
>>>>
>>>> Attached is a tentative patch. I've verified no regression for
>>>> mysql. There should be no regression for sqlite as the code path is
>>>> identical.
>>>>
>>>> Are you able to test for pgsql? As mentioned by Akie it will break
>>>> for PostgresSql < 9.5 but probably it was not working anyway due to
>>>> duplicate keys. Whether this is a wider problem depends on whether
>>>> the insert code is being used for other purposes too.
>>>>
>>>> If you or someone can verify it works on PostgresSql >= 9.5, then
>>>> the next step will be to make it conditional on the version.
>>>>
>>> Thank you very much John for your patch, that's fantastic. I am on
>>> OpenBSD 6.5 and will recompile dovecot from the ports by adding your
>>> patch to it, I hope that works and will let you know if I managed.
>>> If I understand correctly the relevant binary file I need to replace
>>> is the following right:
>>>
>>> /usr/local/lib/dovecot/dict/libdriver_pgsql.so
>>>
>>> or are there any others I also need to replace in order to test? I
>>> am planning to test live by just replacing the relevant file(s) so
>>> that I hopefully don't need to re-install the whole dovecot package.
>>
>> I'm not sure how the source compilation works on OpenBSD, when I do
>> it on linux and run "make install" it installs all relevant
>> binaries/libraries.
>>
>> I saw one issue with the fix though, it does not correctly pull out
>> the username field. I'm wondering if the query can be rewritten not
>> to mention the name of the field that fails the constraint
>>
>> John
>>
> so basically if this works just as well:
>
> INSERT INTO last_logins (last_login,username,domain) VALUES
> (1558273000,'u...@domain.tld <mailto:u...@domain.tld>','domain.tld')
> ON CONFLICT DO UPDATE SET last_login=1558273000,domain='domain.tld';
>
> then the fix can be altered to attached file which is more similar to
> the MYSQL syntax and does not require extra logic to get the username
> field.
>
> John
>
>
So looking into this with a postgresql databse to work with: the above
query does not work. You have to specify either the column name or the
constraint name that you expect to be violated in order for the update
to take place.

With a map like this one you're using

|map { pattern = shared/last-login/$user/$domain table = last_login
value_field = last_login value_type = uint fields { username = $user
domain = $domain } }|

there's no field name that is obviously the primary key. I've reworked
the patch to use the postgres default primary key constraint name
(tablename_pkey).

The attached fix should work in that case, although I feel it's not
general enough.

John

--- dict-sql-private.h.orig 2019-05-19 19:00:12.395887496 +0200
+++ dict-sql-private.h  2019-05-19 19:04:00.147601310 +0200
@@ -13,6 +13,7 @@
HASH_TABLE(const char *, struct sql_prepared_statement *) 
prep_stmt_hash;
 
bool has_on_duplicate_key:1;
+   bool has_on_conflict_do_update:1;
 };
 
 #endif
--- dict-sql.c.orig 2019-05-19 18:58:02.435194691 +0200
+++ dict-sql.c  2019-05-20 00:25:56.558251260 +0200
@@ -105,8 +105,10 @@
i_zero(&sql_set);
sql_set.driver = driver->name;
sql_set.connect_string = dict->set->connect;
-   /* currently pgsql and sqlite don't support "ON DUPLICATE KEY" */
+   /* pgsql and sqlite don't support "ON DUPLICATE KEY" */
+   /* mysql and sqlite don't support "ON CONFLICT DO UPDATE" */
dict->has_on_duplicate_key = strcmp(driver->name, "mysql") == 0;
+   dict->has_on_conflict_do_update = strcmp(driver->name, "pgsql") == 0;
 
if (sql_db_cache_new(dict_sql_db_cache, &sql_set, &dict->db, error_r) < 
0) {
pool_unref(&pool);
@@ -1108,12 +1110,15 @@
 
str_append_str(prefix, suffix);
str_append_c(prefix, ')');
-   if (!dict->has_on_duplicate_key) {
+   if (dict->has_on_duplicate_key ) {
+   str_append(prefix, " ON DUPLICATE KEY UPDATE ");
+   } else if(dict->has_on_conflict_do_update)  {
+   str_printfa(prefix, " ON CONFLICT ON CONSTRAINT %s_pkey DO 
UPDATE SET ",fields[0].map->table);
+   } else {
*stmt_r = sql_dict_transaction_stmt_init(ctx, str_c(prefix), 
¶ms);
return 0;
}
 
-   str_append(prefix, " ON DUPLICATE KEY UPDATE ");
for (i = 0; i < field_count; i++) {
const char *first_value_field =
t_strcut(fields[i].map->value_field, ',');

Re: Further issues on FTS engine

[John Fawcett via dovecot]

On 20/05/2019 08:27, Joan Moreau via dovecot wrote:
>
> Hi,
>
> Additionally to the long list of problem on the FTS previously
> discussed, here a new:
>
>
> WHen I reset the indexes, the indexer-worker seems paralelleizing the
> indexing (which is good), however, the number available in "ps aux |
> grep dove" shows that it does not move:
>
> dovecot 28549 0.0 0.0 8620 3920 ? S 06:20 0:00 dovecot/indexer [0
> clients, 3 requests]
> mailuse+ 28550 98.6 0.1 167412 86916 ? R 06:20 5:28
> dovecot/indexer-worker [j...@grosjo.net  - 800/37755]
>
>
> Looking further, if I put a tracer in teh backend, it treats the
> *same* message several times in parallel, and therefore does not move
> very fast on the global indexing of the box
>
> ANy clue ?
>
> THanks
>
>
Hi Joan

which version of dovecot are you using? I tried running ps aux | grep
dove on latest version but I don't see this behaviour.

Can you post some evidence of the same message being indexed serveral
times in parallel?

John

Re: Dict issue with PostgreSQL for last_login plugin (duplicate key)

[John Fawcett via dovecot]

On 21/05/2019 15:45, mabi via dovecot wrote:
> ‐‐‐ Original Message ‐‐‐
> On Monday, May 20, 2019 12:37 AM, John Fawcett via dovecot
>  wrote:
>
>> So looking into this with a postgresql databse to work with: the
>> above query does not work. You have to specify either the column name
>> or the constraint name that you expect to be violated in order for
>> the update to take place.
>>
>> With a map like this one you're using
>>
>> |map { pattern = shared/last-login/$user/$domain table = last_login
>> value_field = last_login value_type = uint fields { username = $user
>> domain = $domain } }|
>>
>> there's no field name that is obviously the primary key. I've
>> reworked the patch to use the postgres default primary key constraint
>> name (tablename_pkey).
>>
> So as you mention the new query you adapted which includes the primary
> key works, I tested it manually against PostgreSQL 10.5.
>>
>> The attached fix should work in that case, although I feel it's not
>> general enough.
>>
> Unfortunately my compiling skills are quite poor and I did not manage
> to patch and recompile Dovecot on OpenBSD.
>
> Do you think your patch will make it into the Dovecot code?
>
I feel confident that the patch works as the query has been manually
verified and the code change is not complex to validate.

The last_login plugin does not work at the moment with PostgreSql and
probably does not work with Sqlite, given that the only logic that tries
an update when insert fails seems to be a MySQL specific extension to
standard Sql. So I think that it's clear that support for PostgreSql and
Sqlite  needs to be implemented. The same issue likely exist in other
plugins too, for example expire.

My doubts are around the right solution to adopt. Initially I thought
that there was a PostgreSql syntax similar to MySQL which could be
easily added to the code, but closer inspection shows that the
PostgreSql syntax requires specification of either a constraint name or
the index column(s) for the primary/unique keys.

Constraint names are nowhere specified in the dictionary map syntax and
it's not possible either to identify with 100% certainty the primary key
column(s).

The solution I adopted in the latest version of the patch is to use the
default primary key constraint name derived from the table name, but
that won't help if people define custom constraint names. That may be an
unlikely scenario so the fix is certainly better than AS-IS. However it
is not perfect and added to that is the fact that the PostgreSql
extension is available only from 9.5.

I have no issues to submit the patch officially, as long as Dovecot
developers agree. However it may be worthwhile reflecting on a more
structural change

1) logic which always tries to update and falls back to insert if the
update fails (or viceversa) for all sql dictionaries.

2) updates to the map syntax so that either the constraint name or
primary key columns can be specified.

Ideas are welcome.

John

Re: more generic approach as for userdb? (was: Dict issue with PostgreSQL for last_login plugin (duplicate key))

[John Fawcett via dovecot]

On 23/05/2019 07:49, Steffen Kaiser via dovecot wrote:
> On Wed, 22 May 2019, John Fawcett via dovecot wrote:
>
> > an update when insert fails seems to be a MySQL specific extension to
> > standard Sql. So I think that it's clear that support for PostgreSql and
> > Sqlite  needs to be implemented. The same issue likely exist in other
> > plugins too, for example expire.
>
> > My doubts are around the right solution to adopt. Initially I thought
> > that there was a PostgreSql syntax similar to MySQL which could be
> > easily added to the code, but closer inspection shows that the
> > PostgreSql syntax requires specification of either a constraint name or
> > the index column(s) for the primary/unique keys.
>
> You mean the "target" in ON CONFLICT target action, right?
> http://www.postgresqltutorial.com/postgresql-upsert/
>
Yes, whereas MySQL uses a generic syntax not requiring specific info, as
far as I am aware PostgreSql requires the target. I tried without and
got an error.

> > Constraint names are nowhere specified in the dictionary map syntax and
> > it's not possible either to identify with 100% certainty the primary key
> > column(s).
>
> One could dive into Postgres-specifics to get it, but there are other
> SQLs, too; the quota plugin advertises to use TRIGGERs to turn an
> INSERT into an UPDATE silently, which is no general approach either.
> https://wiki2.dovecot.org/Quota/Dict
>
> > 1) logic which always tries to update and falls back to insert if the
> > update fails (or viceversa) for all sql dictionaries.
>
> > 2) updates to the map syntax so that either the constraint name or
> > primary key columns can be specified.
>
> > Ideas are welcome.
>
> Maybe, one should drop the automatic at all and let the user specify
> the commands manually like with the userdb/passwd. Hence, the generic
> SQL preparation code is already present. There could/should/would be
> documented lots of "best practice" settings for various backends.
>
> In fact, this approach would better fit into the open and more
> "general" base idea Dovecot uses in other places, IMHO.
>
thanks for that suggestion, it would mean moving away from a syntax
where other dictionary types use a map statement and sql wouldn't.
> Kind regards,
>
> -- Steffen Kaiser

Re: last login user tracking

[John Fawcett via dovecot]

On 21/06/2019 23:13, David Mehler via dovecot wrote:
> Hello,
>
> I'm trying to get last login user tracking going, using Dovecot and
> mysql. I'm using:
>
> https://docs.iredmail.org/track.user.last.login.html
>
> and it is working, I am seeing an entry and it is updated, the problem
> is the last login value does not mean anything to me. I was wondering
> if it's possible to have the value be of a last login date and time,
> and also a remote ip would be helpful so I can see from where users
> are logging in from.
>
> Thanks.
> Dave.

You can easily view the value in a readable format with the
FROM_UNIXTIME mysql funciton

https://dev.mysql.com/doc/refman/8.0/en/date-and-time-functions.html#function_from-unixtime

To get remote ip

define the rip field in your table and add it to the primary key

add %r to the last_login_key

last_login_key = last-login/%u/%d/%r

add $rip to the pattern and fields section in your
/etc/dovecot/dovecot-last-login.conf

map {
    pattern = shared/last-login/$user/$domain/$rip
    table = last_login
    value_field = last_login
    value_type = uint

    fields {
    username = $user
    domain = $domain
   rip = $rip
    }
}

By adding the rip field to the primary key of the table, you can end up
with multiple rows per username each with different ips, but if you
don't do that, the rip field is never updated after the first insert.
John

Re: last login user tracking

[John Fawcett via dovecot]

On 22/06/2019 16:59, Lefteris Tsintjelis via dovecot wrote:
> On 22/6/2019 16:07, John Fawcett via dovecot wrote:
>> You can easily view the value in a readable format with the
>> FROM_UNIXTIME mysql funciton
>>
>> https://dev.mysql.com/doc/refman/8.0/en/date-and-time-functions.html#function_from-unixtime
>>
>> To get remote ip
>>
>> define the rip field in your table and add it to the primary key
>>
>> add %r to the last_login_key
>>
>> last_login_key = last-login/%u/%d/%r
>>
>> add $rip to the pattern and fields section in your
>> /etc/dovecot/dovecot-last-login.conf
>>
>> map {
>>     pattern = shared/last-login/$user/$domain/$rip
>>     table = last_login
>>     value_field = last_login
>>     value_type = uint
>>
>>     fields {
>>     username = $user
>>     domain = $domain
>>    rip = $rip
>>     }
>> }
>>
>> By adding the rip field to the primary key of the table, you can end up
>> with multiple rows per username each with different ips, but if you
>> don't do that, the rip field is never updated after the first insert.
> Does it work with IPv4 as well as IPv6? What is the optimal way to
> declare the rip field as?

yes IPv4 and IPv6. The value is in a string format. You can define a
VARCHAR(45) to hold it since no IPv4 or IPv6 address will be longer than
that.

John

Re: Error: quota-status: Client sent invalid recipient address: Invalid character in path

[John Fawcett via dovecot]

On 28/06/2019 02:01, Lefteris Tsintjelis via dovecot wrote:
> Any idea what this is?
>
> Jun 26 23:16:34
> quota-status(u...@example.com)<4975>: Error:
> quota-status: Client sent invalid recipient address: Invalid character
> in path
>
> I have this enabled in my config
>
> service quota-status {
>   client_limit = 1
>   executable = quota-status -p postfix
>   unix_listener /var/spool/postfix/private/dovquota {
> mode = 0666
> user = postfix
>   }
> }
>
> Lefteris

this is given by the address validation routines (for example see
lib-smtp/smtp-address.c as a starting point).

Maybe it would be easier to approach it by looking at the email address
that gave the error.

Is the email address compliant to smtp standards? Does it contain utf8
characters? Does the email address correspond to a mailbox of the same
name with a quota to be checked?

John

Re: Dovecot/MSQL issue

[John Fawcett via dovecot]

On 30/06/2019 13:36, Aki Tuomi via dovecot wrote:
>>
>> Hello, I'm trying to work through an issue that cropped up on a server I've 
>> been working on and haven't found a very good workaround.
>>
>> Dovecot is operating in a jailed environment.
>>
>> The configuration in dovecot-sql.conf.ext has been set appropriately with 
>> the host=127.0.0.1 (which works from a jailed environment) and when dovecot 
>> attempts to auth it appears to perform a reverse dns lookup and overwrites 
>> the host with the result localhost before using the msql credentials 
>> 'user'@'localhost' which then fails after timing out. 
>>
>> I'm currently running version 2.2.33.2 of dovecot.
>>
>> The workaround seems to be to have two credentials for the same user. 
>> 'user@'localhost' and 'user'@'127.0.0.1'; 
>> postfix operates in a jail as well and works around the jail issue in the 
>> same way but doesn't overwrite the specified host with a reverse dns lookup.
>>
>> Anyone have any suggestions on how to handle this issue better? or maybe 
>> there's a way to force different behavior which I haven't yet found in the 
>> documentation?
>>
>> Any help is much appreciated.
>>
>> Best Regards.
> Hi!
>
> The 127.0.0.1 issue is actually a MySQL issue, unfortunately. 
>
> " On Unix, MySQL programs treat the host name localhost specially, in a way 
> that is likely different from what you expect compared to other network-based 
> programs: the client connects using a Unix socket file. The --socket option 
> or the MYSQL_UNIX_PORT environment variable may be used to specify the socket 
> name. "
>
> So not really something we could do much about.
>
> Aki

Aki

it is possible to force the use of tcp socket instead of unix socket by
specifying an ip instead of the hostname localhost. As I understood it
Nathan is specifying 127.0.0.1 so that will connect with tcp socket.

John

Re: Dovecot/MSQL issue

[John Fawcett via dovecot]

On 01/07/2019 09:48, lorek via dovecot wrote:
> Actually, it seems I may have been wrong in initial assumption that
> the issue with the client was that it was being identified to mysql as
> coming from localhost when connecting via tcp.
> This is what syslog indicated as a reason for the failure but its not
> the whole picture.
>
> As John mentioned I am trying to have dovecot connect over TCP to
> mysql (not using the socket), and the issue looked like the cause was
> the identified by portion of mysql being read by either mysql
> incorrectly or the domain portion being overwritten on dovecot's end
> (I don't know about the internals enough to say for sure where).
>
> Just as due dilligence, I added credentials for a mysql user
> identified by localhost and removed the jail since the dovecot error
> was stating that it failed for connection by user@'localhost' (where
> there weren't credentials).
> After adding the credentials, I performed all the usual mysql tests
> before moving testing up to dovecot and still get an auth failure. The
> log seems to be a bit of a red herring or at the minimum doesn't show
> the whole picture.
>
> Replacing the connection string host with the socket (host=localhost)
> and everything works, and using an external IP that's not 127.0.0.1
> works as expected as well. (confirmed by standing up two isolated
> mysql and dovecot containers and setting auth up over the bridge).
>
> If the issue was caused by user@'localhost' creating the credentials
> should have resolved it, and it didn't. So something weird is going on.
> I've got the environment built up in a dockerfile I can provide if
> anyone wants to dig into what's causing it.
>
> In the meantime due to time constraints, I'll just be working with the
> socket file from now for hosts running most of the mail stack all in one.
>
> Best Regards,
> Lorek.

Lorek

If you have user@localhost as a user in mysql you will be able to
connect with either of these options:

- a tcp socket via an ip address that resolves to localhost

- a unix socket via hostname localhost.

If you have user@127.0.0.1 as a user in mysql you will be able to
connect only via:

- a tcp socket using ip address 127.0.0.1.

John

Re: Dovecot with MySQL over SSL.

[John Fawcett via dovecot]

On 18/07/2019 23:24, Reio Remma via dovecot wrote:
> Hello!
>
> I'm attempting to get Dovecot working with MySQL user database on
> another machine. I can connect to the MySQL (5.7.26) instance with SSL
> enabled:
>
>  mysql -h db.mrst.ee --ssl-ca=/etc/dovecot/ca.pem
> --ssl-cert=/etc/dovecot/client-cert.pem
> --ssl-key=/etc/dovecot/client-key.pem --ssl-cipher=DHE-RSA-AES256-SHA
> -u vmail -p
>
> However if I use the same values in dovecot-sql.conf.ext, I get the
> following error:
>
> Jul 19 00:20:18 turin dovecot: master: Dovecot v2.3.7 (494d20bdc)
> starting up for imap, lmtp, sieve (core dumps disabled)
> Jul 19 00:20:18 turin dovecot: auth-worker(82996): Error:
> mysql(db.mrst.ee): Connect failed to database (vmail): SSL connection
> error: protocol version mismatch - waiting for 1 seconds before retry
> Jul 19 00:20:18 turin dovecot: auth-worker(82996): Error:
> mysql(db.mrst.ee): Connect failed to database (vmail): SSL connection
> error: protocol version mismatch - waiting for 1 seconds before retry
> Jul 19 00:20:19 turin dovecot: auth-worker(82996): Error:
> mysql(db.mrst.ee): Connect failed to database (vmail): Connections
> using insecure transport are prohibited while
> --require_secure_transport=ON. - waiting for 5 seconds before retry
> Jul 19 00:20:19 turin dovecot: auth-worker(82996): Error:
> mysql(db.mrst.ee): Connect failed to database (vmail): Connections
> using insecure transport are prohibited while
> --require_secure_transport=ON. - waiting for 5 seconds before retry
>
> Database connection string:
>
> connect = host=db.mrst.ee dbname=vmail user=vmail password=stuff \
>     ssl_ca=/etc/dovecot/ca.pem \
>     ssl_cert=/etc/dovecot/client-cert.pem \
>     ssl_key=/etc/dovecot/client-key.pem \
>     ssl_cipher=DHE-RSA-AES256-SHA
>
> If I leave the ssl_cipher unset, I get:
>
> Jul 19 00:23:41 turin dovecot: auth-worker(83069): Error:
> mysql(db.mrst.ee): Connect failed to database (vmail): SSL connection
> error: Failed to set ciphers to use - waiting for 1 seconds before retry
>
> Any ideas?
>
> Thanks!
> Reio

One difference between your testing manually with mysql client and the
same configuration in dovecot is the "ssl_verify_server_cert" parameter.
Dovecot is setting it if it is not specified. So to make the tests the
same you should either specify the --ssl_verify_server_cert parameter to
mysql or set it to no in the dovecot configuration.

John

Re: Pigeonhole Sieve only runs scripts if in users' home directories, or if Sieve is purposely broken

[John Fawcett via dovecot]

On 15/05/2024 18:27, Douglas Morse via dovecot wrote:

My apologies, a small inaccuracy:

Below where I mention .svbin files being created, I incorrectly described what 
is occurring. If the `sieve` is set to the default of users’ home directories 
etc., the file `.dovecot.svbin` is created in the user’s home directory. If 
`sieve` is set to the centralized location, then there is no `.dovecot.svbin` 
but rather a `sieve.svbin` created or updated there. Also a bit confusing.


maybe you have a sieve_default setting that is affecting this.


Am 15.05.2024 um 18:18 schrieb Douglas Morse:

Hi,


Dovecot version is 2.3.19.1 (9b53102964) running on fully up-to-date Debian 12 
(Bookworm) operating system.


For the life of me I cannot figure out how to get sieve scripts working 
correctly in a central directory location. Calling them from users’ home 
directories works fine (as described below).

(Note: This system has recently been migrated from Cyrus IMAP, so some of the 
directories for a central location are very similar to what would find in a 
standard Cyrus IMAP installation.)


If I set `sieve` to its default value of `file:~/sieve;active=~/.dovecot.sieve` 
in `conf.d/90-sieve.conf`, sieve works as expected. I have a file 
`~/sieve/default.sieve` and, as per documentation, I make a symbolic link 
`~/.dovecot.sieve` that points to that file. The sieve script runs, and I see a 
`default.svbin` either created or updated in the ~/sieve directories as emails 
arrive.

Setting `sieve` to `file:~/lib/sieve;active=~/.dovecot.sieve` also works as 
expected. So this verifies I can change the directory where sieve scripts are 
stored.

However, if I then set `sieve` to 
`file:/var/spool/dovecot/home/%1n/user/%n/sieve;active=file:/var/spool/dovecot/home/%1n/user/%n/.dovecot.sieve`,
 again a `default.sieve` in each of these user’s centralized `sieve` 
directories, sieve scripts silently fail to run. No messages of any kind found 
in any log files.


I don't believe that the active option can take the "file:" prefix.

John



So, with the same exact same configuration as one that is the default and as 
proven to working, with the single exception of changing the top-level 
directory (TLD) for the scripts, results in silent failure. Why?

What I find particularly odd is that if I keep `sieve` set to this central 
directory location (i.e., no config changes), and `mv sieve/default.sieve 
sieve.tmp; rmdir sieve; mv sieve.tmp`, that is, have a *file* called `sieve` 
where the setting `sieve` points to (which, per docs and experience, should be 
a *directory*), the users’ sieve scripts *do* run, but with a warning along the 
lines of `Warning: sieve: file storage: Explicitly specified active script path 
`file:/var/spool/dovecot/home/m/user/morse/.dovecot.sieve' is ignored; storage 
path `/var/spool/dovecot/home/m/user/morse/sieve' is not a directory`.

In the docs I see that `sieve` used to refer to a file (PidgeonHole Sieve 
v.0.31 or prior, I believe), and there is mention of a a `sieve_dir` setting 
(which I have *never* set or used). So my guess is that sieve working in this 
centralized TLD with `sieve` as a file rather than directory is due to support 
for some deprecated config / functionality.

I have fiddle around with this for some hours, including setting and not 
setting `mail_home` to the central directory location (i.e., `mail_home` in 
`conf.d/10-mail.conf` set to the same value as `sieve` in 
`conf.d/90-sieve.conf`),to no avail. No matter what I do, I cannot get sieve 
scripts to run in a centralized directory location as they do under `/home`. 
The only exception is as just described in the preceding paragraph. But the 
solution only allows a user to have a single sieve script, and I imagine 
`managesieve` will not work properly is sieve is set up as described in the 
preceding paragraph.

What am I missing or doing incorrectly?

Any and all assistance most appreciated.

Doug


___
dovecot mailing list --dovecot@dovecot.org
To unsubscribe send an email todovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Pigenohole bad commands limits and proposed minor patch

[John Fawcett via dovecot]

Hi

I was looking into Pigeonhole behaviour in the case of managesieve 
receiving invalid input either before or after login. I can see that 
there are hard coded limits after 3 sequential bad commands prior to 
authentication and 20 after authentication.


I was wondering if there is a reason for those values. Sieve is a 
protocol that is used with software and except for the limited use case 
of testing it is not used by people typing manually.


I don't know of a lot of sieve clients. Unfortunately the one I used in 
Thunderbird is no longer maintained, so I only have roundcube. But do 
legitimate sieve clients in general make a lot of mistakes?


An additional doubt about errors that I am seeing is that differently to 
imap and pop3 there doen't appear to be  a dedicated ssl port. I only 
have starttls configured. I do see what look like people trying to 
connect with ssl directly on port 4190, which with my configuration is 
never going to be valid.


I am attaching a very simple proposed patch to make the error limits 
configurable via the following settings in conf.d/20-managesieve.conf 
with defaults as per the existing hard coded ones:


  managesieve_max_command_errors = 20
  managesieve_login_max_command_errors = 3

I have similar doubts about imap error limits but I'll start with sieve 
ones.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: sasl database

[John Fawcett via dovecot]

On 01/06/2024 13:22, Jeff P via dovecot wrote:

Hello,

Is there a guide for using a customized SASL database for dovecot?
for example, SQLite.

Thanks in advance.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Hi Jeff

I've no experience with using sqlite with dovecot but there is 
documentation here:


https://doc.dovecot.org/configuration_manual/authentication/sql/#sqlite

The same page also contains configurations for other db servers.

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

[Patch] blocklist access control for dovecot pop3/imap and pigeonhole managesieve

[John Fawcett via dovecot]

Hi

Just in case this is useful more generally, I'm posting it to the list.

While Dovecot has an access control via allow_nets, it is a user 
database field that applies only at the authentication stage to deny 
access for the specific user when there is a connection attempt from an 
unauthorized ip for that user.


https://doc.dovecot.org/configuration_manual/authentication/allow_nets/

I don't believe there is anything that checks access at connect time to 
deny unwanted traffic prior to authentication, for example from 
compromised machines, botnets etc. Though failed connection attempts do 
not appear to be a significant issue, maybe better to add some safety 
net for the future.


The attached patch is proof of concept code that introduces the 
parameters rbl_check and rbl_check_timeout (msecs) to the protocol 
section. Tested for imap, pop3 and sieve. The following is an example 
for sieve.


protocol sieve {
    rbl_check = zen.spamhaus.net=127.0.0.4

    rbl_check_timeout = 5000

}

If the lookup results in a hit the client is disconnected with a BYE 
"Disconnected for policy." message and the logs report:


    Jun 09 12:00:56 server.example.com dovecot[977650]: 
managesieve-login: Disconnected: Policy (disconnected before auth was 
ready, waited 1 secs): user=<>, service=sieve, rip=n.n.n.n, lip=n.n.n.n


The patch also makes the number of pre-login errors and post-login 
errors configurable (max_login_command_errors and max_command_errors 
respectively) for pop3, imap and sieve protocols .


protocol sieve {

  max_command_errors = 1
  max_login_command_errors = 1

}

A potential extension to the logic would be "allow_nets" and 
"disallow_nets" parameters or maybe something more sophisticated to 
allow ips/networks that would otherwise be blocked or deny additional 
ips/networks.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: [Patch] blocklist access control for dovecot pop3/imap and pigeonhole managesieve

[John Fawcett via dovecot]

The files themselves didn't make it through the list server. I'll try 
with .txt



On 09/06/2024 15:42, John Fawcett via dovecot wrote:

Hi

Just in case this is useful more generally, I'm posting it to the list.

While Dovecot has an access control via allow_nets, it is a user 
database field that applies only at the authentication stage to deny 
access for the specific user when there is a connection attempt from 
an unauthorized ip for that user.


https://doc.dovecot.org/configuration_manual/authentication/allow_nets/

I don't believe there is anything that checks access at connect time 
to deny unwanted traffic prior to authentication, for example from 
compromised machines, botnets etc. Though failed connection attempts 
do not appear to be a significant issue, maybe better to add some 
safety net for the future.


The attached patch is proof of concept code that introduces the 
parameters rbl_check and rbl_check_timeout (msecs) to the protocol 
section. Tested for imap, pop3 and sieve. The following is an example 
for sieve.


protocol sieve {
    rbl_check = zen.spamhaus.net=127.0.0.4

    rbl_check_timeout = 5000

}

If the lookup results in a hit the client is disconnected with a BYE 
"Disconnected for policy." message and the logs report:


    Jun 09 12:00:56 server.example.com dovecot[977650]: 
managesieve-login: Disconnected: Policy (disconnected before auth was 
ready, waited 1 secs): user=<>, service=sieve, rip=n.n.n.n, lip=n.n.n.n


The patch also makes the number of pre-login errors and post-login 
errors configurable (max_login_command_errors and max_command_errors 
respectively) for pop3, imap and sieve protocols .


protocol sieve {

  max_command_errors = 1
  max_login_command_errors = 1

}

A potential extension to the logic would be "allow_nets" and 
"disallow_nets" parameters or maybe something more sophisticated to 
allow ips/networks that would otherwise be blocked or deny additional 
ips/networks.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.orgdiff -x '*.log' -x '*.status' -x '*.deps' -x '*.a' -x '*.o' -ur 
dovecot-2.3.21-orig/src/config/all-settings.c 
dovecot-2.3.21/src/config/all-settings.c
--- dovecot-2.3.21-orig/src/config/all-settings.c   2023-09-14 
15:18:20.0 +0200
+++ dovecot-2.3.21/src/config/all-settings.c2024-06-09 11:31:45.820319492 
+0200
@@ -1750,6 +1750,7 @@
 struct pop3_settings {
bool verbose_proctitle;
const char *rawlog_dir;
+   unsigned int max_bad_commands;
 
/* pop3: */
bool pop3_no_flag_updates;
@@ -1849,6 +1850,10 @@
 
unsigned int mail_max_userip_connections;
 
+   const char *rbl_check;
+   unsigned int rbl_check_timeout;
+   unsigned int max_login_command_errors;
+
/* generated: */
char *const *log_format_elements_split;
 };
@@ -1907,6 +1912,7 @@
 struct imap_settings {
bool verbose_proctitle;
const char *rawlog_dir;
+   unsigned int max_command_errors;
 
/* imap: */
uoff_t imap_max_line_length;
@@ -3277,6 +3283,7 @@
 static const struct setting_define pop3_setting_defines[] = {
DEF(BOOL, verbose_proctitle),
DEF(STR_VARS, rawlog_dir),
+   DEF(UINT, max_bad_commands),
 
DEF(BOOL, pop3_no_flag_updates),
DEF(BOOL, pop3_enable_last),
@@ -3295,6 +3302,7 @@
 static const struct pop3_settings pop3_default_settings = {
.verbose_proctitle = FALSE,
.rawlog_dir = "",
+   .max_bad_commands = 20,
 
.pop3_no_flag_updates = FALSE,
.pop3_enable_last = FALSE,
@@ -4183,6 +4191,10 @@
 
DEF(UINT, mail_max_userip_connections),
 
+   DEF(STR, rbl_check),
+   DEF(TIME_MSECS, rbl_check_timeout),
+   DEF(UINT, max_login_command_errors),
+
SETTING_DEFINE_LIST_END
 };
 static const struct login_settings login_default_settings = {
@@ -4209,7 +4221,11 @@
.auth_debug = FALSE,
.verbose_proctitle = FALSE,
 
-   .mail_max_userip_connections = 10
+   .mail_max_userip_connections = 10,
+
+   .rbl_check = "",
+   .rbl_check_timeout = 10*1000,
+   .max_login_command_errors = 3
 };
 const struct setting_parser_info login_setting_parser_info = {
.module_name = "login",
@@ -4656,6 +4672,7 @@
 static const struct setting_define imap_setting_defines[] = {
DEF(BOOL, verbose_proctitle),
DEF(STR_VARS, rawlog_dir),
+   DEF(UINT, max_command_errors),
 
DEF(SIZE, imap_max_line_length),
DEF(TIME, imap_idle_notify_interval),
@@ -4677,6 +4694,7 @@
 static const struct imap_settings imap_default_settings = {
.verbose_proctitle = FALSE,
.rawlog_dir = "",
+   .max_command_errors = 20,
 
/* RFC-2683 recommends at least 8000 by

Re: [Patch] blocklist access control for dovecot pop3/imap and pigeonhole managesieve

[John Fawcett via dovecot]

On 09/06/2024 18:39, Aki Tuomi via dovecot wrote:

On 09/06/2024 16:50 EEST John Fawcett via dovecot  wrote:

  
The files themselves didn't make it through the list server. I'll try

with .txt


On 09/06/2024 15:42, John Fawcett via dovecot wrote:

Hi

Just in case this is useful more generally, I'm posting it to the list.

While Dovecot has an access control via allow_nets, it is a user
database field that applies only at the authentication stage to deny
access for the specific user when there is a connection attempt from
an unauthorized ip for that user.

https://doc.dovecot.org/configuration_manual/authentication/allow_nets/

I don't believe there is anything that checks access at connect time
to deny unwanted traffic prior to authentication, for example from
compromised machines, botnets etc. Though failed connection attempts
do not appear to be a significant issue, maybe better to add some
safety net for the future.

The attached patch is proof of concept code that introduces the
parameters rbl_check and rbl_check_timeout (msecs) to the protocol
section. Tested for imap, pop3 and sieve. The following is an example
for sieve.

protocol sieve {
     rbl_check = zen.spamhaus.net=127.0.0.4

     rbl_check_timeout = 5000

}

If the lookup results in a hit the client is disconnected with a BYE
"Disconnected for policy." message and the logs report:

     Jun 09 12:00:56 server.example.com dovecot[977650]:
managesieve-login: Disconnected: Policy (disconnected before auth was
ready, waited 1 secs): user=<>, service=sieve, rip=n.n.n.n, lip=n.n.n.n

The patch also makes the number of pre-login errors and post-login
errors configurable (max_login_command_errors and max_command_errors
respectively) for pop3, imap and sieve protocols .

protocol sieve {

   max_command_errors = 1
   max_login_command_errors = 1

}

A potential extension to the logic would be "allow_nets" and
"disallow_nets" parameters or maybe something more sophisticated to
allow ips/networks that would otherwise be blocked or deny additional
ips/networks.

John


How is this different than using weakforced which already supports rbl? Or 
doing this in auth lua?


Hi Aki

I'm not that familiar with weakforced or auth lua, but I guess those 
intervene at the authentication stage, rather than the connection stage.


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Debian Bookworm packages, please !

[John Fawcett via dovecot]

On 26/06/2024 20:48, pgnd via dovecot wrote:

for anyone interested, for dovecot v2.3.14+ @ Fedora,

https://src.fedoraproject.org/rpms/dovecot/blob/rawhide/f/dovecot-2.3.14-opensslv3.patch 




Until this discussion started I didn't realize that I've been using the 
unsupported version of openssl 3 for quite some time with dovecot 2.3.21 
on Fedora 39 and probably previous versions of Fedora too, without any 
issues. As others have mentioned it may depend on which features are in use.


I actually also compiled a vanilla 2.3.21 (i.e. without the fedora 
patches) for development work and didn't see any issues, though I 
wouldn't use it for a live system without the patches.


I can understand that with 2.4 getting quite close (and after having 
originally an earlier plan for it) that Ox wasn't planning to invest in 
backporting stuff to 2.3 branches. There is nothing stopping the 
community from doing that where needed. But given the issues mentioned, 
can anyone point to reproducable issue reports with the current packaged 
versions in Debian? (apart from the non working mail crypt plugin 
mentioned by Aki).


Thanks

John



___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

On 29/06/2024 10:08, postfix_dovecot--- via dovecot wrote:

Host: VMware Workstation 14.1.8

OS: Debian 12 (Bookworm)

Dovecot: 2.3.19.1

Postfix: 3.7.10
Mailclient: Outlook 2016

  


Hello,

I am currently working on a new mail server to replace my Debian 10 mail 
server. For preparation, I use VMware Workstation to learn and test the 
installation steps. When I'm eventually done, I'll rebuild my root server from 
scratch.

This time, my problem is getting the IMAPSieve plugin working to trigger rspamd 
if mail gets moved to the junk folder. Sieve runs well – if rspam recognizes a 
spam mail, it will be transferred to the junk folder.



For me it looks like the IMAPSieve plugin recognizes the move but will not 
running the script behind. I can see some actions in the log but nothing 
happens on the rspamd side where the log is also open to view.
Moving a mail with Outlook produces the following log entry:

imap(mail@test.example)<1797>: Debug: Module loaded: 
/usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so

imap(mail@test.example)<1797>: Debug: imapsieve: mailbox 
INBOX/Spam: APPEND event

imap(mail@test.example)<1797>: Debug: sieve: Pigeonhole 
version 0.5.19 (4eae2f79) initializing

imap(mail@test.example)<1797>: Debug: sieve: include: 
sieve_global is not set; it is currently not possible to include `:global' scripts.

imap(mail@test.example)<1797>: Debug: sieve: Sieve imapsieve 
plugin for Pigeonhole version 0.5.19 (4eae2f79) loaded

imap(mail@test.example)<1797>: Debug: sieve: Sieve 
Extprograms plugin for Pigeonhole version 0.5.19 (4eae2f79) loaded

imap(mail@test.example)<1797>: Debug: imapsieve: Static mailbox 
rule [1]: mailbox=`Spam' from=`*' causes=(COPY APPEND) => 
before=`file:/usr/lib/dovecot/sieve/report-spam.sieve' after=(none)

imap(mail@test.example)<1797>: Debug: imapsieve: Static mailbox 
rule [2]: mailbox=`*' from=`Spam' causes=(COPY APPEND) => 
before=`file:/usr/lib/dovecot/sieve/report-ham.sieve' after=(none)

imap(mail@test.example)<1795>: Debug: imapsieve: mailbox 
INBOX: FLAG event (changed flags: \Deleted)

imap(mail@test.example)<1795>: Debug: sieve: Pigeonhole 
version 0.5.19 (4eae2f79) initializing

imap(mail@test.example)<1795>: Debug: sieve: include: 
sieve_global is not set; it is currently not possible to include `:global' scripts.

imap(mail@test.example)<1795>: Debug: sieve: Sieve imapsieve 
plugin for Pigeonhole version 0.5.19 (4eae2f79) loaded

imap(mail@test.example)<1795>: Debug: sieve: Sieve 
Extprograms plugin for Pigeonhole version 0.5.19 (4eae2f79) loaded

imap(mail@test.example)<1795>: Debug: imapsieve: Static mailbox 
rule [1]: mailbox=`Spam' from=`*' causes=(COPY APPEND) => 
before=`file:/usr/lib/dovecot/sieve/report-spam.sieve' after=(none)

imap(mail@test.example)<1795>: Debug: imapsieve: Static mailbox 
rule [2]: mailbox=`*' from=`Spam' causes=(COPY APPEND) => 
before=`file:/usr/lib/dovecot/sieve/report-ham.sieve' after=(none)

imap(mail@test.example)<1795>: Debug: imapsieve: mailbox 
INBOX: FLAG event (changed flags: \Seen)

  ..


Hi Jens

I can't give you any expert advise about that plugin as I haven't used 
it. However, looking at the debug info you posted I don't see where the 
rules are being matched.


That debugging looks like it is reading in the configuration, so you can 
see that dovecot recognises the rules configuration, but nothing in the 
debugging suggests that those rules are being matched or that you sieve 
scripts are being executed.


So it seems that the event that is being recognized "mailbox INBOX/Spam: 
APPEND event" doesn't match your defined rules. Maybe it's to do with 
name of the mailbox in the event not matching the one in your rule 1. 
It's a wild guess on my part, but maybe changing the rules to use 
"INBOX/Spam" might just solve it.


When the rules match you should see a message like "Matched static 
mailbox rule n" where n=1,2 etc in the debugging output.


John
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: AW: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

On 30/06/2024 19:57, postfix_dovecot--- via dovecot wrote:

Tried it now in every known combination. Nothing changes. It's my 7th evening 
about this and I'm starting to despair :(


Hi Jens

any chance of posting the debugging from that  test in case it gives any 
clues.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Sieve Symlink Error

[John Fawcett via dovecot]

On 30/06/2024 07:17, Benjamin Rose via dovecot wrote:

Hello,

I'm in the process of moving our mail server from RHEL 6 to RHEL 9. We 
will be moving to:


# dovecot --version
2.3.16 (7e2e900c1a)

My issue is that sieve does not appear to work on the new setup, where 
it does work on the old one. I made a simple filter rule:


# cat /u/mail0test/.sieve/ingo.sieve
# Sieve Filter
# Generated by Ingo (http://www.horde.org/apps/ingo/) (06/28/2024, 
11:14:52 PM)

require "fileinto";
# Test
if header :comparator "i;ascii-casemap" :contains "Subject" 
"filtertest"  {

    fileinto "Fun";
    stop;
}

Upon sending an email to this test account, the following appears in 
/var/log/maillog:


Jun 29 23:19:56 mail5 dovecot[3066980]: 
lda(mail0test)<3066980>: Warning: sieve: file 
storage: Active sieve script symlink /u/mail0test/.dovecot.sieve is 
broken: Invalid/unknown path to storage (points to /u/mail0test/.sieve).
Jun 29 23:19:56 mail5 dovecot[2987026]: 
doveadm(mail0test)<3066983>: Warning: sieve: 
file storage: Active sieve script symlink /u/mail0test/.dovecot.sieve 
is broken: Invalid/unknown path to storage (points to 
/u/mail0test/.sieve).
Jun 29 23:19:56 mail5 dovecot[2987026]: 
doveadm(mail0test)<3067016>: Warning: sieve: 
file storage: Active sieve script symlink /u/mail0test/.dovecot.sieve 
is broken: Invalid/unknown path to storage (points to 
/u/mail0test/.sieve).


Yet:

# ll /u/mail0test/.dovecot.sieve
lrwxrwxrwx. 1 mail0test sysguest 17 Jun 28 23:26 
/u/mail0test/.dovecot.sieve -> .sieve/ingo.sieve

# file /u/mail0test/.sieve/ingo.sieve
/u/mail0test/.sieve/ingo.sieve: ASCII text

That is the filter file I've pasted above.

I've set the following directives in /etc/dovecot/conf.d/90-sieve.conf 
via puppet:


augeas {
  "dovecot_sieve_settings":
    context => "/files/etc/dovecot/conf.d/90-sieve.conf",
    changes => [
      "set plugin/sieve_dir ~/.sieve",
  "set plugin/sieve_user_log ~/.sieve/log"
    ],
    require => Package["dovecot"],
    notify => Service["dovecot"];
}

The full configuration dump is attached.

/u in our environment is the path for user homedirs, which is an NFS 
mount to a NetApp. The OS is Springdale Linux 9.2, a clone of RedHat 
from before the IBM license change. It will soon be RHEL 9.4 as we 
have obtained a license, but for all intents and purposes, Springdale 
9.2 and RHEL 9.2 should be considered bug-for-bug compatible. The arch 
is x86_64 with both machines mail5 and mail6 (replicated) having 
Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz and 768gb of memory. I have 
the same issue with SELinux in both enforcing and permissive modes, so 
this is not a permissions error due to SELinux.


Am I doing something wrong, or is this a bug? I've seen that there 
have been some previous issues similar to this that ended up being 
bugs in pigeonhole, so here I am.


Thanks,
Ben


Hi Ben

what version of Pigeonhole are you using?

I read here that sieve_dir is deprecated since v0.3.1

https://doc.dovecot.org/settings/pigeonhole/#pigeonhole_setting-sieve_dir

In any case these settings look as though they don't really match up. Is 
the correct directory .sieve or sieve?


sieve = file:~/sieve;active=~/.dovecot.sieve
sieve_dir = ~/.sieve

Also, I was curious if your inboxes are really under /var/spool/mail/%u 
and your indexes under /home/%u/indexes?


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Sieve Symlink Error

[John Fawcett via dovecot]

On 01/07/2024 03:32, Benjamin Rose via dovecot wrote:


On 6/30/24 16:48, John Fawcett via dovecot wrote:


On 30/06/2024 07:17, Benjamin Rose via dovecot wrote:

Hello,

I'm in the process of moving our mail server from RHEL 6 to RHEL 9. 
We will be moving to:


# dovecot --version
2.3.16 (7e2e900c1a)

My issue is that sieve does not appear to work on the new setup, 
where it does work on the old one. I made a simple filter rule:


# cat /u/mail0test/.sieve/ingo.sieve
# Sieve Filter
# Generated by Ingo (http://www.horde.org/apps/ingo/) (06/28/2024, 
11:14:52 PM)

require "fileinto";
# Test
if header :comparator "i;ascii-casemap" :contains "Subject" 
"filtertest"  {

    fileinto "Fun";
    stop;
}

Upon sending an email to this test account, the following appears in 
/var/log/maillog:


Jun 29 23:19:56 mail5 dovecot[3066980]: 
lda(mail0test)<3066980>: Warning: sieve: 
file storage: Active sieve script symlink 
/u/mail0test/.dovecot.sieve is broken: Invalid/unknown path to 
storage (points to /u/mail0test/.sieve).
Jun 29 23:19:56 mail5 dovecot[2987026]: 
doveadm(mail0test)<3066983>: Warning: sieve: 
file storage: Active sieve script symlink 
/u/mail0test/.dovecot.sieve is broken: Invalid/unknown path to 
storage (points to /u/mail0test/.sieve).
Jun 29 23:19:56 mail5 dovecot[2987026]: 
doveadm(mail0test)<3067016>: Warning: sieve: 
file storage: Active sieve script symlink 
/u/mail0test/.dovecot.sieve is broken: Invalid/unknown path to 
storage (points to /u/mail0test/.sieve).


Yet:

# ll /u/mail0test/.dovecot.sieve
lrwxrwxrwx. 1 mail0test sysguest 17 Jun 28 23:26 
/u/mail0test/.dovecot.sieve -> .sieve/ingo.sieve

# file /u/mail0test/.sieve/ingo.sieve
/u/mail0test/.sieve/ingo.sieve: ASCII text

That is the filter file I've pasted above.

I've set the following directives in 
/etc/dovecot/conf.d/90-sieve.conf via puppet:


augeas {
  "dovecot_sieve_settings":
    context => "/files/etc/dovecot/conf.d/90-sieve.conf",
    changes => [
      "set plugin/sieve_dir ~/.sieve",
  "set plugin/sieve_user_log ~/.sieve/log"
    ],
    require => Package["dovecot"],
    notify => Service["dovecot"];
}

The full configuration dump is attached.

/u in our environment is the path for user homedirs, which is an NFS 
mount to a NetApp. The OS is Springdale Linux 9.2, a clone of RedHat 
from before the IBM license change. It will soon be RHEL 9.4 as we 
have obtained a license, but for all intents and purposes, 
Springdale 9.2 and RHEL 9.2 should be considered bug-for-bug 
compatible. The arch is x86_64 with both machines mail5 and mail6 
(replicated) having Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz and 
768gb of memory. I have the same issue with SELinux in both 
enforcing and permissive modes, so this is not a permissions error 
due to SELinux.


Am I doing something wrong, or is this a bug? I've seen that there 
have been some previous issues similar to this that ended up being 
bugs in pigeonhole, so here I am.


Thanks,
Ben


Hi Ben

what version of Pigeonhole are you using?

I read here that sieve_dir is deprecated since v0.3.1

https://doc.dovecot.org/settings/pigeonhole/#pigeonhole_setting-sieve_dir 



In any case these settings look as though they don't really match up. 
Is the correct directory .sieve or sieve?


sieve = file:~/sieve;active=~/.dovecot.sieve
sieve_dir = ~/.sieve

Also, I was curious if your inboxes are really under 
/var/spool/mail/%u and your indexes under /home/%u/indexes?


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org



Hello,

Thank you! Adding the line to puppet to enforce that this exists in 
/etc/dovecot/conf.d/90-sieve.conf:


sieve=file:~/.sieve;active=~/.dovecot.sieve

has solved the problem. Filters now work as expected!

To answer your questions, I am using 
dovecot-pigeonhole-2.3.16-8.el9.x86_64, and yes, user mail spools live 
under /var/spool/mail (NFS-mounted mbox files) and indexes live under 
/home (local disk - soon to be SSD). That's only for users who are 
using mbox format / pine / mutt. Most users are using only modern 
clients and in this case their storage is mdbox and entirely kept 
inside of /home. This is configured on a per-user basis inside of an 
LDAP value named mailMessageStore. Either it exists such as 
"mdbox:/home//mail", or it does not exist at all, in which 
case, delivery falls back to old-style mbox format. If they are on 
mbox format, only INBOX is kept in /var/spool/mail, all other folders 
are kept in ~/mail (/u//mail/).


Ben


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Hi Ben

as far as I can see, dovecot-pigeonhole-2.3.16-8.el9.x86_64 uses 
pigeonhole 

Re: AW: AW: AW: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

On 01/07/2024 22:09, postfix_dovecot--- via dovecot wrote:

Hi John,

  


the prefix is ​​just a sign of my desperation - I tried all sorts of variations 
yesterday and now forgot to undo it.

  


There’s a very detailed tutorial available (German language) with Debian 10. 
Just the sieve scripts are little different. I spend the time to raise a Debian 
10 with this tutorial – and with exact the same result as with my up to date 
configuration.


So it would be interesting for me to know if there is anyone here who has 
managed to get this to work on Debian?

  


Jens



Hi Jens

changing random things is rarely a good way to solve these kinds of 
issues. My advice, if you don't need the inbox namespace prefix set for 
a specific reason, would be to go with the default prefix (ie. blank) 
and then set up the imapsieve rules as per your original post and repeat 
the test. If that does not work then post the debug logging and 
configuration from that test.


There could be multiple places where this is failing but the very first 
point is to have your sieve rules match. They will show something like 
the following in the logging. Notice the "Matched static mailbox rule" 
message. If that is not happening it's pointless to look further down 
the line into issues in the scripts themselves.


Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: imapsieve: 
mailbox INBOX: MOVE event
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: Mailbox Spam: 
UID 2: Expunge requested
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: duplicate db: 
Initialize
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: 
Pigeonhole version 0.5.21 (f6cd4b8e) initializing
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: include: 
sieve_global is not set; it is currently not possible to include 
`:global' scripts.
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: Sieve 
imapsieve plugin for Pigeonhole version 0.5.21 (f6cd4b8e) loaded
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: Sieve 
Extprograms plugin for Pigeonhole version 0.5.21 (f6cd4b8e) loaded
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: imapsieve: 
Static mailbox rule [1]: mailbox=`Spam' from=`*' causes=(COPY) => 
before=`file:/usr/lib/dovecot/sieve/report-spam.sieve' after=(none)
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: imapsieve: 
Static mailbox rule [2]: mailbox=`*' from=`Spam' causes=(COPY APPEND) => 
before=`file:/usr/lib/dovecot/sieve/report-ham.sieve' after=(none)
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: imapsieve: 
Matched static mailbox rule [2]


As for Debian, sorry I can't help on that, I'm using Fedora :-)

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: AW: AW: AW: AW: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

demo.example)<1827>: Debug: imapsieve: Static mailbox 
rule [2]: mailbox=`*' from=`Junk' causes=(COPY) => 
before=`file:/usr/lib/dovecot/sieve/report-ham.sieve' after=(none)
imap(info@demo.example)<1827>: Debug: Mailbox INBOX/Junk: 
Mailbox opened
imap(info@demo.example)<1825>: Debug: imapsieve: mailbox 
INBOX: FLAG event (changed flags: \Deleted)
imap(info@demo.example)<1825>: Debug: Mailbox INBOX: UID 1: 
Expunge requested
imap(info@demo.example)<1825>: Debug: Mailbox INBOX: UID 1: 
Mail expunged

# doveconf -n
# 2.3.19.1 (9b53102964): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.19 (4eae2f79)
# OS: Linux 6.1.0-21-amd64 x86_64 Debian 12.5
# Hostname: ServerIV-home.demo.example
auth_mechanisms = plain login
mail_debug = yes
mail_location = maildir:~/Maildir
mail_privileged_group = mail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date index ihave duplicate 
mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve
namespace inbox {
   hidden = no
   ignore_on_failure = no
   inbox = yes
   list = yes
   location =
   mailbox Drafts {
 auto = subscribe
 special_use = \Drafts
   }
   mailbox Junk {
 auto = subscribe
 special_use = \Junk
   }
   mailbox Sent {
 auto = subscribe
 special_use = \Sent
   }
   mailbox "Sent Messages" {
 auto = subscribe
 special_use = \Sent
   }
   mailbox Trash {
 auto = subscribe
 special_use = \Trash
   }
   prefix = INBOX/
   separator = /
   subscriptions = yes
   type = private
}
passdb {
   driver = pam
}
passdb {
   args = scheme=CRYPT username_format=%u /etc/dovecot/users
   driver = passwd-file
}
plugin {
   imapsieve_mailbox1_before =file:/usr/lib/dovecot/sieve/report-spam.sieve
   imapsieve_mailbox1_causes = COPY
   imapsieve_mailbox1_name = Junk
   imapsieve_mailbox2_before =file:/usr/lib/dovecot/sieve/report-ham.sieve
   imapsieve_mailbox2_causes = COPY
   imapsieve_mailbox2_from = Junk
   imapsieve_mailbox2_name = *
   sieve =file:~/sieve;active=~/.dovecot.sieve
   sieve_after = /etc/dovecot/conf.d/custom-sieve/global_after.sieve
   sieve_before = /etc/dovecot/conf.d/custom-sieve/global_before.sieve
   sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment 
+vnd.dovecot.debug
   sieve_pipe_bin_dir = /usr/lib/dovecot/sieve
   sieve_plugins = sieve_imapsieve sieve_extprograms
}
protocols = imap lmtp sieve
service auth {
   unix_listener /var/spool/postfix/private/auth {
 mode = 0666
   }
}
service lmtp {
   inet_listener lmtp {
 address = 127.0.0.1 ::1
 port = 24
   }
}
ssl_cert = Von: John Fawcett via dovecot  
Gesendet: Montag, 1. Juli 2024 23:16

An:dovecot@dovecot.org
Betreff: Re: AW: AW: AW: IMAPSieve plugin will not run rspamd script

Hi Jens

changing random things is rarely a good way to solve these kinds of issues. My 
advice, if you don't need the inbox namespace prefix set for a specific reason, 
would be to go with the default prefix (ie. blank) and then set up the 
imapsieve rules as per your original post and repeat the test. If that does not 
work then post the debug logging and configuration from that test.

There could be multiple places where this is failing but the very first point is to have 
your sieve rules match. They will show something like the following in the logging. 
Notice the "Matched static mailbox rule"
message. If that is not happening it's pointless to look further down the line 
into issues in the scripts themselves.

Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: imapsieve: mailbox 
INBOX: MOVE event
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: Mailbox Spam: UID 2: 
Expunge requested
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: duplicate db: Initialize
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: Pigeonhole 
version 0.5.21 (f6cd4b8e) initializing
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: include: 
sieve_global is not set; it is currently not possible to include `:global' scripts.
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: Sieve imapsieve 
plugin for Pigeonhole version 0.5.21 (f6cd4b8e) loaded
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: Sieve 
Extprograms plugin for Pigeonhole version 0.5.21 (f6cd4b8e) loaded
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: imapsieve: Static mailbox 
rule [1]: mailbox=`Spam' from=`*' causes=(COPY) => 
before=`file:/

Re: AW: AW: AW: AW: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

Hi Jens

just one update and some more insight looking at the code

Your causes should be COPY APPEND (whether applying proposed solution 
(1) or (2). I updated inline below for solution (2).


Logic for that: I was testing from Thunderbird which is generating a 
MOVE event. In the imapsieve plugin code MOVE and COPY are treated 
almost the same, so my rule with COPY fire on MOVE too. In your case the 
event you're getting is APPEND, but none of your rules have that as a 
cause so won't match even if the mailbox matches.


John

On 02/07/2024 12:07, John Fawcett via dovecot wrote:

Hi Jens

that log looks to me like it is for moving a message from Inbox to 
Junk rather than Junk to Inbox.


The issue I see is that how you defined the rules does not match your 
namespace naming scheme. Event being received is the following. In 
that event you can see the name of the mailbox is "INBOX/Junk".


imap(info@demo.example)<1827>: Debug: imapsieve: 
mailbox INBOX/Junk: APPEND event


However the rule you defined has the mailbox as "Junk".

Static mailbox rule [1]: mailbox=`Junk' from=`*' causes=(COPY) => 
before=`file:/usr/lib/dovecot/sieve/report-spam.sieve' after=(none)


The logging you show confirms that the rule is not matching, else you 
would get a log line like the one I showed with the text "Debug: 
imapsieve: Matched static mailbox rule". So I have two possible 
alternative solutions, that in my tests I found both working:


(1) you define your inbox namespace with blank prefix instead of 
prefix=INBOX/


OR

(2) you adjust the rules as follows

  imapsieve_mailbox1_before 
=file:/usr/lib/dovecot/sieve/report-spam.sieve

  imapsieve_mailbox1_causes = COPY APPEND
  imapsieve_mailbox1_name = INBOX/Junk
  imapsieve_mailbox2_before =file:/usr/lib/dovecot/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = COPY APPEND
  imapsieve_mailbox2_from = INBOX/Junk
  imapsieve_mailbox2_name = *

If you apply one of those solutions and repeat the test 
unsuccessfully, it would be good to see the logging from that test.

John

On 02/07/2024 11:28, postfix_dovecot--- via dovecot wrote:

Hi John!

You are of course absolutely right - wild experimentation is rarely a 
good approach. But you are surely familiar with the situation when 
you have done everything logical and then, in your desperation, you 
start to change things that you would not otherwise change ;)


I've actually gotten to the point where I'm going to give up on the 
idea and run a cron job at night. Then the spam won't be detected in 
real time - but what the heck?


As I wrote yesterday, I implemented a very popular tutorial for 
Debian 10 in Germany 1:1. The result was the same. Nevertheless, I 
reset my VM, where I do everything with the current releases, and 
attaching the log here. I would be very happy if it still works.


To be sure, I deleted the whole inbox and recreated the account on 
Outlook to create a fresh, new folder structure given by the Dovecot 
configuration. I also added the plugin vnd.dovecot.debug to the 
sieve_global_extensions line and added >debug_log "report-spam.sieve 
was running!";< to the report-spam.sieve file. So this string should 
appear in the log if the script were running at all. But it doesn't. 
So I still guess, it's not getting triggered.


That's the log if I move a mail from Junk (Spam) to Inbox:

# journalctl -u dovecot -f
imap-login: Login: user=, method=PLAIN, 
rip=192.168.30.26, lip=192.168.30.34, mpid=1827, TLS, 
session=
imap(info@demo.example)<1827>: Debug: Loading 
modules from directory: /usr/lib/dovecot/modules
imap(info@demo.example)<1827>: Debug: Module 
loaded: /usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so
imap(info@demo.example)<1827>: Debug: Effective 
uid=1, gid=1, home=/var/mail/vhosts/demo.example/info
imap(info@demo.example)<1827>: Debug: 
open(/proc/self/io) failed: Permission denied
imap(info@demo.example)<1827>: Debug: Namespace 
inbox: type=private, prefix=INBOX/, sep=/, inbox=yes, hidden=no, 
list=yes, subscriptions=yes location=maildir:~/Maildir
imap(info@demo.example)<1827>: Debug: maildir++: 
root=/var/mail/vhosts/demo.example/info/Maildir, index=, indexpvt=, 
control=, inbox=/var/mail/vhosts/demo.example/info/Maildir, alt=
imap(info@demo.example)<1827>: Debug: Namespace : 
type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, 
subscriptions=no location=fail::LAYOUT=none
imap(info@demo.example)<1827>: Debug: none: root=, 
index=, indexpvt=, control=, inbox=, alt=
imap(info@demo.example)<1827>: Debug: Mailbox 
INBOX/Junk: Mailbox opened
imap(info@demo.example)<1827>: Debug: imapsieve: 
mailbox INBOX/Junk: APPEND event
imap(info@demo.example)<1827>: Debug: duplicate db: 
Initialize
imap(info@demo.example)<1827>: Debug: sieve: 
Pigeonhole version 0.5.19 (4eae2f79) initializing
i

Re: AW: AW: AW: AW: AW: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

(Resending because of size limit exceeded on previous post)

Hi Jens

that looks much better, though not yet completely solved. What you're 
now getting is both rules firing when you move a message to Junk, 
whereas you only want the first one firing. I guess you will have no 
rules firing when you move from Junk to elsewhere. My suspicion is that 
for APPEND event the target and destination mailboxes are set to the 
same value.  i.e. APPEND event for moving to Junk will have 
mailbox=INBOX/Junk and from=INBOX/Junk which fires both rules, whereas 
when moving from Junk to INBOX mailbox=INBOX and from=INBOX, leading to 
KO for rule one on "mailbox" and KO on rule 2 for "from".


Static mailbox rule [1]: mailbox=`INBOX/Junk' from=`*' causes=(COPY 
APPEND) => before=`file:/usr/lib/dovecot/sieve/report-spam.sieve' 
after=(none)


Static mailbox rule [2]: mailbox=`*' from=`INBOX/Junk' causes=(COPY APPEND) => 
before=`file:/usr/lib/dovecot/sieve/report-ham.sieve' after=(none)

Whenever I do the move of the message I get a MOVE (which is treated 
like COPY). I don't see these APPEND events. Not sure why your email 
client generates APPEND instead of MOVE. That could be a point to 
investigate. If you're going to have to manage APPEND events you'll 
probably need different rules. If you'll going to manage MOVE/COPY and 
APPEND events then you can probably combine sets of rules (adjusting the 
sequential numbering).


The following work for MOVE/COPY events

  imapsieve_mailbox1_before =file:/usr/lib/dovecot/sieve/report-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = INBOX/Junk
  imapsieve_mailbox2_before =file:/usr/lib/dovecot/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = INBOX/Junk
  imapsieve_mailbox2_name = *

The following should work for APPEND events, but it won't support 
wildcards fo rule 2 else it will fire when moving to Junk as well as 
from Junk. I'm pretty sure you can't specify negation e.g. 
imapsieve_mailbox2_name = !INBOX/Junk, which is what would be needed to 
make it work for moving from Junk to any mailbox. As written rule 2 
fires for moving from Junk to INBOX


  imapsieve_mailbox1_before =file:/usr/lib/dovecot/sieve/report-spam.sieve
  imapsieve_mailbox1_causes = APPEND
  imapsieve_mailbox1_name = INBOX/Junk
  imapsieve_mailbox2_before =file:/usr/lib/dovecot/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = APPEND
  imapsieve_mailbox2_name = INBOX

John

On 02/07/2024 14:51, postfix_dovecot--- via dovecot wrote:

Gotcha

It looks like you nailed it John! (Log and config attached)

  


The log looks now very different and my log entry is also shown.

Now I can continue working at this point. Adjusting the events i.e.

  


The reason was actually simple - but I spent a few evenings searching and was 
blind. I could have figured it out on my own! Thank you so much John for took 
such care of this issue!

Jens

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

login_access_sockets

[John Fawcett via dovecot]

Hi

I've used Dovecot for a long time but I never stop learning about the 
depth of features I've never used I just discovered the 
login_access_sockets setting when reading this page:


https://doc.dovecot.org/admin_manual/login_processes/

The compilation using  --with-libwrap configure setting was not 
immediate, since on Fedora which I am using there is no 
tcp_wrappers-devel package so tcpd.h wasn't installed in /usr/include 
and the library name libwrap.so that was expected by Dovecot needed a 
symlink to the library installed by tcp_wrappers package.


So differently to what I previously understood, Dovecot does have a way 
of controlling access at connect time via tcp_wrappers hosts.allow and 
hosts.deny files. But now I also realize that this discovery will be 
short lived, since I understand this is going away in 3.0 and also 2.4 
though I can't find a page on that:


https://doc.dovecotpro.com/3.0/installation/upgrading/2.3.x-3.x.html

I can understand the logic about slimming down the code to leave out 
less used features, but this part of the code looks well written and 
hasn't required much maintenance over the years. I know there's zero 
chance of reversing the removal of login_access_sockets, but I just 
wondered whether there could be the chance to introduce a LUA hook at 
connect time rather than waiting for authentication to be done. Also I 
have no idea what the overhead of LUA script is, does it spawn 
additional processes per connection?


Thanks

John




___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: AW: AW: AW: AW: AW: AW: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

On 03/07/2024 20:53, postfix_dovecot--- via dovecot wrote:

Hi John,

again you're hitting the point!

I put a "report-spam.sieve fired!" and "report-ham.sieve fired!" debug message in the 
scripts and monitored with "journalctl -u dovecot -f | grep fired".
The APPEND event (move to junk) triggers BOTH scripts while the APPEND (move 
from junk to elsewhere) do nothing.

With your second listing it runs as expected. Now I will crawling some RFCs 
about the commands and a possible solution.

But I wonder am I the only one with this problem? I expected Outlook, even the 
older 2016, to be very popular?!

Jens


Hi Jens

after posting I was thinking over the rule for APPENDing to Inbox and I 
did have my doubts on that. I think it will fire even if people move 
email from some other folder to Inbox or even potentially on receiving 
new email, so probably not an ideal solution. I think the only real 
solution is to have negative matching rules (which at the moment there 
isn't)


I don't have access to Outlook 2016. I have tried it with Outlook 
1.2024.701.200 (which seems a very strange version number, but that's 
what it says) and I got a MOVE event just like Thunderbird. I think that 
a mail client should generate a MOVE event when moving a message between 
folders that are on the same server. I would only expect APPEND events 
if the move was happening for example between two different servers or 
between server and local folders. If you want to investigate further 
maybe the rawlog_dir feature would help to capture more info.


best of luck!

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: One-way synchronization with doveadm/TCP

[John Fawcett via dovecot]

On 04/07/2024 09:35, Дилян Палаузов via dovecot wrote:

Hello,

the replication procedure described at 
https://doc.dovecot.org/configuration_manual/replication/ suggests to open a 
server port 12345 on two systems, and then point each system to the other one.  
This creates bi-directional synchronization.

I want to synchronize only from server A to server B, but not in the opposite 
direction.  Is it sufficient on server A to set

   service doveadm {
 inet_listener {
   port = 12345
 }
   }
   doveadm_password = secret

and server B to set

   doveadm_port = 12345
   doveadm_password = secret

?  In other words, when a doveadm client (system B) connects to a server 
(system A) for replication purposes, does it only read data? Or does it 
instruct the server also to delete data?

To start the replication I had to executed «doveadm replicator replicate '*'».  
Is in necessary to run this periodically?

Thanks for your answer,
   Дилян


Hi Дилян

in case you had not seen it, from my understanding of previous posts on 
the list, the replication feature will be removed in 2.4 release.  I 
know that doesn't help with the detailed questions, but thought I'd 
mention it before you decide to spend time on setting it up.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: AW: AW: AW: AW: AW: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

demo.example): Debug: sieve: uid=3: Commit storing into mailbox 
'INBOX/Junk'

imap(info@demo.example): Debug: sieve: uid=3: left message in mailbox 
'INBOX/Junk'

imap(info@demo.example): Debug: sieve: uid=3: Finish implicit keep action

imap(info@demo.example): Debug: sieve: uid=3: Finishing actions

imap(info@demo.example): Debug: sieve: uid=3: Finish pipe action

imap(info@demo.example): Debug: sieve: uid=3: Finish pipe action

imap(info@demo.example): Debug: sieve: uid=3: Finished executing result (final, 
status=ok, keep=yes)

imap(info@demo.example): Debug: sieve: multi-script: Sequence finished 
(status=ok, keep=yes)

imap(info@demo.example): Debug: sieve: multi-script: Destroy

imap(info@demo.example): Debug: Mailbox INBOX/Junk: Mailbox opened

imap(info@demo.example)<2107>: Debug: imapsieve: mailbox 
INBOX: FLAG event (changed flags: \Deleted)

imap(info@demo.example)<2107>: Debug: Mailbox INBOX: UID 3: 
Expunge requested

imap(info@demo.example)<2107>: Debug: Mailbox INBOX: UID 3: 
Mail expunged

  

  


#dovecot -n
# 2.3.19.1 (9b53102964): /etc/dovecot/dovecot.conf

# Pigeonhole version 0.5.19 (4eae2f79)

# OS: Linux 6.1.0-21-amd64 x86_64 Debian 12.5

# Hostname: ServerIV-home.demo.example

auth_mechanisms = plain login

mail_debug = yes

mail_location = maildir:~/Maildir

mail_privileged_group = mail

managesieve_notify_capability = mailto

managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date index ihave duplicate 
mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve

namespace inbox {

   hidden = no

   ignore_on_failure = no

   inbox = yes

   list = yes

   location =

   mailbox Drafts {

 auto = subscribe

 special_use = \Drafts

   }

   mailbox Junk {

 auto = subscribe

 special_use = \Junk

   }

   mailbox Sent {

 auto = subscribe

 special_use = \Sent

   }

   mailbox "Sent Messages" {

 auto = subscribe

 special_use = \Sent

   }

   mailbox Trash {

 auto = subscribe

 special_use = \Trash

   }

   prefix = INBOX/

   separator = /

   subscriptions = yes

   type = private

}

passdb {

   driver = pam

}

passdb {

   args = scheme=CRYPT username_format=%u /etc/dovecot/users

   driver = passwd-file

}

plugin {

   imapsieve_mailbox1_before =file:/usr/lib/dovecot/sieve/report-spam.sieve

   imapsieve_mailbox1_causes = COPY APPEND

   imapsieve_mailbox1_name = INBOX/Junk

   imapsieve_mailbox2_before =file:/usr/lib/dovecot/sieve/report-ham.sieve

   imapsieve_mailbox2_causes = COPY APPEND

   imapsieve_mailbox2_from = INBOX/Junk

   imapsieve_mailbox2_name = *

   sieve =file:~/sieve;active=~/.dovecot.sieve

   sieve_after = /etc/dovecot/conf.d/custom-sieve/global_after.sieve

   sieve_before = /etc/dovecot/conf.d/custom-sieve/global_before.sieve

   sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment 
+vnd.dovecot.debug

   sieve_pipe_bin_dir = /usr/lib/dovecot/sieve

   sieve_plugins = sieve_imapsieve sieve_extprograms

}

protocols = imap lmtp sieve

service auth {

   unix_listener /var/spool/postfix/private/auth {

 mode = 0666

   }

}

service lmtp {

   inet_listener lmtp {

 address = 127.0.0.1 ::1

 port = 24

   }

}

ssl_cert =   

  

  


-Ursprüngliche Nachricht-
Von: John Fawcett via dovecot  
Gesendet: Dienstag, 2. Juli 2024 12:34

An:dovecot@dovecot.org
Betreff: Re: AW: AW: AW: AW: IMAPSieve plugin will not run rspamd script

  


Hi Jens

  


just one update and some more insight looking at the code

  


Your causes should be COPY APPEND (whether applying proposed solution

(1) or (2). I updated inline below for solution (2).

  


Logic for that: I was testing from Thunderbird which is generating a MOVE 
event. In the imapsieve plugin code MOVE and COPY are treated almost the same, 
so my rule with COPY fire on MOVE too. In your case the event you're getting is 
APPEND, but none of your rules have that as a cause so won't match even if the 
mailbox matches.

  


John

___
dovecot mailing list --dovecot@dovecot.org
To unsubscribe send an email todovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dsync will be removed in 2.4 - why?

[John Fawcett via dovecot]

On 05/07/2024 11:51, Aki Tuomi via dovecot wrote:

On 05/07/2024 12:34 EEST Oliver Krone via dovecot  wrote:

  
Hi there,


I recently learned that dsync (mail replication) functionality will be
removed in the upcoming releases of dovecot. What is the reason and will
there be a replacement? How can I synchronize my e-mails  in the future?

Thanks a lot.

Regards
      Oliver


You have learned slightly wrong. Replicator is going to be removed, not doveadm 
sync (dsync).

You can still synchronize your emails by running doveadm sync periodically.

Aki
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


That's good to know (also thanks to the person that wrote me directly to 
say so too), I will look into this. I had completely disabled 
replication in order to get used to the future.


Just a doubt, when scheduling doveadm sync at short intervals. Will 
doveadm sync work ok if more than one job is overlapping (i.e. new cron 
job starts but old one is still running) or should I define some locking 
mechanism to ensure that there is no more than one sync job running?


Thanks

John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Pread error over smb3

[John Fawcett via dovecot]

Hi Joan

not sure what OS you're using, so just a guess: but maybe this is 
selinux related or something similar. When it's the OS providing the 
error code to dovecot, it's very unlikely to be anything in dovecot itself.


On 06/07/2024 12:51, Joan Moreau via dovecot wrote:

No error on the error side

the error occurs only with dovecot. all other soft do nto complains
about the smb3 protocol

I get also
Jul 6 10:49:45 gjserver dovecot[4220]:
lmtp(ad...@grosjo.net)<4355>: Error:
rename(/net/mails/grosjo.net/admin/storage/dovecot.map.index.tmp,
/net/mails/grosjo.net/admin/storage/dovecot.map.index) failed:
Permission denied



On Tue, 2024-07-02 at 08:00 +0300, Aki Tuomi via dovecot wrote:

Ok. But the error is coming from kernel, so not much Dovecot can do
about it. Maybe try turning on some debugging in your server to see
what is going on?

Aki


On 02/07/2024 07:54 EEST Joan Moreau via dovecot
 wrote:

  
Permissions on the server are very fine


The problem occurs ONLY with dovecot


On Tue, 2024-07-02 at 07:49 +0300, Aki Tuomi wrote:

On 02/07/2024 02:05 EEST Joan Moreau via dovecot
 wrote:

  
Hi


I am trying to move my storage of email on a smb3 mounted
volume.

I am getting the following error :
Error: pread(/net/.../storage/dovecot.map.index.log) failed:
Permission
denied (euid=1004(mailusers) egid=12(mail) UNIX perms appear ok
(ACL/MAC wrong?))

How to resolve that ?

Thank you


This seems to be some kind of smb3 ACL problem, check permissions
on
the server?

Aki

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot-lda: timeout for external scripts called by sieve

[John Fawcett via dovecot]

On 09/07/2024 09:02, Meinhard Schneider via dovecot wrote:

Hello,

I have a small problem with dovecot-lda. I am using Debian 12 Bookworm:

root@lnxmail01:~# dovecot --version
2.3.13 (89f716dc2)


Historically grown (probably still from sendmail times, long time ago) 
I always used procmail for local e-mail delivery. This also included 
the call for spamc for SpamAssassin.


I have now changed this and set up a route via sieve. There is a small 
script for this:


root@lnxmail01:~# cat /usr/lib/dovecot/sieve-filter/spamc-filter.sh
#!/bin/bash

/usr/bin/spamc


My sieve configuration says (in extracts):

root@lnxmail01:~# head -12 ~meini/.dovecot.sieve
## Generated by Roundcube Webmail SieveRules Plugin ##

require 
["body","comparator-i;ascii-numeric","copy","envelope","fileinto","imap4flags","include","regex","relational","spamtestplus","vacation","variables","spamtest"];


include :global "spamc";

# rule:[Spam]
if spamtest :value "ge" :comparator "i;ascii-numeric" "5"
{
    fileinto "Spam";
    stop;
}


In general, this has been working for many months without any 
problems. But sometimes the whole thing fails due to a timeout:


Jul  7 22:00:13 lnxmail01 dovecot: 
lda(meini)<766014>: Error: program 
exec:/usr/lib/dovecot/sieve-filter/spamc-filter.sh (766016): Execution 
timed out (> 1 msecs)
Jul  7 22:00:13 lnxmail01 dovecot: 
lda(meini)<766014>: Error: program 
exec:/usr/lib/dovecot/sieve-filter/spamc-filter.sh (766016): Forcibly 
terminated with signal 15
Jul  7 22:00:13 lnxmail01 dovecot: 
lda(meini)<766014>: Panic: output stream (temp 
iostream in /tmp/dovecot.lda. for (program client seekable output)) is 
missing error handling
Jul  7 22:00:13 lnxmail01 dovecot: 
lda(meini)<766014>: Error: Raw backtrace: 
/usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42) 
[0x7ff00aee04e2] -> 
/usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7ff00aee05fe] 
-> /usr/lib/dovecot/libdovecot.so.0(+0xfc49b) [0x7ff00aeec49b] -> 
/usr/lib/dovecot/libdovecot.so.0(+0xfc501) [0x7ff00aeec501] -> 
/usr/lib/dovecot/libdovecot.so.0(+0x53aee) [0x7ff00ae43aee] -> 
/usr/lib/dovecot/libdovecot.so.0(o_stream_unref+0x5e) [0x7ff00af1003e] 
-> /usr/lib/dovecot/libdovecot.so.0(program_client_destroy+0x8e) 
[0x7ff00ae6ee6e] -> 
/usr/lib/dovecot/modules/sieve/lib90_sieve_extprograms_plugin.so(sieve_extprogram_destroy+0x18) 
[0x7ff00aacd8f8] -> 
/usr/lib/dovecot/modules/sieve/lib90_sieve_extprograms_plugin.so(+0x556d) 
[0x7ff00aacb56d] -> 
/usr/lib/dovecot/libdovecot-sieve.so.0(sieve_interpreter_continue+0xea) 
[0x7ff00ab4c9ba] -> 
/usr/lib/dovecot/libdovecot-sieve.so.0(ext_include_execute_include+0x2c7) 
[0x7ff00ab72e17] -> 
/usr/lib/dovecot/libdovecot-sieve.so.0(sieve_interpreter_continue+0xea) 
[0x7ff00ab4c9ba] -> 
/usr/lib/dovecot/libdovecot-sieve.so.0(sieve_multiscript_run+0x82) 
[0x7ff00ab62b32] -> 
/usr/lib/dovecot/modules/lib90_sieve_plugin.so(+0x37a3) 
[0x7ff00abd47a3] -> 
/usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver+0x176) 
[0x7ff00b11f906] -> /usr/lib/dovecot/dovecot-lda(main+0x879) 
[0x55ef085e4fb9] -> 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) 
[0x7ff00ac3ad0a] -> /usr/lib/dovecot/dovecot-lda(_start+0x2a) 
[0x55ef085e56ea]




The worst thing is that the entire mail is lost (/dev/null)!

Then I tried to set the timeout higher (than the current 10 seconds):

root@lnxmail01:~# doveconf | grep -A 12 ^plugin
plugin {
  sieve = file:~/sieve;active=~/.dovecot.sieve
  sieve_execute_exec_timeout = 60s
  sieve_extension_exec_timeout = 60s
  sieve_extensions = +vnd.dovecot.filter +spamtest +spamtestplus
  sieve_filter_bin_dir = /usr/lib/dovecot/sieve-filter
  sieve_filter_socket_dir = sieve-filter
  sieve_global = /usr/lib/dovecot/sieve-global
  sieve_plugins = sieve_extprograms
  sieve_spamtest_max_value = 5
  sieve_spamtest_status_header = X-Spam-Level
  sieve_spamtest_status_type = strlen
}


I have set both the values for "sieve_execute_exec_timeout" and 
"sieve_extension_exec_timeout" to 60 sec, but according to the log, 
this does not seem to affect the 1 msec mentioned therein.


The SpamAssassin daemon has enough child processes to handle the 
requests (most of them were idle at the time of the error).


How can I configure that the external script gets more time for 
execution? And what can I do so that - should the script terminate 
contrary to expectations or be killed in the timeout - the mail is not 
lost but simply processed further?


Many thanks & best regards
Meinhard



Hi Mainhard

I can see the error in the log lines you posted above, but I don't see 
anything in your configuration that invokes 
/usr/lib/dovecot/sieve-filter/spamc-filter.sh script.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot-lda: timeout for external scripts called by sieve

[John Fawcett via dovecot]

On 10/07/2024 09:40, John Fawcett via dovecot wrote:


On 09/07/2024 09:02, Meinhard Schneider via dovecot wrote:

Hello,


I have set both the values for "sieve_execute_exec_timeout" and 
"sieve_extension_exec_timeout" to 60 sec, but according to the log, 
this does not seem to affect the 1 msec mentioned therein.


The SpamAssassin daemon has enough child processes to handle the 
requests (most of them were idle at the time of the error).


How can I configure that the external script gets more time for 
execution? And what can I do so that - should the script terminate 
contrary to expectations or be killed in the timeout - the mail is 
not lost but simply processed further?


Many thanks & best regards
Meinhard



Hi Mainhard

I can see the error in the log lines you posted above, but I don't see 
anything in your configuration that invokes 
/usr/lib/dovecot/sieve-filter/spamc-filter.sh script.


John

___

Just one thing I noticed which could well be relevant to the issue. 
Seems there is no setting called "|sieve_extension_exec_timeout|".


|For sieve_extension_socket_dir|and |sieve_extension_bin_dir| you 
correctly substituted extension with filter, but you did not do the same 
with sieve_extension_exec_timeout which should be sieve_pipe_exec_timeout


https://doc.dovecot.org/configuration_manual/sieve/plugins/extprograms/

   The three extensions introduced by this plugin
   -|vnd.dovecot.pipe|,|vnd.dovecot.filter|and|vnd.dovecot.execute|-
   each have separate but similar configuration. The following
   configuration settings are used, for which “” in the
   setting name is replaced by
   either|pipe|,|filter|or|execute|depending on which extension is
   being configured:

   |sieve_extension_socket_dir|=

   Points to a directory relative to the Dovecot base_dir where the
   plugin looks for script service sockets.

   |sieve_extension_bin_dir|=

   Points to a directory where the plugin looks for programs (shell
   scripts) to execute directly and pipe messages to.

   |sieve_extension_exec_timeout|= 10s

   Configures the maximum execution time after which the program is
   forcibly terminated.

John
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot-lda: timeout for external scripts called by sieve

[John Fawcett via dovecot]

Corrected a typo in the previous message, should have read 
sieve_filter_exec_timeout and not sieve_pipe_exec_timeout


On 10/07/2024 10:41, John Fawcett wrote:



On 10/07/2024 09:40, John Fawcett via dovecot wrote:


On 09/07/2024 09:02, Meinhard Schneider via dovecot wrote:

Hello,


I have set both the values for "sieve_execute_exec_timeout" and 
"sieve_extension_exec_timeout" to 60 sec, but according to the log, 
this does not seem to affect the 1 msec mentioned therein.


The SpamAssassin daemon has enough child processes to handle the 
requests (most of them were idle at the time of the error).


How can I configure that the external script gets more time for 
execution? And what can I do so that - should the script terminate 
contrary to expectations or be killed in the timeout - the mail is 
not lost but simply processed further?


Many thanks & best regards
Meinhard



Hi Mainhard

I can see the error in the log lines you posted above, but I don't 
see anything in your configuration that invokes 
/usr/lib/dovecot/sieve-filter/spamc-filter.sh script.


John

___

Just one thing I noticed which could well be relevant to the issue. 
Seems there is no setting called "|sieve_extension_exec_timeout|".


|For sieve_extension_socket_dir|and |sieve_extension_bin_dir| you 
correctly substituted extension with filter, but you did not do the 
same with sieve_extension_exec_timeout which should be 
sieve_filter_exec_timeout


https://doc.dovecot.org/configuration_manual/sieve/plugins/extprograms/

The three extensions introduced by this plugin
-|vnd.dovecot.pipe|,|vnd.dovecot.filter|and|vnd.dovecot.execute|-
each have separate but similar configuration. The following
configuration settings are used, for which “” in the
setting name is replaced by
either|pipe|,|filter|or|execute|depending on which extension is
being configured:

|sieve_extension_socket_dir|=

Points to a directory relative to the Dovecot base_dir where
the plugin looks for script service sockets.

|sieve_extension_bin_dir|=

Points to a directory where the plugin looks for programs
(shell scripts) to execute directly and pipe messages to.

|sieve_extension_exec_timeout|= 10s

Configures the maximum execution time after which the program
is forcibly terminated.

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: New Member Introduction

[John Fawcett via dovecot]

On 10/07/2024 07:37, Selena Thomas via dovecot wrote:

Hi everyone,

I am new to this forum and excited to be here.
I'm interested in learning more about Dovecot and its features, and I'm eager 
to participate in the discussions here.
Could someone please guide me on how to ask questions here ?
Where should I post if I have a query ?

Looking forward to your advice and connecting with you all.

Thanks,
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Hi Selena

here is some info  about posting to the list:

https://www.dovecot.org/mailing-lists/

The mailing list archives and the dovecot documentation are good 
starting points for finding info and resolving issues.


https://dovecot.org/mailman3/hyperkitty/list/dovecot@dovecot.org/

https://doc.dovecot.org/

Otherwise if you've got some specific issue or question just post it to 
the list. For configuration issues it helps if you post your 
configuration "dovecot -n" output and relevant logging. Hope that helps


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot replication

[John Fawcett via dovecot]

On 12/07/2024 13:05, Jeff Pang via dovecot wrote:

Hello,

Does the community version of dovecot have the replication feature?
When one dovecot was down, another one could take over the tasks.

Thanks.

_


Jeff

Replication is in the current dovecot version but will go away in 2.4.

The doveadm sync feature is staying. So with some work you can set it up 
what you are requesting.


I used to use replication and now I'm thinking about using sync but have 
not implemented it. The following are thoughs on it.


There are some points to be addressed that are outside dovecot. I think 
you'd have to make sure that your sync happened frequently enough that 
you could live with losing the emails that arrives bewteen syncs for 
example. That would tend to lead to a requirement to sync more 
frequently and reduce risk of email loss. But then you'd need to avoid 
more than one sync being active simultaneously (that is my assumption 
that this would not work, but I don't know if it is a real problem).


The failover would require you to stop people accessing the old server, 
stop syncing and start the backup instance. When the main instance is 
available to come back on line, you'd need to stop the backup instance, 
sync in the opposite direction and then start dovecot and re-enable the 
sync mechansim towards the backup.


You need to ensure people don't connect simultaneously to both 
instances. So some thought would be needed about those cases where the 
main node goes and then comes backup to ensure that your sync is not 
still active at that point and replicates the old state onto the backup 
server and to ensure then people don't start connecting to it again 
without being able to control it.


It's a disaster recover rather than high availability solution, but I 
think it can work. Others may already have got working implementations 
and thought through some of the implications.


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot replication

[John Fawcett via dovecot]

On 12/07/2024 17:38, James Cook via dovecot wrote:

Replication is in the current dovecot version but will go away in 2.4.

The doveadm sync feature is staying. So with some work you can set it 
up what you are requesting.


I used to use replication and now I'm thinking about using sync but 
have not implemented it. The following are thoughs on it.


There are some points to be addressed that are outside dovecot. I 
think you'd have to make sure that your sync happened frequently 
enough that you could live with losing the emails that arrives 
bewteen syncs for example.


I have been thinking of writing a hacky delivery script that passes 
the email on to dovecot-lda, then runs doveadm sync, and only returns 
success after the sync is done (or after a timeout). No idea what 
problems I will run into, but the idea is to never accept an email 
until it's guaranteed replicated.


That would tend to lead to a requirement to sync more frequently and 
reduce risk of email loss. But then you'd need to avoid more than one 
sync being active simultaneously (that is my assumption that this 
would not work, but I don't know if it is a real problem).


Doesn't the -l option protect against simultaneous syncs? Just based 
on reading the doveadm-sync man page. (I guess it could cause a 
problem if you start sync processes more quickly than they can finish.)


NB I'm just running a one-person email server so don't take my ideas 
too seriously :)



Hi James

I want to avoid the -1 parameter because it doesn't do deletes in the 
target. It might not be an issue, but I'd like to keep the target the 
same as the source. I don't know if it would protect against 
simultaneous jobs, but I dont' know even if that is an issue. I was 
making the assumption it could be so I plan to avoid it somehow.


As for the lda to doveadm sync script, I have been wondering too about 
how to close the gap for emails arriving between syncs, even though the 
risk might not be so significant.


With delivering to two dovecot servers before accepting the email, 
either one going down will stop email delivery. That's an inconvenience 
but without necessarily introducing risks for losing the emails. 
Ultimately it depends on where those undelivered emails are being kept 
in the meantime  (presumably MTA queue) and whether they are safer there 
than being delivered to a single instance. Though that is related only 
to the downtime. When both are up the risk would be mitigated by the 
solution.


Other idea I was thinking of is a replicated file system. That could 
make emails available in real time to both dovecot instances assuming a 
functionality to sync to disk on both/multiple nodes before replying 
back to dovecot to accept the email. I believe there would still need to 
be only one dovecot instance active at a time. However, I have heard of 
but don't know personally of horror stories about email outages on 
clustered file systems, which leads me to believe that there would still 
be sufficient residual risk to require a backup copy of the mailboxes 
with doveadm sync or other tools.


John

John





___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot replication

[John Fawcett via dovecot]

On 12/07/2024 21:14, James Cook wrote:

On Fri, Jul 12, 2024 at 06:28:13PM GMT, John Fawcett via dovecot wrote:

Hi James

I want to avoid the -1 parameter because it doesn't do deletes in the 
target.


-l, not -1.


Thanks I missed that - so locking can be done within Dovecot

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: doveadm auth lookup fails for system user

[John Fawcett via dovecot]

On 12/07/2024 21:47, Christian H. Kuhn via dovecot wrote:

Hi all,

next step with my auth problem with dovecot.

I want to authenticate a system user. The user exists, can log in, can 
sudo -i etc.pp. SASL with sql passdb and userdb works fine.


root@bywater /etc/dovecot/conf.d # doveadm user qno
field   value
uid 1001
gid 1001
home    /home/qno
mail    maildir:~/Maildir
system_groups_user  qno

But:
root@bywater /etc/dovecot/conf.d # doveadm auth lookup qno
passdb lookup: user qno doesn't exist

And no surprise:
root@bywater /etc/dovecot/conf.d # doveadm auth test qno
Password:
passdb: qno auth failed
extra fields:
  user=qno


Hi QNo

I can't see why this happening. Only suggestion I have is to run those 
commands with -D to get more detailed info.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: doveadm auth lookup fails for system user

[John Fawcett via dovecot]

10_doveadm_sieve_plugin.so
Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() 
failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: 
undefined symbol: lucene_index_iter_deinit (this is usually 
intentional, so just ignore this message)
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: 
undefined symbol: fts_user_get_language_list (this is usually 
intentional, so just ignore this message)
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() 
failed: 
/usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: 
undefined symbol: mail_crypt_box_get_pvt_digests (this is usually 
intentional, so just ignore this message)
Jul 15 19:45:44 Debug: Loading modules from directory: 
/usr/lib/dovecot/modules/doveadm
Jul 15 19:45:44 Debug: Skipping module doveadm_acl_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: 
undefined symbol: acl_user_module (this is usually intentional, so 
just ignore this message)
Jul 15 19:45:44 Debug: Skipping module doveadm_quota_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: 
undefined symbol: quota_user_module (this is usually intentional, so 
just ignore this message)
Jul 15 19:45:44 Debug: Skipping module doveadm_fts_lucene_plugin, 
because dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: 
undefined symbol: lucene_index_iter_deinit (this is usually 
intentional, so just ignore this message)
Jul 15 19:45:44 Debug: Skipping module doveadm_fts_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: 
undefined symbol: fts_user_get_language_list (this is usually 
intentional, so just ignore this message)
Jul 15 19:45:44 Debug: Skipping module doveadm_mail_crypt_plugin, 
because dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: 
undefined symbol: mail_crypt_box_get_pvt_digests (this is usually 
intentional, so just ignore this message)
Jul 15 19:45:44 Debug: auth-master: passdb lookup(qno): Started passdb 
lookup
Jul 15 19:45:44 Debug: auth-master: conn 
unix:/run/dovecot/auth-userdb: Connecting
Jul 15 19:45:44 Debug: auth-master: conn unix:/run/dovecot/auth-userdb 
(pid=2542,uid=0): Client connected (fd=9)
Jul 15 19:45:44 Debug: auth-master: passdb lookup(qno): auth PASS 
input: user=qno
Jul 15 19:45:44 Debug: auth-master: passdb lookup(qno): Finished 
passdb lookup (user=qno )

passdb: qno
  user  : qno

Jul 15 19:45:44 Debug: auth-master: conn unix:/run/dovecot/auth-userdb 
(pid=2542,uid=0): Disconnected: Connection closed (fd=9)



Am 13.07.2024 um 13:01 schrieb John Fawcett via dovecot:


On 12/07/2024 21:47, Christian H. Kuhn via dovecot wrote:

Hi all,

next step with my auth problem with dovecot.

I want to authenticate a system user. The user exists, can log in, 
can sudo -i etc.pp. SASL with sql passdb and userdb works fine.


root@bywater /etc/dovecot/conf.d # doveadm user qno
field   value
uid 1001
gid 1001
home    /home/qno
mail    maildir:~/Maildir
system_groups_user  qno

But:
root@bywater /etc/dovecot/conf.d # doveadm auth lookup qno
passdb lookup: user qno doesn't exist

And no surprise:
root@bywater /etc/dovecot/conf.d # doveadm auth test qno
Password:
passdb: qno auth failed
extra fields:
  user=qno


Hi QNo

I can't see why this happening. Only suggestion I have is to run 
those commands with -D to get more detailed info.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dictionary with %d variable in sql stops insert and select domain part - dovecot 2.3.17+

[John Fawcett via dovecot]

On 16/07/2024 08:53, Mariusz Piasecki via dovecot wrote:

Hi,

unfortunately no :(

In sql query instead of variable %d I get user

UPDATE used_domain_quota SET bytes=bytes+4359,messages=messages+1 
WHERE domain

= 'h.u...@example.com';

and domain can receive emails when over quota.

On 15.07.2024 20:15, yoh2010--- via dovecot wrote:

Mariusz Piasecki, hello.

I have a similar problem after migrating from Debian 10 (Dovecot 
2.3.4) to Debian 12 (Dovecot 2.3.19). Have you found a way to solve 
the problem?


Best regards, Evgeniy Rusachenko
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Hi looks like a bug introduced as an unintended regression between 
2.3.16 and 2.3.17. In 2.3.17 the username field is always taken as the 
default i.e. username. The %d override, though it is read when parsing 
the config is not used for the lookup.


The quota dict is going away in 2.4 which is a strong indication that no 
patch would be accepted, since the code is no longer present in the main 
branch. You'll probably want to start looking into quota count plugin 
and quota clone plugin if you still want to publish the quota to mysql 
tables. I don't know if it's possible to use a domain quota with quota 
count plugin. I didn't see an explicitly documented way.


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Invalid/Unmapped Key with Last Login Plugin

[John Fawcett via dovecot]

On 17/07/2024 12:28, Adam Miller via dovecot wrote:

OS: Ubuntu 22.04.1
Dovecot: 2.3.16 (7e2e900c1a)

Hello all! I have the last login plugin working with the IMAP service 
however with LMTP, it errors. I do not necessarily need / want the 
LMTP service and did not explicitly enable it but it is throwing an 
error in my logs.


Jul 17 10:22:14 lmtp(184051):Error: last_login_dict: Failed to write 
value: dict-server returned failure: sql dict set: Invalid/unmapped 
key:shared/last-login/lmtp/usern...@domain.com/ (reply took 0.234 secs 
(0.001 in dict wait, 0.033 in other ioloops, 0.030 in locks, as


ync-id reply 0.000 secs ago, started on dict-server 0.220 secs ago, 
took 0.000 secs))




It would be nice to get rid of this error one or another: either 
disable last login for LMTP or get it working.


Current Configuration from doveconf -n

# 2.3.16 (7e2e900c1a): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.16 (09c29328)
# OS: Linux 6.5.0-1020-aws x86_64 Ubuntu 22.04.4 LTS nfs4
# Hostname: ec2-us-east-1a-arda-mail-001.ardaemail.com

mail_plugins = " quota mailbox_alias last_login"

protocol imap {
  mail_plugins = " quota mailbox_alias last_login imap_sieve 
imap_quota last_login"

}
protocol lmtp {
  mail_plugins = " quota mailbox_alias last_login sieve"
}

In the configuration above, I do not have "last_login" in my actual 
config for "protocol lmtp", something is adding it.




Hi Adam

maybe you do have last_login enabled for lmtp without realizing it. What 
is your actual setting for mail_plugins in the protocol lmtp section?


If it something like:

    mail_plugins = $mail_plugins sieve

then you are adding the following definition too from the general definition

mail_plugins = " quota mailbox_alias last_login"

So it would make sense to remove last_login from the general 
mail_plugins setting and add it only to the protocol specific 
mail_plugins section for the protocols where you want it be used.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Invalid/Unmapped Key with Last Login Plugin

[John Fawcett via dovecot]

On 17/07/2024 16:03, dovecot--- via dovecot wrote:
Hello all! I have the last login plugin working with the IMAP service 
however with LMTP, it errors.
I do not necessarily need / want the LMTP service and did not 
explicitly enable it but it is throwing an error in my logs.


Jul 17 10:22:14 lmtp(184051):Error: last_login_dict: Failed to write 
value: dict-server returned failure: sql dict set: Invalid/unmapped 
key:shared/last-login/lmtp/usern...@domain.com/ (reply took 0.234 
secs (0.001 in dict wait, 0.033 in other ioloops, 0.030 in locks, as 
ync-id reply 0.000 secs ago, started on dict-server 0.220 secs ago, 
took 0.000 secs))


It would be nice to get rid of this error one or another: either 
disable last login for LMTP or get it working.


Current Configuration from doveconf -n

...

protocols = imap lmtp

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0666
    user = postfix
  }
}



But LMTP is being explicitly enabled in your config. Remove it from 
the protocols= line. And optionally remove the service {...} block for 
it. However just taking it out of protocols should be good enough.



Hi Adam

My recommendation, before disabling lmtp, would be to investigate and 
maybe revise this point "I do not necessarily need / want the LMTP 
service and did not explicitly enable it". That error is being generated 
by a login to lmtp. So what is logging in despite you not needing lmtp?


If for example you are using lmtp to deliver email from postfix to 
dovecot, disabling lmtp will stop the flow of new email.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Invalid/Unmapped Key with Last Login Plugin

[John Fawcett via dovecot]

On 17/07/2024 22:59, Adam Miller via dovecot wrote:
Yikes, this is my bad! I AM using the LMTP service for delivery. What 
I meant was that I do not necessarily need to track the logins to the 
LMTP service. I suppose it wouldn’t hurt to track them though. Any 
idea why it would be throwing this error? Which key is it expecting to 
be mapped? Am I missing a configuration specifically for this? Should 
the last login for LMTP use the same dict as the IMAP configuration?



Hi Adam

not sure if you saw it but in my earlier response I gave the info about 
excluding last_login plugin for services where it is not needed. You 
have it configured globally at the moment.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Invalid/Unmapped Key with Last Login Plugin

[John Fawcett via dovecot]

On 17/07/2024 23:35, Adam Miller via dovecot wrote:
Yes, I did see that, thank you. If I did want to get this working 
globally, do you know what the issue is here and how to resolve it? I 
think it has something to do with my database table but I am not 
positive what the issue is. Here’s my database table (MySQL):


CREATE TABLE `mailboxLastLogin` (
  `email` varchar(192) NOT NULL DEFAULT ‘’,
  `service` varchar(16) NOT NULL DEFAULT ‘’,
  `timestamp` bigint(20) NOT NULL,
  `ipaddress` varchar(40) NOT NULL,
  `createdAt` datetime NOT NULL DEFAULT current_timestamp(),
  PRIMARY KEY (`email`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

I also have two triggers on this table so when a new row is inserted 
or an existing row is updated, a new row is inserted into another 
tables named `mailboxHistory`. This is working fine and should not be 
causing any issues with LMTP trying to query itself.


BEGIN
    DECLARE mailboxUuid VARCHAR(36);

    SELECT uuid INTO mailboxUuid FROM mailbox WHERE email = NEW.email;

    INSERT INTO mailboxHistory (`mailboxUuid`, `service`, `command`, 
`ipaddress`, `uuid`)

    VALUES (mailboxUuid, NEW.service, 'login', NEW.ipaddress, uuidv4());
END

BEGIN
    DECLARE mailboxUuid VARCHAR(36);

    SELECT uuid INTO mailboxUuid FROM mailbox WHERE email = NEW.email;

    INSERT INTO mailboxHistory (`mailboxUuid`, `service`, `command`, 
`ipaddress`, `uuid`)

    VALUES (mailboxUuid, NEW.service, 'login', NEW.ipaddress, uuidv4());
END

CREATE TABLE `mailboxHistory` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `uuid` varchar(36) DEFAULT ‘’,
  `mailboxUuid` varchar(36) NOT NULL DEFAULT ‘’,
  `service` varchar(16) NOT NULL,
  `command` varchar(36) DEFAULT NULL,
  `extra` longtext DEFAULT NULL,
  `ipaddress` varchar(40) NOT NULL,
  `createdAt` datetime NOT NULL DEFAULT current_timestamp(),
  PRIMARY KEY (`id`),
  UNIQUE KEY `UUID` (`uuid`)
) ENGINE=InnoDB AUTO_INCREMENT=2047 DEFAULT CHARSET=utf8mb4 
COLLATE=utf8mb4_general_ci;


Thank you,

Adam



Hi Adam

the solution is to change

mail_plugins = " quota mailbox_alias last_login"

to

mail_plugins = quota mailbox_alias

(without the last_login)

The last_login should be configured inside the protocols where you need 
it, like imap. I don't think it's useful to have the plugin setup for 
services different to imap, pop3 and smtp where you have users logging 
in. If you want to have last_login for lmtp without that error, I'd need 
a bit more info


What are the contents of /etc/dovecot/inc.d/inc.lastlogin.sql.conf 
(masking any passwords) and what is the output of doveconf 
plugin/last_login_key


John



___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: doveadm: SSL handshake failed - wrong version number

[John Fawcett via dovecot]

On 18/07/2024 14:41, Rachel Roch via dovecot wrote:

When attempting a doveadm backup, e.g. :

doveadm -D backup -Ru user tcp:foo.example.com:12345


Hi Rachel

maybe that should be with tcps

doveadm -D backup -Ru user tcps:foo.example.com:12345

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Sieve duplicate detection not working as expected

[John Fawcett via dovecot]

On 18/07/2024 15:37, Rosario Esposito via dovecot wrote:


Hi,

On a dovecot server (2.3.21) I have a simple sieve script:

require ["duplicate"];
# rule:[duplicate]
if duplicate {
    discard;
    stop;
}


I use virtual users with mail aliases on a LDAP backend.

I have user "resposit" having 2 mail aliases:

- respo...@na.infn.it
- rosario.espos...@na.infn.it

If I send a single message to both aliases I still receive 2 copies as 
you can see from the attached log.


Is the "duplicate" sieve extension supposed to work this way ?

Any help would be much appreciated !



Hi Rosario

I guess the duplicate db lookup key includes both message id and 
username so the same message id for different users will not be flagged 
as a duplicate.


If you define the aliases in the mta rather than via the dovecot ldap 
lookup, so that the username that comes into dovecot is the same one for 
both aliases I suspect it will work.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: sieve mail date change

[John Fawcett via dovecot]

On 22/07/2024 21:03, Marc via dovecot wrote:

I am a bit rookie with sieve. Some time ago I created a sieve pipe that would 
sometimes alter alter the subject and move a message to a different mailbox. 
Currently this results in that a date changes of this message while I don't see 
anything in the source altering a date.

What could be causing this and how to work around this?
___


Hi Marc

can you show info that might be useful to understand the issue you are 
facing? For example, what are the contents of the sieve script? When is 
the script being executed (for example on mail delivery by lmtp/lda)?  
When you say the date changes of the message can you give an example of 
what you are seeing?


John
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Sieve duplicate detection not working as expected

[John Fawcett via dovecot]

On 24/07/2024 10:48, Rosario Esposito via dovecot wrote:



Hi John,

I'm not sure the DB lookup key includes the username ("resposit" in my 
case)


I tried to see what gets written in .dovecot.lda-dupes but it looks 
like a binary file (doveadm dump does not help). This is what I see 
after sending the same message to both aliases:


[root@imap1 ~]# hexdump -C /mail/resposit/.dovecot.lda-dupes
  02 00 00 00 5f c9 a0 66  10 00 00 00 1b 00 00 00 
|_..f|
0010  bc 34 2f 26 82 bd 69 6d  e7 a0 79 da 28 d7 4a 82 
|.4/&..im..y.(.J.|
0020  72 6f 73 61 72 69 6f 2e  65 73 70 6f 73 69 74 6f 
|rosario.esposito|
0030  40 6e 61 2e 69 6e 66 6e  2e 69 74 5f c9 a0 66 10 
|@na.infn.it_..f.|
0040  00 00 00 13 00 00 00 bc  34 2f 26 82 bd 69 6d e7 
|4/&..im.|
0050  a0 79 da 28 d7 4a 82 72  65 73 70 6f 73 69 74 40 
|.y.(.J.resposit@|

0060  6e 61 2e 69 6e 66 6e 2e  69 74 |na.infn.it|
006a


It doesn't seem "resposit" appears in it.


Yes, you're right, it was the point I was trying to make. The dovecot 
user (which in this case is equivalent to the email address arriving in 
the MAIL FROM command of the lmtp session) is what is stored in the 
duplicates db. Dovecot is not programmed to realize that these users can 
both read each others messages and will not recognize them as duplicates 
no more than if they were two completely separate users with no mail 
location in common.




Doveadm fetch shows the messages has the same message-id but I still 
get two copies with different guid/uid:


[root@imap1 ~]# doveadm -f table fetch -u resposit 'guid uid 
hdr.message-id date.received' mailbox INBOX | grep 2024-07-24
60525f254fbba066622191d3cd3c 18010 
<8803feb5-7c3e-4f04-b8ea-62761787e...@gmail.com> 2024-07-24 10:29:03
28e634264fbba066622191d3cd3c 18011 
<8803feb5-7c3e-4f04-b8ea-62761787e...@gmail.com> 2024-07-24 10:29:03



Doing what you suggest (manage aliases in the MTA rather than 
dovecot/ldap) would not be so easy in my case as it requires a lot of 
changes on a production system.


Currently I have a Proxmox Mail Gatway server (based on Postfix) which 
gets mail from outside, applies filters (greylisting, antispam, 
antivirus etc...) and delivers anything directed to my domain to a 
Dovecot server via LMTP transport. Dovecot knows LDAP aliases/usernames.


Do you think there might be any other workarounds to make the 
duplicate sieve extension work properly ?


Thanks,

Rosario


Il 19/07/2024 22:37, John Fawcett via dovecot ha scritto:


On 18/07/2024 15:37, Rosario Esposito via dovecot wrote:


Hi,

On a dovecot server (2.3.21) I have a simple sieve script:

require ["duplicate"];
# rule:[duplicate]
if duplicate {
    discard;
    stop;
}


I use virtual users with mail aliases on a LDAP backend.

I have user "resposit" having 2 mail aliases:

- respo...@na.infn.it
- rosario.espos...@na.infn.it

If I send a single message to both aliases I still receive 2 copies 
as you can see from the attached log.


Is the "duplicate" sieve extension supposed to work this way ?

Any help would be much appreciated !




I don't know how well Dovecot supports aliases where (I am presuming) 
you use the same mail location for multiple email addresses. I 
personally have all aliases defined at MTA level so all email addresses 
arriving in lmtp are one to one mappings with dovecot users.


I dont' know your configuration so I am a bit reluctant to suggest any 
changes. My only suggestion would be to look into returning a "user" 
field from your ldap userdb lookup to override the one received from 
lmtp. If you define the same override for both aliases (presumably 
resposit), that will be used as the user in the duplciates db and 
hopefully produce a match regardless of which alias is used.


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot-lda: timeout for external scripts called by sieve

[John Fawcett via dovecot]

On 27/07/2024 21:26, Meinhard Schneider via dovecot wrote:


Hello,

I have now adjusted the values accordingly:

root@lnxmail01:~# doveconf | grep -A 13 ^plugin
plugin {
  sieve = file:~/sieve;active=~/.dovecot.sieve
  sieve_execute_exec_timeout = 60s
  sieve_extensions = +vnd.dovecot.filter +spamtest +spamtestplus
  sieve_filter_bin_dir = /usr/lib/dovecot/sieve-filter
  sieve_filter_exec_timeout = 60s
  sieve_filter_socket_dir = sieve-filter
  sieve_global = /usr/lib/dovecot/sieve-global
  sieve_pipe_exec_timeout = 60s
  sieve_plugins = sieve_extprograms
  sieve_spamtest_max_value = 5
  sieve_spamtest_status_header = X-Spam-Level
  sieve_spamtest_status_type = strlen
}

I will observe whether the problems continue to occur.

I guess that the possibility of problems happening is still there since 
whatever timeout you put, it could be exceeded. It doesn't seem like a 
robust way of filtering.




Nevertheless, I would like to know if there is a way to prevent the 
e-mails from being completely discarded if an error occurs.


I would expect Dovecot and Pigeonhole not to discard emails in the case 
of errors. My expectation would be that if the script returns an error 
code then the email will be kept by default. Do you have any evidence of 
discarding email in the case of errors, where the scripts is returning 
an error to dovecot but it is ignored?


As for discarding emails when the script times out, I believe you are 
right about that. On the face of it it seems the wrong thing to do but I 
guess there may be an explanation that I don't know as to why it works 
that way.




The program /usr/lib/dovecot/sieve-filter/spamc-filter.sh is executed 
because there is an include in the file .dovecot.sieve:

include :global “spamc”;
This is sufficient to start the filter “spamc-filter.sh” (I got this 
from a guide on the Internet, but I can't find the link to it now).


My understanding is that this will include another sieve script called 
"spamc", not directly call an external comamnd.




The problem is that only spamc is started in the script and STDIN and 
STDOUT are connected to dovecot-lda. It would be easy to abort the 
call of spamc after X seconds (man 'timeout(1)'). But then the input 
provided by dovecot-lda and piped into spamc would be lost. You would 
therefore have to cache all input in the script that comes via STDIN 
in a file. If spamc is aborted, the script would have to recognize 
this and output the contents of this file again via STDOUT. spamc must 
be called using 'timeout' and the timeout used must be less than what 
was set for sieve_filter_exec_timeout. This would then save the mail 
from /dev/null.


Unfortunately, I currently see no other way to solve this problem.

Or does anyone have another idea how to tackle this issue?


Maybe you could work out how to return an error code from the script 
after a timeout (before the dovecot timeout kicks in and kills it). I'm 
not convinced you'd need to worry about STDIN and STDOUT. You could 
check this easily by having a test script which returns a non zero value 
(e.g. exit 1).


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: lmtp Fatal: setresgid Operation not permitted

[John Fawcett via dovecot]

On 28/07/2024 18:41, Rachel Roch via dovecot wrote:



I am getting the following error when Postfix attempts LMTP delivery :

"lmtp(REDACTED)<32674><7Jm0BSVopmaifwAAbW4UVQ>: Fatal:
setresgid(121(mailbox_user),121(mailbox_user),8(mail)) failed with
euid=111(mailbox_user): Operation not permitted"

mailbox_user is uid 111, gid 121 on my system.


Not related to low uid/gid numbers? I think newer OS have their limits 
increased to above 1000 or so.


Well, my OS is Debian Bookworm which should be "newer" enough ? ;-)

User/group were originally created via :

adduser --system --disabled-login --shell /usr/sbin/nologin mailbox_user
addgroup --system mailbox_user

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Hi Rachel

Could it be that user mailbox_user is not a member of group mailbox_user?

What does grep mailbox_user /etc/passwd show?

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Event Export (auth_request_finished)

[John Fawcett via dovecot]

On 04/08/2024 13:17, Serhii via dovecot wrote:
I am trying to implement logging of all failed authentication attempts 
to catch bruteforce automatically. Currently, I have the following 
configuration:

...



But for me it doesn't look like what is specified in docs[1]:


Field | Description
---
error | Set when error happens
success | yes, when authentication succeeded
policy_penalty | Time of penalty added by policy server
policy_result | Values: ok, delayed, refused


Why I don't see neither "success" and "error" field in logs? Also, why 
policy_result is ok despite I am logging only failed authentication 
attempts? From postfix I can see that those attempts were actually 
failed:




Hi Serhii

the way the code currently works is that "success: yes" is the only 
possible value. When the authentication is not successful the "success" 
is not present. i.e. there is no "success: no".


You're not seeing any "success" values since the code only produces 
"success: yes" and you've filtered that out.


As to why you're not seeing any error, my suspicion is that it is 
unintentional. If I am right about that then the following patch in the 
function auth_request_fail_with_reply(...) could solve it. It now logs 
error: authentication failed.


--- dovecot-2.3.21-orig/src/auth/auth-request.c    2023-09-14 
15:17:46.0 +0200
+++ dovecot-2.3.21/src/auth/auth-request.c    2024-08-04 
14:43:03.837000812 +0200

@@ -303,7 +303,7 @@
     stats = auth_request_stats_get(request);
     stats->auth_failure_count++;
 }
-
+    request->failed = TRUE;
 auth_request_set_state(request, AUTH_REQUEST_STATE_FINISHED);
 auth_request_refresh_last_access(request);
 auth_request_log_finished(request);

The need for something like this also seems to be warranted by the fact 
that internal failures on authentication only get reported by the event 
logging if request->failed is set and I couldn't see anywhere that 
happens. With the above patch these will also now be logged if there is 
a call to the function auth_request_internal_failure(...)


I also think that the above patch may not deal with all the cases where 
there is an internal failure in authentication, but those are a bit 
harder to test. There may still be some cases where there is neither a 
success or error, and those cases should still be treated as failures or 
subject to further patching.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: username_filter settings in passdb

[John Fawcett via dovecot]

Hi

in my opinion you found a bug. In the function passdb_preinit(..) in 
file src/auth/passdb.c the username_filters are copied from 
set->username_filter to passdb->username_filter. However that code only 
gets to be executed if this line returns NULL


passdb = passdb_find(set->driver, set->args, &idx);

For some reason this code finds a match (i.e. != NULL) for your second 
passdb (postmasterfilter) so it never reaches the code to setup the 
filter correctly. Strange to say for the third db it doesn't find a 
match and does setup the filter.


I would guess that the main reason this hasn't been noticed is that most 
use cases of multiple passdb's use different drivers. Either this should 
be treated as a bug or the documentation should be updated to state that 
multiple passdb's should not use the same driver.


As a workaround to this problem, given that you are handing off 
authentication to a php script, my suggestion would be to update that 
script so that it handles all the use cases directly. In Dovecot you 
will only need to define one passdb that calls the script.


John

On 01/08/2024 15:17, zaxwat93--- via dovecot wrote:

added names:
passdb allusers {
  ...
}
passdb postmasterfilter {
   ...
   username_filter =us...@postmaster.local.one  
   ...

}
passdb user2filter {
   
   username_filter =us...@user2.local.one  
   ,,,

}

try "doveadm auth testus...@user2.local.one" and got logs:
Aug  1 16:08:02 auth: Debug: auth client connected (pid=0)
Aug  1 16:08:02 auth: Debug: client in: AUTH   1   PLAIN   service=doveadm 
debug   resp=
Aug  1 16:08:02 auth: Debug: allusers(us...@user2.local.one): Performing passdb 
lookup
Aug  1 16:08:02 auth: Debug: allusers(us...@user2.local.one): execute: 
/usr/local/bin/php /usr/local/dovecot/bin/checkpassword.php 
/usr/local/libexec/dovecot/checkpassword-reply
Aug  1 16:08:02 auth: Debug: allusers(us...@user2.local.one): Received input:
Aug  1 16:08:02 auth: Debug: allusers(us...@user2.local.one): exit_status=1
Aug  1 16:08:02 auth: allusers(us...@user2.local.one): Login failed (status=1)
Aug  1 16:08:02 auth: Debug: allusers(us...@user2.local.one): Finished passdb 
lookup
Aug  1 16:08:02 auth: Debug: postmasterfilter(us...@user2.local.one): 
Performing passdb lookup
Aug  1 16:08:02 auth: Debug: postmasterfilter(us...@user2.local.one): username 
changedus...@user2.local.one  ->us...@local.one
Aug  1 16:08:02 auth: Debug: postmasterfilter(us...@local.one): execute: 
/usr/local/bin/php /usr/local/dovecot/bin/checkpassword.php 
/usr/local/libexec/dovecot/checkpassword-reply
Aug  1 16:08:02 auth: Debug: postmasterfilter(us...@local.one): Received 
input:user=us...@local.oneuserdb_home=/var/spool/mail/
userdb_uid=dovecot  userdb_gid=dovecot
Aug  1 16:08:02 auth: Debug: postmasterfilter(us...@local.one): exit_status=0
Aug  1 16:08:02 auth: Debug: postmasterfilter(us...@local.one): username 
changedus...@local.one  ->postmas...@local.one
Aug  1 16:08:02 auth: Debug: postmasterfilter(postmas...@local.one): Finished 
passdb lookup
Aug  1 16:08:02 auth: Debug: auth(postmas...@local.one): Auth request finished
Aug  1 16:08:02 auth: Debug: client passdb out: OK 
1user=postmas...@local.one original_user=us...@user2.local.one

Same result: it should skip postmasterfilter passdb and work with user2filter, 
but it didn't
___
dovecot mailing list --dovecot@dovecot.org
To unsubscribe send an email todovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: username_filter settings in passdb

[John Fawcett via dovecot]

On 05/08/2024 10:00, zaxwat93--- via dovecot wrote:

Thank you for explanation. Unfortunately it is better not to touch this script, 
so I will wait for the fix
___
dovecot mailing list --dovecot@dovecot.org
To unsubscribe send an email todovecot-le...@dovecot.org


Hi

I stated one thing wrong in my summary "Strange to say for the third db 
it doesn't find a match and does setup the filter."


That part was not correct since I had maybe a typo in my config. But 
that gave me a clue for another workaround. If the "args" parameter is 
different in each passdb, the current code seems to be working.


You can duplicate the script under different names or even create 2 
symlinks to the same script. I didn't try it with php which I don't have 
installed.


passdb db1 {
  driver = checkpassword
  args = /usr/local/libexec/dovecot/checkpassword
}

passdb db2 {
  driver = checkpassword
  args = /usr/local/libexec/dovecot/checkpassword1
  username_filter = us...@postmaster.local.one
  default_fields = domain=local.one
  override_fields = user=postmas...@local.one
}

passdb db3 {
  driver = checkpassword
  args = /usr/local/libexec/dovecot/checkpassword2
  username_filter = us...@user2.local.one
  default_fields = domain=local.one
  override_fields = user=us...@local.one
}

John
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Dovecot refuses to install/run on machine without IPv6

[John Fawcett via dovecot]

On 06/08/2024 21:08, Kurt Fitzner via dovecot wrote:

Hi,

I just tried to install Dovecot (version 2.3.19.1 9b53102964) on a 
Debian 12 server I'm building.  It failed because Dovecot's default 
listen address is explicitly "*, ::" and it appears to have no logic 
to determine if there actually is an IPv6-enabled interface or that 
IPv6 is enabled on the target machine before it tries to listen on 
it.  If dovecot wants to listen on IPv6 by default, that's neither 
here nor there, but if this is default behaviour it should check first.


I'm curious if it's the same behaviour for machines without IPv4.

    Kurt Fitzner 


Kurt

I can understand the issues you mention about running Dovecot with a 
configuration that does not match up with the network configuration of 
the server. I was curious about what errors you were finding on the 
install. I am not sure why Dovecot would refuse to install.


Strange to say even though I do have ipv6 configured on the network 
interface I have your same issue about running Dovecot (and not just 
Dovecot) on a system that uses systemd. The default systemd scripts 
don't work for me when rebooting, since the ipv6 interfaces come up 
later than the ipv4 ones and I never worked out if there was a 
dependency I could add in the scripts to ensure that Dovecot is started 
only when the ipv6 interfaces were up. So on every reboot Dovecot 
startup fails because it doesn't find the ipv6 address, but when started 
later it works since by that time ipv6 is available.


If Dovecot was modified in the way you were suggesting, it would cause a 
problem for me. Since then it would check for ipv6 (not yet up) and only 
bind to ipv4. It would be running but without the configured ipv6. 
Current behaviour where it refuses to run, is a much better situation, 
since I just have a cron job that checks if it is not running and starts 
dovecot by which time ipv6 is ready.


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: lda not sending email on sieve correctly

[John Fawcett via dovecot]

On 08/08/2024 10:15, Eduardo Diaz Rodriguez via dovecot wrote:
Hi to all I was using dovecot like 15 years on multiples debian 
places, and I have the same issue on debian 12 and debian 11.


on my configuration I use exim with 2 process to send email, and at 
the same time use mailscanner to scan all the email that runs on the 
server.


exim1 get the email, and put on the mailscanner directory and 
mailscanner scan the email and put on another directory to be send by 
exim2...


Debian-+  242594   1  0 Jul17 ?    00:00:15 /usr/sbin/exim4 
-bd -oY -odq -oX 587:25 -oP /var/run/exim4/exim.pid
Debian-+  242597   1  0 Jul17 ?    00:00:01 /usr/sbin/exim4 
-oP /run/exim4/eximqr.pid -q15m -oY -DOUTGOING



I enable vacation the users, and works ok, but the issue is related 
the email sent by dovecot…


The mail sent by dovecot lacks of the headers and the SPF and DKMI 
headers but if I sent the email form my command line using 
/usr/sbin/sendmail the email has the all the headers…. The mail issue 
here is with the norms for spam from google the nobody get the 
vacations emails reply.


as far I see on the logs of sendmail I see only the email on the exim4 
-DOutgoing (log) (I don't know why) because if I send the mail using 
send mail go to the whole process


I don't know why dovecot send the mail to exim -DOUTGOING and not use 
sendmail on command line..


There any way to create a warper to capture the mail and sending "like 
this one" /usr/local/bin/dovecot-sendmail-wrapper


I enable the debugn on send mail to see what is doing dovect but no luck

Aug  8 02:13:27 servidorabm dovecot: 
lda(luis.alvarez)<870407>: Debug: program 
exec:/usr/sbin/exim: Created
Aug  8 02:13:27 servidorabm dovecot: 
lda(luis.alvarez)<870407>: Debug: program 
exec:/usr/sbin/exim: Establishing connection
Aug  8 02:13:27 servidorabm dovecot: 
lda(luis.alvarez)<870407>: Debug: program 
exec:/usr/sbin/exim: Forked child process
Aug  8 02:13:27 servidorabm dovecot: 
lda(luis.alvarez)<870407>: Debug: program 
exec:/usr/sbin/exim (870409): Connected to program
Aug  8 02:13:27 servidorabm dovecot: 
lda(luis.alvarez)<870407>: Debug: program 
exec:/usr/sbin/exim (870409): Finished streaming payload to program
Aug  8 02:13:27 servidorabm dovecot: 
lda(luis.alvarez)<870407>: Debug: program 
exec:/usr/sbin/exim (870409): Finished input to program
Aug  8 02:13:27 servidorabm dovecot: 
lda(luis.alvarez)<870407>: Debug: program 
exec:/usr/sbin/exim (870409): Disconnected
Aug  8 02:13:27 servidorabm dovecot: 
lda(luis.alvarez)<870407>: Debug: program 
exec:/usr/sbin/exim (870409): Waiting for program to finish after 0 
msecs (timeout = 3 msecs)
Aug  8 02:13:27 servidorabm dovecot: 
lda(luis.alvarez)<870407>: Debug: program 
exec:/usr/sbin/exim (870409): Child process ended
Aug  8 02:13:27 servidorabm dovecot: 
lda(luis.alvarez)<870407>: Debug: program 
exec:/usr/sbin/exim (870409): Destroy
Aug  8 02:13:27 servidorabm dovecot: 
lda(luis.alvarez)<870407>: sieve: 
msgid=<2118db23-b529-4b96-8d36-4a9ed7596...@pk25.com>: vacation 
action: sent vacation response to 


...


Hi Eduardo

would you mind showing your configuration, in particular it would be 
interesting to know the output of these:


doveconf sendmail_path
doveconf submission_host

What is the way you want to deliver email to from dovecot lda to exim, 
via sendmail command, via port 25 or something else?


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Userdb empty

[John Fawcett via dovecot]

On 05/08/2024 16:02, Cam Ellison via dovecot wrote:
I have shifted from a system-based email system to virtual for a 
website I manage. The MTA is exim4, and user data are in a MySQL 
database.


It will receive email, but with one exception the userdb is empty and 
emails go out with no username in the email address. Sample error 
message from exim:


2024-08-05 05:21:43 1sawid-00065N-0R Error while reading message with 
no usable sender address (R=1sawiX-00065C-1a): at least one malformed 
recipient address: @scasl.ca - no local part



Hi Cam

if your issue is about sending emails and you are using dovecot only as 
an imap server and using exim to send emails, what is the relation 
between your problem and the dovecot configuration? I didn't see one, 
but maybe I'm misunderstanding something about the problem you are seeing.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot-2.3.2.1 and dovecot-pigeonhole-0.5.2 bug?

[John Fawcett via dovecot]

On 12/08/2024 19:06, skraw--- via dovecot wrote:

As I already said, this setup is way too complex to think about an update.
The cause of this problem cannot be a problem of the same size. If somebody 
(probably you or Stephan) already found the cause I could concentrate on 
backporting a solution.
And future readers of the thread may be interested as well ...
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Hi

this is what I found

https://dovecot.org/list/dovecot/2019-April/115674.html

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: LMTP error reading configuration

[John Fawcett via dovecot]

On 07/10/2024 05:08, DaLynX via dovecot wrote:

Hello,

I am getting the following error on my dovecot LMTP server, upon delivery from 
my postfix mailserver:

Oct  6 03:31:09 m dovecot[36551]: lmtp: Fatal: Error reading configuration: 
read(/var/run/dovecot/config) failed: read(size=8192) failed: Interrupted 
system call - Also failed to read config by executing doveconf: 
/var/run/dovecot/config is a UNIX socket (path is from CONFIG_FILE environment)

This results in mail being deferred by postfix (different host), and 
successfully delivered at the next try:

Oct  6 03:31:10 mailer postfix/lmtp[44835]: 46F0BA329: to=, 
orig_to=, relay=192.168.1.101[192.168.1.101]:24, delay=17, 
delays=0.2/0.01/16/0, dsn=4.4.2, status=deferred (lost connection with 
192.168.1.101[192.168.1.101] while receiving the initial server greeting)

Any idea where it comes from?

I am running dovecot 2.3.21.1 (d492236fa0) on FreeBSD 14.1-RELEASE-p5.

Here is my configuration:

...


Looks like a tricky problem. Are there any other dovecot processes 
logging errors or warnings (e.g. the config process)? Is this happening 
on all messages or only occasionally and is there any pattern to it? If 
you look at the full lmtp logging from connect to when it gives the 
error, is there some pattern there that is different for when the 
problem happens compared to when it doesn't?


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: last_login plugin does not update IP

[John Fawcett via dovecot]

On 30/09/2024 15:49, Maciej Polewczyński via dovecot wrote:
When using the last_login plugin I noticed that the IP address 
information is not updated. If there is no entry in the sql database 
for a given user, the first time the login date and time information 
and the IP address will be saved. Each subsequent login only updates 
the date and time. The IP address remains unchanged.


Example configuration for pop3 service:

map {
 pattern = shared/last-login/pop3/$user/$rip
 table = last_login
 value_field = last_pop3_access
 value_type = uint

 fields {
   userid = $user
   last_pop3_ip = $rip
 }
}

This is translated into the following sql command:

INSERT INTO last_login (last_pop3_access,userid,last_pop3_ip) VALUES 
(1727691183673,'test','1.2.3.4') ON DUPLICATE KEY UPDATE 
last_pop3_access=1727691183673


You can see that after "ON DUPLICATE KEY" only the time is updated.

Is there any way to influence the behavior of the plugin and the sql 
query so that the IP address is also updated? I'm using Dovecot v2.3.4


One solution to this would be to redefine the primary key so that it is 
composed from multiple columns userid + last_pop3_id. That way you would 
get a row inserted per userid and pop3_ip combination and the timestamp 
would be the last time that userid and pop3_ip logged in.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Verifying LDAP configuration from command line ?

[John Fawcett via dovecot]

On 05/06/2025 22:12, BuzzSaw Code via dovecot wrote:

But for the life of me I can't find out how to test the configuration
file without having to actually update my dovecot configuration to use
it.  I'm looking for something like the Postfix 'postmap' tool that I
can provide my configuration file for the LDAP lookup, give a username
input to and it will show me the results from the configuration,
including the return values.


Hi

not sure if it will work for your use case, but have you looked into 
"doveadm auth test" command?


I know it requires a password but not sure what happens if you don't 
need one.


There is a -o setting that can override configurations from your 
existing config.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org