Re: json_parse_number broken by compiler optimization

[Christian Ehrhardt]

On Wed, Mar 31, 2021 at 8:46 AM Christian Ehrhardt
 wrote:
>
> On Tue, Mar 30, 2021 at 9:21 PM Josef 'Jeff' Sipek
>  wrote:
> >
> > On Tue, Mar 30, 2021 at 13:34:54 -0400, Josef 'Jeff' Sipek wrote:
> > > On Tue, Mar 30, 2021 at 17:53:27 +0200, Christian Ehrhardt wrote:
> > > > Hi,
> > > > the recent Ubuntu (re)builds uncovered an issue with dovecot 
> > > > 1:2.3.13+dfsg1-1
> > > > build log: 
> > > > https://launchpadlibrarian.net/529849650/buildlog_ubuntu-hirsute-amd64.dovecot_1%3A2.3.13+dfsg1-1build1_BUILDING.txt.gz
> > > > A coworker tried 2.3.14 but got the same result.
> > > >
> > > > What fails is the json_parser build time test like:
> > > >   test-json-parser.c:161: Assert(#25) failed:
> > > > null_strcmp(json_output[pos].value, value) == 0
> > > >
> > > > I was looking into that a bit more and what I found is that it is
> > > > dependent on the new toolchain
> > > > of gcc 10.2.0-1.
> > >
> > > FWIW, I managed to reproduce it on FreeBSD with gcc 11, so the good news 
> > > for
> > > you is that it isn't Ubuntu specific :)
> > >
> > > I'll debug further.
> >
> > The culprit seems to be LTO.  If you disable LTO, everything should work
> > just fine.
>
> I've had LTO disabled and it has still shown the same effect (with my
> gcc 10.2.0-1).
> I'll give it a non-LTO retry and double check if it really changed the
> compile options accordingly.
> I'll let you know about that later on.

Indeed, I wonder what I tried yesterday in regard to LTO then .. :-/
I can confirm that disabling LTO fixes the issue for me as well and
for now that should be a good mitigation until the root cause is found
and fixed.

Since it might help debugging the underlying problem with LTO here is
another data point.
With LTO enabled (and skipping the json-parser issues with my
optimization trick) there is another testcase later that fails (but
works with LTO disabled):

test-istream-attachment.c:354: Assert failed: memcmp(data +
sizeof(BINARY_TEXT_LONG)-1, BINARY_TEXT_SHORT,
strlen(BINARY_TEXT_SHORT)) == 0
istream attachment ... : FAILED
Panic: file test-istream-attachment.c: line 395
(test_istream_attachment_extractor_one): assertion failed: (size >=
prefix_len && memcmp(data, mail_broken_input_body_prefix, prefix_len)
== 0)
Error: Raw backtrace: ./test-istream-attachment(+0x4cd95)
[0x55c0db91bd95] -> ./test-istream-attachment(backtrace_get+0x75)
[0x55c0db91bf65] -> ./test-istream-attachment(+0x2a7fb)
[0x55c0db8f97fb] -> ./test-istream-attachment(+0x2a837)
[0x55c0db8f9837] -> ./test-istream-attachment(+0x13c5c)
[0x55c0db8e2c5c] -> ./test-istream-attachment(+0x12d39)
[0x55c0db8e1d39] -> ./test-istream-attachment(+0x1cca3)
[0x55c0db8ebca3] -> ./test-istream-attachment(+0x2424d)
[0x55c0db8f324d] -> ./test-istream-attachment(test_run+0x63)
[0x55c0db8f32f3] ->
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xd5)
[0x7f60d232d565] -> ./test-istream-attachment(_start+0x2e)
[0x55c0db8e7c2e]
/bin/bash: line 1: 1650909 Aborted (core dumped) ./$bin


> >  So, I think that'll be the "official" workaround - and a much
> > better one than disabling optimization completely.
>
> Well, "completely" is a bit hard, as I only disabled it on a single
> function and not the full build :-)
> But yeah if it really turns out to be LTO then disabling that will be
> fine as an avoidance until we've found the underlying root cause.
>
> > Now, the big question is, is something in the test breaking or is the parser
> > itself somehow triggering this.
> >
> > Jeff.
> >
> > >
> > > Thanks again for the report,
> > >
> > > Jeff.
> > >
> > > >
> > > > Not all calls to json_parse_* fail, e.g. the first one looks all good 
> > > > and passes
> > > > I was iterating the tests using a report function defined like
> > > >
> > > > (gdb) define repcon
> > > > >c
> > > > >p pos
> > > > >p json_output[pos].type
> > > > >p type
> > > > >p json_output[pos].value
> > > > >p value
> > > > >call null_strcmp(json_output[pos].value, value)
> > > > >end
> > > >
> > > > The first one to be bad was:
> > > > Breakpoint 2, test_json_parser_success (full_size=) at
> > > > test-json-parser.c:161
> > > > 161 test_assert_idx(null_strcmp(json_output[pos].value, value) == 0, 
> > > > pos);
> > > > $84 = 25
> > > > $85 = JSON_TYPE_NUMBER
> > > > $86 = JSON_TYPE_NUMBER
> > > > $87 = 0x55633b25 "-12.456"
> > > > $88 = 0x55693110 ""
> > > > $89 = 45
> > > >
> > > > Earlier and later parsing was happy, for example
> > > >
> > > > Breakpoint 2, test_json_parser_success (full_size=) at
> > > > test-json-parser.c:161
> > > > 161 test_assert_idx(null_strcmp(json_output[pos].value, value) == 0, 
> > > > pos);
> > > > $90 = 27
> > > > $91 = JSON_TYPE_NUMBER
> > > > $92 = JSON_TYPE_NUMBER
> > > > $93 = 0x55633b32 "12.456e9"
> > > > $94 = 0x55693110 "12.456e9"
> > > > $95 = 0
> > > > (gdb)
> > > >
> > > >
> > > > We have two things we compare here.
> > > > 1. json_output[] which is a static define and for this value is

Temporary disable push notifications while restoring a mailbox

[Ralf Becker]

We use "doveamd import" to restore mailboxes from a snapshot.

Since we use push with Dovecot 2.3.13:

mail_plugins = $mail_plugins mail_lua notify push_notification 
push_notification_lua


plugin {
  push_notification_driver = lua:file=/etc/dovecot/dovecot-push.lua
  push_lua_url = http://172.17.0.1/

the import creates a flood of push-notifications.

Therefore the questions: is there a possibility to disable push for 
"doveadm import" on the command line (not in general)?


Kind regards

Ralf

--
Ralf Becker
EGroupware GmbH [www.egroupware.org]
Handelsregister HRB Kaiserslautern 3587
Geschäftsführer Birgit und Ralf Becker
Leibnizstr. 17, 67663 Kaiserslautern, Germany
Telefon +49 631 31657-0

RE: Dovecot 2.3.14 - Invalid mailbox name: Name must not have '/'

[Markus Valentin]

> On 03/30/2021 8:25 PM Jürgen Edner  wrote:
> Hi @all, 
>  can anyone please shed some light on the changes of v2.3.14 and what need to 
> be done to get the shared folders working again as in v2.3.13. 
>  I cannot see any invalid characters in the folder names. I would also 
> appreciate any advise on how I can debug the program any further. 

Hi,

can you provide us with the output of `doveconf -n` please? 

Markus

Re: Temporary disable push notifications while restoring a mailbox

[Aki Tuomi]

> On 31/03/2021 11:00 Ralf Becker  wrote:
> 
>  
> We use "doveamd import" to restore mailboxes from a snapshot.
> 
> Since we use push with Dovecot 2.3.13:
> 
> mail_plugins = $mail_plugins mail_lua notify push_notification 
> push_notification_lua
> 
> plugin {
>    push_notification_driver = lua:file=/etc/dovecot/dovecot-push.lua
>    push_lua_url = http://172.17.0.1/
> 
> the import creates a flood of push-notifications.
> 
> Therefore the questions: is there a possibility to disable push for 
> "doveadm import" on the command line (not in general)?
> 
> Kind regards
> 
> Ralf
> 

doveadm -o plugin/push_notification_driver= import might work...

alternatively you can create /etc/dovecot/import.conf and use it with doveadm 
-c /etc/dovecot/import.conf import

Aki

Re: Dovecot 2.3.14 - Invalid mailbox name: Name must not have '/'

[Juergen Edner]

Hi Markus,

>> Hi @all, 
>>  can anyone please shed some light on the changes of v2.3.14 and what need 
>> to be done to get the shared folders working again as in v2.3.13. 
>>  I cannot see any invalid characters in the folder names. I would also 
>> appreciate any advise on how I can debug the program any further. 
> 
> Hi,
> 
> can you provide us with the output of `doveconf -n` please? 

attached you will find the desired command output.

Thanks
Juergen

-- 
Mail: juer...@eisfair.org
# 2.3.14 (cee3cbc0d): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.14 (1b5c82b2)
# OS: Linux 4.9.261-eisfair-64-VIRT x86_64  
# Hostname: farragut.privatnet.lan
auth_mechanisms = plain login cram-md5
base_dir = /run/dovecot
default_internal_group = trusted
default_internal_user = exim
default_login_user = exim
listen = *,
log_path = /var/log/dovecot.log
mail_location = maildir:~/.imapmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date index ihave duplicate 
mime foreverypart extracttext
namespace {
  location = maildir:/home/imappublic:INDEXPVT=~/.imapmail/public
  prefix = "#Public/"
  separator = /
  subscriptions = no
  type = public
}
namespace {
  location = maildir:/home/imapshared:INDEXPVT=~/.imapmail/shared
  prefix = "#Shared/"
  separator = /
  subscriptions = no
  type = shared
}
namespace inbox {
  inbox = yes
  location = 
  mailbox Drafts {
auto = subscribe
special_use = \Drafts
  }
  mailbox Sent {
auto = subscribe
special_use = \Sent
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Spam {
auto = subscribe
special_use = \Junk
  }
  mailbox Trash {
auto = subscribe
special_use = \Trash
  }
  prefix = 
  separator = /
  type = private
}
passdb {
  args = scheme=cram-md5 /etc/cram-md5.pwd
  driver = passwd-file
}
plugin {
  mail_log_fields = uid box msgid size from
  sieve = file:~/sieve;active=~/.dovecot.sieve
}
pop3_uidl_format = %08Xv%08Xu
protocols = pop3 imap
service auth {
  unix_listener auth-exim {
group = trusted
mode = 0600
user = exim
  }
}
service imap-login {
  inet_listener imap {
port = 143
  }
  inet_listener imaps {
port = 993
ssl = yes
  }
}
service pop3-login {
  inet_listener pop3 {
port = 110
  }
  inet_listener pop3s {
port = 995
ssl = yes
  }
}
service submission-login {
  inet_listener submission {
port = 587
  }
}
ssl_cert = 

Re: json_parse_number broken by compiler optimization

[Josef 'Jeff' Sipek]

On Wed, Mar 31, 2021 at 09:07:28 +0200, Christian Ehrhardt wrote:
> On Wed, Mar 31, 2021 at 8:46 AM Christian Ehrhardt 
>  wrote:
> > On Tue, Mar 30, 2021 at 9:21 PM Josef 'Jeff' Sipek 
> >  wrote:
...
> > > The culprit seems to be LTO.  If you disable LTO, everything should work
> > > just fine.
> >
> > I've had LTO disabled and it has still shown the same effect (with my
> > gcc 10.2.0-1).
> > I'll give it a non-LTO retry and double check if it really changed the
> > compile options accordingly.
> > I'll let you know about that later on.
> 
> Indeed, I wonder what I tried yesterday in regard to LTO then .. :-/
> I can confirm that disabling LTO fixes the issue for me as well and
> for now that should be a good mitigation until the root cause is found
> and fixed.

Sounds good.  Thanks for the confirmation.

> Since it might help debugging the underlying problem with LTO here is
> another data point.
> With LTO enabled (and skipping the json-parser issues with my
> optimization trick) there is another testcase later that fails (but
> works with LTO disabled):
> 
> test-istream-attachment.c:354: Assert failed: memcmp(data +
> sizeof(BINARY_TEXT_LONG)-1, BINARY_TEXT_SHORT,
> strlen(BINARY_TEXT_SHORT)) == 0
> istream attachment ... : 
> FAILED

Yep, fails here as well.

...
> > >  So, I think that'll be the "official" workaround - and a much
> > > better one than disabling optimization completely.
> >
> > Well, "completely" is a bit hard, as I only disabled it on a single
> > function and not the full build :-)

Agreed ;)

Jeff.

Replicator: getting full resync requests to sync

[Norman King]

Hello.

I'm getting the following output.

root@mail:~# doveadm replicator status
Queued 'sync' requests    0
Queued 'high' requests    0
Queued 'low' requests 0
Queued 'failed' requests  0
Queued 'full resync' requests 61
Waiting 'failed' requests 4


How can i get the full resync requests to actually sync?

Is there a configurable resync delay time that i can use to force a 
resync of these mailboxes? Also, is there a way using either doveadm 
replicator or doveadm sync that i can see all the full resync users?



Thanks.


--

regards
Norman King
Compass Foundation sysadmin/tech support

Phone, 856-974-5335 ext 221

supp...@compassfoundation.io 

Website 

Re: Mass Stripping Attachments by Directory, Age, Size

[justina colmena ~biz]

Well ain't that rich? To use an allegory of sorts, we're going to have start 
using staples rather than paperclips 📎🖇️  with our email attachments, and one 
unified digital signature on the whole message as sent rather than a separate 
signature for each enclosure as commonly "done" with PGP, GnuPG, etc.

On March 30, 2021 7:39:02 PM AKDT, Plutocrat  wrote:
>Still can't find the magic solution to this.
>
>- My PERL isn't good enough to re-purpose strip-attachments.pl so it
>works on individual emails.
>- ripmime works to extract attachments only
>- altermime looked good and would delete all attachments from a
>directory of emails. However it messed up the structure somehow so they
>wouldn't display in an email client (Thunderbird, Roundcube).
>- mimeDEFANG looked possible, but couldn't figure out how to use that
>as a standalone script.
>- PHP solutions including the promising
>https://github.com/php-mime-mail-parser/php-mime-mail-parser seem only
>to be able to save attachments from the email, not delete it.
>
>I'll keep going I guess. I can't believe I'm the only person in the
>world to want to do this though ...
>
>P.
>
>On 19/03/2021 07.31, Joseph Tam wrote:
>> On Thu, 18 Mar 2021, Plutocrat wrote:
>> 
>>> I've been looking around for a solution to this problem. I want to
>prune down the attachments on a server before a migration. Some of the
>emails are 7 years old and have 40Mb attachments, so this seems like a
>good opportunity to rationalize things. So perhaps I'd like to "Remove
>all attachments from emails older than 2 years, in the .Sent
>directory", or "Attachments over 10Mb anywhere in the mail tree"
>>>
>>> I've found the strip_attachments.pl script here
>
>which works fine on mbox (as tested on my local Thunderbird mboxes),
>but not on maildir which is on the dovecot server. My Perl isn't strong
>enough to re-purpose it.
>> 
>> It you have anything that works on mbox, it will probably work on
>Maildir
>> as each file can be considered a single message mbox.  You can
>combine
>> the script with
>> 
>>  find ~user/MailDir -type f ... -exec /path/to/mbox-strip {} \;
>> 
>> The ... can be replaced with more file tests (like minimum size or
>age
>> or only within */cur/) to cut down on processing.
>> 
>> I wrote a gawk script to slim down a multi-Gb Outlook mbox
>> for a user, but it wasn't really complicated, just matching for
>> /^Content-Transfer-Encoding:.*base64/i header (virtually all bulky
>data
>> will be encoded this way), buffering the base64 data part, then
>outputting
>> it if it was small, or deleting/replacing/extracting it otherwise.
>> 
>> It was a one-off discarded tool but I can hunt for it if you're hard
>up.
>> 
>>> I've looked at ripmime and mpack/munpack, and although they seem
>like useful tools to do the job of deconstructing the mail into its
>constituent parts, it doesn't seem to help in re-building the email. I
>think they could be used with a bit of study into mail MIME structure,
>and used with a helper script.
>>>
>>> So before I take a deep dive into scripting my own solution, I just
>wanted to check if anyone else on the list has been through this and
>has some resources or pointers they can share, or maybe even someone to
>tell me "Duh, you can do it with doveadm of course".
>> 
>> MIMEDefang may help.
>> 
>> Joseph Tam 

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.