Hi! I can see the crash now, thanks.
For 2.3, I would suggest as workaround to enable the fetch-size imapc_feature. Aki > On 20/03/2024 09:58 EET John van der Kamp via dovecot <dovecot@dovecot.org> > wrote: > > > See attached script I used. If you get EOF then you've hit the crash. > > > John > > > > > -----Original message----- > From: Aki Tuomi <aki.tu...@open-xchange.com> > Sent: Wednesday, 20th March 2024, 8:40 > To: John van der Kamp <jk...@amazon.nl>; John van der Kamp via dovecot > <dovecot@dovecot.org> > Subject: RE: Crash in dovecot snippet when using imapc > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you can confirm the sender and know the > content is safe. > > > > Could you provide some simple way to reproduce this, minimal config etc? > > Aki > > > On 19/03/2024 17:44 EET John van der Kamp via dovecot <dovecot@dovecot.org> > > wrote: > > > > > > Hi, sorry for the late reply. > > > > The commit you've pointed at before is the commit introducing code for the > > snippets. > > > > Your claim that main is fixed is incorrect: I've bisected through the git > > history, and the commit that "fixes" it, is the one flipping imapc features > > to negatives: > > https://github.com/dovecot/core/commit/7810b38d30b7dbb2155f78873fe760bc9e2e6212 > > > > <https://github.com/dovecot/core/commit/7810b38d30b7dbb2155f78873fe760bc9e2e6212> > > However, the default imapc_features value stays the same, so all the > > "negative" features are suddenly enabled. > > > > I've reset the defaults in the frontend config to what it was before: > > > > imapc_features = no-fetch-size no-fetch-headers no-search no-modseq > > no-delay-login no-fetch-bodystructure no-acl > > > > and then dovecot starts crashing again in the described scenario. It is the > > "no-fetch-size" flag, and if I use "rfc822.size" feature on a 2.3 branch it > > stops crashing. > > > > > > Turns out this same feature adds some filter that seems to be meant for > > some exchange email side-effect: > > https://github.com/dovecot/core/blob/main/src/lib-storage/index/imapc/imapc-mail-fetch.c#L596 > > > > <https://github.com/dovecot/core/blob/main/src/lib-storage/index/imapc/imapc-mail-fetch.c#L596> > > where this filter tries to remove any X-Message-Flag header. This is > > weird, because it could have been an normally received header as well as > > something that was tacked on later by exchange. > > > > > > The main bug is not fixed by just removing that filter: chaining filters is > > probably very broken when using the imapc backend, and it might be broken > > in other unknown scenarios. > > > > > > Regards, > > > > > > John > > > > > > > > -----Original message----- > > From: Aki Tuomi via dovecot <dovecot@dovecot.org> > > Sent: Friday, 19th January 2024, 8:37 > > To: Aki Tuomi via dovecot <dovecot@dovecot.org>; John van der Kamp > > <jk...@amazon.nl> > > Subject: RE: Crash in dovecot snippet when using imapc > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and know > > the content is safe. > > > > > > > > Sorry, the provided patch link was wrong, it's already in 2.3.21, my bad. > > Anyways, it is still fixed in main, since it does not happen there. > > > > Aki > > > > > On 19/01/2024 09:13 EET Aki Tuomi via dovecot <dovecot@dovecot.org> wrote: > > > > > > > > > Hi! > > > > > > I was able to reproduce this issue with 2.3.21, but it seems to have been > > > fixed in main. I think > > > https://github.com/dovecot/core/commit/1c1b77dbf9a548aac788efb76973ce2d0fa6c732.patch > > > will fix this. > > > > > > Aki > > > > > > > On 18/01/2024 22:51 EET John van der Kamp via dovecot > > > > <dovecot@dovecot.org> wrote: > > > > > > > > > > > > Hello, > > > > > > > > > > > > I've found a crash in a very specific setup. A dovecot server with > > > > imapc connection needs to receive an email with no body contents for > > > > the intent of generating a preview/snippet. It crashes somewhere deep > > > > in the jungle of istream and snapshots. I've included a script which > > > > sets up the systems to reproduce the crash. > > > > > > > > > > > > I've tested this with several versions. 2.3.16 doesn't seem to be > > > > affected, but 2.3.20 and 2.3.21 are affect. > > > > > > > > > > > > For me it produces a traceback like this, using the ubuntu version from > > > > here: https://packages.ubuntu.com/noble/dovecot-core > > > > > > > > > > > > (gdb) bt > > > > #0 __pthread_kill_implementation (no_tid=0, signo=6, > > > > threadid=140530132887360) at ./nptl/pthread_kill.c:44 > > > > #1 __pthread_kill_internal (signo=6, threadid=140530132887360) at > > > > ./nptl/pthread_kill.c:78 > > > > #2 __GI___pthread_kill (threadid=140530132887360, signo=signo@entry=6) > > > > at ./nptl/pthread_kill.c:89 > > > > #3 0x00007fcfb8842476 in __GI_raise (sig=sig@entry=6) at > > > > ../sysdeps/posix/raise.c:26 > > > > #4 0x00007fcfb88287f3 in __GI_abort () at ./stdlib/abort.c:79 > > > > #5 0x00007fcfb8b37fe5 in default_fatal_finish (status=0, > > > > type=LOG_TYPE_PANIC) at ../lib/failures.c:465 > > > > #6 fatal_handler_real (ctx=<optimized out>, format=<optimized out>, > > > > args=<optimized out>) at ../lib/failures.c:477 > > > > #7 0x00007fcfb8be50d7 in i_internal_fatal_handler (ctx=<optimized > > > > out>, format=<optimized out>, args=<optimized out>) at > > > > ../lib/failures.c:879 > > > > #8 0x00007fcfb8b37eea in i_panic (format=0x7fcfb8c29020 "file %s: line > > > > %d (%s): assertion failed: (%s)") at ../lib/failures.c:530 > > > > #9 0x00007fcfb8b3387b in i_stream_header_filter_snapshot_free > > > > (_snapshot=<optimized out>) at ../lib-mail/istream-header-filter.c:663 > > > > #10 i_stream_header_filter_snapshot_free (_snapshot=0x55dabe297a60) at > > > > ../lib-mail/istream-header-filter.c:655 > > > > #11 0x00007fcfb8bf25ac in i_stream_snapshot_free > > > > (_snapshot=_snapshot@entry=0x55dabe29b0d0) at ../lib/istream.c:253 > > > > #12 0x00007fcfb8bf2654 in i_stream_unref (stream=0x7ffc16cc7fa0) at > > > > ../lib/istream.c:66 > > > > #13 0x00007fcfb8d96baa in index_mail_write_body_snippet > > > > (mail=0x55dabe292058) at index/index-mail.c:1151 > > > > #14 0x00007fcfb8d97e48 in index_mail_parse_bodystructure > > > > (mail=0x55dabe292058, field=MAIL_CACHE_BODY_SNIPPET) at > > > > index/index-mail.c:1551 > > > > #15 0x00007fcfb8d97fe2 in index_mail_fetch_body_snippet > > > > (value_r=0x7ffc16cc8050, mail=0x55dabe292058) at index/index-mail.c:1602 > > > > #16 index_mail_get_special (_mail=0x55dabe292058, field=<optimized > > > > out>, value_r=0x7ffc16cc8050) at index/index-mail.c:1730 > > > > #17 0x00007fcfb8d16ffe in mail_get_special > > > > (mail=mail@entry=0x55dabe292058, > > > > field=field@entry=MAIL_FETCH_BODY_SNIPPET, > > > > value_r=value_r@entry=0x7ffc16cc8050) > > > > at > > > > /home/ubuntu/dovecot/new/dovecot-2.3.21+dfsg1/src/lib-storage/mail.c:418 > > > > #18 0x000055dabc52645c in fetch_snippet (ctx=0x55dabe26e050, > > > > mail=0x55dabe292058, preview=0x55dabe28f1f8) at > > > > ./src/imap/imap-fetch-body.c:615 > > > > #19 0x000055dabc52b5cc in imap_fetch_more_int (ctx=0x55dabe26e050, > > > > cancel=false) at ./src/imap/imap-fetch.c:562 > > > > #20 0x000055dabc52b8ad in imap_fetch_more (ctx=0x55dabe26e050, > > > > cmd=0x55dabe26de98) at ./src/imap/imap-fetch.c:617 > > > > #21 0x000055dabc51fd07 in cmd_fetch (cmd=0x55dabe26de98) at > > > > ./src/imap/cmd-fetch.c:382 > > > > #22 0x000055dabc528af4 in command_exec (cmd=0x55dabe26de98) at > > > > ./src/imap/imap-commands.c:201 > > > > #23 0x000055dabc52e9e2 in client_command_input (cmd=<optimized out>) at > > > > ./src/imap/imap-client.c:1237 > > > > #24 0x000055dabc52ea96 in client_command_input (cmd=<optimized out>) at > > > > ./src/imap/imap-client.c:1307 > > > > #25 0x000055dabc52eeed in client_handle_next_command > > > > (remove_io_r=<synthetic pointer>, client=0x55dabe26d2c8) at > > > > ./src/imap/imap-client.c:1349 > > > > #26 client_handle_input (client=0x55dabe26d2c8) at > > > > ./src/imap/imap-client.c:1363 > > > > #27 0x000055dabc52f2c4 in client_input (client=0x55dabe26d2c8) at > > > > ./src/imap/imap-client.c:1407 > > > > #28 0x00007fcfb8bfe27d in io_loop_call_io (io=0x55dabe26e660) at > > > > ../lib/ioloop.c:737 > > > > #29 0x00007fcfb8bff81a in io_loop_handler_run_internal > > > > (ioloop=0x55dabe243fd0) at ../lib/ioloop-epoll.c:222 > > > > #30 0x00007fcfb8bff8d4 in io_loop_handler_run (ioloop=0x55dabe243fd0) > > > > at ../lib/ioloop.c:789 > > > > #31 0x00007fcfb8bffa90 in io_loop_run (ioloop=0x55dabe243fd0) at > > > > ../lib/ioloop.c:762 > > > > #32 0x00007fcfb8b6ce57 in master_service_run (service=0x55dabe243e20, > > > > callback=callback@entry=0x55dabc533210 <client_connected>) at > > > > ../lib-master/master-service.c:878 > > > > #33 0x000055dabc51ad37 in main (argc=<optimized out>, argv=<optimized > > > > out>) at ./src/imap/main.c:575 > > > > > > > > > > > > John > > > > > > > > > > > > > > > > Hello, > > > > > > > > I've found a crash in a very specific setup. A dovecot server with imapc > > > > connection needs to receive an email with no body contents for the > > > > intent of > > > > generating a preview/snippet. It crashes somewhere deep in the jungle of > > > > istream and snapshots. I've included a script which sets up the systems > > > > to > > > > reproduce the crash. > > > > > > > > I've tested this with several versions. 2.3.16 doesn't seem to be > > > > affected, but > > > > 2.3.20 and 2.3.21 are affect. > > > > > > > > For me it produces a traceback like this, using the ubuntu version from > > > > here: > > > > https://packages.ubuntu.com/noble/dovecot-core > > > > > > > > (gdb) bt > > > > #0 __pthread_kill_implementation (no_tid=0, signo=6, > > > > threadid=140530132887360) > > > > at ./nptl/pthread_kill.c:44 > > > > #1 __pthread_kill_internal (signo=6, threadid=140530132887360) at > > > > ./nptl/ > > > > pthread_kill.c:78 > > > > #2 __GI___pthread_kill (threadid=140530132887360, signo=signo@entry=6) > > > > at ./ > > > > nptl/pthread_kill.c:89 > > > > #3 0x00007fcfb8842476 in __GI_raise (sig=sig@entry=6) at > > > > ../sysdeps/posix/ > > > > raise.c:26 > > > > #4 0x00007fcfb88287f3 in __GI_abort () at ./stdlib/abort.c:79 > > > > #5 0x00007fcfb8b37fe5 in default_fatal_finish (status=0, > > > > type=LOG_TYPE_PANIC) > > > > at ../lib/failures.c:465 > > > > #6 fatal_handler_real (ctx=<optimized out>, format=<optimized out>, > > > > args=<optimized out>) at ../lib/failures.c:477 > > > > #7 0x00007fcfb8be50d7 in i_internal_fatal_handler (ctx=<optimized out>, > > > > format=<optimized out>, args=<optimized out>) at ../lib/failures.c:879 > > > > #8 0x00007fcfb8b37eea in i_panic (format=0x7fcfb8c29020 "file %s: line > > > > %d > > > > (%s): assertion failed: (%s)") at ../lib/failures.c:530 > > > > #9 0x00007fcfb8b3387b in i_stream_header_filter_snapshot_free > > > > (_snapshot=<optimized out>) at ../lib-mail/istream-header-filter.c:663 > > > > #10 i_stream_header_filter_snapshot_free (_snapshot=0x55dabe297a60) at > > > > ../lib- > > > > mail/istream-header-filter.c:655 > > > > #11 0x00007fcfb8bf25ac in i_stream_snapshot_free > > > > (_snapshot=_snapshot@entry=0x55dabe29b0d0) at ../lib/istream.c:253 > > > > #12 0x00007fcfb8bf2654 in i_stream_unref (stream=0x7ffc16cc7fa0) at > > > > ../lib/ > > > > istream.c:66 > > > > #13 0x00007fcfb8d96baa in index_mail_write_body_snippet > > > > (mail=0x55dabe292058) > > > > at index/index-mail.c:1151 > > > > #14 0x00007fcfb8d97e48 in index_mail_parse_bodystructure > > > > (mail=0x55dabe292058, > > > > field=MAIL_CACHE_BODY_SNIPPET) at index/index-mail.c:1551 > > > > #15 0x00007fcfb8d97fe2 in index_mail_fetch_body_snippet > > > > (value_r=0x7ffc16cc8050, mail=0x55dabe292058) at index/index-mail.c:1602 > > > > #16 index_mail_get_special (_mail=0x55dabe292058, field=<optimized out>, > > > > value_r=0x7ffc16cc8050) at index/index-mail.c:1730 > > > > #17 0x00007fcfb8d16ffe in mail_get_special > > > > (mail=mail@entry=0x55dabe292058, > > > > field=field@entry=MAIL_FETCH_BODY_SNIPPET, > > > > value_r=value_r@entry=0x7ffc16cc8050) > > > > at > > > > /home/ubuntu/dovecot/new/dovecot-2.3.21+dfsg1/src/lib-storage/mail.c:418 > > > > #18 0x000055dabc52645c in fetch_snippet (ctx=0x55dabe26e050, > > > > mail=0x55dabe292058, preview=0x55dabe28f1f8) at > > > > ./src/imap/imap-fetch-body.c: > > > > 615 > > > > #19 0x000055dabc52b5cc in imap_fetch_more_int (ctx=0x55dabe26e050, > > > > > > cancel=false) at ./src/imap/imap-fetch.c:562 > > > > #20 0x000055dabc52b8ad in imap_fetch_more (ctx=0x55dabe26e050, > > > > cmd=0x55dabe26de98) at ./src/imap/imap-fetch.c:617 > > > > #21 0x000055dabc51fd07 in cmd_fetch (cmd=0x55dabe26de98) at > > > > ./src/imap/cmd- > > > > fetch.c:382 > > > > #22 0x000055dabc528af4 in command_exec (cmd=0x55dabe26de98) at > > > > ./src/imap/imap- > > > > commands.c:201 > > > > #23 0x000055dabc52e9e2 in client_command_input (cmd=<optimized out>) at > > > > ./src/ > > > > imap/imap-client.c:1237 > > > > #24 0x000055dabc52ea96 in client_command_input (cmd=<optimized out>) at > > > > ./src/ > > > > imap/imap-client.c:1307 > > > > #25 0x000055dabc52eeed in client_handle_next_command > > > > (remove_io_r=<synthetic > > > > pointer>, client=0x55dabe26d2c8) at ./src/imap/imap-client.c:1349 > > > > #26 client_handle_input (client=0x55dabe26d2c8) at > > > > ./src/imap/imap-client.c: > > > > 1363 > > > > #27 0x000055dabc52f2c4 in client_input (client=0x55dabe26d2c8) at > > > > ./src/imap/ > > > > imap-client.c:1407 > > > > #28 0x00007fcfb8bfe27d in io_loop_call_io (io=0x55dabe26e660) at ../lib/ > > > > ioloop.c:737 > > > > #29 0x00007fcfb8bff81a in io_loop_handler_run_internal > > > > (ioloop=0x55dabe243fd0) > > > > at ../lib/ioloop-epoll.c:222 > > > > #30 0x00007fcfb8bff8d4 in io_loop_handler_run (ioloop=0x55dabe243fd0) > > > > at ../ > > > > lib/ioloop.c:789 > > > > #31 0x00007fcfb8bffa90 in io_loop_run (ioloop=0x55dabe243fd0) at ../lib/ > > > > ioloop.c:762 > > > > #32 0x00007fcfb8b6ce57 in master_service_run (service=0x55dabe243e20, > > > > callback=callback@entry=0x55dabc533210 <client_connected>) at > > > > ../lib-master/ > > > > master-service.c:878 > > > > #33 0x000055dabc51ad37 in main (argc=<optimized out>, argv=<optimized > > > > out>) at > > > > ./src/imap/main.c:575 > > > > > > > > John > > > > > > > > _______________________________________________ > > > > dovecot mailing list -- dovecot@dovecot.org > > > > To unsubscribe send an email to dovecot-le...@dovecot.org > > > _______________________________________________ > > > dovecot mailing list -- dovecot@dovecot.org > > > To unsubscribe send an email to dovecot-le...@dovecot.org > > _______________________________________________ > > dovecot mailing list -- dovecot@dovecot.org > > To unsubscribe send an email to dovecot-le...@dovecot.org > > > > > > > > Amazon Development Center (Netherlands) B.V., Johanna Westerdijkplein 1, > > NL-2521 EN The Hague, Registration No. Chamber of Commerce 56869649, VAT: > > NL 852339859B01 > > > > > > > > Hi, sorry for the late reply. > > The commit you've pointed at before is the commit introducing code for the > > snippets. > > Your claim that main is fixed is incorrect: I've bisected through the git > > history, and the commit that "fixes" it, is the one flipping imapc features > > to > > negatives: https://github.com/dovecot/core/commit/ > > 7810b38d30b7dbb2155f78873fe760bc9e2e6212 However, the default imapc_features > > value stays the same, so all the "negative" features are suddenly enabled. > > I've reset the defaults in the frontend config to what it was before: > > imapc_features = no-fetch-size no-fetch-headers no-search no-modseq > > no-delay- > > login no-fetch-bodystructure no-acl > > and then dovecot starts crashing again in the described scenario. It is the > > "no-fetch-size" flag, and if I use "rfc822.size" feature on a 2.3 branch it > > stops crashing. > > > > Turns out this same feature adds some filter that seems to be meant for some > > exchange email side-effect: > > https://github.com/dovecot/core/blob/main/src/lib- > > storage/index/imapc/imapc-mail-fetch.c#L596 where this filter tries to > > remove > > any X-Message-Flag header. This is weird, because it could have been an > > normally received header as well as something that was tacked on later by > > exchange. > > > > The main bug is not fixed by just removing that filter: chaining filters is > > probably very broken when using the imapc backend, and it might be broken in > > other unknown scenarios. > > > > Regards, > > > > John > > > > > > -----Original message----- > > From: Aki Tuomi via dovecot <dovecot@dovecot.org> > > Sent: Friday, 19th January 2024, 8:37 > > To: Aki Tuomi via dovecot <dovecot@dovecot.org>; John van der Kamp > > <jk...@amazon.nl> > > Subject: RE: Crash in dovecot snippet when using imapc > > > > CAUTION: This email originated from outside of the organization. Do > > not click links or open attachments unless you can confirm the sender > > and know the content is safe. > > > > > > > > Sorry, the provided patch link was wrong, it's already in 2.3.21, my > > bad. Anyways, it is still fixed in main, since it does not happen > > > there. > > > > Aki > > > > > On 19/01/2024 09:13 EET Aki Tuomi via dovecot <dovecot@dovecot.org> > > wrote: > > > > > > > > > Hi! > > > > > > I was able to reproduce this issue with 2.3.21, but it seems to > > > have been fixed in main. I think https://github.com/dovecot/core/ > > > commit/1c1b77dbf9a548aac788efb76973ce2d0fa6c732.patch will fix this. > > > > > > Aki > > > > > > > On 18/01/2024 22:51 EET John van der Kamp via dovecot > > <dovecot@dovecot.org> wrote: > > > > > > > > > > > > Hello, > > > > > > > > > > > > I've found a crash in a very specific setup. A dovecot server > > > with imapc connection needs to receive an email with no body contents > > for the intent of generating a preview/snippet. It crashes somewhere > > deep in the jungle of istream and snapshots. I've included a script > > which sets up the systems to reproduce the crash. > > > > > > > > > > > > I've tested this with several versions. 2.3.16 doesn't seem to be > > affected, but 2.3.20 and 2.3.21 are affect. > > > > > > > > > > > > For me it produces a traceback like this, using the ubuntu > > version from here: https://packages.ubuntu.com/noble/dovecot-core > > > > > > > > > > > > > (gdb) bt > > > > #0 __pthread_kill_implementation (no_tid=0, signo=6, > > threadid=140530132887360) at ./nptl/pthread_kill.c:44 > > > > #1 __pthread_kill_internal (signo=6, threadid=140530132887360) > > at ./nptl/pthread_kill.c:78 > > > > #2 __GI___pthread_kill (threadid=140530132887360, > > signo=signo@entry=6) at ./nptl/pthread_kill.c:89 > > > > #3 0x00007fcfb8842476 in __GI_raise (sig=sig@entry=6) at ../ > > sysdeps/posix/raise.c:26 > > > > #4 0x00007fcfb88287f3 in __GI_abort () at ./stdlib/abort.c:79 > > > > #5 0x00007fcfb8b37fe5 in default_fatal_finish (status=0, > > type=LOG_TYPE_PANIC) at ../lib/failures.c:465 > > > > #6 fatal_handler_real (ctx=<optimized out>, format=<optimized > > out>, args=<optimized out>) at ../lib/failures.c:477 > > > > #7 0x00007fcfb8be50d7 in i_internal_fatal_handler > > (ctx=<optimized out>, format=<optimized out>, args=<optimized out>) > > at ../lib/failures.c:879 > > > > #8 0x00007fcfb8b37eea in i_panic (format=0x7fcfb8c29020 "file > > %s: line %d (%s): assertion failed: (%s)") at ../lib/failures.c:530 > > > > #9 0x00007fcfb8b3387b in i_stream_header_filter_snapshot_free > > (_snapshot=<optimized out>) at ../lib-mail/istream-header-filter.c: > > 663 > > > > #10 i_stream_header_filter_snapshot_free > > (_snapshot=0x55dabe297a60) at ../lib-mail/istream-header-filter.c:655 > > > > #11 0x00007fcfb8bf25ac in i_stream_snapshot_free > > (_snapshot=_snapshot@entry=0x55dabe29b0d0) at ../lib/istream.c:253 > > > > #12 0x00007fcfb8bf2654 in i_stream_unref (stream=0x7ffc16cc7fa0) > > at ../lib/istream.c:66 > > > > #13 0x00007fcfb8d96baa in index_mail_write_body_snippet > > (mail=0x55dabe292058) at index/index-mail.c:1151 > > > > #14 0x00007fcfb8d97e48 in index_mail_parse_bodystructure > > (mail=0x55dabe292058, field=MAIL_CACHE_BODY_SNIPPET) at index/index- > > mail.c:1551 > > > > #15 0x00007fcfb8d97fe2 in index_mail_fetch_body_snippet > > (value_r=0x7ffc16cc8050, mail=0x55dabe292058) at index/index-mail.c: > > 1602 > > > > #16 index_mail_get_special (_mail=0x55dabe292058, > > field=<optimized out>, value_r=0x7ffc16cc8050) at index/index-mail.c: > > 1730 > > > > #17 0x00007fcfb8d16ffe in mail_get_special > > (mail=mail@entry=0x55dabe292058, > > field=field@entry=MAIL_FETCH_BODY_SNIPPET, > > value_r=value_r@entry=0x7ffc16cc8050) > > > > at /home/ubuntu/dovecot/new/dovecot-2.3.21+dfsg1/src/lib- > > > storage/mail.c:418 > > > > #18 0x000055dabc52645c in fetch_snippet (ctx=0x55dabe26e050, > > mail=0x55dabe292058, preview=0x55dabe28f1f8) at ./src/imap/imap- > > fetch-body.c:615 > > > > #19 0x000055dabc52b5cc in imap_fetch_more_int > > (ctx=0x55dabe26e050, cancel=false) at ./src/imap/imap-fetch.c:562 > > > > #20 0x000055dabc52b8ad in imap_fetch_more (ctx=0x55dabe26e050, > > cmd=0x55dabe26de98) at ./src/imap/imap-fetch.c:617 > > > > #21 0x000055dabc51fd07 in cmd_fetch (cmd=0x55dabe26de98) at ./ > > src/imap/cmd-fetch.c:382 > > > > #22 0x000055dabc528af4 in command_exec (cmd=0x55dabe26de98) at ./ > > src/imap/imap-commands.c:201 > > > > #23 0x000055dabc52e9e2 in client_command_input (cmd=<optimized > > out>) at ./src/imap/imap-client.c:1237 > > > > #24 0x000055dabc52ea96 in client_command_input (cmd=<optimized > > out>) at ./src/imap/imap-client.c:1307 > > > > #25 0x000055dabc52eeed in client_handle_next_command > > (remove_io_r=<synthetic pointer>, client=0x55dabe26d2c8) at ./src/ > > imap/imap-client.c:1349 > > > > #26 client_handle_input (client=0x55dabe26d2c8) at ./src/imap/ > > imap-client.c:1363 > > > > #27 0x000055dabc52f2c4 in client_input (client=0x55dabe26d2c8) at > > ./src/imap/imap-client.c:1407 > > > > #28 0x00007fcfb8bfe27d in io_loop_call_io (io=0x55dabe26e660) at > > ../lib/ioloop.c:737 > > > > #29 0x00007fcfb8bff81a in io_loop_handler_run_internal > > (ioloop=0x55dabe243fd0) at ../lib/ioloop-epoll.c:222 > > > > #30 0x00007fcfb8bff8d4 in io_loop_handler_run > > (ioloop=0x55dabe243fd0) at ../lib/ioloop.c:789 > > > > #31 0x00007fcfb8bffa90 in io_loop_run (ioloop=0x55dabe243fd0) at > > ../lib/ioloop.c:762 > > > > #32 0x00007fcfb8b6ce57 in master_service_run > > (service=0x55dabe243e20, callback=callback@entry=0x55dabc533210 > > <client_connected>) at ../lib-master/master-service.c:878 > > > > #33 0x000055dabc51ad37 in main (argc=<optimized out>, > > argv=<optimized out>) at ./src/imap/main.c:575 > > > > > > > > > > > > John > > > > > > > > > > > > > > > > Hello, > > > > > > > > I've found a crash in a very specific setup. A dovecot server > > > with imapc > > > > connection needs to receive an email with no body contents for > > the intent of > > > > generating a preview/snippet. It crashes somewhere deep in the > > jungle of > > > > istream and snapshots. I've included a script which sets up the > > systems to > > > > reproduce the crash. > > > > > > > > I've tested this with several versions. 2.3.16 doesn't seem to be > > affected, but > > > > 2.3.20 and 2.3.21 are affect. > > > > > > > > For me it produces a traceback like this, using the ubuntu > > version from here: > > > > https://packages.ubuntu.com/noble/dovecot-core > > > > > > > > (gdb) bt > > > > #0 __pthread_kill_implementation (no_tid=0, signo=6, > > threadid=140530132887360) > > > > at ./nptl/pthread_kill.c:44 > > > > #1 __pthread_kill_internal (signo=6, threadid=140530132887360) > > at ./nptl/ > > > > pthread_kill.c:78 > > > > #2 __GI___pthread_kill (threadid=140530132887360, > > signo=signo@entry=6) at ./ > > > > nptl/pthread_kill.c:89 > > > > #3 0x00007fcfb8842476 in __GI_raise (sig=sig@entry=6) at ../ > > sysdeps/posix/ > > > > raise.c:26 > > > > #4 0x00007fcfb88287f3 in __GI_abort () at ./stdlib/abort.c:79 > > > > #5 0x00007fcfb8b37fe5 in default_fatal_finish (status=0, > > type=LOG_TYPE_PANIC) > > > > at ../lib/failures.c:465 > > > > #6 fatal_handler_real (ctx=<optimized out>, format=<optimized > > out>, > > > > args=<optimized out>) at ../lib/failures.c:477 > > > > #7 0x00007fcfb8be50d7 in i_internal_fatal_handler > > (ctx=<optimized out>, > > > > format=<optimized out>, args=<optimized out>) at ../lib/ > > failures.c:879 > > > > #8 0x00007fcfb8b37eea in i_panic (format=0x7fcfb8c29020 "file > > %s: line %d > > > > (%s): assertion failed: (%s)") at ../lib/failures.c:530 > > > > #9 0x00007fcfb8b3387b in i_stream_header_filter_snapshot_free > > > > (_snapshot=<optimized out>) at ../lib-mail/istream-header- > > filter.c:663 > > > > #10 i_stream_header_filter_snapshot_free > > (_snapshot=0x55dabe297a60) at ../lib- > > > > mail/istream-header-filter.c:655 > > > > #11 0x00007fcfb8bf25ac in i_stream_snapshot_free > > > > (_snapshot=_snapshot@entry=0x55dabe29b0d0) at ../lib/istream.c: > > 253 > > > > #12 0x00007fcfb8bf2654 in i_stream_unref (stream=0x7ffc16cc7fa0) > > at ../lib/ > > > > istream.c:66 > > > > #13 0x00007fcfb8d96baa in index_mail_write_body_snippet > > (mail=0x55dabe292058) > > > > at index/index-mail.c:1151 > > > > #14 0x00007fcfb8d97e48 in index_mail_parse_bodystructure > > (mail=0x55dabe292058, > > > > field=MAIL_CACHE_BODY_SNIPPET) at index/index-mail.c:1551 > > > > #15 0x00007fcfb8d97fe2 in index_mail_fetch_body_snippet > > > > (value_r=0x7ffc16cc8050, mail=0x55dabe292058) at index/index- > > mail.c:1602 > > > > #16 index_mail_get_special (_mail=0x55dabe292058, > > field=<optimized out>, > > > > value_r=0x7ffc16cc8050) at index/index-mail.c:1730 > > > > #17 0x00007fcfb8d16ffe in mail_get_special > > (mail=mail@entry=0x55dabe292058, > > > > field=field@entry=MAIL_FETCH_BODY_SNIPPET, > > > > value_r=value_r@entry=0x7ffc16cc8050) > > > > at /home/ubuntu/dovecot/new/dovecot-2.3.21+dfsg1/src/lib- > > > storage/mail.c:418 > > > > #18 0x000055dabc52645c in fetch_snippet (ctx=0x55dabe26e050, > > > > mail=0x55dabe292058, preview=0x55dabe28f1f8) at ./src/imap/imap- > > fetch-body.c: > > > > 615 > > > > #19 0x000055dabc52b5cc in imap_fetch_more_int > > (ctx=0x55dabe26e050, > > > > cancel=false) at ./src/imap/imap-fetch.c:562 > > > > #20 0x000055dabc52b8ad in imap_fetch_more (ctx=0x55dabe26e050, > > > > cmd=0x55dabe26de98) at ./src/imap/imap-fetch.c:617 > > > > #21 0x000055dabc51fd07 in cmd_fetch (cmd=0x55dabe26de98) at ./ > > src/imap/cmd- > > > > fetch.c:382 > > > > #22 0x000055dabc528af4 in command_exec (cmd=0x55dabe26de98) at ./ > > src/imap/imap- > > > > commands.c:201 > > > > #23 0x000055dabc52e9e2 in client_command_input (cmd=<optimized > > out>) at ./src/ > > > > imap/imap-client.c:1237 > > > > #24 0x000055dabc52ea96 in client_command_input (cmd=<optimized > > out>) at ./src/ > > > > imap/imap-client.c:1307 > > > > #25 0x000055dabc52eeed in client_handle_next_command > > (remove_io_r=<synthetic > > > > pointer>, client=0x55dabe26d2c8) at ./src/imap/imap-client.c:1349 > > > > #26 client_handle_input (client=0x55dabe26d2c8) at ./src/imap/ > > imap-client.c: > > > > 1363 > > > > #27 0x000055dabc52f2c4 in client_input (client=0x55dabe26d2c8) at > > ./src/imap/ > > > > imap-client.c:1407 > > > > #28 0x00007fcfb8bfe27d in io_loop_call_io (io=0x55dabe26e660) at > > ../lib/ > > > > ioloop.c:737 > > > > #29 0x00007fcfb8bff81a in io_loop_handler_run_internal > > (ioloop=0x55dabe243fd0) > > > > at ../lib/ioloop-epoll.c:222 > > > > #30 0x00007fcfb8bff8d4 in io_loop_handler_run > > (ioloop=0x55dabe243fd0) at ../ > > > > lib/ioloop.c:789 > > > > #31 0x00007fcfb8bffa90 in io_loop_run (ioloop=0x55dabe243fd0) at > > ../lib/ > > > > ioloop.c:762 > > > > #32 0x00007fcfb8b6ce57 in master_service_run > > (service=0x55dabe243e20, > > > > callback=callback@entry=0x55dabc533210 <client_connected>) at ../ > > lib-master/ > > > > master-service.c:878 > > > > #33 0x000055dabc51ad37 in main (argc=<optimized out>, > > argv=<optimized out>) at > > > > ./src/imap/main.c:575 > > > > > > > > John > > > > > > > > _______________________________________________ > > > > dovecot mailing list -- dovecot@dovecot.org > > > > To unsubscribe send an email to dovecot-le...@dovecot.org > > > _______________________________________________ > > > dovecot mailing list -- dovecot@dovecot.org > > > To unsubscribe send an email to dovecot-le...@dovecot.org > > _______________________________________________ > > dovecot mailing list -- dovecot@dovecot.org > > To unsubscribe send an email to dovecot-le...@dovecot.org > > > > > > > > Amazon Development Center (Netherlands) B.V., Johanna Westerdijkplein > > 1, NL-2521 EN The Hague, Registration No. Chamber of Commerce > > 56869649, VAT: NL 852339859B01 > > _______________________________________________ > > dovecot mailing list -- dovecot@dovecot.org > > To unsubscribe send an email to dovecot-le...@dovecot.org > > > > Amazon Development Center (Netherlands) B.V., Johanna Westerdijkplein 1, > NL-2521 EN The Hague, Registration No. Chamber of Commerce 56869649, VAT: NL > 852339859B01 > > > > See attached script I used. If you get EOF then you've hit the crash. > > John > > > -----Original message----- > From: Aki Tuomi <aki.tu...@open-xchange.com> > Sent: Wednesday, 20th March 2024, 8:40 > To: John van der Kamp <jk...@amazon.nl>; John van der Kamp via > dovecot <dovecot@dovecot.org> > Subject: RE: Crash in dovecot snippet when using imapc > > CAUTION: This email originated from outside of the organization. Do > not click links or open attachments unless you can confirm the sender > and know the content is safe. > > > > Could you provide some simple way to reproduce this, minimal config > etc? > > Aki > > > On 19/03/2024 17:44 EET John van der Kamp via dovecot > <dovecot@dovecot.org> wrote: > > > > > > Hi, sorry for the late reply. > > > > The commit you've pointed at before is the commit introducing code > for the snippets. > > > > Your claim that main is fixed is incorrect: I've bisected through > the git history, and the commit that "fixes" it, is the one flipping > imapc features to negatives: https://github.com/dovecot/core/commit/ > 7810b38d30b7dbb2155f78873fe760bc9e2e6212 <https://github.com/dovecot/ > core/commit/7810b38d30b7dbb2155f78873fe760bc9e2e6212> However, the > default imapc_features value stays the same, so all the "negative" > features are suddenly enabled. > > > > I've reset the defaults in the frontend config to what it was > before: > > > > imapc_features = no-fetch-size no-fetch-headers no-search no-modseq > no-delay-login no-fetch-bodystructure no-acl > > > > and then dovecot starts crashing again in the described scenario. > It is the "no-fetch-size" flag, and if I use "rfc822.size" feature on > a 2.3 branch it stops crashing. > > > > > > Turns out this same feature adds some filter that seems to be meant > for some exchange email side-effect: https://github.com/dovecot/core/ > blob/main/src/lib-storage/index/imapc/imapc-mail-fetch.c#L596 <https: > //github.com/dovecot/core/blob/main/src/lib-storage/index/imapc/ > imapc-mail-fetch.c#L596> where this filter tries to remove any X- > Message-Flag header. This is weird, because it could have been an > normally received header as well as something that was tacked on > later by exchange. > > > > > > The main bug is not fixed by just removing that filter: chaining > filters is probably very broken when using the imapc backend, and it > might be broken in other unknown scenarios. > > > > > > Regards, > > > > > > John > > > > > > > > -----Original message----- > > From: Aki Tuomi via dovecot <dovecot@dovecot.org> > > Sent: Friday, 19th January 2024, 8:37 > > To: Aki Tuomi via dovecot <dovecot@dovecot.org>; John van der Kamp > <jk...@amazon.nl> > > Subject: RE: Crash in dovecot snippet when using imapc > > > > CAUTION: This email originated from outside of the organization. Do > not click links or open attachments unless you can confirm the sender > and know the content is safe. > > > > > > > > Sorry, the provided patch link was wrong, it's already in 2.3.21, > my bad. Anyways, it is still fixed in main, since it does not happen > there. > > > > Aki > > > > > On 19/01/2024 09:13 EET Aki Tuomi via dovecot > <dovecot@dovecot.org> wrote: > > > > > > > > > Hi! > > > > > > I was able to reproduce this issue with 2.3.21, but it seems to > have been fixed in main. I think https://github.com/dovecot/core/ > commit/1c1b77dbf9a548aac788efb76973ce2d0fa6c732.patch will fix this. > > > > > > Aki > > > > > > > On 18/01/2024 22:51 EET John van der Kamp via dovecot > <dovecot@dovecot.org> wrote: > > > > > > > > > > > > Hello, > > > > > > > > > > > > I've found a crash in a very specific setup. A dovecot server > with imapc connection needs to receive an email with no body contents > for the intent of generating a preview/snippet. It crashes somewhere > deep in the jungle of istream and snapshots. I've included a script > which sets up the systems to reproduce the crash. > > > > > > > > > > > > I've tested this with several versions. 2.3.16 doesn't seem to > be affected, but 2.3.20 and 2.3.21 are affect. > > > > > > > > > > > > For me it produces a traceback like this, using the ubuntu > version from here: https://packages.ubuntu.com/noble/dovecot-core > > > > > > > > > > > > (gdb) bt > > > > #0 __pthread_kill_implementation (no_tid=0, signo=6, > threadid=140530132887360) at ./nptl/pthread_kill.c:44 > > > > #1 __pthread_kill_internal (signo=6, threadid=140530132887360) > at ./nptl/pthread_kill.c:78 > > > > #2 __GI___pthread_kill (threadid=140530132887360, > signo=signo@entry=6) at ./nptl/pthread_kill.c:89 > > > > #3 0x00007fcfb8842476 in __GI_raise (sig=sig@entry=6) at ../ > sysdeps/posix/raise.c:26 > > > > #4 0x00007fcfb88287f3 in __GI_abort () at ./stdlib/abort.c:79 > > > > #5 0x00007fcfb8b37fe5 in default_fatal_finish (status=0, > type=LOG_TYPE_PANIC) at ../lib/failures.c:465 > > > > #6 fatal_handler_real (ctx=<optimized out>, format=<optimized > out>, args=<optimized out>) at ../lib/failures.c:477 > > > > #7 0x00007fcfb8be50d7 in i_internal_fatal_handler > (ctx=<optimized out>, format=<optimized out>, args=<optimized out>) > at ../lib/failures.c:879 > > > > #8 0x00007fcfb8b37eea in i_panic (format=0x7fcfb8c29020 "file > %s: line %d (%s): assertion failed: (%s)") at ../lib/failures.c:530 > > > > #9 0x00007fcfb8b3387b in i_stream_header_filter_snapshot_free > (_snapshot=<optimized out>) at ../lib-mail/istream-header-filter.c: > 663 > > > > #10 i_stream_header_filter_snapshot_free > (_snapshot=0x55dabe297a60) at ../lib-mail/istream-header-filter.c:655 > > > > #11 0x00007fcfb8bf25ac in i_stream_snapshot_free > (_snapshot=_snapshot@entry=0x55dabe29b0d0) at ../lib/istream.c:253 > > > > #12 0x00007fcfb8bf2654 in i_stream_unref > (stream=0x7ffc16cc7fa0) at ../lib/istream.c:66 > > > > #13 0x00007fcfb8d96baa in index_mail_write_body_snippet > (mail=0x55dabe292058) at index/index-mail.c:1151 > > > > #14 0x00007fcfb8d97e48 in index_mail_parse_bodystructure > (mail=0x55dabe292058, field=MAIL_CACHE_BODY_SNIPPET) at index/index- > mail.c:1551 > > > > #15 0x00007fcfb8d97fe2 in index_mail_fetch_body_snippet > (value_r=0x7ffc16cc8050, mail=0x55dabe292058) at index/index-mail.c: > 1602 > > > > #16 index_mail_get_special (_mail=0x55dabe292058, > field=<optimized out>, value_r=0x7ffc16cc8050) at index/index-mail.c: > 1730 > > > > #17 0x00007fcfb8d16ffe in mail_get_special > (mail=mail@entry=0x55dabe292058, > field=field@entry=MAIL_FETCH_BODY_SNIPPET, > value_r=value_r@entry=0x7ffc16cc8050) > > > > at /home/ubuntu/dovecot/new/dovecot-2.3.21+dfsg1/src/lib- > storage/mail.c:418 > > > > #18 0x000055dabc52645c in fetch_snippet (ctx=0x55dabe26e050, > mail=0x55dabe292058, preview=0x55dabe28f1f8) at ./src/imap/imap- > fetch-body.c:615 > > > > #19 0x000055dabc52b5cc in imap_fetch_more_int > (ctx=0x55dabe26e050, cancel=false) at ./src/imap/imap-fetch.c:562 > > > > #20 0x000055dabc52b8ad in imap_fetch_more (ctx=0x55dabe26e050, > cmd=0x55dabe26de98) at ./src/imap/imap-fetch.c:617 > > > > #21 0x000055dabc51fd07 in cmd_fetch (cmd=0x55dabe26de98) at ./ > src/imap/cmd-fetch.c:382 > > > > #22 0x000055dabc528af4 in command_exec (cmd=0x55dabe26de98) at > ./src/imap/imap-commands.c:201 > > > > #23 0x000055dabc52e9e2 in client_command_input (cmd=<optimized > out>) at ./src/imap/imap-client.c:1237 > > > > #24 0x000055dabc52ea96 in client_command_input (cmd=<optimized > out>) at ./src/imap/imap-client.c:1307 > > > > #25 0x000055dabc52eeed in client_handle_next_command > (remove_io_r=<synthetic pointer>, client=0x55dabe26d2c8) at ./src/ > imap/imap-client.c:1349 > > > > #26 client_handle_input (client=0x55dabe26d2c8) at ./src/imap/ > imap-client.c:1363 > > > > #27 0x000055dabc52f2c4 in client_input (client=0x55dabe26d2c8) > at ./src/imap/imap-client.c:1407 > > > > #28 0x00007fcfb8bfe27d in io_loop_call_io (io=0x55dabe26e660) > at ../lib/ioloop.c:737 > > > > #29 0x00007fcfb8bff81a in io_loop_handler_run_internal > (ioloop=0x55dabe243fd0) at ../lib/ioloop-epoll.c:222 > > > > #30 0x00007fcfb8bff8d4 in io_loop_handler_run > (ioloop=0x55dabe243fd0) at ../lib/ioloop.c:789 > > > > #31 0x00007fcfb8bffa90 in io_loop_run (ioloop=0x55dabe243fd0) > at ../lib/ioloop.c:762 > > > > #32 0x00007fcfb8b6ce57 in master_service_run > (service=0x55dabe243e20, callback=callback@entry=0x55dabc533210 > <client_connected>) at ../lib-master/master-service.c:878 > > > > #33 0x000055dabc51ad37 in main (argc=<optimized out>, > argv=<optimized out>) at ./src/imap/main.c:575 > > > > > > > > > > > > John > > > > > > > > > > > > > > > > Hello, > > > > > > > > I've found a crash in a very specific setup. A dovecot server > with imapc > > > > connection needs to receive an email with no body contents for > the intent of > > > > generating a preview/snippet. It crashes somewhere deep in the > jungle of > > > > istream and snapshots. I've included a script which sets up the > systems to > > > > reproduce the crash. > > > > > > > > I've tested this with several versions. 2.3.16 doesn't seem to > be affected, but > > > > 2.3.20 and 2.3.21 are affect. > > > > > > > > For me it produces a traceback like this, using the ubuntu > version from here: > > > > https://packages.ubuntu.com/noble/dovecot-core > > > > > > > > (gdb) bt > > > > #0 __pthread_kill_implementation (no_tid=0, signo=6, > threadid=140530132887360) > > > > at ./nptl/pthread_kill.c:44 > > > > #1 __pthread_kill_internal (signo=6, threadid=140530132887360) > at ./nptl/ > > > > pthread_kill.c:78 > > > > #2 __GI___pthread_kill (threadid=140530132887360, > signo=signo@entry=6) at ./ > > > > nptl/pthread_kill.c:89 > > > > #3 0x00007fcfb8842476 in __GI_raise (sig=sig@entry=6) at ../ > sysdeps/posix/ > > > > raise.c:26 > > > > #4 0x00007fcfb88287f3 in __GI_abort () at ./stdlib/abort.c:79 > > > > #5 0x00007fcfb8b37fe5 in default_fatal_finish (status=0, > type=LOG_TYPE_PANIC) > > > > at ../lib/failures.c:465 > > > > #6 fatal_handler_real (ctx=<optimized out>, format=<optimized > out>, > > > > args=<optimized out>) at ../lib/failures.c:477 > > > > #7 0x00007fcfb8be50d7 in i_internal_fatal_handler > (ctx=<optimized out>, > > > > format=<optimized out>, args=<optimized out>) at ../lib/ > failures.c:879 > > > > #8 0x00007fcfb8b37eea in i_panic (format=0x7fcfb8c29020 "file > %s: line %d > > > > (%s): assertion failed: (%s)") at ../lib/failures.c:530 > > > > #9 0x00007fcfb8b3387b in i_stream_header_filter_snapshot_free > > > > (_snapshot=<optimized out>) at ../lib-mail/istream-header- > filter.c:663 > > > > #10 i_stream_header_filter_snapshot_free > (_snapshot=0x55dabe297a60) at ../lib- > > > > mail/istream-header-filter.c:655 > > > > #11 0x00007fcfb8bf25ac in i_stream_snapshot_free > > > > (_snapshot=_snapshot@entry=0x55dabe29b0d0) at ../lib/istream.c: > 253 > > > > #12 0x00007fcfb8bf2654 in i_stream_unref > (stream=0x7ffc16cc7fa0) at ../lib/ > > > > istream.c:66 > > > > #13 0x00007fcfb8d96baa in index_mail_write_body_snippet > (mail=0x55dabe292058) > > > > at index/index-mail.c:1151 > > > > #14 0x00007fcfb8d97e48 in index_mail_parse_bodystructure > (mail=0x55dabe292058, > > > > field=MAIL_CACHE_BODY_SNIPPET) at index/index-mail.c:1551 > > > > #15 0x00007fcfb8d97fe2 in index_mail_fetch_body_snippet > > > > (value_r=0x7ffc16cc8050, mail=0x55dabe292058) at index/index- > mail.c:1602 > > > > #16 index_mail_get_special (_mail=0x55dabe292058, > field=<optimized out>, > > > > value_r=0x7ffc16cc8050) at index/index-mail.c:1730 > > > > #17 0x00007fcfb8d16ffe in mail_get_special > (mail=mail@entry=0x55dabe292058, > > > > field=field@entry=MAIL_FETCH_BODY_SNIPPET, > > > > value_r=value_r@entry=0x7ffc16cc8050) > > > > at /home/ubuntu/dovecot/new/dovecot-2.3.21+dfsg1/src/lib- > storage/mail.c:418 > > > > #18 0x000055dabc52645c in fetch_snippet (ctx=0x55dabe26e050, > > > > mail=0x55dabe292058, preview=0x55dabe28f1f8) at ./src/imap/ > imap-fetch-body.c: > > > > 615 > > > > #19 0x000055dabc52b5cc in imap_fetch_more_int > (ctx=0x55dabe26e050, > > > > > > cancel=false) at ./src/imap/imap-fetch.c:562 > > > > #20 0x000055dabc52b8ad in imap_fetch_more (ctx=0x55dabe26e050, > > > > cmd=0x55dabe26de98) at ./src/imap/imap-fetch.c:617 > > > > #21 0x000055dabc51fd07 in cmd_fetch (cmd=0x55dabe26de98) at ./ > src/imap/cmd- > > > > fetch.c:382 > > > > #22 0x000055dabc528af4 in command_exec (cmd=0x55dabe26de98) at > ./src/imap/imap- > > > > commands.c:201 > > > > #23 0x000055dabc52e9e2 in client_command_input (cmd=<optimized > out>) at ./src/ > > > > imap/imap-client.c:1237 > > > > #24 0x000055dabc52ea96 in client_command_input (cmd=<optimized > out>) at ./src/ > > > > imap/imap-client.c:1307 > > > > #25 0x000055dabc52eeed in client_handle_next_command > (remove_io_r=<synthetic > > > > pointer>, client=0x55dabe26d2c8) at ./src/imap/imap-client.c: > 1349 > > > > #26 client_handle_input (client=0x55dabe26d2c8) at ./src/imap/ > imap-client.c: > > > > 1363 > > > > #27 0x000055dabc52f2c4 in client_input (client=0x55dabe26d2c8) > at ./src/imap/ > > > > imap-client.c:1407 > > > > #28 0x00007fcfb8bfe27d in io_loop_call_io (io=0x55dabe26e660) > at ../lib/ > > > > ioloop.c:737 > > > > #29 0x00007fcfb8bff81a in io_loop_handler_run_internal > (ioloop=0x55dabe243fd0) > > > > at ../lib/ioloop-epoll.c:222 > > > > #30 0x00007fcfb8bff8d4 in io_loop_handler_run > (ioloop=0x55dabe243fd0) at ../ > > > > lib/ioloop.c:789 > > > > #31 0x00007fcfb8bffa90 in io_loop_run (ioloop=0x55dabe243fd0) > at ../lib/ > > > > ioloop.c:762 > > > > #32 0x00007fcfb8b6ce57 in master_service_run > (service=0x55dabe243e20, > > > > callback=callback@entry=0x55dabc533210 <client_connected>) at > ../lib-master/ > > > > master-service.c:878 > > > > #33 0x000055dabc51ad37 in main (argc=<optimized out>, > argv=<optimized out>) at > > > > ./src/imap/main.c:575 > > > > > > > > John > > > > > > > > _______________________________________________ > > > > dovecot mailing list -- dovecot@dovecot.org > > > > To unsubscribe send an email to dovecot-le...@dovecot.org > > > _______________________________________________ > > > dovecot mailing list -- dovecot@dovecot.org > > > To unsubscribe send an email to dovecot-le...@dovecot.org > > _______________________________________________ > > dovecot mailing list -- dovecot@dovecot.org > > To unsubscribe send an email to dovecot-le...@dovecot.org > > > > > > > > Amazon Development Center (Netherlands) B.V., Johanna > Westerdijkplein 1, NL-2521 EN The Hague, Registration No. Chamber of > Commerce 56869649, VAT: NL 852339859B01 > > > > > > > > Hi, sorry for the late reply. > > The commit you've pointed at before is the commit introducing code > for the > > snippets. > > Your claim that main is fixed is incorrect: I've bisected through > the git > > history, and the commit that "fixes" it, is the one flipping imapc > features to > > negatives: https://github.com/dovecot/core/commit/ > > 7810b38d30b7dbb2155f78873fe760bc9e2e6212 However, the default > imapc_features > > value stays the same, so all the "negative" features are suddenly > enabled. > > I've reset the defaults in the frontend config to what it was > before: > > imapc_features = no-fetch-size no-fetch-headers no-search no-modseq > no-delay- > > login no-fetch-bodystructure no-acl > > and then dovecot starts crashing again in the described scenario. > It is the > > "no-fetch-size" flag, and if I use "rfc822.size" feature on a 2.3 > branch it > > stops crashing. > > > > Turns out this same feature adds some filter that seems to be meant > for some > > exchange email side-effect: https://github.com/dovecot/core/blob/ > main/src/lib- > > storage/index/imapc/imapc-mail-fetch.c#L596 where this filter > tries to remove > > any X-Message-Flag header. This is weird, because it could have > been an > > normally received header as well as something that was tacked on > later by > > exchange. > > > > The main bug is not fixed by just removing that filter: chaining > filters is > > probably very broken when using the imapc backend, and it might be > broken in > > other unknown scenarios. > > > > Regards, > > > > John > > > > > > -----Original message----- > > From: Aki Tuomi via dovecot <dovecot@dovecot.org> > > Sent: Friday, 19th January 2024, 8:37 > > To: Aki Tuomi via dovecot <dovecot@dovecot.org>; John van der > Kamp > > <jk...@amazon.nl> > > Subject: RE: Crash in dovecot snippet when using imapc > > > > CAUTION: This email originated from outside of the > organization. Do > > not click links or open attachments unless you can confirm the > sender > > and know the content is safe. > > > > > > > > Sorry, the provided patch link was wrong, it's already in > 2.3.21, my > > bad. Anyways, it is still fixed in main, since it does not > happen > > there. > > > > Aki > > > > > On 19/01/2024 09:13 EET Aki Tuomi via dovecot > <dovecot@dovecot.org> > > wrote: > > > > > > > > > Hi! > > > > > > I was able to reproduce this issue with 2.3.21, but it seems > to > > have been fixed in main. I think https://github.com/dovecot/ > core/ > > commit/1c1b77dbf9a548aac788efb76973ce2d0fa6c732.patch will fix > this. > > > > > > Aki > > > > > > > On 18/01/2024 22:51 EET John van der Kamp via dovecot > > <dovecot@dovecot.org> wrote: > > > > > > > > > > > > Hello, > > > > > > > > > > > > I've found a crash in a very specific setup. A dovecot > server > > with imapc connection needs to receive an email with no body > contents > > for the intent of generating a preview/snippet. It crashes > somewhere > > deep in the jungle of istream and snapshots. I've included a > script > > which sets up the systems to reproduce the crash. > > > > > > > > > > > > I've tested this with several versions. 2.3.16 doesn't > seem to be > > affected, but 2.3.20 and 2.3.21 are affect. > > > > > > > > > > > > For me it produces a traceback like this, using the ubuntu > > version from here: https://packages.ubuntu.com/noble/dovecot- > core > > > > > > > > > > > > (gdb) bt > > > > #0 __pthread_kill_implementation (no_tid=0, signo=6, > > threadid=140530132887360) at ./nptl/pthread_kill.c:44 > > > > #1 __pthread_kill_internal (signo=6, > threadid=140530132887360) > > at ./nptl/pthread_kill.c:78 > > > > #2 __GI___pthread_kill (threadid=140530132887360, > > signo=signo@entry=6) at ./nptl/pthread_kill.c:89 > > > > #3 0x00007fcfb8842476 in __GI_raise (sig=sig@entry=6) at > ../ > > sysdeps/posix/raise.c:26 > > > > #4 0x00007fcfb88287f3 in __GI_abort () at ./stdlib/ > abort.c:79 > > > > #5 0x00007fcfb8b37fe5 in default_fatal_finish (status=0, > > type=LOG_TYPE_PANIC) at ../lib/failures.c:465 > > > > #6 fatal_handler_real (ctx=<optimized out>, > format=<optimized > > out>, args=<optimized out>) at ../lib/failures.c:477 > > > > #7 0x00007fcfb8be50d7 in i_internal_fatal_handler > > (ctx=<optimized out>, format=<optimized out>, args=<optimized > out>) > > at ../lib/failures.c:879 > > > > #8 0x00007fcfb8b37eea in i_panic (format=0x7fcfb8c29020 > "file > > %s: line %d (%s): assertion failed: (%s)") at ../lib/ > failures.c:530 > > > > #9 0x00007fcfb8b3387b in > i_stream_header_filter_snapshot_free > > (_snapshot=<optimized out>) at ../lib-mail/istream-header- > filter.c: > > 663 > > > > #10 i_stream_header_filter_snapshot_free > > (_snapshot=0x55dabe297a60) at ../lib-mail/istream-header- > filter.c:655 > > > > #11 0x00007fcfb8bf25ac in i_stream_snapshot_free > > (_snapshot=_snapshot@entry=0x55dabe29b0d0) at ../lib/ > istream.c:253 > > > > #12 0x00007fcfb8bf2654 in i_stream_unref > (stream=0x7ffc16cc7fa0) > > at ../lib/istream.c:66 > > > > #13 0x00007fcfb8d96baa in index_mail_write_body_snippet > > (mail=0x55dabe292058) at index/index-mail.c:1151 > > > > #14 0x00007fcfb8d97e48 in index_mail_parse_bodystructure > > (mail=0x55dabe292058, field=MAIL_CACHE_BODY_SNIPPET) at index/ > index- > > mail.c:1551 > > > > #15 0x00007fcfb8d97fe2 in index_mail_fetch_body_snippet > > (value_r=0x7ffc16cc8050, mail=0x55dabe292058) at index/index- > mail.c: > > 1602 > > > > #16 index_mail_get_special (_mail=0x55dabe292058, > > field=<optimized out>, value_r=0x7ffc16cc8050) at index/index- > mail.c: > > 1730 > > > > #17 0x00007fcfb8d16ffe in mail_get_special > > (mail=mail@entry=0x55dabe292058, > > field=field@entry=MAIL_FETCH_BODY_SNIPPET, > > value_r=value_r@entry=0x7ffc16cc8050) > > > > at /home/ubuntu/dovecot/new/dovecot-2.3.21+dfsg1/src/ > lib- > > storage/mail.c:418 > > > > #18 0x000055dabc52645c in fetch_snippet > (ctx=0x55dabe26e050, > > mail=0x55dabe292058, preview=0x55dabe28f1f8) at ./src/imap/ > imap- > > fetch-body.c:615 > > > > #19 0x000055dabc52b5cc in imap_fetch_more_int > > (ctx=0x55dabe26e050, cancel=false) at ./src/imap/imap-fetch.c: > 562 > > > > #20 0x000055dabc52b8ad in imap_fetch_more > (ctx=0x55dabe26e050, > > cmd=0x55dabe26de98) at ./src/imap/imap-fetch.c:617 > > > > #21 0x000055dabc51fd07 in cmd_fetch (cmd=0x55dabe26de98) > at ./ > > src/imap/cmd-fetch.c:382 > > > > #22 0x000055dabc528af4 in command_exec > (cmd=0x55dabe26de98) at ./ > > src/imap/imap-commands.c:201 > > > > #23 0x000055dabc52e9e2 in client_command_input > (cmd=<optimized > > out>) at ./src/imap/imap-client.c:1237 > > > > #24 0x000055dabc52ea96 in client_command_input > (cmd=<optimized > > out>) at ./src/imap/imap-client.c:1307 > > > > #25 0x000055dabc52eeed in client_handle_next_command > > (remove_io_r=<synthetic pointer>, client=0x55dabe26d2c8) at ./ > src/ > > imap/imap-client.c:1349 > > > > #26 client_handle_input (client=0x55dabe26d2c8) at ./src/ > imap/ > > imap-client.c:1363 > > > > #27 0x000055dabc52f2c4 in client_input > (client=0x55dabe26d2c8) at > > ./src/imap/imap-client.c:1407 > > > > #28 0x00007fcfb8bfe27d in io_loop_call_io > (io=0x55dabe26e660) at > > ../lib/ioloop.c:737 > > > > #29 0x00007fcfb8bff81a in io_loop_handler_run_internal > > (ioloop=0x55dabe243fd0) at ../lib/ioloop-epoll.c:222 > > > > #30 0x00007fcfb8bff8d4 in io_loop_handler_run > > (ioloop=0x55dabe243fd0) at ../lib/ioloop.c:789 > > > > #31 0x00007fcfb8bffa90 in io_loop_run > (ioloop=0x55dabe243fd0) at > > ../lib/ioloop.c:762 > > > > #32 0x00007fcfb8b6ce57 in master_service_run > > (service=0x55dabe243e20, > callback=callback@entry=0x55dabc533210 > > <client_connected>) at ../lib-master/master-service.c:878 > > > > #33 0x000055dabc51ad37 in main (argc=<optimized out>, > > argv=<optimized out>) at ./src/imap/main.c:575 > > > > > > > > > > > > John > > > > > > > > > > > > > > > > Hello, > > > > > > > > I've found a crash in a very specific setup. A dovecot > server > > with imapc > > > > connection needs to receive an email with no body contents > for > > the intent of > > > > generating a preview/snippet. It crashes somewhere deep in > the > > jungle of > > > > istream and snapshots. I've included a script which sets > up the > > systems to > > > > reproduce the crash. > > > > > > > > I've tested this with several versions. 2.3.16 doesn't > seem to be > > affected, but > > > > 2.3.20 and 2.3.21 are affect. > > > > > > > > For me it produces a traceback like this, using the ubuntu > > version from here: > > > > https://packages.ubuntu.com/noble/dovecot-core > > > > > > > > (gdb) bt > > > > #0 __pthread_kill_implementation (no_tid=0, signo=6, > > threadid=140530132887360) > > > > at ./nptl/pthread_kill.c:44 > > > > #1 __pthread_kill_internal (signo=6, > threadid=140530132887360) > > at ./nptl/ > > > > pthread_kill.c:78 > > > > #2 __GI___pthread_kill (threadid=140530132887360, > > signo=signo@entry=6) at ./ > > > > nptl/pthread_kill.c:89 > > > > #3 0x00007fcfb8842476 in __GI_raise (sig=sig@entry=6) at > ../ > > sysdeps/posix/ > > > > raise.c:26 > > > > #4 0x00007fcfb88287f3 in __GI_abort () at ./stdlib/ > abort.c:79 > > > > #5 0x00007fcfb8b37fe5 in default_fatal_finish (status=0, > > type=LOG_TYPE_PANIC) > > > > at ../lib/failures.c:465 > > > > #6 fatal_handler_real (ctx=<optimized out>, > format=<optimized > > out>, > > > > args=<optimized out>) at ../lib/failures.c:477 > > > > #7 0x00007fcfb8be50d7 in i_internal_fatal_handler > > (ctx=<optimized out>, > > > > format=<optimized out>, args=<optimized out>) at ../lib/ > > failures.c:879 > > > > #8 0x00007fcfb8b37eea in i_panic (format=0x7fcfb8c29020 > "file > > %s: line %d > > > > (%s): assertion failed: (%s)") at ../lib/failures.c:530 > > > > #9 0x00007fcfb8b3387b in > i_stream_header_filter_snapshot_free > > > > (_snapshot=<optimized out>) at ../lib-mail/istream-header- > > filter.c:663 > > > > #10 i_stream_header_filter_snapshot_free > > (_snapshot=0x55dabe297a60) at ../lib- > > > > mail/istream-header-filter.c:655 > > > > #11 0x00007fcfb8bf25ac in i_stream_snapshot_free > > > > (_snapshot=_snapshot@entry=0x55dabe29b0d0) at ../lib/ > istream.c: > > 253 > > > > #12 0x00007fcfb8bf2654 in i_stream_unref > (stream=0x7ffc16cc7fa0) > > at ../lib/ > > > > istream.c:66 > > > > #13 0x00007fcfb8d96baa in index_mail_write_body_snippet > > (mail=0x55dabe292058) > > > > at index/index-mail.c:1151 > > > > #14 0x00007fcfb8d97e48 in index_mail_parse_bodystructure > > (mail=0x55dabe292058, > > > > field=MAIL_CACHE_BODY_SNIPPET) at index/index-mail.c:1551 > > > > #15 0x00007fcfb8d97fe2 in index_mail_fetch_body_snippet > > > > (value_r=0x7ffc16cc8050, mail=0x55dabe292058) at index/ > index- > > mail.c:1602 > > > > #16 index_mail_get_special (_mail=0x55dabe292058, > > field=<optimized out>, > > > > value_r=0x7ffc16cc8050) at index/index-mail.c:1730 > > > > #17 0x00007fcfb8d16ffe in mail_get_special > > (mail=mail@entry=0x55dabe292058, > > > > field=field@entry=MAIL_FETCH_BODY_SNIPPET, > > > > value_r=value_r@entry=0x7ffc16cc8050) > > > > at /home/ubuntu/dovecot/new/dovecot-2.3.21+dfsg1/src/ > lib- > > storage/mail.c:418 > > > > #18 0x000055dabc52645c in fetch_snippet > (ctx=0x55dabe26e050, > > > > mail=0x55dabe292058, preview=0x55dabe28f1f8) at ./src/ > imap/imap- > > fetch-body.c: > > > > 615 > > > > #19 0x000055dabc52b5cc in imap_fetch_more_int > > (ctx=0x55dabe26e050, > > > > cancel=false) at ./src/imap/imap-fetch.c:562 > > > > #20 0x000055dabc52b8ad in imap_fetch_more > (ctx=0x55dabe26e050, > > > > cmd=0x55dabe26de98) at ./src/imap/imap-fetch.c:617 > > > > #21 0x000055dabc51fd07 in cmd_fetch (cmd=0x55dabe26de98) > at ./ > > src/imap/cmd- > > > > fetch.c:382 > > > > #22 0x000055dabc528af4 in command_exec > (cmd=0x55dabe26de98) at ./ > > src/imap/imap- > > > > commands.c:201 > > > > #23 0x000055dabc52e9e2 in client_command_input > (cmd=<optimized > > out>) at ./src/ > > > > imap/imap-client.c:1237 > > > > #24 0x000055dabc52ea96 in client_command_input > (cmd=<optimized > > out>) at ./src/ > > > > imap/imap-client.c:1307 > > > > #25 0x000055dabc52eeed in client_handle_next_command > > (remove_io_r=<synthetic > > > > pointer>, client=0x55dabe26d2c8) at ./src/imap/imap- > client.c:1349 > > > > #26 client_handle_input (client=0x55dabe26d2c8) at ./src/ > imap/ > > imap-client.c: > > > > 1363 > > > > #27 0x000055dabc52f2c4 in client_input > (client=0x55dabe26d2c8) at > > ./src/imap/ > > > > imap-client.c:1407 > > > > #28 0x00007fcfb8bfe27d in io_loop_call_io > (io=0x55dabe26e660) at > > ../lib/ > > > > ioloop.c:737 > > > > #29 0x00007fcfb8bff81a in io_loop_handler_run_internal > > (ioloop=0x55dabe243fd0) > > > > at ../lib/ioloop-epoll.c:222 > > > > #30 0x00007fcfb8bff8d4 in io_loop_handler_run > > (ioloop=0x55dabe243fd0) at ../ > > > > lib/ioloop.c:789 > > > > #31 0x00007fcfb8bffa90 in io_loop_run > (ioloop=0x55dabe243fd0) at > > ../lib/ > > > > ioloop.c:762 > > > > #32 0x00007fcfb8b6ce57 in master_service_run > > (service=0x55dabe243e20, > > > > callback=callback@entry=0x55dabc533210 <client_connected>) > at ../ > > lib-master/ > > > > master-service.c:878 > > > > #33 0x000055dabc51ad37 in main (argc=<optimized out>, > > argv=<optimized out>) at > > > > ./src/imap/main.c:575 > > > > > > > > John > > > > > > > > _______________________________________________ > > > > dovecot mailing list -- dovecot@dovecot.org > > > > To unsubscribe send an email to dovecot-le...@dovecot.org > > > _______________________________________________ > > > dovecot mailing list -- dovecot@dovecot.org > > > To unsubscribe send an email to dovecot-le...@dovecot.org > > _______________________________________________ > > dovecot mailing list -- dovecot@dovecot.org > > To unsubscribe send an email to dovecot-le...@dovecot.org > > > > > > > > Amazon Development Center (Netherlands) B.V., Johanna > Westerdijkplein > > 1, NL-2521 EN The Hague, Registration No. Chamber of Commerce > > 56869649, VAT: NL 852339859B01 > > _______________________________________________ > > dovecot mailing list -- dovecot@dovecot.org > > To unsubscribe send an email to dovecot-le...@dovecot.org > > > > Amazon Development Center (Netherlands) B.V., Johanna Westerdijkplein > 1, NL-2521 EN The Hague, Registration No. Chamber of Commerce > 56869649, VAT: NL 852339859B01 > _______________________________________________ > dovecot mailing list -- dovecot@dovecot.org > To unsubscribe send an email to dovecot-le...@dovecot.org _______________________________________________ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
