I will give it a try after back at home if the other reconstruct problem under load still exists i try to get back release version to release version until it is away and want the commits after upwards
that sort of regressions was solved for sure because we both spent a lot of hours last year and finally there where load tests over hours without a single mismatch However, if we really find a guilty commit the interesting question will be what assumption leaded to that exactly code and how to prevent similar bugs - Autotest are great but can't replay real world mixed load, even my stress tests are synthetic at the end of the day and can't really simulate 50 different clients at the same time acting on completely different messages and what no test ever will reproduce is the memory state with possible leaks part of the game after several days of running the race bug on the other hand seems to be worst after restart the service - very strange -------- Ursprüngliche Nachricht -------- Von: Paul J Stevens <p...@nfg.nl> Gesendet: 20. Juli 2014 17:00:15 MESZ An: DBMail mailinglist <dbmail@dbmail.org> Betreff: Re: [Dbmail] 3.1.16 regression -> Re: DBMail 3.1.16 released (Re: corrupted multi-mime messages) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I can't believe this passed the tests, since this exact message is part of the test-suite. Seems the test itself was broken :-( I just pushed a fix. *Please* test! On 20-07-14 14:12, Reindl Harald wrote: > CAUTION > > 3.1.16 at least has a regression in case of the commit below that's > a different problem and the testmessage with downgrading is > reconstructed proper > http://git.dbmail.eu/paul/dbmail/commit/?h=dbmail_3_1&id=8a042214ae1d120581740020f4e73c3cf8d3a6c0 > > however, the mime-part-id problem is still the same under load > > i sent Paul some debug data offlist because they contain real > messages i am allowed by the mailowner to forward only that way, > not in public > > Regards Harry > > Am 20.07.2014 13:38, schrieb Reindl Harald: >> *attached* a compressed example *diff* broken / non-broken copy >> while compare the same mail IMAP / POP3 in a loop >> >> 33 out of 1000 always identically broken >> >> it don't matter on which test message i start the loop the result >> is the same in case of multi-mime mails >> >> fact is there are times while running mixed load for a minute not >> a single one get broken and in random time windows any multimime >> message for some seconds is damaged the same way >> _______________________________________________________________ >> >> that below is the temp-folder where the loop stores diffs look at >> the number at the end, that's the loop-comuter >> >> at the begin of the loop a horrible result with 19 broken copies >> in serie and later 118-125 are again 8 in serie and they all have >> the same difference as "example-diff.zip" >> >> the mixed load now get "multipart_message_3" instead >> "multipart_message_6" most of the time broken >> _______________________________________________________________ >> >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_002.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_003.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_004.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_005.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_006.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_007.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_009.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_010.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_011.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_012.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_013.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_014.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_015.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_016.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_017.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_018.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_019.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_020.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_021.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_084.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_089.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_118.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_119.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_120.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_121.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_122.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_123.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_124.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_125.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_335.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_364.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_380.diff >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_707.diff >> _______________________________________________ >> >> Jul 20 13:26:56 POP3 33: EXPECTED: >> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >> morning) Jul 20 13:26:56 IMAP 24: EXPECTED: >> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND: >> 97039dbf12052e527808ca16188971a405fcb529 (multipart_message_3) >> Jul 20 13:26:56 IMAP 33: EXPECTED: >> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >> morning) Jul 20 13:27:23 POP3 33: EXPECTED: >> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >> morning) Jul 20 13:27:23 IMAP 24: EXPECTED: >> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND: >> 97039dbf12052e527808ca16188971a405fcb529 (multipart_message_3) >> Jul 20 13:27:23 IMAP 33: EXPECTED: >> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >> morning) Jul 20 13:27:52 POP3 33: EXPECTED: >> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >> morning) Jul 20 13:27:52 IMAP 24: EXPECTED: >> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND: >> 97039dbf12052e527808ca16188971a405fcb529 (multipart_message_3) >> Jul 20 13:27:52 IMAP 33: EXPECTED: >> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >> morning) Jul 20 13:28:15 POP3 33: EXPECTED: >> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >> morning) Jul 20 13:28:15 IMAP 24: EXPECTED: >> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND: >> 97039dbf12052e527808ca16188971a405fcb529 (multipart_message_3) >> Jul 20 13:28:15 IMAP 33: EXPECTED: >> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >> morning) Am 20.07.2014 12:56, schrieb Reindl Harald: >>> Hi Paul >>> >>> sorry but still unconfirmed >>> >>> see attached screenshot of debug-interface as well as >>> "stress-test.txt" which normally should not output anything >>> because it prcatically makes a lot of noise while check again >>> and aagin the same messages against known checksums >>> >>> the "Ostrava Tuesday morning" and "multipart_message_6" which >>> is from the dbmail-autotests imported on my IMAP seems to get >>> most likely broken, but others to randomly >>> >>> "Ostrava Tuesday morning is the one with context to >>> "8a042214ae1d120581740020f4e73c3cf8d3a6c0" and was the first >>> customer complaint >>> >>> Regards Harry >>> >>> Am 20.07.2014 12:30, schrieb Paul J Stevens: >>>> It was a buffer overrun which has now been fixed to the best >>>> of my abilities. >>>> >>>> I've done another round of coverity analysis on the master >>>> branch, and this doesn't trigger any new warnings with regard >>>> to mentioned fix which is also in the dbmail_3_1 branch. >>>> >>>> I've just tagged and thereby released 3.1.16 which contains >>>> just this fix. >>>> >>>> Being on vacation, I don't have full access to my usual >>>> release scripts (home-network is unreachable), so a tag will >>>> have to do. >>>> >>>> But I've also uploaded the tar file seperately to >>>> www.dbmail.org, so all the usual download locations should >>>> work. >>>> >>>> On 18-07-14 23:05, Reindl Harald (mobile) wrote: >>>>> Interesting - your attachment showed exactly the same bug I >>>>> have only randomly for whatever type of messages and only >>>>> under load >>>>> >>>>> I would sell my soul for a predictable reproducer because >>>>> in that case debug logging / gdb / strace could shed some >>>>> light and since it's always the same want finally happens >>>>> it's pretty sure only a few lines of code in dbmail to fix >>>>> it >>>>> >>>>> I am still sure it's exactly the same code path in both of >>>>> our cases and only the trigger to went that path is >>>>> different >>>>> >>>>> Maybe Paul has some idea how you can provide debug info's >>>>> since you are able to isolate what happens while my trigger >>>>> ends in a large mess of mixed debug logs >>>>> >>>>> >>>>> -------- Ursprüngliche Nachricht -------- Von: Guilherme >>>>> Souza <gslso...@gmail.com> Gesendet: 18. Juli 2014 21:20:35 >>>>> MESZ An: DBMail mailinglist <dbmail@dbmail.org> Betreff: >>>>> Re: [Dbmail] corrupted multi-mime messages >>>>> >>>>> I was no able to reply to your message. In my case is not >>>>> random. >>>>> >>>>> Every message html formatted with an attached file sent by >>>>> ensignia webmail shows the same missing boundaries. >>>>> >>>>> Every message sent by roundcube or thunderbird shows >>>>> correctly. >>>>> >>>>> >>>>> On Fri, Jul 18, 2014 at 2:31 PM, Reindl Harald (mobile) >>>>> <h.rei...@thelounge.net> wrote: >>>>>> That is exactly what i reported a few days ago - that's >>>>>> for sure a race condition in reconstruction because i get >>>>>> predictable identical broken messages in a loop while >>>>>> mixed load on my testserver - at the same machine 1000 >>>>>> loops receive the same message without other clients no >>>>>> broken one > > > > > _______________________________________________ DBmail mailing > list DBmail@dbmail.org > http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail > - -- ________________________________________________________________ Paul J Stevens pjstevns @ gmail, twitter, github, linkedin www.nfg.nl/i...@nfg.nl/+31.85.877.99.97 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlPL2X8ACgkQ8iITvBH4zTGatACePuAvKXmihNJxvLbq2l66If3j XV4An0CldZCEP3L29ubcrUry6A4K1ru2 =0rVS -----END PGP SIGNATURE----- _______________________________________________ DBmail mailing list DBmail@dbmail.org http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail -- Reindl Harald (mobile) the lounge interactive design GmbH A-1060 Vienna, Hofmühlgasse 17 CTO / CISO / Software-Development +43 (676) 40 221 40 http://www.thelounge.net _______________________________________________ DBmail mailing list DBmail@dbmail.org http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
