i am out of ideas, customer complaint that two messages
with a pdf invoice are broken and i really don't know
that to answer - i can't tell a customer that get
incoming mails undamaged is some sort of luck
that's exactly the reconstruct problem "--" without mime-idDate: Mon, 21 Jul 2014 10:43:17 +0200 Date: Mon, 21 Jul 2014 10:43:33 +0200 _______________________________________________________ -- Content-Type: text/html; -- Content-Type: application/pdf; _______________________________________________________ Am 20.07.2014 20:56, schrieb Reindl Harald: > Am 20.07.2014 20:47, schrieb Paul J Stevens: >> If you can, test the master-HEAD. It contains all coverity fixes I >> could find. I will not run coverity against 3.1 > > the problem is that i can't push MASTER to production and the > problems are affecting dbmail 3.1 stable > > i did not mean fix anything coverity lists but maybe > with the diffs what happening to messages coverity > can give a educated hint > > since when it happens it's always the same result > having lines with "--" instead "-- mimeid" it is > likely a single specific bug :-( > >> On 20-07-14 19:06, Reindl Harald wrote: >>> 77d17ca7724c32dba27eb187d4abfa129ccf6c4e is *way better* but still >>> not error free, can you please fire coverity again because i think >>> it's more effective than me building and testing a lot of versions >>> over hours - smells like one remaining buffer overflow / underrun >> >>> * "Ostrava Tuesday morning" no longer broken * "Panel: E-Mail-Debug >>> - IMAP-POP3-Download" 4 out of 1000 broken * i have attached the 4 >>> diff-files * all 4 still always identical broken * two times POP3 * >>> two times IMAP >> >>> additional info: >> >>> * "Ostrava Tuesday morning" with 23,40 KB *no error* on 100 loops * >>> "Panel: E-Mail-Debug - IMAP-POP3-Download" is 380 KB large >> >>> so it seems larger messages are more likely to reprocude the >>> problem, but that's a error rate which let me put >>> 77d17ca7724c32dba27eb187d4abfa129ccf6c4e in production because >>> better than current 8a042214ae1d120581740020f4e73c3cf8d3a6c0 >>> ___________________________________________________ >> >>> my stress-test complained within 25 minutes 5 times >> >>> "multipart_message_3", "message_rfc822" and "multipart_message_2" >>> are from the dbmail-testsuite, "Test 5" is one of my past problem >>> cases >> >>> Jul 20 18:37:13 POP3 24: EXPECTED: >>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND: >>> 97039dbf12052e527808ca16188971a405fcb529 (multipart_message_3) >> >>> Jul 20 18:41:48 POP3 28: EXPECTED: >>> 60dd43bb886f6c47d89b5dc3ab3f7f764b68eb2e / FOUND: >>> b99c389d3479d8d407d43045bcd99219e51a8977 (message_rfc822) >> >>> Jul 20 18:45:53 POP3 25: EXPECTED: >>> 0e2409bcf6921c5da801e1bf4ca4f57be9f07775 / FOUND: >>> 3ef1aef632e08eef66d77e5a97091797640f3192 (multipart_message_2) >> >>> Jul 20 18:47:00 POP3 01: EXPECTED: >>> 04073992ce1c1e75d9bd88e0388486dcc80df627 / FOUND: >>> af6836092d12edbb8be1396ad5af08c804fad2fb (Test 5) >> >>> Jul 20 18:54:47 POP3 01: EXPECTED: >>> 04073992ce1c1e75d9bd88e0388486dcc80df627 / FOUND: >>> af6836092d12edbb8be1396ad5af08c804fad2fb (Test 5) >> >>> Am 20.07.2014 17:36, schrieb Reindl Harald (mobile): >>>> I will give it a try after back at home >>>> >>>> if the other reconstruct problem under load still exists i try to >>>> get back release version to release version until it is away and >>>> want the commits after upwards >>>> >>>> that sort of regressions was solved for sure because we both >>>> spent a lot of hours last year and finally there where load tests >>>> over hours without a single mismatch >>>> >>>> However, if we really find a guilty commit the interesting >>>> question will be what assumption leaded to that exactly code and >>>> how to prevent similar bugs - Autotest are great but can't replay >>>> real world mixed load, even my stress tests are synthetic at the >>>> end of the day and can't really simulate 50 different clients at >>>> the same time acting on completely different messages and what no >>>> test ever will reproduce is the memory state with possible leaks >>>> part of the game after several days of running >>>> >>>> the race bug on the other hand seems to be worst after restart >>>> the service - very strange >>>> >>>> >>>> -------- Ursprüngliche Nachricht -------- Von: Paul J Stevens >>>> <p...@nfg.nl> Gesendet: 20. Juli 2014 17:00:15 MESZ An: DBMail >>>> mailinglist <dbmail@dbmail.org> Betreff: Re: [Dbmail] 3.1.16 >>>> regression -> Re: DBMail 3.1.16 released (Re: corrupted >>>> multi-mime messages) >>>> >>>> I can't believe this passed the tests, since this exact message >>>> is part of the test-suite. Seems the test itself was broken :-( >>>> >>>> I just pushed a fix. *Please* test! >>>> >>>> On 20-07-14 14:12, Reindl Harald wrote: >>>>> CAUTION >>>> >>>>> 3.1.16 at least has a regression in case of the commit below >>>>> that's a different problem and the testmessage with downgrading >>>>> is reconstructed proper >>>>> http://git.dbmail.eu/paul/dbmail/commit/?h=dbmail_3_1&id=8a042214ae1d120581740020f4e73c3cf8d3a6c0 >>>> >>>>> >>>>> >> however, the mime-part-id problem is still the same under load >>>> >>>>> i sent Paul some debug data offlist because they contain real >>>>> messages i am allowed by the mailowner to forward only that >>>>> way, not in public >>>> >>>>> Regards Harry >>>> >>>>> Am 20.07.2014 13:38, schrieb Reindl Harald: >>>>>> *attached* a compressed example *diff* broken / non-broken >>>>>> copy while compare the same mail IMAP / POP3 in a loop >>>>>> >>>>>> 33 out of 1000 always identically broken >>>>>> >>>>>> it don't matter on which test message i start the loop the >>>>>> result is the same in case of multi-mime mails >>>>>> >>>>>> fact is there are times while running mixed load for a minute >>>>>> not a single one get broken and in random time windows any >>>>>> multimime message for some seconds is damaged the same way >>>>>> _______________________________________________________________ >>>>>> >>>>>> >>>>>> >> that below is the temp-folder where the loop stores diffs look at >>>>>> the number at the end, that's the loop-comuter >>>>>> >>>>>> at the begin of the loop a horrible result with 19 broken >>>>>> copies in serie and later 118-125 are again 8 in serie and >>>>>> they all have the same difference as "example-diff.zip" >>>>>> >>>>>> the mixed load now get "multipart_message_3" instead >>>>>> "multipart_message_6" most of the time broken >>>>>> _______________________________________________________________ >>>>>> >>>>>> >> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_002.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_003.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_004.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_005.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_006.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_007.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_009.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_010.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_011.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_012.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_013.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_014.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_015.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_016.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_017.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_018.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_019.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_020.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_021.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_084.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_089.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_118.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_119.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_120.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_121.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_122.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_123.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_124.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_125.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_335.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_364.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_380.diff >>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_707.diff >>>>>> _______________________________________________ >>>>>> >>>>>> Jul 20 13:26:56 POP3 33: EXPECTED: >>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >>>>>> morning) Jul 20 13:26:56 IMAP 24: EXPECTED: >>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND: >>>>>> 97039dbf12052e527808ca16188971a405fcb529 >>>>>> (multipart_message_3) Jul 20 13:26:56 IMAP 33: EXPECTED: >>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >>>>>> morning) Jul 20 13:27:23 POP3 33: EXPECTED: >>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >>>>>> morning) Jul 20 13:27:23 IMAP 24: EXPECTED: >>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND: >>>>>> 97039dbf12052e527808ca16188971a405fcb529 >>>>>> (multipart_message_3) Jul 20 13:27:23 IMAP 33: EXPECTED: >>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >>>>>> morning) Jul 20 13:27:52 POP3 33: EXPECTED: >>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >>>>>> morning) Jul 20 13:27:52 IMAP 24: EXPECTED: >>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND: >>>>>> 97039dbf12052e527808ca16188971a405fcb529 >>>>>> (multipart_message_3) Jul 20 13:27:52 IMAP 33: EXPECTED: >>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >>>>>> morning) Jul 20 13:28:15 POP3 33: EXPECTED: >>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >>>>>> morning) Jul 20 13:28:15 IMAP 24: EXPECTED: >>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND: >>>>>> 97039dbf12052e527808ca16188971a405fcb529 >>>>>> (multipart_message_3) Jul 20 13:28:15 IMAP 33: EXPECTED: >>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND: >>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday >>>>>> morning) Am 20.07.2014 12:56, schrieb Reindl Harald: >>>>>>> Hi Paul >>>>>>> >>>>>>> sorry but still unconfirmed >>>>>>> >>>>>>> see attached screenshot of debug-interface as well as >>>>>>> "stress-test.txt" which normally should not output >>>>>>> anything because it prcatically makes a lot of noise while >>>>>>> check again and aagin the same messages against known >>>>>>> checksums >>>>>>> >>>>>>> the "Ostrava Tuesday morning" and "multipart_message_6" >>>>>>> which is from the dbmail-autotests imported on my IMAP >>>>>>> seems to get most likely broken, but others to randomly >>>>>>> >>>>>>> "Ostrava Tuesday morning is the one with context to >>>>>>> "8a042214ae1d120581740020f4e73c3cf8d3a6c0" and was the >>>>>>> first customer complaint >>>>>>> >>>>>>> Regards Harry >>>>>>> >>>>>>> Am 20.07.2014 12:30, schrieb Paul J Stevens: >>>>>>>> It was a buffer overrun which has now been fixed to the >>>>>>>> best of my abilities. >>>>>>>> >>>>>>>> I've done another round of coverity analysis on the >>>>>>>> master branch, and this doesn't trigger any new warnings >>>>>>>> with regard to mentioned fix which is also in the >>>>>>>> dbmail_3_1 branch. >>>>>>>> >>>>>>>> I've just tagged and thereby released 3.1.16 which >>>>>>>> contains just this fix. >>>>>>>> >>>>>>>> Being on vacation, I don't have full access to my usual >>>>>>>> release scripts (home-network is unreachable), so a tag >>>>>>>> will have to do. >>>>>>>> >>>>>>>> But I've also uploaded the tar file seperately to >>>>>>>> www.dbmail.org, so all the usual download locations >>>>>>>> should work. >>>>>>>> >>>>>>>> On 18-07-14 23:05, Reindl Harald (mobile) wrote: >>>>>>>>> Interesting - your attachment showed exactly the same >>>>>>>>> bug I have only randomly for whatever type of messages >>>>>>>>> and only under load >>>>>>>>> >>>>>>>>> I would sell my soul for a predictable reproducer >>>>>>>>> because in that case debug logging / gdb / strace could >>>>>>>>> shed some light and since it's always the same want >>>>>>>>> finally happens it's pretty sure only a few lines of >>>>>>>>> code in dbmail to fix it >>>>>>>>> >>>>>>>>> I am still sure it's exactly the same code path in both >>>>>>>>> of our cases and only the trigger to went that path is >>>>>>>>> different >>>>>>>>> >>>>>>>>> Maybe Paul has some idea how you can provide debug >>>>>>>>> info's since you are able to isolate what happens while >>>>>>>>> my trigger ends in a large mess of mixed debug logs >>>>>>>>> >>>>>>>>> >>>>>>>>> -------- Ursprüngliche Nachricht -------- Von: >>>>>>>>> Guilherme Souza <gslso...@gmail.com> Gesendet: 18. Juli >>>>>>>>> 2014 21:20:35 MESZ An: DBMail mailinglist >>>>>>>>> <dbmail@dbmail.org> Betreff: Re: [Dbmail] corrupted >>>>>>>>> multi-mime messages >>>>>>>>> >>>>>>>>> I was no able to reply to your message. In my case is >>>>>>>>> not random. >>>>>>>>> >>>>>>>>> Every message html formatted with an attached file sent >>>>>>>>> by ensignia webmail shows the same missing boundaries. >>>>>>>>> >>>>>>>>> Every message sent by roundcube or thunderbird shows >>>>>>>>> correctly. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jul 18, 2014 at 2:31 PM, Reindl Harald (mobile) >>>>>>>>> <h.rei...@thelounge.net> wrote: >>>>>>>>>> That is exactly what i reported a few days ago - >>>>>>>>>> that's for sure a race condition in reconstruction >>>>>>>>>> because i get predictable identical broken messages >>>>>>>>>> in a loop while mixed load on my testserver - at the >>>>>>>>>> same machine 1000 loops receive the same message >>>>>>>>>> without other clients no broken one
signature.asc
Description: OpenPGP digital signature
_______________________________________________ DBmail mailing list DBmail@dbmail.org http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
