Hello, as this gets quite long a short overiew in advance: After upgrading to exim 4.90_1 (Ubuntu 18.04), I observe message corruption in the queue in rare cases. All corrupted messages are from linux kernel mailing list. I can eliminate file system as reason for corruption.
Whole story: Since updating exim to version 4.90_1 (Ubuntu upgrade to 18.04), in rare cases (less than 1 per 10000 messages deliveries) messages get stuck in the queue (with message "spool format error: size=6045" when I list the queue). It seems, that files in /var/spool/exim4/input get corrupted (header and body parts get mixed up). I can eliminate filesystems corruption as root cause (no other indication for file corruption, filesystem checks w/o errors). So far I only observe corruption on linux kernel mailing list posts (example is attached), all are patch messages (mailers are git-send-email in various versions and tip-git-log-daemon). --cut exim mainlog excerpt: 2018-05-31 12:47:37.521 [4547] 1fOL7J-0001BL-DC SA: Debug: SAEximRunCond expand returned: '1' 2018-05-31 12:47:37.522 [4547] 1fOL7J-0001BL-DC SA: Debug: check succeeded, running spamc 2018-05-31 12:47:39.330 [4547] 1fOL7J-0001BL-DC SA: Action: scanned but message isn't spam: score=-15.2 required=5.0 (scanned in 2/2 secs | Message-Id: [email protected]). From <[email protected]> (host=vger.kernel.org [209.132.180.67]) for linux-kernel@XXX 2018-05-31 12:47:39.368 [4547] 1fOL7J-0001BL-DC <= [email protected] H=vger.kernel.org [209.132.180.67]:37639 I=[1.2.3.4]:25 P=esmtp K S=7580 M8S=8 [email protected] from <[email protected]> for XXX 2018-05-31 12:47:39.381 [4612] 1fOL7J-0001BL-DC Format error in spool file 1fOL7J-0001BL-DC-H: size=6763 --cut --cut mail.log excerpt: May 31 12:47:37 deep-thought spamd[4493]: spamd: processing message <[email protected]> for Debian-exim:116 May 31 12:47:39 deep-thought spamd[4493]: spamd: result: . -15 - BAYES_00,GREYLIST_ISWHITE,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI scantime=1.8,size=5135,user=Debian-exim,uid=116,required_score=5.0,rhost=localhost.localdomain,raddr=127.0.0.1,rport=36468,mid=<[email protected]>,bayes=0.000000,autolearn=ham autolearn_force=no --cut I run the Ubuntus exim4-daemon-heavy flavor with sa_exim for greylisting/spamassassin integration and ClamAV for basic malware scanning. SPF checking is enabled. All corrupted messages were accepted without errors or warnings in exim and other (ClamAV, spamassassin, ...) logs. No external message altering in /var/spool/exim4/input (where corrupted messages get stuck). --cut "exim4 --version" Exim version 4.90_1 #2 built 14-Feb-2018 16:01:14 Copyright (c) University of Cambridge, 1995 - 2017 (c) The Exim Maintainers and contributors in ACKNOWLEDGMENTS file, 2007 - 2017 Berkeley DB: Berkeley DB 5.3.28: (September 9, 2013) Support for: crypteq iconv() IPv6 PAM Perl Expand_dlfunc GnuTLS move_frozen_messages Content_Scanning DKIM DNSSEC Event OCSP PRDR PROXY SOCKS TCP_Fast_Open Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmjz dbmnz dnsdb dsearch ldap ldapdn ldapm mysql nis nis0 passwd pgsql sqlite Authenticators: cram_md5 cyrus_sasl dovecot plaintext spa tls Routers: accept dnslookup ipliteral iplookup manualroute queryprogram redirect Transports: appendfile/maildir/mailstore/mbx autoreply lmtp pipe smtp Fixed never_users: 0 Configure owner: 0:0 Size of off_t: 8 Configuration file is /var/lib/exim4/config.autogenerated --cut Any hints comments highly appreciated, I can provide more examples or details if necessary. Best regards, Thomas
1fOL7J-0001BL-DC-D Commit-ID: 18a7057420f8b67f15d17087bf5c0863db752c8b Gitweb: https://git.kernel.org/tip/18a7057420f8b67f15d17087bf5c0863db752c8b Author: Arnaldo Carvalho de Melo <[email protected]> AuthorDate: Fri, 25 May 2018 16:37:36 -0300 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Wed, 30 May 2018 15:40:26 -0300 perf tools: Fix perf.data format description of NRCPUS header In the perf.data HEADER_CPUDESC feadure header we store first the number of available CPUs in the system, then the number of CPUs at the time of writing the header, not the other way around. Reported-by: Thomas-Mich Richter <[email protected]> Acked-by: Andi Kleen <[email protected]> Cc: Adrian Hunter <[email protected]> Cc: David Ahern <[email protected]> Cc: He Kuang <[email protected]> Cc: Hendrik Brueckner <[email protected]> Cc: Jin Yao <[email protected]> Cc: Jiri Olsa <[email protected]> Cc: Kim Phillips <[email protected]> Cc: Lakshman Annadorai <[email protected]> Cc: Namhyung Kim <[email protected]> Cc: Simon Que <[email protected]> Cc: Stephane Eranian <[email protected]> Cc: Wang Nan <[email protected]> Link: https://lkml.kernel.org/n/[email protected] Signed-off-by: Arnaldo Carvalho de Melo <[email protected]> --- tools/perf/Documentation/perf.data-file-format.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/perf/Documentation/perf.data-file-format.txt b/tools/perf/Documentation/perf.data-file-format.txt index c57904a526ce..dfb218feaad9 100644 --- a/tools/perf/Documentation/perf.data-file-format.txt +++ b/tools/perf/Documentation/perf.data-file-format.txt @@ -111,8 +111,8 @@ A perf_header_string with the CPU architecture (uname -m) A structure defining the number of CPUs. struct nr_cpus { - uint32_t nr_cpus_online; uint32_t nr_cpus_available; /* CPUs not yet onlined */ + uint32_t nr_cpus_online; }; HEADER_CPUDESC = 8,
1fOL7J-0001BL-DC-H Debian-exim 116 123 <[email protected]> 1527763657 0 -received_time_usec .409222 -helo_name vger.kernel.org -host_address 209.132.180.67.37639 -host_name vger.kernel.org -interface_address 1.2.3.4.25 -active_hostname XXX -received_protocol esmtp -body_linecount 47 -max_received_linelength 116 -deliver_firsttime XX 1 linux-kernel@XXX 240P Received: from vger.kernel.org ([209.132.180.67]:37639) by deep-thought with esmtp (Exim 4.90_1) (envelope-from <[email protected]>) id 1fOL7J-0001BL-DC for linux-kernel@XXX; Thu, 31 May 2018 12:47:39 +0200 187P Received: ([email protected]) by vger.kernel.org via listexpand id S1754462AbeEaKqZ (ORCPT <rfc822;linux-kernel@XXX>); Thu, 31 May 2018 06:46:25 -0400 271P Received: from terminus.zytor.com ([198.137.202.136]:48299 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754562AbeEaKqX (ORCPT <rfc822;[email protected]>); Thu, 31 May 2018 06:46:23 -0400 256P Received: from terminus.zytor.com (localhost [127.0.0.1]) by terminus.zytor.com (8.15.2/8.15.2) with ESMTPS id w4VAk69D3525797 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 31 May 2018 03:46:06 -0700 147P Received: (from tipbot@localhost) by terminus.zytor.com (8.15.2/8.15.2/Submit) id w4VAk6Qv3525794; Thu, 31 May 2018 03:46:06 -0700 040 Date: Thu, 31 May 2018 03:46:06 -0700 093 X-Authentication-Warning: terminus.zytor.com: tipbot set sender to [email protected] using -f 064F From: tip-bot for Arnaldo Carvalho de Melo <[email protected]> 058I Message-ID: <[email protected]> 460C Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] 484R Reply-To: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] 042T To: [email protected] 090* Subject: [tip:perf/urgent] perf tools: Fix perf.data format description of NRCPUS header 056 Git-Commit-ID: 18a7057420f8b67f15d17087bf5c0863db752c8b 029 X-Mailer: tip-git-log-daemon 035 Robot-ID: <tip-bot.git.kernel.org> 089 Robot-Unsubscribe: Contact <mailto:[email protected]> to get blacklisted from these emails 018 MIME-Version: 1.0 032 Content-Transfer-Encoding: 8bit 040 Content-Type: text/plain; charset=UTF-8 028 Content-Disposition: inline 177* X-Spam-Status: No, score=1.4 required=5.0 tests=ALL_TRUSTED,BAYES_00, DATE_IN_FUTURE_48_96,FREEMAIL_FORGED_REPLYTO autolearn=no autolearn_force=no version=3.4.1 016* X-Spam-Level: * 078* X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on terminus.zytor.com 043S Sender: [email protected] 017 Precedence: bulk 040 List-ID: <linux-kernel.vger.kernel.org> 045 X-Mailing-List: [email protected] 116 Received-SPF: none client-ip=209.132.180.67; [email protected]; helo=vger.kernel.org 037 X-SA-Exim-Connect-IP: 209.132.180.67 048 X-SA-Exim-Rcpt-To: linux-kernel@XXX 056 X-SA-Exim-Mail-From: [email protected] 093 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on deep-thought.ursa-minor-beta.org 015 X-Spam-Level: 175 X-Spam-Status: No, score=-15.2 required=5.0 tests=BAYES_00,GREYLIST_ISWHITE, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=3.4.1 031 X-Spam-Relay-Country: US US ** 090 Subject: [tip:perf/urgent] perf tools: Fix perf.data format description of NRCPUS header 065 X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) 066 X-SA-Exim-Scanned: Yes (on s-Mich Richter <[email protected]> 042 Acked-by: Andi Kleen <[email protected]> 044 Cc: Adrian Hunter <[email protected]> 036 Cc: David Ahern <[email protected]> 034 Cc: He Kuang <[email protected]> 053 Cc: Hendrik Brueckner <[email protected]> 038 Cc: Jin Yao <[email protected]> 033 Cc: Jiri Olsa <[email protected]> 040 Cc: Kim Phillips <[email protected]> 046 Cc: Lakshman Annadorai <[email protected]> 039 Cc: Namhyung Kim <[email protected]> 034 Cc: Simon Que <[email protected]> 042 Cc: Stephane Eranian <[email protected]> 035 Cc: Wang Nan <[email protected]> 076 Link: https://lkml.kernel.org/n/[email protected] 058 Signed-off-by: Arnaldo Carvalho de Melo <[email protected]> 110 --- tools/perf/Documentation/perf.data-file-format.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) 001 117 diff --git a/tools/perf/Documentation/perf.data-file-format.txt b/tools/perf/Documentation/perf.data-file-format.txt 040 index c57904a526ce..dfb218feaad9 100644 057 --- a/tools/perf/Documentation/perf.data-file-format.txt 057 +++ b/tools/perf/Documentation/perf.data-file-format.txt 140 @@ -111,8 +111,8 @@ A perf_header_string with the CPU architecture (uname -m) A structure defining the number of CPUs. struct nr_cpus { 096 - uint32_t nr_cpus_online; uint32_t nr_cpus_available; /* CPUs not yet onlined */ 061 + uint32_t nr_cpus_online; }; HEADER_CPUDESC = 8, 009 _addr)); 053 + pydict_set_item_string_decref(dict_sample, "addr", 167 + PyLong_FromUnsignedLongLong(sample->addr)); set_sample_read_in_dict(dict_sample, sample, evsel); pydict_set_item_string_decref(dict, "sample", dict_sample); 066 at a single location. Because of that we need to follow the same 067 + * heuristic with the head, i.e we increment it before using its 068 + * value. Otherwise the first element of the packet queue is not 011 + * used. 006 + */ 113 + decoder->head = (decoder->head + 1) & (MAX_BUFFER - 1); *packet = decoder->packet_buffer[decoder->head]; 058 - decoder->head = (decoder->head + 1) & (MAX_BUFFER - 1); 043 - decoder->packet_count--; return 1; 010 m-Status)
-- ## List details at https://lists.exim.org/mailman/listinfo/exim-users ## Exim details at http://www.exim.org/ ## Please use the Wiki with this list - http://wiki.exim.org/
