notforwarded 691469 forwarded 691469 https://marc.info/?l=postfix-users&m=135129541824896&w=2 thanks
Update the broken gmane link to the postfix user mailing list. TLDR; sum up status of this issue. Fix would probably be to switch to maildir as default. Still, this would require to know if any Debian mail clients relies on /var/spool/mail/<user> to be in the mbox format. A short term option could be to document this mboxo spurious corruption in the postfix README. All in all this is not a data corruption that renders the emails unreadeable. It merely confuses the reader by changing the quote state of line starting with ">From " if his mail client unquote the quoted From_ lines per mboxo specifications. ------------ With mboxo, the email body: >From 1 >From 2 >>From 3 becomes: >From 1 >From 2 >>From 3 A simple upgrade path exists to mboxrd >From 1 >>From 2 >>>From 3 That was deemed a mostly wontfix in 2012 at least for breaking backward compatibilty. https://marc.info/?l=postfix-users&m=135145804930171&w=2 (ie ok but an optin) Though it makes no sense to me. A program that read mboxo can also read mboxrd, the only difference is that with the mboxo the program will get an inconsistent content (well corrupted content) while with mboxrd the message could be restored to its intial state (by removoing one '>' from each line starting with any number of '>' followed by "From ". I hope the reason for this wontfix is that no analysis of the changes in behavior was made. This http://jdebp.info/FGA/mail-mbox-formats.html also conclude the same, even if it makes no sense to me: > Conversely, when an "mboxo" reader is used, less message corruption will be observed in the final results if the messages were written by an "mboxo" tool than if they were written by an "mboxrd" tool. To me if an mboxo reader cannot unquote any better than an mboxrd reader would. The difference would be for a reader that does not unescape mbox quoted "From " lines. Emails that had no lines starting with "From " but any starting with any number of '>' followed by "From " would get an extra '>'. This still would require analysis and testing, as all actors tell taht the behavior of mobox readers would change (but they do not give their sources, so the work has to be done anew even if they are right). I still do not see how mboxrd would make mboxo only readers badly behave. Though I believe this should be brought to the upstream mailing list, I am for now trying to update this bug and remove the confusion out of it. Also it was told in 2019 that postfix does not use mboxo for local transport (though it would require checking the code, the documentation is an intent not a proof of the implementation) https://marc.info/?l=postfix-users&m=156158209919449&w=2 " List: postfix-users Subject: Re: mbox format? From: Viktor Dukhovni <postfix-users () dukhovni ! org> Date: 2019-06-26 20:47:23 Message-ID: 20190626204723.GK84864 () straasha ! imrryr ! org [Download RAW message or body] On Wed, Jun 26, 2019 at 04:27:06PM -0400, Wietse Venema wrote: > Viktor Dukhovni: > > > On Wed, Jun 26, 2019 at 12:39:02PM -0700, Ronald F. Guilmette wrote: > > > > > When Postfix hands a message to something... say a script invoked via > > > some ~/.forward file... which one of these four formats will the message > > > be in? > > > > See: > > https://github.com/vdukhovni/postfix/blob/master/postfix/src/global/mail_copy.c#L235 > > > > So the answer is mboxo. > > According to 'man 8 local', section 'EXTERNAL COMMAND DELIVERY': > The local(8) daemon prepends a "From sender time_stamp" envelope header > to each message, prepends an X-Original-To: header with the recipient > address as given to Postfix, prepends an optional Delivered-To: header > with the final recipient envelope address, prepends a Return-Path: > header with the sender envelope address, and appends no empty line. Oops, sorry, my answer was about delivery by local(8) directly to a mailbox, not a command invoked via a pipe. " >From an inspection of the code the local daemon call mail_copy: https://github.com/vdukhovni/postfix/blob/0abb45ca0732fc039e0f43b66ed441d283eb2565/postfix/src/local/mailbox.c#L212 the same as pipe:https://github.com/vdukhovni/postfix/blob/e5e46ec03f77e0bf837b1ad00cba2663d74a92c3/postfix/src/global/pipe_command.c#L587 mail_copy which translate to mboxo if the MAIL_COPY_QUOTE flag is set. This flag is included in the MAIL_COPY_MBOX one. https://github.com/vdukhovni/postfix/blob/0abb45ca0732fc039e0f43b66ed441d283eb2565/postfix/src/local/mailbox.c#L188 though the pipe only has MAIL_COPY_QUOTE if the pipe '>' flag is set in the "flags=" pipe attribute. https://github.com/vdukhovni/postfix/blob/master/postfix/src/pipe/pipe.c#L913 https://www.postfix.org/pipe.8.html An alternative to mboxrd is client MIME quoted/printable for "From " lines in the body https://marc.info/?l=postfix-users&m=135161915820930&w=2 , though it does not fix all thes mboxo "From " issues https://marc.info/?l=postfix-users&m=135161915820931&w=2. In https://bugzilla.gnome.org/show_bug.cgi?id=686363#c2 I believe Milan Crha is wrong in that he tells there is no need to quote the quoted (prefixed) "From " in an email body (ie ">From "). But if these are stored in an mbox file then on mbox decode these quoted ">From " lines then will be unqutoted to become "From ", which is also a corruption of the email body. The best course of action would probably be to have at least the local transport store in the maildir format. It seems that /var/spool/mail/user directories as maildir is valid a setup. Still I don't know if it would break other tools that might expect /var/spool/mail/user to be mbox. dovecot handling (based on content length): https://doc.dovecot.org/3.0/admin_manual/mailbox_formats/mbox/ " In mboxes a new mail always begins with a “From ” line, commonly referred to as From_-line. To avoid confusion, lines beginning with “From ” in message bodies are usually prefixed with ‘>’ character while the message is being written to in mbox. Dovecot doesn’t currently do this escaping however. Instead it prevents this confusion by adding Content-Length headers so it knows later where the next message begins. Dovecot also doesn’t remove the ‘>’ characters before sending the data to clients. " Dovecot stores mbox as mboxcl2 https://doc.dovecot.org/3.0/admin_manual/mailbox_formats/mbox/. It also doesn’t remove the ‘>’ characters before sending the data to clients as it does not add it in its mbox storage mboxcl2 variant. So the format for /var/spool/mail/user mbox mailboxes does not seems to be set in stone. mboxcl2 introduces issues as already stated by Christoph: https://marc.info/?l=postfix-users&m=135145702029753&w=2 " I'd agree that mboxcl2 is not suited for postfix, cause when a client doesn't expect it, one will get huge corruptions and "phantom messages". mboxcl (v1) doesn't make much sense,.. IMHO. " Google Takeout uses mboxrd from what I read. I disagree with https://marc.info/?l=postfix-users&m=135161915820930&w=2 that: > Well, if you'd looked at the date of your sources, you'd have known that others have failed establishing alternatives to what DJB or Rahul Dhesi or whoever dubbed "mboxo" in nearly two decades. It seems to me that everybody switched to one of the alternatives. What is true is they did not settle for one alternative, they made have settled for three of them. The fact that there is not only one descent of mboxo does not mean that no alternative was established. As Matthies himslef admit in https://marc.info/?l=postfix-users&m=135161915820935&w=2: > As stated earlier, there is no single established mbox flavour, and has not been for nearly two decades. Seems the only agreed upon standard is not to use mboxo. I do not see how this call for to keep mboxo as default. He also state that switching to mboxrd would: > Incompatibility with documentation - such as local(8) - and existing Postfix practice of the past sixteen-or-so years. While as shown above the documentation for "local" transport states that it does not follow mboxo. When it in fact does. https://en.wikipedia.org/wiki/Mbox All these variants have the problem that the content of the message sometimes must be modified to remove ambiguities, as shown below, so that applications have to know which quoting rule has been used to perform the correct reversion, which turned out to be impractical. Using MIME and choosing a content-transfer-encoding that quotes "From_" lines in a standard-compliant fashion ensures that message content doesn't need to be changed, but only their MIME representation. The wikipedia citation could call for to keep with the mboxo format so clients could know how to quote "From " lines in an mbox file. But the fact is the clients already have to faces the variants when opening an mbox. So staying with mboxo does not help clients. And mboxo does not As told above the MIME quote of "From " only means that clients do not need to unquote "From " lines anymore. But to be backward compatible they have to keep unquoting. So they will still unquote ">From " lines even though they should not because this is an non mboxo quoted '>From " line. If an mbox ends up mixing mboxo and MIME quoted From_ lines clients will still be unable to know how to decode it. The way out would be for the mbox format to settle on the MIME quoting of From_ lines and clients to stop unquoting the From_ lines. But we might as well settle on hte mboxrd then. The issue is the same, ie standardization. So MIME only had to the confusion. Only mboxrd should be backward compatible with mboxo clients (at least from its spec). As mboxo clients are already expected to unquote ">From " the only issue us that they are not ready to unquote From_ lines with more than one '>'. So mboxrd clients will have the correct output. mboxo clients will have incorrect output only when quoted From_ lines exists. Which they already incorrectly handles when there was a single quote. Though only mboxcl2 and MIME preserves checksums for S/MIME and PGP. But I do not see how keeping mboxo helps in this regards as it changes the message content alike as mboxrd. Switching to mboxrd is not a regression (except for more message will be changed than, with mboxo as it quote more From_ lines). So the switch to mboxrd could be done for the default mbox local store. And a setting could be added to optin the MIME quoted mbox storage. Or have mboxrd and MIME as optin but with a default that could be set at build time to let distribution choose. Cheers, Alban
