Re: corrupted attachments

[Vincent Lefevre]

On 2017-06-12 18:13:58 -0500, Derek Martin wrote:
> I'm not opposed to using libmagic for this, but I actually think the
> heueristic code should be removed entirely from Mutt.  Message bodies
> should be assumed to be text, and attachments are assumed to be NOT
> text, by virtue of them not being message bodies.  Their types should
> be determined via the standard MIME mechanisms.  If those mechanisms
> fail--for whatever reason--assume application/octet-stream and move
> on.

I disagree: those mechanisms will typically fail on what is
actually arbitrary text, because text does not correspond to
a file format. Very often it does not have a file extension.

> That guarantees the file will arrive unmolested and obviates the
> need for some mostly pointless code in Mutt to try to guess what the
> user was thinking.

Even if binary data is sent as valid text (BTW, any binary data
can be regarded as valid text in some user-defined character set),
it should not arrive as unmolested. I mean that in any case, the
user should be able to choose text/plain for "Content-Type" and
be sure that no corruption occurs. It is up to Mutt to choose an
adequate "Content-Transfer-Encoding".

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: [Mutt] #3950: Prune unusable keys from PGP key selection menu

[Mutt]

#3950: Prune unusable keys from PGP key selection menu
-+-
  Reporter:  madduck |  Owner:  mutt-dev
  Type:  | Status:  new
  enhancement|
  Priority:  major   |  Milestone:
 Component:  crypto  |Version:  1.8.0
Resolution:  |   Keywords:  pgp unusable key menu selection
 |  expired revoked gpg recipient
-+-

Comment (by kevin8t8):

 I believe in your case, mutt is still prompting because the key is not
 fully trusted.  When you choose it, does mutt prompt you to confirm,
 saying "ID has undefined validity.", "ID is not valid.", or "ID is only
 marginally valid."?

--
Ticket URL: 
Mutt 
The Mutt mail user agent

Re: [Mutt] #3950: Prune unusable keys from PGP key selection menu

[Mutt]

#3950: Prune unusable keys from PGP key selection menu
-+-
  Reporter:  madduck |  Owner:  mutt-dev
  Type:  | Status:  new
  enhancement|
  Priority:  major   |  Milestone:
 Component:  crypto  |Version:  1.8.0
Resolution:  |   Keywords:  pgp unusable key menu selection
 |  expired revoked gpg recipient
-+-

Comment (by Kevin McCarthy ):

 In [changeset:"26cbf2cac701062cc6fad45920219b8df359baec"
 7084:26cbf2cac701]:
 {{{
 #!CommitTicketReference repository=""
 revision="26cbf2cac701062cc6fad45920219b8df359baec"
 Make GPGME key selection behavior the same as classic-PGP. (see #3950)

 Classic-PGP key selection auto-selected a match if there was a single
 strong, valid, address-match result, even if there were other weak
 matches.

 GPGME was prompting in the same situation, if there were other weak
 matches.

 Change GPGME to match the classic behavior, as this is more useful and
 matches the manual description.
 }}}

--
Ticket URL: 
Mutt 
The Mutt mail user agent

Re: corrupted attachments

[Andries E. Brouwer]

On Mon, Jun 12, 2017 at 06:13:58PM -0500, Derek Martin wrote:
> On Tue, Jun 06, 2017 at 02:03:34AM +0200, Andries E. Brouwer wrote:
> > On Tue, Jun 06, 2017 at 09:15:23AM +1000, Cameron Simpson wrote:
> > > On 05Jun2017 22:12, Andries E. Brouwer  wrote:
> > >> A moment ago I sent 8 images as attachment with mutt.
> > >> Three were garbled on arrival.
> > > 
> > > Just for some more context, did the files lack a useful file extension?
> > 
> > No mime-type was known. No mechanism using mime-types could work.
> 
> That can't be true.  You said you attached the files... they had to be
> on disk and had to have names.  Even if your image format is
> proprietary and lacks an official standard, you can manually set its
> MIME type to application/octet-stream.

You seem unable to read or have no imagination.
Here are files on a 22-year old filesystem.
I have no idea about the contents. I forward the files to
someone interested. They arrive corrupted.
I investigate the reason for the corruption and find out
that some silly guessing is going on, so that even files
with many NUL bytes can be seen by mutt as text files.

Of course I might have read the mutt source first, I might
have seen that mutt is buggy and does this silly guessing,
and could have corrected the mime type by hand.
There are lots of other things I could have done. But I just
expected the mutt default to work. It didn't.

> > Using information when it is available is a separate discussion.
> > In the present discussion one only has the file contents.
> 
> This also is not true.  When you asked Mutt to attach the file, you 
> had to tell it the name of the file.  If you specified a bad filename
> or didn't configure your mime types properly that's user error.

What a strange ideas some people have...

Please read the bug report, and shut up if you have nothing helpful to 
contribute.
I know that you are an active contributor to mutt.
Still it doesnt help when people encounter a bug to deny that there is a 
problem,
or to reply telling the reporter that his bug report is inaccurate,
where in fact it is precise and correct.

Andries

Re: corrupted attachments

[Derek Martin]

On Tue, Jun 13, 2017 at 09:11:24AM +0200, Vincent Lefevre wrote:
> On 2017-06-12 18:13:58 -0500, Derek Martin wrote:
> > I'm not opposed to using libmagic for this, but I actually think
> > the heueristic code should be removed entirely from Mutt.  Message
> > bodies should be assumed to be text, and attachments are assumed
> > to be NOT text, by virtue of them not being message bodies.  Their
> > types should be determined via the standard MIME mechanisms.  If
> > those mechanisms fail--for whatever reason--assume
> > application/octet-stream and move on.
> 
> I disagree: those mechanisms will typically fail on what is
> actually arbitrary text, because text does not correspond to
> a file format.

Which matters exactly zero if you use application/octet-stream.

> Very often it does not have a file extension.

"Arbitrary text" has a MIME type and associated file extension;
The choice to not use it is the user's choice, but like all choices
that has consequences.  Whatever Mutt does is just a guess, even if
it's a good guess, and we know some mail systems translate end-of-line
chars, and we know they won't when an application/octet-stream
attachment is used.  So if you want to send text attachments, and you
want them to be properly identified as such automatically, you should
use the associated file extension; or else be prepared to set them
manually.

If you still choose not to, then application/octet-stream is fine, and
you can change it manually if not.  Small price to pay for your
stubborn refusal to adhere to existing standards.  Your recipients may
need to save the file to view it--a minor annoyance caused by YOU for
not following established norms--but the content will arrive
unmolested.

> Even if binary data is sent as valid text (BTW, any binary data
> can be regarded as valid text in some user-defined character set),

If it's user-defined, it's not standards-compliant, and needs to be
treated specially regardless; i.e. the recipient won't be able to
display it without some special effort, such as sending it to a
display filter (which will require a suitable MIME type to work
automatically).  This case isn't interesting.

> it should not arrive as unmolested. I mean that in any case, the
> user should be able to choose text/plain for "Content-Type" and
> be sure that no corruption occurs. It is up to Mutt to choose an
> adequate "Content-Transfer-Encoding".

A reasonable point, but Mutt should not have to jump through hoops to
make that happen, and it should be fast--i.e. take negligible time, so
that the user does not have to wait perceptibly for the file to be
attached.  I think the existing heueristics do that just fine, so long
as the user doesn't try something unreasonable. Although, perhaps
Mutt should resort to quoted printable more readily, to reduce the
chances of producing incorrect messages... 

However, the case that sparked this thread was someone attaching IMAGE
FILES, which are not text, and REQUIRE a suitable MIME type.  If the
user won't name them properly, or if they truly are a non-standard
type that has no established MIME type, then the user will need to do
what I described before: either define a fake type with a uniform
extension and add it to mime.types, or manually set it in Mutt's
attachment menu to something that won't get molested by the mail
system.

Mutt should encourage following standards, and should not cater to
those who refuse.

-- 
Derek D. Martinhttp://www.pizzashack.org/   GPG Key ID: 0xDFBEAD02
-=-=-=-=-
This message is posted from an invalid address.  Replying to it will result in
undeliverable mail due to spam prevention.  Sorry for the inconvenience.



pgpBJ1dGcUbeG.pgp
Description: PGP signature

majordomo

[Andries E. Brouwer]

On Tue, Jun 13, 2017 at 10:44:41AM -0500, Derek Martin wrote:
[something I did not read]

Just for your amusement:

I sent an unsubscribe request to majord...@mutt.org with the line
that it had told me to use when I subscribed.

The following happened:

   - The following addresses had permanent fatal errors -

(reason: 550 High probability of spam)

   - Transcript of session follows -
... while talking to mx10.guard.email.:
>>> DATA
<<< 550 High probability of spam
554 5.0.0 Service unavailable


Final-Recipient: RFC822; majord...@mutt.org
Action: failed
Status: 5.2.0
Remote-MTA: DNS; mx10.guard.email
Diagnostic-Code: SMTP; 550 High probability of spam
Last-Attempt-Date: Tue, 13 Jun 2017 20:59:54 +0200

Re: corrupted attachments

[Derek Martin]

On Tue, Jun 13, 2017 at 08:12:36PM +0200, Andries E. Brouwer wrote:
> Please read the bug report, and shut up if you have nothing helpful
> to contribute.

You apparently stopped reading before the part where I supplied a
simple and efficient solution to the problem:

On Mon, Jun 12, 2017 at 06:13:58PM -0500, Derek Martin wrote:
> I'm not opposed to using libmagic for this, but I actually think the
> heueristic code should be removed entirely from Mutt.  Message bodies
> should be assumed to be text, and attachments are assumed to be NOT
> text, by virtue of them not being message bodies.  Their types should
> be determined via the standard MIME mechanisms.  If those mechanisms
> fail--for whatever reason--assume application/octet-stream and move
> on.  That guarantees the file will arrive unmolested and obviates the
> need for some mostly pointless code in Mutt to try to guess what the
> user was thinking.

On Tue, Jun 13, 2017 at 08:12:36PM +0200, Andries E. Brouwer wrote:
> I know that you are an active contributor to mutt.  Still it doesnt
> help when people encounter a bug to deny that there is a problem, or
> to reply telling the reporter that his bug report is inaccurate,
> where in fact it is precise and correct.

I didn't deny there was a problem dude, I provided a solution.  I did
however outline your possible contribution to the problem, and explain
how it's supposed to work.

-- 
Derek D. Martinhttp://www.pizzashack.org/   GPG Key ID: 0xDFBEAD02
-=-=-=-=-
This message is posted from an invalid address.  Replying to it will result in
undeliverable mail due to spam prevention.  Sorry for the inconvenience.



pgpDY1UZtaSkm.pgp
Description: PGP signature

Re: corrupted attachments

[Martin Mares]

Hello, world!\n

> "Arbitrary text" has a MIME type and associated file extension;

No, it does not. People use lots of different extensions for plain
text files and also often no extension at all (e.g., "README"
or "COPYING").

Using libmagic to detect the MIME type sounds reasonable, but inferring
anything from the file's extension (or lack thereof) sounds not.

Have a nice fortnight
-- 
Martin `MJ' Mares http://mj.ucw.cz/
Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth
"A semicolon. Another line ends in the dance of camel." -- Kabir Ahuja

Re: corrupted attachments

[Vincent Lefevre]

On 2017-06-13 10:44:41 -0500, Derek Martin wrote:
> On Tue, Jun 13, 2017 at 09:11:24AM +0200, Vincent Lefevre wrote:
> > On 2017-06-12 18:13:58 -0500, Derek Martin wrote:
> > > I'm not opposed to using libmagic for this, but I actually think
> > > the heueristic code should be removed entirely from Mutt.  Message
> > > bodies should be assumed to be text, and attachments are assumed
> > > to be NOT text, by virtue of them not being message bodies.  Their
> > > types should be determined via the standard MIME mechanisms.  If
> > > those mechanisms fail--for whatever reason--assume
> > > application/octet-stream and move on.
> > 
> > I disagree: those mechanisms will typically fail on what is
> > actually arbitrary text, because text does not correspond to
> > a file format.
> 
> Which matters exactly zero if you use application/octet-stream.

But the goal is to have text/plain on text files.

> > Very often it does not have a file extension.
> 
> "Arbitrary text" has a MIME type and associated file extension;

Of course, this is completely wrong! Just look at all the files
without a file extension (or with a non-standard one) in the Mutt
repository, for instance.

> If you still choose not to, then application/octet-stream is fine, and
> you can change it manually if not.  Small price to pay for your
> stubborn refusal to adhere to existing standards.

Again, your remark in completely stupid. The standard is that some
files may not have an extension. And usual software regards them as
text if they are obviously text.

[snipped non-sense]

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: corrupted attachments

[Vincent Lefevre]

On 2017-06-13 23:14:49 +0200, Martin Mares wrote:
> Using libmagic to detect the MIME type sounds reasonable, but inferring
> anything from the file's extension (or lack thereof) sounds not.

I don't think that libmagic is a good idea. It has too many false
positives; by that, I mean that it can say that some file has
some specific MIME type (e.g. text/x-c) while it is actually
just text that may have some lines of C in a small part of it.
Many bugs have been reported and most of them are still not fixed.

IMHO, the rules should be (step by step):

0. Some Mutt variable may refer to an external program that will
   determine the MIME type, possibly with a charset, or fail.
   The expected output should be the same as "file -bi", but
   an empty output is also possible for a fallback (as the
   following steps). Execute this program when this variable is
   defined.
   Note: though "file -bi" could be used as the default value,
   I would not recommend it as explained above.

1. If the filename has a known file extension, then deduce the
   MIME type from it in the usual way (/etc/mime.types system).

2. If the file is valid UTF-8 and contains no blacklisted control
   characters (to be defined, but this should include null bytes),
   then MIME type = text/plain.

3. MIME type = application/octet-stream.

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: corrupted attachments

[Vincent Lefevre]

On 2017-06-14 00:18:44 +0200, Vincent Lefevre wrote:
> 1. If the filename has a known file extension, then deduce the
>MIME type from it in the usual way (/etc/mime.types system).

On this point, according to the Mutt manual, Mutt currently does:

   When you add an attachment to your mail message, Mutt searches your
   personal mime.types file at $HOME/.mime.types, and then the system
   mime.types file at /usr/local/share/mutt/mime.types or /etc/mime.types

I think that this hard-coded rule is a bad idea because it leaves
no way for the end user who is not an admin to remove some setting.
The problem is that there are many clashes in the file extensions
(a file extension being used for completely different usages),
partly because of the lack of a standard. In practice, various
file extensions that can be found in /etc/mime.types may be wrong
for the user, who may prefer text detection. An example:

application/x-trash ~ % bak old sik

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: corrupted attachments

[Derek Martin]

On Wed, Jun 14, 2017 at 12:18:44AM +0200, Vincent Lefevre wrote:
> I don't think that libmagic is a good idea. It has too many false
> positives; by that, I mean that it can say that some file has
> some specific MIME type (e.g. text/x-c) while it is actually
> just text that may have some lines of C in a small part of it.
> Many bugs have been reported and most of them are still not fixed.

I agree... you're getting to the crux of my issue with all of this,
but there's more...

> 2. If the file is valid UTF-8 and contains no blacklisted control
>characters (to be defined, but this should include null bytes),
>then MIME type = text/plain.

What if the attachment is 10GB?

-- 
Derek D. Martinhttp://www.pizzashack.org/   GPG Key ID: 0xDFBEAD02
-=-=-=-=-
This message is posted from an invalid address.  Replying to it will result in
undeliverable mail due to spam prevention.  Sorry for the inconvenience.



pgp8wIaGot27F.pgp
Description: PGP signature

Re: corrupted attachments

[Vincent Lefevre]

On 2017-06-13 17:46:44 -0500, Derek Martin wrote:
> On Wed, Jun 14, 2017 at 12:18:44AM +0200, Vincent Lefevre wrote:
> > 2. If the file is valid UTF-8 and contains no blacklisted control
> >characters (to be defined, but this should include null bytes),
> >then MIME type = text/plain.
> 
> What if the attachment is 10GB?

Several comments about this:

1. Scanning the attachment for the above detection will take much
   less time than sending the mail. So, I don't think that this is
   really an issue.

2. Sending such a huge attachment is not a common thing.
   I think that such corner cases should not have an influence
   on the default behavior.

3. Such huge files are probably not text anyway. So, this means
   that in practice, Mutt will detect very early that the file
   is binary. There could be another rule to say that above some
   configurable limit, the file is regarded as binary, but I really
   don't think that this is necessary.

Mutt should also warn the user when sending such a big attachment
(all users make mistake...). When Mutt is run without the UI, it
should probably refuse to send the mail unless some --force option
is used or something like that.

BTW, is there a way to specify the MIME type with the -a option?

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)