[EMAIL] Serious network crash bug for commons email 2 library

2024-07-25 Thread Adrian Mikula
Hi everyone,

I am a senior software engineer working on a TomEE project and as part of our 
recent upgrade to Jakarta 10 and Java 17, we also upgraded the Apache Commons 
email library to version 2 which you just released.

We encountered a very critical bug which only occurs with certain hardware and 
drivers, but it has very serious impacts on the host OS and would be 
catastrophic if it were to happen on production systems.  I suggest you look 
into this as a matter of urgency because I would not want anyone else to 
encounter this bug as it is very bad, but also very obscure and could easily be 
missed during testing.


PROBLEM DESCRIPTION:
On certain physical hardware, with a specific OS network driver installed, if 
the Apache commons email2 library is used and an email is sent, the email fails 
to send, and the OS network driver crashes and the computer loses all network 
access.  The network access can only be restored by restarting the computer, 
sometimes multiple times.


HARDWARE & OS:
Computer: Optiplex 7000
Windows 11
Ethernet driver: Intel 12.19.2.45

SOFTWARE:
Java 17
Jakarta 10
TomEE 9.1.2

These dependencies caused the issue:
commons-mail = { module = "org.apache.commons:commons-email2-jakarta", version 
= "2.0.0-M1" }
commons-mail-core = { module = "org.apache.commons:commons-email2-core", 
version = "2.0.0-M1" }

We tried both the snapshot version from a few months ago, and the final release 
version from a few weeks ago.
Snapshot repo: https://repository.apache.org/content/repositories/snapshots/
Release repo: 
https://mvnrepository.com/artifact/org.apache.commons/commons-email2-jakarta/2.0.0-M1

In the Java code:
import org.apache.commons.mail2.core.EmailException;
import org.apache.commons.mail2.jakarta.HtmlEmail;


STEPS TO REPRODUCE:

  1.  Make sure you are testing on a computer with the same physical 
hardware/network card
  2.  Install the exact same version of the Intel ethernet driver. This issue 
does not occur for older/newer versions of the Intel driver
  3.  Run TomEE9 with the apache-commons-email2 library in the libs folder of a 
WAR deployment.
  4.  Send an email.
  5.  The email will never get sent across the network.
  6.  The computer will lose all network access.
  7.  Network access will only be restored after restarting the computer.

DEBUG NOTES:

While debugging the Java stack trace, we found that if the commons-email2 
libraries are used, somehow one of the Java classes in geronimo-mail is called 
which is part of the TomEE/libs and this crashes the computer network. So it 
might be an issue with commons-email2 communicating with certain 3rd-party 
libraries in combination with certain hardware/drivers.


STEPS TO FIX:

If we add an excludes for the Geronimo library, and swap apache-commons-email 
with a different email implementation library, it seems fine.



Please also note if I update my network driver to the latest Intel version, 
2.0.0-M1 doesn't crash my network. It also doesn't crash with older driver 
versions, only version 12.19.2.45.  This is not an issue for Linux systems.

Please let me know if you need any more information about how to reproduce the 
issue.

Regards,
Adrian


RE: [EMAIL] Serious network crash bug for commons email 2 library

2024-08-01 Thread Adrian Mikula
Hi Gary,

Sorry, for security reasons we can't share the full stack trace.  But I did 
find some more details we recorded while debugging this:

By debugging, we worked out that the problem manifests when we get to the Java 
code that builds and sends the email, but before any email is actually sent 
out.  It seems to occur around the time when the Tomee9 classloaders are trying 
to load a bunch of jakarta.mail classes and failing to find them. We also 
worked out that the org.apache.commons.email2 library calls jakarta.mail, which 
in turn calls the apache.geronimo library.  We also found that after the 
problem manifests, there are a couple of extra RMI TCP threads created which 
have a HTTP connection and have some kind of resource lock.

Regards,
Adrian

-Original Message-
From: Gary Gregory  
Sent: Friday, July 26, 2024 7:41 PM
To: Commons Developers List 
Subject: Re: [EMAIL] Serious network crash bug for commons email 2 library

What is the stack trace? That's a critical element you left out. It could just 
be a class path entry ordering issue.

Gary

On Fri, Jul 26, 2024, 7:39 AM Elliotte Rusty Harold 
wrote:

> Interesting. Thanks,
>
> Since commons-mail is pure Java, there's no way this can be completely 
> caused by bugs in commons-mail. though it sounds like some detail in 
> commons-mail is tickling other bugs elsewhere in the chain. My guess 
> is the network driver is at fault since that's what crashes and it 
> only happens in one version of the driver, but it could be the JDK, 
> the OS, or all three.
>
>
> On Fri, Jul 26, 2024 at 4:52 AM Adrian Mikula 
>  wrote:
> >
> > Hi everyone,
> >
> > I am a senior software engineer working on a TomEE project and as 
> > part
> of our recent upgrade to Jakarta 10 and Java 17, we also upgraded the 
> Apache Commons email library to version 2 which you just released.
> >
> > We encountered a very critical bug which only occurs with certain
> hardware and drivers, but it has very serious impacts on the host OS 
> and would be catastrophic if it were to happen on production systems.  
> I suggest you look into this as a matter of urgency because I would 
> not want anyone else to encounter this bug as it is very bad, but also 
> very obscure and could easily be missed during testing.
> >
> >
> > PROBLEM DESCRIPTION:
> > On certain physical hardware, with a specific OS network driver
> installed, if the Apache commons email2 library is used and an email 
> is sent, the email fails to send, and the OS network driver crashes 
> and the computer loses all network access.  The network access can 
> only be restored by restarting the computer, sometimes multiple times.
> >
> >
> > HARDWARE & OS:
> > Computer: Optiplex 7000
> > Windows 11
> > Ethernet driver: Intel 12.19.2.45
> >
> > SOFTWARE:
> > Java 17
> > Jakarta 10
> > TomEE 9.1.2
> >
> > These dependencies caused the issue:
> > commons-mail = { module = 
> > "org.apache.commons:commons-email2-jakarta",
> version = "2.0.0-M1" }
> > commons-mail-core = { module = 
> > "org.apache.commons:commons-email2-core",
> version = "2.0.0-M1" }
> >
> > We tried both the snapshot version from a few months ago, and the 
> > final
> release version from a few weeks ago.
> > Snapshot repo:
> https://repository.apache.org/content/repositories/snapshots/
> > Release repo:
> https://mvnrepository.com/artifact/org.apache.commons/commons-email2-j
> akarta/2.0.0-M1
> >
> > In the Java code:
> > import org.apache.commons.mail2.core.EmailException;
> > import org.apache.commons.mail2.jakarta.HtmlEmail;
> >
> >
> > STEPS TO REPRODUCE:
> >
> >   1.  Make sure you are testing on a computer with the same physical
> hardware/network card
> >   2.  Install the exact same version of the Intel ethernet driver. 
> > This
> issue does not occur for older/newer versions of the Intel driver
> >   3.  Run TomEE9 with the apache-commons-email2 library in the libs
> folder of a WAR deployment.
> >   4.  Send an email.
> >   5.  The email will never get sent across the network.
> >   6.  The computer will lose all network access.
> >   7.  Network access will only be restored after restarting the computer.
> >
> > DEBUG NOTES:
> >
> > While debugging the Java stack trace, we found that if the
> commons-email2 libraries are used, somehow one of the Java classes in 
> geronimo-mail is called which is part of the TomEE/libs and this 
> crashes the computer network. So it might be an issue with 
> commons-email2 communicating with certain 3rd-par