Mark,
Thanks for the comments. I have somewhat of a solution. I just want to
document what I learned in case anyone else has this problem. From what
I have been able to determine, you are correct about it being an Amazon
problem. I don't really understand what is happening. But here's what
I found... After a day of testing all types of scenarios, I found that
it only happens the first time after installing a new ami (image) on a
brand new EC2 and bringing up the EC2 and auto-starting TC for the
first time. But it fails EVERY time with this scenario. Bouncing TC
fixed it every time. Bouncing the EC2 never causes it to come back.
Only booting a brand new EC2 instance causes it. I tried removing the
tomcat autostart, ran the ami EC2 creation, waited a bit, and went to
the console and manually started tomcat. No problem.
So all I can deduce from this is that there's a timing issue. I'm
suspecting that AWS is starting up the EC2 before it is fully created
(likely something not complete in the file system setup). I added a
2-minute sleep in rc.local to delay the startup of tomcat long enough
for whatever wasn't finished yet to complete. With a 2-min sleep, it now
works every time. Pretty ugly fix IMHO. But working ugly beats not
working pretty. I'll probably play around with the 2-min sleep to see
if I can reduce that. I may try to talk to Amazon about it. I may also
try banging my head against the wall, probably with the same results as
talking to amazon about this.
The takeaway from this is that AWS appears to be trying to shave a
little performance delay by jumping the gun on the EC2 boot. The result
in this case is TC saying the crypto file doesn't exist, but it does
exist a minute or two later. Likely nothing to do specifically with the
crypto file. That's just likely the first file needed that wasn't ready
yet.
Moving on the next fire that needs to be put out...
Thx
Jerry
On 12/20/2021 5:22 AM, Mark Thomas wrote:
On 20/12/2021 06:59, Jerry Malcolm wrote:
I'm adding a slight variation to the error I get at times (see bottom
of stack trace below)
This is the code that throws the root exception:
if (!Files.isDirectory(cryptoPolicyPath)
|| !Files.isReadable(cryptoPolicyPath)) {
throw new SecurityException(
"Can't read cryptographic policy directory: " +
cryptoPolicyProperty);
}
That points very strongly to a file system issue. My recommendation is
to take this up with AWS support.
Mark
On 12/19/2021 11:55 PM, Jerry Malcolm wrote:
I have a production environment of 10+ AWS EC2 servers all built
from a single EC2 snapshot image. The master EC2 works fine. But
when we propagated the image to all of the production servers, we
started getting "Can't read cryptographic policy directory:
unlimited" errors when I try to get a jdbc connection object from
the pool. Full stack trace is below. The three critical lines in
the trace are:
java.lang.ExceptionInInitializerError
Caused by: java.lang.SecurityException: Can not initialize
cryptographic mechanism
Caused by: java.lang.SecurityException: Can't read cryptographic
policy directory: unlimited
On some machines rebooting the tomcat service fixes the problem and
it continues to work fine after that. On other machines, the
problem is still there after rebooting. But even on the machines
that are fixed after TC bounce, if I bounce the full EC2, the
problem is back.
We've had this environment working for almost two years. I did a
minor version upgrade to TC 8.5.73 a month ago. And I upgraded to
java 11 probably a year ago. Otherwise, no changes that I'm aware
of. Definitely nothing in the past few days before the error hit
yesterday.
Everything I can find on google about the error messages says to
make sure the 'unlimited' folder is present and accessible. The
folder is there and has been there untouched. And I know it's not
disappearing and reappearing to become accessible after the TC
reboot that sometimes fixes the problem.
The mySQL RDS all of these instances talk to hasn't changed. And
other servers that aren't part of this image distribution have no
problems accessing it. It just makes no sense that the same EC2
image that works on one machine started failing on a bunch of
identical configuration EC2s yesterday.
I know this call goes through layers of tomcat, layers of the mysql
driver, and then layers of the jvm before it occurs. But can anyone
help me understand what TC/mySQL might be trying to do with this
call to JVM crypto code and why it has started failing on all of my
servers? Any Java crypto gurus out there?
Stack trace of error:
java.lang.ExceptionInInitializerError
at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540)
at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:185)
at
java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCipher.java:483)
at java.base/sun.security.ssl.SSLCipher.<init>(SSLCipher.java:472)
at java.base/sun.security.ssl.SSLCipher.<clinit>(SSLCipher.java:81)
at java.base/sun.security.ssl.CipherSuite.<clinit>(CipherSuite.java:67)
at
java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCipherSuites(SSLContextImpl.java:348)
at
java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.<clinit>(SSLContextImpl.java:580)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:315)
at
java.base/java.security.Provider$Service.getImplClass(Provider.java:1918)
at
java.base/java.security.Provider$Service.newInstance(Provider.java:1894)
at
java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
at
java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
at java.base/javax.net.ssl.SSLContext.getInstance(SSLContext.java:168)
at
com.mysql.cj.protocol.ExportControlled.getSSLContext(ExportControlled.java:565)
at
com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:302)
at
com.mysql.cj.protocol.StandardSocketFactory.performTlsHandshake(StandardSocketFactory.java:188)
at
com.mysql.cj.protocol.a.NativeSocketConnection.performTlsHandshake(NativeSocketConnection.java:99)
at
com.mysql.cj.protocol.a.NativeProtocol.negotiateSSLConnection(NativeProtocol.java:331)
at
com.mysql.cj.protocol.a.NativeAuthenticationProvider.negotiateSSLConnection(NativeAuthenticationProvider.java:777)
at
com.mysql.cj.protocol.a.NativeAuthenticationProvider.proceedHandshakeWithPluggableAuthentication(NativeAuthenticationProvider.
at
com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:202)
at
com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1348)
at com.mysql.cj.NativeSession.connect(NativeSession.java:163)
at
com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:947)
at
com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:817)
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:447)
at
com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:237)
at
com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:199)
at
org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52)
at
org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:365)
at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:116)
at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:665)
at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:547)
at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:747)
at jwm.db.DBData.getConnection(DBData.java:443)
...
Caused by: java.lang.SecurityException: Can not initialize
cryptographic mechanism
at java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120)
... 81 more
Caused by: java.lang.SecurityException: Can't read cryptographic
policy directory: unlimited
at
java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:326)
at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111)
at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:107)
.. 81 more
================= This is a variation of the error I've also found in
the logs. NoClassDefFoundError that is random just adds to the
strangeness of this problem...
java.lang.NoClassDefFoundError: Could not initialize class
sun.security.ssl.SSLContextImpl$TLSContext
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:315)
at
java.base/java.security.Provider$Service.getImplClass(Provider.java:1918)
at
java.base/java.security.Provider$Service.newInstance(Provider.java:1894)
at
java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
at
java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
at java.base/javax.net.ssl.SSLContext.getInstance(SSLContext.java:168)
at
com.mysql.cj.protocol.ExportControlled.getSSLContext(ExportControlled.java:565)
at
com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:302)
at
com.mysql.cj.protocol.StandardSocketFactory.performTlsHandshake(StandardSocketFactory.j
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
