Hi Chris, To cut a long story short openssl helped. Using openssl -connect showed the error below. When I added the -tls1 flag the error went away.
2104:error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac:.\ssl\s3_pkt.c:532: Configuring Tomcat with the JVM parameter below resolved the issue. -Dhttps.protocols="TLSv1" This may limit our client Web application but I have very little influence over the server-side application, so it'll have to do as a solution for now. So the issue seems to be a Java handshake error whereby it has a problem downgrading from TLSv1.2 to TLSv1 during the handshake. Why it works in our non-prod environment with the same versions of everything is a mystery, but I can speculate at some difference in underlying O/S settings, or perhaps a Java bug. Many thanks for your patience and advice. Diarmuid On 28 Aug 2015 20:46, "Christopher Schultz" <ch...@christopherschultz.net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Diarmuid, > > On 8/28/15 3:25 PM, dmccrthy wrote: > > Thanks again for replying. I agree it doesn't look like a Tomcat > > issue and would be happy to close the issue. I've included > > responses to your points below prefixed with [DMC] for my initials, > > but think I need to focus on the Java side of things and perhaps > > open an Oracle ticket. > > Feel free to continue the conversation here. I just marked it as "off > topic" because it's not directly-related to Tomcat. This could have > happened in any other application server, for instance. > > There's nothing wrong with having an off-topic discussion, here. I > just wanted everyone to know that this wasn't directly-related to Tomcat > . > > See below. > > > On 8/27/15 4:31 PM, dmccrthy wrote: > >>>> Thanks for responding so quickly. My apologies, I should have > >>>> been clearer on the topology. We have a Tomcat instance with > >>>> a 3rd party web app deployed on it (the "Tomcat client") > >>>> running on Windows 2008 . This connects via HTTPS to a 3rd > >>>> party service running behind IBM Http Server on Solaris 10. > > > > Okay, so, just to confirm: > > > > Tomcat (app) -> HTTPS -> 3rd party service > > > > Configurations and results: > > > > 0. prod Tomcat -> prod 3rd-party service FAILS with > > TLS_RSA_WITH_AES_128_CBC_SHA > > > >> [DMC] Correct > > Ok. > > > (do other ciphers work? or are you completely dead in the water?) > > > >> [DMC] No, this is the next step. Our production environment is > >> completely locked down and we have to exhaust all other avenues > >> before making speculative changes. > > I suppose what I meant was: do 100% of your connections to this > 3rd-party service fail because this one cipher suite seems to be > broken, or do you have some connections that work and others that do > not? If some work, do they have anything in common with each other? > > > 2. prod Tomcat -> prod 3rd-party service with lower version of > > Tomcat and/or Java WORKS with TLS_RSA_WITH_AES_128_CBC_SHA > > > >> [DMC] Correct > > So, downgrading does in fact improve the situation. Can you try > upgrading Java in one of your working production environments but > leave Tomcat alone? That would definitely rule-out Tomcat as a problem. > > > 1. test Tocmat -> test 3rd-party service WORKS with > > TLS_RSA_WITH_AES_128_CBC_SHA > > > >> [DMC] Correct > > Good. What version was this running again? Same Java/Tomcat versions > as the production server which seems to fail? > > > 2. test client (command-line?) on production host -> prod 3rd > > parts service WORKS with TLS_RSA_WITH_AES_128_CBC_SHA > > > >> [DMC] Correct. The command line test client is a simple Java > >> class that just opens an InputStream to the 3rd party service > > So: same Java, same host, same client library: works. But within > Tomcat on that host with that Java with that same client library: > fails? :( > > > No changes are known/expected, other than Tomcat and Java version. > > > >> [DMC] It's a greenfield app and no changes are planned bar > >> functional patches. The same version of operating system, java, > >> tomcat, and the application is in our non-prod environment and > >> works ok > > There is obviously a difference. We just have to figure out what it is. > > What are your JVM parameters to launch Tomcat? You can find those out > using "jinfo" from the command-line if you have the process id (at > least I *think* that works on Windows). Check the running production > application as well as the CLI client running on the same machine. > Quadruple-check the Java versions of those processes. > > >>>> Two additional pieces of analysis that came out of our > >>>> investigations today are: > >>>> > >>>> 1) we are using NextGen Cisco devices. These are application > >>>> and protocol aware. This means that the firewall might allow > >>>> allow our standalone Java client to connect using HTTPS, but > >>>> interfere with the HTTPS SOAP requests from our Tomcat/3rd > >>>> party app "Tomcat client". > > > > How can the device be "application aware"? How does the > > applicaiton identify itself to the device? How is that > > authenticated? What is stopping EvilApp.exe from claiming to be > > InnocentApp.exe? > > > >> [DMC] the firewall devices check the tcp connections. I don't > >> know how. I do know we had endless trouble from overzealous > >> firewall guys putting restrictive rules in place. The impact of > >> that was that, for example, we could telnet through the firewall > >> but not establish a Sybase connection. In any case, we compared > >> all the non-prod and prod firewall rules and they are configured > >> the same way. We also had several debug sessions with the > >> firewall guys and no traffic is being blocked. So at least we've > >> ruled that out. > > Good. > > >>>> 2) there are some anecdotes of Java 7 clients failing to > >>>> downgrade from TLSv1.2 to TLSv1. For example > >>>> http://stackoverflow.com/questions/31579587/java-tls-1-2-downgrade- > to- > > > >>>> > tls-1-0-sometimes-error. > > > > This > >>>> > > is not a "piece of analysis". This is something you found online > > that might be related. But you haven't collected any information > > to support the idea that it's related. I note that the experience > > is different in that SO article: the user is getting a different > > error at a different part of the handshake. > > > >> [DMC] I agree. But in the spirit of trying different cipher > >> suites, this is something that may (or may not) be relevant > > Okay. Conceptually, it's not hard to black-list a cipher suite. Your > client library might not have that capability, though. Having done > some of that work recently, I know that it takes miles of plumbing > code so that the user can just call > client.setAllowedCipherSuites(String[]). The client may simply not > support it. > > >>>> Some takeaway actions for me to explore are below. I'll > >>>> welcome any other thoughts that you may have. > >>>> > >>>> 1) compare the firewall rules on our working non-production > >>>> client with the rules governing the failing production > >>>> client > > > > Firewalls are usually pretty dumb. I'd be surprised if the firewall > > is breaking anything, here. > > > >>>> 2) use openssl from our client server and see what it shows > > > > Yes, please do that. Why wasn't that the first thing you tried? > > > >> [DMC] capacity issues, to be honest. Analysing this type of issue > >> in my organisation involves numerous teams across multiple > >> timezones and the communication overheads are enormous. Plus our > >> production servers are strictly controlled so I can't install it > >> myself. It's taken me a week to get this far and it's been on my > >> list all that time. > > Well, you can use "openssl s_client" from your own workstation, right? > That's a step: find out if the server just can't handle that cipher, > TLS1.2 handshakes, etc. No need to touch your own production servers. > > Also, you can download openssl and install it as a user: it's an > all-in-one .exe on Windows: no administrative privileges are required. > > (Also, I'm pretty sure that Tomcat ships with openssl.exe. Check that > out, too.) > > >>>> 3) consider forcing the Tomcat/3rd party Tomcat client to > >>>> use TLSv1 > > > > This is a possibility, but you shouldn't have to hamper your own > > application because there are some servers out there that are so > > old they don't support TLSv1.2: you should always offer the best > > encryption you can, and only fall-back when necessary. > > > >> [DMC] The 3rd party service only supports TLSv1. So as a > >> diagnostic step we have nothing to lose. > > If the 3rd-party service only supports TLSv1, then why is their client > library trying to connect using TLSv1.2? > > >>>> 4) check other logs on the server side to see if other > >>>> clients are failing TLS handshakes > > > > You didn't do this already? > > > >> [DMC] our server-side partners are under the same production > >> limitations that we are and insisted on discounting all possible > >> infrastructure and client possibilities before increasing > >> diagnostics. They did that today and found that the connections > >> are being received but are being terminated by a generic SSL > >> handshake error. > > Is that all connections from your servers, or only from those running > the upgraded Tomcat/Java versions? > > >>>> 6) consider upgrading Tomcat (as a last resort) > > > > I thought the initial problem was that an upgrade to Tomcat "broke > > something". I think it's more likely to be the Java upgrade that > > did something, and not Tomcat itself. > > > >> [DMC] No, sorry for not being clear. This production instance is > >> a greenfield installation and has never successfully connected > >> (browser and standalone tests excepted) > > > > Are you sure that the standalone test client is using the exact > > same version of Java as the Tomcat server is? > > > >> [DMC] 100%. There's only one version of java installed > > Okay. > > I'm interested to find out how this all ends. > > - -chris > -----BEGIN PGP SIGNATURE----- > Comment: GPGTools - http://gpgtools.org > > iQIcBAEBCAAGBQJV4LpoAAoJEBzwKT+lPKRYkdwP/jC08lLoa5P25ACD64wbv1ZA > VBnVjfcg00gGE8UmRYNxdk6Aheqoj2o8UlPXM7HenrwUAPFg8XUEv1Mj63mt6d0A > 1fLFn5fb5/KmozCNiOb/VwWKG5TnGvqHmfxbYgUHmzA5r1OV1rLzkYYYk6ed2ynw > XJi+za4LWRcr2RDrEQkgcqFx1aF1EWy7PPxCmiuNAKDv53zCnXfUyCe19bDmGJgo > IcT+bRA9mRYC5I7lzjLJwSThG3qyEHXR/YAfZHUt0Sd3Iv/7EacOoUyaUdmOsrAY > 5n9Hy98Pckeb9XvdmrhxRhPr7kQINC4F0AuJF+mKN0m84jPJgrQjYogvXhiIJTMM > WDHtjqbcM4xmd2OUKfbvFgYxHhBbOeSBDgKoV0Qpa8ydk1nPkCvyyjSHY3djyQod > mo5wQociKmSLZhMC/R/A2beGnC+lYl2mI9Zz0qUPIspq7ZoNOgr2iDfjPgKaeIZW > FhJ2jks0PoLt4AWwrs2A2Ia4/hSVOmlmW6d1eTt0uQIT9C4Zw6S33w4qWRNo8hrn > quQPZUsLY+snA+Gc1Et/ksH3GBGDZOgMlxd6qDmcSMFmSS/qEQepDYKryuvLtyqq > YPul5dwNgWCR6/j7cDBxgyqaTcNdScbxh1r6QIzu5BNLOfQtDl/BD3+x5u2WuP+G > aFlTGsFRQKpXPrtbZiDH > =92uM > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
