Apologies, this reply got away from me... :-)
Perhaps grab a fresh $BEVERAGE before reading. ----- Original Message ----- > From: "Christopher Schultz" <ch...@christopherschultz.net> > To: users@tomcat.apache.org > Sent: Friday, December 6, 2019 9:51:52 AM > Subject: Re: Tomcat - No Fork for debugging? > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Alex, > > On 12/5/19 18:05, Alex Scheel wrote: > > ----- Original Message ----- > >> From: "Christopher Schultz" <ch...@christopherschultz.net> To: > >> users@tomcat.apache.org Sent: Thursday, December 5, 2019 5:05:42 > >> PM Subject: Re: Tomcat - No Fork for debugging? > >> > > Alex, > > > > On 12/5/19 15:18, Alex Scheel wrote: > >>>> Hi all, > >>>> > >>>> I've tried searching to no avail. > >>>> > >>>> I'm working on a(nother) SSL adapter. However, I've had some > >>>> issues with it. There's a native component and I'm trying to > >>>> tease apart its relationship with why the client won't > >>>> handshake. The stack traces aren't overly helpful and I'd > >>>> love to attach gdb to this and set a few specific environment > >>>> variables. > >>>> > >>>> Query: > >>>> > >>>> Does Tomcat have a mode where it won't fork to a different > >>>> user and will run with a limited number of threads? That'd > >>>> greatly improve my ability to debug. Something similar to > >>>> `radiusd -X` or `sshd -d`? > > > > You can accomplish this in two steps: > > > > 1. Change your <Connector> to have an <Executor> with only a > > single thread, or set maxThreads="1" on the <Connector> if you > > aren't separately configuring an <Executor> > > > > 2. From the command line, run: > > > > $ bin/catalina.sh run > > > >> I think that behaves better. Thanks! > > No problem. > > > This will run Tomcat directly in the terminal window (or, on > > Windows, open a second terminal window where it will run). > > > > You can see stdout in that terminal, and you can press CTRL-C to > > kill the process. > > > >>>> Rationale: > >>>> > >>>> NSS has support for logging calls to its PKCS#11 interface to > >>>> a file, based on the presence of environment variables. When > >>>> I set these environment variables and directly call the JVM > >>>> to start Tomcat: > >>>> > >>>> # java -classpath $CLASSPATH $FLAGS > >>>> org.apache.catalina.startup.Bootstrap start > >>>> > >>>> I see it logging calls when the JDK starts up, but when I hit > >>>> it with wget on the TLS port, the resulting PKCS#11 calls > >>>> aren't logged. When launching in gdb, I get an error about > >>>> /sbin/nologin doesn't understand the -c option, which to me > >>>> says that Bootstrap is trying to fork and create a new shell > >>>> (I'm running as root in a VM and it wants to launch as the > >>>> tomcat user), dropping my environmental variables I want. > >>>> > >>>> Ideally (for debugging) I'd like to simplify this. Is there a > >>>> more direct entry route I can use perhaps? > > > > Is there a JSSE wrapper for NSS? You can just plug-in the crypto > > provider for the SSLContext instead of writing your own connector. > > Wait. You said "adapter". What kind of "adapter" are you writing? > > > >> Ah sorry, my terminology was loose there. I'm writing another > >> org.apache.tomcat.util.net.SSLImplementation implementation. > >> Heh. > > > >> I'm the maintainer of JSS, a NSS wrapper in Java. This is mostly > >> utilized by the Dogtag project. > > > >> I also co-maintain TomcatJSS, a JSS adapter for Tomcat < 8.5. > >> When used with Tomcat >= 8.5, it uses JSSE's SSLEngine but > >> initializes JSS based on configuration in server.xml and use it > >> for the keystore. Mostly this is because we've not yet gotten > >> around to adding a SSLEngine to JSS (but will likely finish that > >> work soon -- JSS was started at Netscape back in the late 90s and > >> predated the SSLEngine being added to JVM interfaces in JDK > >> version 5). RHEL 7 still ships with Tomcat 7, but RHEL 8 only > >> has Tomcat 9, so the hard requirement for a SSLEngine is new. > >> We're getting there :) > > > >> With FIPS certification requirements though, we can't use JSSE > >> (since it isn't FIPS certified). > > I was pretty sure Java's crypto implementation was FIPS certified. I > haven't looked in a while, but a quick good search turned up: > > https://www2.cs.duke.edu/csed/java/jdk1.7/technotes/guides/security/jsse > /FIPS.html > > and > > https://docs.oracle.com/javase/6/docs/technotes/guides/security/enhancem > ents.html > > That last one suggests that Oracle Java already contains a JSSE > provider which uses NSS. DO I misunderstad? Also relevant: https://docs.oracle.com/javase/8/docs/technotes/guides/security/p11guide.html#NSS Ugh... This is where I confess I'm not an expert on FIPS stuff. Here's my best understanding: We're talking about the same provider (that's good!). The subtle point I think you're missing is that FIPS isn't the _default_ configuration and it imposes more restrictions when in FIPS mode. It also needs special configuration (which in RHEL looks like): fips.provider.1=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.fips.cfg fips.provider.2=sun.security.provider.Sun fips.provider.3=sun.security.ec.SunEC fips.provider.4=com.sun.net.ssl.internal.ssl.Provider SunPKCS11-NSS-FIPS <--- "FIPS" In comparison, the default (non-FIPS mode providers) are: security.provider.1=sun.security.provider.Sun security.provider.2=sun.security.rsa.SunRsaSign security.provider.3=sun.security.ec.SunEC security.provider.4=com.sun.net.ssl.internal.ssl.Provider <--- "Non-FIPS" security.provider.5=com.sun.crypto.provider.SunJCE security.provider.6=sun.security.jgss.SunProvider security.provider.7=com.sun.security.sasl.Provider security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI security.provider.9=sun.security.smartcardio.SunPCSC security.provider.10=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg Now, where my understanding gets more fuzzy... RHEL ships OpenJDK and upstream itself doesn't do the FIPS certification; it is Red Hat that does the certification of NSS on RHEL. JDK itself, under the FIPS configuration above, now does no crypto. All the crypto is done via PKCS#11 to NSS. So there's nothing (in OpenJDK) to certify any more. Unlike in the non-FIPS mode above, there's lots of crypto that would need to be certified. Oracle, in their JDK, might ship their own PKCS#11 provider and/or certify NSS, I don't know. I know IBM ships their own FIPS crypto provider, and probably the next most popular is BouncyCastle, but I thought that might be losing popularity because they might've stopped FIPS certifying it, who knows. But my key point is there's no "OpenJDK FIPS Certificate" on the FIPS page: https://csrc.nist.gov/projects/cryptographic-module-validation-program/validated-modules/search (Search Java) But there is one for NSS ("NSS" - and then its "NSS Cryptographic Module" on the results page). So yes, the SunJSSE provider is still used but the key differences are: 1) The crypto is done by SunPKCS11-NSS-FIPS provider, 2) The NSS that is used is FIPS certified, 3) It places additional restrictions (more about that below). If you don't really care 2 (the token "Certificate") and just want to test stuff, 1 and 3 work on any distro (e.g., Ubuntu and Fedora), and NSS everywhere understands FIPS mode. Also the SunPKCS11 / SunPKCS11-NSS-FIPS providers are limited to Linux, iirc. That's how I understand the above and no, you don't seem to misunderstand. :) > >> OpenJDK supports NSS via the Sun-PKCS11 provider and has a > >> SSLEngine implementation under the SunJSSE provider. It has some > >> restrictions on it that means it can't be used from Tomcat's > >> JSSE adapter just by swapping out the provider name (for one, > >> wrapping the KeyManager like Tomcat does isn't allowed). > > Hmm. Maybe there's a better way to do it, then, and you don't have to > write so much code. What's the problem with wrapping a KeyManager? That's an RFE I've filed internally against the JDK team. To me, it makes no sense. It fails now with Tomcat's JSSEImplementation : Dec 06, 2019 10:20:04 AM org.apache.catalina.util.LifecycleBase handleSubClassException SEVERE: Failed to initialize component [Connector[HTTP/1.1-8443]] org.apache.catalina.LifecycleException: Protocol handler initialization failed at org.apache.catalina.connector.Connector.initInternal(Connector.java:935) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.core.StandardService.initInternal(StandardService.java:530) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:852) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.startup.Catalina.load(Catalina.java:633) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:306) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:491) Caused by: java.lang.IllegalArgumentException: FIPS mode: only SunJSSE KeyManagers may be used at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:114) at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:85) at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:216) at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1043) at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:540) at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:74) at org.apache.catalina.connector.Connector.initInternal(Connector.java:932) ... 12 more Caused by: java.security.KeyManagementException: FIPS mode: only SunJSSE KeyManagers may be used at sun.security.ssl.SSLContextImpl.chooseKeyManager(SSLContextImpl.java:154) at sun.security.ssl.SSLContextImpl.engineInit(SSLContextImpl.java:71) at javax.net.ssl.SSLContext.init(SSLContext.java:282) at org.apache.tomcat.util.net.jsse.JSSESSLContext.init(JSSESSLContext.java:53) at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:112) ... 18 more My config roughly looks like: <Connector name="Secure" port="8443" scheme="https" secure="true" SSLEnabled="true" keyAlias="localhost" keystorePass="keystorePass" keystoreType="PKCS11" keystoreProvider="SunPKCS11-NSS-FIPS" keyManagerAlgorithm="SunX509" truststoreAlgorithm="PKCS11" truststoreProvider="SunJSSE" truststoreType="SunX509" truststorePassword="truststorePassword" SSLProtocol="TLSv1.2" maxThreads="1" /> (Yeah, that's messy -- but it illustrates the point -- you can probably remove a number of those elements -- *including* keyAlias, and it'll still fail). This is because, for all KeyManagers, Tomcat wraps it its own KeyManager: https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/util/net/SSLUtilBase.java#L378 Probably the minimum untested change that I'd have proposed would be to wrap it in an if block like you have ealier: https://github.com/apache/tomcat/blob/master/java/org/apache/tomcat/util/net/SSLUtilBase.java#L195 --- Now the core of the _why_ is because that's how NSS behaves. In FIPS mode, NSS doesn't let you import keys. You can either have keys in a NSS DB, or you can wrap/unwrap them into one (or the temporary, in-memory key store). But you can't just call PK11_ImportSymKey (or the related function for private keys) -- they'll fail! That's because NSS was a FIPS level 2 certified provider, unlike say, OpenSSL which only undergoes FIPS level 1 certification. Some of those restrictions that level 2 places are around key management and handling, including where they're sourced from and how they're protected in transit. One of the RFEs I've been discussing with our JDK team have been to loosen the restrictions of the SunJSSE provider in FIPS mode. Restricting the origin of the KeyManager (while having good intentions of preventing the above) means that some otherwise valid use cases -- like selecting a single key alias from the server.xml config -- won't work. Really they should only care that the key used is from the SunPKCS11-NSS-FIPS provider, they don't care how they got it and from which KeyManager it came from. Those changes are still under discussion. --- There's also about ~4 or 5 other weird bugs in this SunJSSE FIPS interface that make it less than usable or cause it to fail to handshake in FIPS mode, but they don't really concern Tomcat. Those are why I'm not proposing a PR yet. :-) > > >> I was trying to see if the JDK's SSLEngine would work for us. On > >> RHEL, we FIPS certify our shipped NSS release so the JDK's > >> Sun-PKCS11-NSS-FIPS provider is FIPS certified transitively > >> (since it does no crypto... :). Since I'm not a Tomcat developer > >> (and am more familiar with TomcatJSS), I stubbed out a new > >> SSLImplementation that uses SunJSSE so I could bypass the parts > >> of the Tomcat JSSEImplementation that don't work with the JDK > >> SunJSSE provider. > > That sounds like it would work, but also sound like a huge amount of > plumbing. What can you tell me about the way Tomcat uses the JSSE API > that precludes FIPS support from the existing tools -- including your > own JSS? Mostly that's discussed above w.r.t. SunJSSE + SunPKCS11-NSS-FIPS. Why JSS? Dogtag is a self-hosted CA using NSS and came out of Netscape's certificate offerings before landing at Red Hat. We support RSA and ECDSA certificates and also have strong HSM support (thanks to NSS). In part, this means we can install a CA and have both the CA and the SSL server certificates for the CA interface never leave the HSM (or the NSS DB). Since (especially in FIPS mode) HSMs generally frown upon extracting keys, that makes using say, BouncyCastle or Tomcat's OpenSSL adapter harder, since we'd need to configure and support HSM access from both NSS and $PROVIDER simultaneously. Far easier to use a PKCS#11 provider with NSS (where we've already set up the HSM and NSS DB). That leaves well, JSS or SunJSSE+SunPKCS11-NSS. For internal reasons, the FIPS mode SunJSSE provider wasn't well... supported until relatively recently. So for most of our time, we've been fine using JSS (via TomcatJSS's BIO adapter on Tomcat < 8.5). Its only recently that we've needed an NIO adapter (so JSS hasn't yet gotten a SSLEngine) -- and only recently that we've considered SunJSSE again. So: 1) We can't use Tomcat's JSSE API because we don't have a SSLEngine in JSS yet, 2) We can't use SunJSSE+SunPKCS11-NSS-FIPS because it doesn't work. Currently we ignore FIPS (on Tomcat >= 8.5) and use the regular, non-FIPS SunJSSE. The only thing we really use TomcatJSS for is to initialize JSS early enough in the Tomcat startup sequence (before the app starts up) -- so that we can use JSS as the keystore for Tomcat and SunJSSE (which the non-FIPS mode adapter is happy to do). For the rest of the app, we do need JSS though. In part because its one of the few Java providers that provide CMAC and (soon) KBKDF, but we also use it for all our certificate issuance operations. I'd like to get to the point where either work but both need a few bug fixes and/or RFEs first. :-) > > > >> However, currently on RHEL the SunJSSE provider is a bit broken, > >> so this FIPSImplementation isn't really useful. If you're > >> interested, I'd like to get Tomcat to a place where it supports > >> FIPS mode JDK without the need for a separate implementation, but > >> first I need a working FIPS-mode JDK (currently it doesn't > >> handshake)... ...and we're probably going the route of finishing > >> JSS anyways. > > Hmm. Is your goal to get a working JSSE+FIPS implementation > specifically with JSS or do you just want to get FIPS working with > Tomcat? The OpenSSL provider can be put into FIPS mode, provided that > it was build with FIPS support of course. I think I covered OpenSSL above. We have considered it and it would be possible. But we'd prefer a NSS backed provider. So either JSSE+SunPKCS11-NSS-FIPS or JSSE+/-JSS. :-) - Alex > > - -chris > -----BEGIN PGP SIGNATURE----- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3qawcACgkQHPApP6U8 > pFg1RBAAi1ANc0Bcf9rlZ/QnjWe4QAg3mA/lftAMjI6/l2QIwbIDf7Pt66a/0e2H > h/Vzv8xK15+HWovknzrvZrIbHX7duisLeYNV4nV6QCjYeCviiw8ZPTKvrOUcdnfC > pvYDyZ+K/lTr2dCSVV94trbzYkexgS5cPS6T4zdZuhszMH8Z6dY/lUj6kl4R9eQt > 6meWPzlXljCOxPamJC2XsgM1dmxwz4MCoI46sj6qPvqNPrpglKEZqPytMl3EXZT1 > UmEYTglYZLixZL9D56WSzF89FyGKKWyYueqRfaI/iM04tupn7baT+k8SVDroRM4g > I7dkIlJUO/t0WpxIEPHRul7dW+R4I/Dr6miAkm/vhKtVMV5n0gsQL5md7G/eOTls > 9QJcBZktkzdtvznyBNniMZ3+QFFXZkftuOzB+l3VUwqJ6K0OvtFxMqiy7ET/ZvVA > ZOyGKnrmIF+JXne+hOAm5Dj80R1DMQl8oITSY3mc/bRPWo96mdfFlc9SYs8ZiDDG > uaxocXWA8zKRrQNQdVkmHmlB2Tr6i4Nz4MulXlhz7FHLXD2J30mrJuaBBYdmS8qk > XTlkhflpOsr9hB7GoMkjWcTNjm+mZtcvpLR9p1VJgd5ewounl4d16Az6IphtSyqV > wXU3KDJ889TSTLlYJoK/s/FCCFsSCM6E3ER9rVbwJsUXEu7C0ow= > =pu3l > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
