Greetings I'm running tomcat v9.0.105 with a CXF based SOAP service (you know, the old school JAXWS services). Within that service, I had a need to retrieve a specific http header and i've been running into some inconsistent results. The tomcat server is sitting behind a nginx proxy if that matters. Sometimes the header was there, sometimes it wasn't. It was more consistent if I send the header twice with different casing. x-forwarded-for vs X-Forwarded-For. (issue 1)
In order to troubleshoot the inconsistency, i wrote a little routine to simple get the servlet request, get the header names, enumerate them and print all the keys and values to the log file for troubleshooting. Then i ran into an even stranger problem... sometimes, but not always, I'm getting a NPE being thrown from tomcat code and I honestly have no idea why (issue 2) First the stack trace for (issue 2)... at org.apache.tomcat.util.http.ValuesEnumerator.findNext(MimeHeaders.java:509) [tomcat-coyote.jar:9.0.105] at org.apache.tomcat.util.http. ValuesEnumerator.nextElement(MimeHeaders.java:525) [tomcat-coyote.jar:9.0.105] at org.apache.tomcat.util.http. ValuesEnumerator.nextElement(MimeHeaders.java:490) [tomcat-coyote.jar:9.0.105] at org.apache.cxf.transport.http.HttpServletRequestSnapshot.getHeader(HttpServletRequestSnapshot.java:124) [cxf-rt-transports-http-3.5.5.jar] Now checking that specific line number in the source code...yields something odd... https://github.com/apache/tomcat/blob/9.0.x/java/org/apache/tomcat/util/http/MimeHeaders.java#L509 and it's @Override line... so i dug a bit further and decompiled the jar in order to understand what's going on here. (issue 3) private void findNext() { this.next=null; for (; this.pos < this.size; this.pos++) { MessageBytes n1 = this.headers.getName(this.pos); //fails here! if (n1.equalsIgnoreCase(this.name)) { this.next = this.headers.getValue(this.pos); break; } } this.pos++; } Apologies for the formatting as i had to retype this by hand. This would seem to imply that there's an extra 9 lines or so of code in the jar i'm using, which was from a docker public container tomcat:9.0-jdk11 Additional info: I did wireshark this and was able to see the message before going to tomcat and it looks ok. Headers all appeared to be there and were well formed, nothing out of the ordinary. Questions/issue summary (1) apparent inconsistent access to a specific http header in tomcat with behind a proxy. Sometimes it's there, something it isn't. (2) NPE being thrown when enumerating headers. Any idea why am i getting a NPE in this case? (3) should be concerned that the git repo's source line numbers do not align with what the jar is reporting? Hoping this isn't some kind of supply chain attack Some theories I've cooked up on questions (1, 2) Since it's a CXF JAXWS SOAP service with springboot servlet, i'm pretty sure the only way to get http headers is via @Resource WebServiceContext injection and then dig down to get the servlet request. I was always under the impression that this should be thread safe, but if it isn't, it's possible that the MimeHeaders class, at least according to the javadocs may have recycled before i got it to, thus making the 'headers' variable be cleared out. Or it could be a bug in cxf (it is an older version but I'm stuck until we can update JREs) or maybe a tomcat bug. I did note that someone reporting this issue many years ago on stack overflow, specifically here: https://stackoverflow.com/questions/37493552 but as i said, the request looks ok and no different than any other setup. The only other thing that might be a factor was that we recently renewed the server certificate key pair. Seems unlikely though. Pretty sure I'm not using the tomcat native libraries for anything related to TLS Anyhow, a long one but hopefully someone will have an idea. TIA
