13 Jul 2023 13:31:29 Dan McLaughlin <d...@danshome.net.INVALID>:
Mark,
Never mind the last message, I thought it was working, but I had
looked at my test balancer config and forgot I had left the
application config using http instead of h2. So it's still busted. I
am running out of time on my end to spend on this at the moment.
Since it seems to be a Windows-specific issue, it will take me a lot
longer to set up a way to reproduce it since I don't have the
flexibility of spinning up docker images. I will be moving back to
mod_jk for now since I know that works. If I can provide you anything
else as far as logs from our environment that might point to what's
going on without me having to create a way to reproduce it for a bug
report I'd be happy to do so, but for now, I have to get back to
working on other things on my plate.
Understood.
Can I make one final request?
Please open a BZ issue and summarise which combinations cause problems
and which don't. OS for each component (https, Tomcat, client) and
whether they are all on the same host is likely relevant as well as the
usual version and config info.
I'll try and reproduce it when I have time. Worst case it will get put
into NEEDINFO state until we can figure out how to trigger the issue.
Thanks,
Mark
--
Thanks,
Dan
On Wed, Jul 12, 2023 at 10:56 PM Dan McLaughlin <d...@danshome.net>
wrote:
Hey Mark,
I found a workaround/fix. On the Tomcat Connector, instead of using
protocol=HTTP/1.1, I changed it to
protocol="org.apache.coyote.http11.Http11Nio2Protocol," I haven't had
a single failure since. Not only that, but our application response
times are noticeably faster.
--
Thanks,
Dan
On Wed, Jul 12, 2023 at 9:58 PM Dan McLaughlin <d...@danshome.net>
wrote:
Well, the deeper I get into the problem, the more complicated it
gets. I thought I was onto something, thinking the size of the JSON
might have something to do with it, so I created a Python script to
call curl POSTs with increasingly larger JSON thinking I would
eventually hit some size limit, but what I'm seeing is that it seems
to fail less with smaller JSON files, although it will fail on just
about any size. I even changed the Python script to retry failed
POSTs, which will work on over half the second or third attempt.
So I decided to try to create a maven project to build the test war,
then start a Tomcat and Apache docker image..and I can't reproduce
the issue.
My worst fear was that I was dealing with something potentially
Windows-specific, back to the drawing board.
--
Thanks,
Dan
On Wed, Jul 12, 2023 at 4:05 PM Dan McLaughlin <d...@danshome.net>
wrote:
Mark,
I'm working on a test case. I've built a simple spring boot war with
a rest API "jsonInput" that accepts any JSON and responds with
{"message":"OK"}. What I've determined so far is that it only
happens when you are proxying the request through Apache using
mod_proxy_http2, and the size of the JSON that you are sending has
something to do with the problem. I can send a large JSON or a
small one directly to Tomcat, which works. If I send a small JSON
through mod_proxy_http2, it also works, but if I send the JSON that
our client apps are sending, which is quite large, it fails.
Before I spend more time on this test case, can you think of any
setting in Tomcat or mod_proxy_http2 that might cause the POST of
the larger JSON to fail?
--
Thanks,
Dan
On Wed, Jul 12, 2023 at 2:36 PM Mark Thomas <ma...@apache.org>
wrote:
12 Jul 2023 13:40:18 Dan McLaughlin <d...@danshome.net.INVALID>:
I can confirm that if I switch h2 to http, everything works as
expected, change it back to h2 or h2c, and it breaks.
That makes me think it is an h2 bug in Tomcat.
Mark, Please let me know if the http2 logs weren't enough to tell
you
what's happening; if not, I'll work on creating a simple
standalone
reproduction using docker.
I've looked through those logs and don't see anything. Enabling
debug
logs for org.apache.tomcat.util.net might help but a reproducible
test
case is probably the easiest for us to work with.
If you can avoid using docker that helps as it is one less thing
for us
to unpick when digging for the root cause but we can work with a
docker
based reproducible test case if needed.
Mark
--
Thanks,
Dan
On Wed, Jul 12, 2023 at 6:00 AM Dan McLaughlin <d...@danshome.net>
wrote:
Hi Mark,
I already provided the output from org.apache.coyote.http2.level
=
FINE in the very first post to this thread. I didn't include
everything because all the header information includes things I
don't
necessarily want to post publicly and because it would take a
while
for me to obfuscate. I will see if it's reproducible with a curl
command and if I can reproduce it in a standalone docker image.
I will also try with mod_proxy_http, as suggested by Chris.
Let me know if there is more logging I truncated that you need to
see
that might tell you where the problem is; if I can provide it, I
will.
--
Thanks,
Dan
On Wed, Jul 12, 2023 at 3:34 AM Mark Thomas <ma...@apache.org>
wrote:
On 11/07/2023 19:10, Dan McLaughlin wrote:
One other note, is I can switch to h2c, and it still fails, and
a
packet
capture shows the entire JSON is delivered to Tomcat, and when
I put
the
JSON from the packet inspection together (Packets 10199 -->
10208)
and
compare it to what the browser says was sent, they are
identical.
There are
no signs of TCP retransmissions or other things you might
expect to
see if
there was a networking related issue.
Hi Dan,
This looks like a possible Tomcat bug to me.
To debug futher I'd suggest the following:
Enable http2 debug logging by adding the following to
$CATALINA_BASE/conf/logging.properties
org.apache.coyote.http2.level = FINE
(that line should already be there, it just needs to be
uncommented).
If you can provide a curl command or similar that triggers this
issue
then please feel free to open a Bugzilla issue and attached the
script
and any relevant configuration snippets for httpd etc and we can
try
and
reproduce it.
Thanks,
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
--
*NOTICE:* This e-mail message and all attachments transmitted with
it are for the sole use of the intended recipient(s) and may
contain
confidential and privileged information. Any unauthorized review,
use,
disclosure, or distribution is strictly prohibited. The contents
of
this
e-mail are confidential and may be subject to work product
privileges.
If
you are not the intended recipient, please contact the sender by
reply
e-mail and destroy all copies of the original message.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org> For
additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
--
*NOTICE:* This e-mail message and all attachments transmitted with
it are for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use,
disclosure, or distribution is strictly prohibited. The contents of
this
e-mail are confidential and may be subject to work product privileges.
If
you are not the intended recipient, please contact the sender by reply
e-mail and destroy all copies of the original message.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org> For
additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
