Hey Mark,
I found a workaround/fix. On the Tomcat Connector, instead of using
protocol=HTTP/1.1, I changed it to
protocol="org.apache.coyote.http11.Http11Nio2Protocol," I haven't had
a single failure since. Not only that, but our application response
times are noticeably faster.
--
Thanks,
Dan
On Wed, Jul 12, 2023 at 9:58 PM Dan McLaughlin <d...@danshome.net> wrote:
>
> Well, the deeper I get into the problem, the more complicated it gets. I
> thought I was onto something, thinking the size of the JSON might have
> something to do with it, so I created a Python script to call curl POSTs with
> increasingly larger JSON thinking I would eventually hit some size limit, but
> what I'm seeing is that it seems to fail less with smaller JSON files,
> although it will fail on just about any size. I even changed the Python
> script to retry failed POSTs, which will work on over half the second or
> third attempt.
>
> So I decided to try to create a maven project to build the test war, then
> start a Tomcat and Apache docker image..and I can't reproduce the issue.
>
> My worst fear was that I was dealing with something potentially
> Windows-specific, back to the drawing board.
>
> --
>
> Thanks,
>
> Dan
>
> On Wed, Jul 12, 2023 at 4:05 PM Dan McLaughlin <d...@danshome.net> wrote:
>>
>> Mark,
>>
>> I'm working on a test case. I've built a simple spring boot war with a rest
>> API "jsonInput" that accepts any JSON and responds with {"message":"OK"}.
>> What I've determined so far is that it only happens when you are proxying
>> the request through Apache using mod_proxy_http2, and the size of the JSON
>> that you are sending has something to do with the problem. I can send a
>> large JSON or a small one directly to Tomcat, which works. If I send a small
>> JSON through mod_proxy_http2, it also works, but if I send the JSON that our
>> client apps are sending, which is quite large, it fails.
>>
>> Before I spend more time on this test case, can you think of any setting in
>> Tomcat or mod_proxy_http2 that might cause the POST of the larger JSON to
>> fail?
>>
>> --
>>
>> Thanks,
>>
>> Dan
>>
>> On Wed, Jul 12, 2023 at 2:36 PM Mark Thomas <ma...@apache.org> wrote:
>>>
>>>
>>> 12 Jul 2023 13:40:18 Dan McLaughlin <d...@danshome.net.INVALID>:
>>>
>>> > I can confirm that if I switch h2 to http, everything works as
>>> > expected, change it back to h2 or h2c, and it breaks.
>>>
>>> That makes me think it is an h2 bug in Tomcat.
>>>
>>> > Mark, Please let me know if the http2 logs weren't enough to tell you
>>> > what's happening; if not, I'll work on creating a simple standalone
>>> > reproduction using docker.
>>>
>>> I've looked through those logs and don't see anything. Enabling debug
>>> logs for org.apache.tomcat.util.net might help but a reproducible test
>>> case is probably the easiest for us to work with.
>>>
>>> If you can avoid using docker that helps as it is one less thing for us
>>> to unpick when digging for the root cause but we can work with a docker
>>> based reproducible test case if needed.
>>>
>>> Mark
>>>
>>> >
>>> > --
>>> >
>>> > Thanks,
>>> >
>>> > Dan
>>> >
>>> > On Wed, Jul 12, 2023 at 6:00 AM Dan McLaughlin <d...@danshome.net>
>>> > wrote:
>>> >>
>>> >> Hi Mark,
>>> >>
>>> >> I already provided the output from org.apache.coyote.http2.level =
>>> >> FINE in the very first post to this thread. I didn't include
>>> >> everything because all the header information includes things I don't
>>> >> necessarily want to post publicly and because it would take a while
>>> >> for me to obfuscate. I will see if it's reproducible with a curl
>>> >> command and if I can reproduce it in a standalone docker image.
>>> >>
>>> >> I will also try with mod_proxy_http, as suggested by Chris.
>>> >>
>>> >> Let me know if there is more logging I truncated that you need to see
>>> >> that might tell you where the problem is; if I can provide it, I will.
>>> >>
>>> >> --
>>> >>
>>> >> Thanks,
>>> >>
>>> >> Dan
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Jul 12, 2023 at 3:34 AM Mark Thomas <ma...@apache.org> wrote:
>>> >>>
>>> >>> On 11/07/2023 19:10, Dan McLaughlin wrote:
>>> >>>> One other note, is I can switch to h2c, and it still fails, and a
>>> >>>> packet
>>> >>>> capture shows the entire JSON is delivered to Tomcat, and when I put
>>> >>>> the
>>> >>>> JSON from the packet inspection together (Packets 10199 --> 10208)
>>> >>>> and
>>> >>>> compare it to what the browser says was sent, they are identical.
>>> >>>> There are
>>> >>>> no signs of TCP retransmissions or other things you might expect to
>>> >>>> see if
>>> >>>> there was a networking related issue.
>>> >>>
>>> >>> Hi Dan,
>>> >>>
>>> >>> This looks like a possible Tomcat bug to me.
>>> >>>
>>> >>> To debug futher I'd suggest the following:
>>> >>>
>>> >>> Enable http2 debug logging by adding the following to
>>> >>> $CATALINA_BASE/conf/logging.properties
>>> >>>
>>> >>> org.apache.coyote.http2.level = FINE
>>> >>>
>>> >>> (that line should already be there, it just needs to be uncommented).
>>> >>>
>>> >>> If you can provide a curl command or similar that triggers this issue
>>> >>> then please feel free to open a Bugzilla issue and attached the
>>> >>> script
>>> >>> and any relevant configuration snippets for httpd etc and we can try
>>> >>> and
>>> >>> reproduce it.
>>> >>>
>>> >>> Thanks,
>>> >>>
>>> >>> Mark
>>> >>>
>>> >>> ---------------------------------------------------------------------
>>> >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> >>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>> >>>
>>> >
>>> > --
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > *NOTICE:* This e-mail message and all attachments transmitted with
>>> > it are for the sole use of the intended recipient(s) and may contain
>>> > confidential and privileged information. Any unauthorized review, use,
>>> > disclosure, or distribution is strictly prohibited. The contents of
>>> > this
>>> > e-mail are confidential and may be subject to work product privileges.
>>> > If
>>> > you are not the intended recipient, please contact the sender by reply
>>> > e-mail and destroy all copies of the original message.
>>> >
>>> >
>>> >
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org> For
>>> > additional commands, e-mail: users-h...@tomcat.apache.org
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>>
--
*NOTICE:* This e-mail message and all attachments transmitted with
it are for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use,
disclosure, or distribution is strictly prohibited. The contents of this
e-mail are confidential and may be subject to work product privileges. If
you are not the intended recipient, please contact the sender by reply
e-mail and destroy all copies of the original message.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
