Hi All,
I have been able to get around my issue by creating more than 1 connection
with the web server/service. I got this clue from a message I posted on
github issues page. Here is a link to the issue in case anyone is
interested:
https://github.com/golang/go/issues/52845
It appears that the HTTP/2 client was blocked in previous requests when I
increased the TPS rate to more than ~2500. This caused those requests
blocked to get timed out as it appears that the timer starts from the time
the HTTP/2 Do method is invoked. And by the time the message being sent out
gets a chance in the queue, it does not have enough time to get written out
to the socket. This causes the "RoundTripper context deadline exceeded"
errors.
On Thu, May 12, 2022 at 5:58 PM envee <neeraj.vaidy...@gmail.com> wrote:
> Hi Robert, do you have any inputs about this ?
>
> On Wednesday, 11 May 2022 at 08:31:24 UTC+10 envee wrote:
>
>> Ok, so you can ignore my comment about the incomplete payload being sent.
>> It is just that it is being sent in 2 frames as "partial entities".
>> But I am still wondering why there is a context deadline exceeded which
>> then results in a RST_STREAM with CANCEL being sent by my client ?
>> it's just 100ms which have elapsed since the last part of the payload was
>> sent in the DATA frame.
>>
>> On Wednesday, 11 May 2022 at 07:47:15 UTC+10 envee wrote:
>>
>>> Thanks for your reply, Robert.
>>>
>>> I enabled GODEBUG for http2 in my client application and noticed the
>>> following sequence of log lines for any particular stream for which I see
>>> my client sending an RST_STREAM frame.
>>>
>>> 2022/05/10 21:59:12.827474 http2: Framer 0xc23f448000: wrote HEADERS
>>> flags=END_HEADERS stream=247467 len=65
>>> 2022/05/10 21:59:12.896018 http2: Framer 0xc23f448000: wrote DATA
>>> stream=247467 len=75 data="{\n\t\"invocationSequenceNumber\":
>>> 1,\n\t\"invocationTimeStamp\": \"2022-05-10T11:59"
>>> 2022/05/10 21:59:12.996612 RoundTrip failure: context deadline exceeded
>>> 2022/05/10 21:59:12.996692 http2: Framer 0xc23f448000: wrote RST_STREAM
>>> stream=247467 len=4 ErrCode=CANCEL
>>>
>>> Usually, the payload that I send in my DATA frame is of size ~2400
>>> bytes. And for such "successful" requests in the logs I can see that it
>>> only prints some part and says "(2153 bytes omitted)". That is fair enough.
>>>
>>> But in this erroneous case, the DATA frame size is shown as 75 bytes and
>>> the log line does not contain any such extra information about "omitting
>>> bytes" from the log.
>>> I am wondering if my client failed to upload the entire JSON payload and
>>> hence this caused it to timeout waiting for Header.
>>>
>>> My client timeout is set to 5seconds, but with the logs being shown for
>>> the particular failing stream, why is there a timeout after just 100
>>> milli-seconds i.e. between sending off the DATA frame and waiting for
>>> HEADER frame ?
>>>
>>> On Tuesday, 10 May 2022 at 22:49:25 UTC+10 ren...@ix.netcom.com wrote:
>>>
>>>> Conn.Error() should give you the underlying error as well.
>>>>
>>>>
>>>> On May 10, 2022, at 7:47 AM, robert engels <ren...@ix.netcom.com>
>>>> wrote:
>>>>
>>>> I would add some logging to
>>>>
>>>> func (cs *clientStream) cleanupWriteRequest(err error)
>>>>
>>>> but you should be getting the underlying error when you perform the
>>>> next write on the stream - maybe you are not processing this error properly
>>>> (there is a code - CANCEL, but also an err).
>>>>
>>>> On May 10, 2022, at 6:59 AM, Robert Engels <ren...@ix.netcom.com>
>>>> wrote:
>>>>
>>>> Set a breakpoint on the RST_STREAM generation and run under the
>>>> debugger.
>>>>
>>>> On May 10, 2022, at 4:18 AM, envee <neeraj....@gmail.com> wrote:
>>>>
>>>> Hi All, I am using a Go HTTP/2 client (no SSL) to send requests at the
>>>> rate of around 3000 transactions per second (TPS) to a Web-server which is
>>>> based on Netty.
>>>>
>>>> At lower rates of say around 500 TPS , I do not see this issue, but at
>>>> rates as high as 2500 - 3000 TPS, I can see that my Go HTTP/2 client sends
>>>> RST_STREAM frame right after transmitting the JSON payload via the DATA
>>>> frame. The Error in the RST_STREAM is seen as CANCEL even though the
>>>> response from the server for the Request has not been received. Also, there
>>>> is no timeout that has occured which might cause this.
>>>>
>>>> This occurs for some requests and not all, maybe about 100 out of the
>>>> 3000 per second.
>>>>
>>>> If you observe the screenshot attached, the HEADER is transmitted in
>>>> packet 118847 in stream id 49765. The DATA frame is in packet 122113.
>>>>
>>>> Right after that, the client transmits RST_STREAM for stream id 49765
>>>> in packet 122277.
>>>>
>>>> And then this results in an error on the response from the Server in
>>>> packet 122483 and 122485 which says "Request stream 49765 is not correct
>>>> for server connection"
>>>>
>>>> Any clues what could be causing this issue of the client sending
>>>> RST_STREAM before receiving the HTTP response from the server ?
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "golang-nuts" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to golang-nuts...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/golang-nuts/56766546-356f-4901-b3d1-827be5bb0947n%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/golang-nuts/56766546-356f-4901-b3d1-827be5bb0947n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>> <Screenshot 2022-05-10 190439.png>
>>>>
>>>>
>>>>
>>>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/ebea4500-f99d-46d7-856c-67d6e01d5cdan%40googlegroups.com
> <https://groups.google.com/d/msgid/golang-nuts/ebea4500-f99d-46d7-856c-67d6e01d5cdan%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CALFCev4zXUi2yrejOn5S41DHYvax%2BH35xzs0-q4LgEnxf8VieA%40mail.gmail.com.
