Perfect! Thanks for checking! Florin
> On Aug 23, 2018, at 7:35 PM, Yu, Ping <ping...@intel.com> wrote: > > Thanks Florin, and this fix works pretty well. > Before the fix, it has problem to handle >500 sessions, and in this patch, it > works fine even >10000 sessions. J > > <> > <>From: vpp-dev@lists.fd.io [mailto:vpp-dev@lists.fd.io] On Behalf Of Florin > Coras > Sent: Friday, August 24, 2018 9:37 AM > To: Yu, Ping <ping...@intel.com> > Cc: vpp-dev@lists.fd.io; Florin Coras (fcoras) <fco...@cisco.com> > Subject: Re: [vpp-dev] TCP retransmission > > Ping, > > Does this [1] fix the issues you’ve encountered? > > Thanks, > Florin > > [1] https://gerrit.fd.io/r/#/c/14453/ <https://gerrit.fd.io/r/#/c/14453/> > > > On Aug 23, 2018, at 6:54 AM, Yu, Ping <ping...@intel.com > <mailto:ping...@intel.com>> wrote: > > By checking these active session, we can see that in client side, these > sessions has sent out 8192 bytes, and the received < 8192 bytes, so echo > client is still waiting for packets. > > Rx fifo: cursize 0 nitems 65536 has_event 0 > head 4284 tail 4284 segment manager 4 > vpp session 62 thread 0 app session 237 thread 0 > ooo pool 1 active elts newest 4294967295 > [2856, 3908], len 1052, next -1, prev -1 > Tx fifo: cursize 0 nitems 65536 has_event 0 > > > While at the server side, we can see even if server has received 8192 bytes > from clients, and even if the tx tail is set to be 8192, but the head is > <8192. > > > Rx fifo: cursize 0 nitems 65536 has_event 0 > head 8192 tail 8192 segment manager 2 > vpp session 4 thread 0 app session 0 thread 0 > ooo pool 0 active elts newest 4294967295 > Tx fifo: cursize 3908 nitems 65536 has_event 0 > head 4284 tail 8192 segment manager 2 > > thanks > Ping > > > From: Yu, Ping > Sent: Thursday, August 23, 2018 4:49 PM > To: Florin Coras <fcoras.li...@gmail.com <mailto:fcoras.li...@gmail.com>> > Cc: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>; Florin Coras (fcoras) > <fco...@cisco.com <mailto:fco...@cisco.com>>; Yu, Ping <ping...@intel.com > <mailto:ping...@intel.com>> > Subject: RE: [vpp-dev] TCP retransmission > > Verbose 1 also shows some info below. Interesting to see the session index > are continuous. > > [#0][T] 10.10.1.1:1111->10.10.1.2:23630 ESTABLISHED 0 > 63056 71 > [#0][T] 10.10.1.1:1111->10.10.1.2:59371 ESTABLISHED 0 > 63056 72 > [#0][T] 10.10.1.1:1111->10.10.1.2:20412 ESTABLISHED 0 > 63056 73 > [#0][T] 10.10.1.1:1111->10.10.1.2:2961 ESTABLISHED 0 > 63056 74 > [#0][T] 10.10.1.1:1111->10.10.1.2:52858 ESTABLISHED 0 > 63056 75 > [#0][T] 10.10.1.1:1111->10.10.1.2:58311 ESTABLISHED 0 > 63056 76 > [#0][T] 10.10.1.1:1111->10.10.1.2:49160 ESTABLISHED 0 > 63056 77 > [#0][T] 10.10.1.1:1111->10.10.1.2:38413 ESTABLISHED 0 > 63056 78 > [#0][T] 10.10.1.1:1111->10.10.1.2:65510 ESTABLISHED 0 > 63056 79 > [#0][T] 10.10.1.1:1111->10.10.1.2:3043 ESTABLISHED 0 > 63056 80 > [#0][T] 10.10.1.1:1111->10.10.1.2:15764 ESTABLISHED 0 > 63056 81 > [#0][T] 10.10.1.1:1111->10.10.1.2:21193 ESTABLISHED 0 > 63056 82 > [#0][T] 10.10.1.1:1111->10.10.1.2:56466 ESTABLISHED 0 > 63056 83 > [#0][T] 10.10.1.1:1111->10.10.1.2:64575 ESTABLISHED 0 > 63056 84 > [#0][T] 10.10.1.1:1111->10.10.1.2:54368 ESTABLISHED 0 > 63056 85 > [#0][T] 10.10.1.1:1111->10.10.1.2:32197 ESTABLISHED 0 > 63056 86 > [#0][T] 10.10.1.1:1111->10.10.1.2:36990 ESTABLISHED 0 > 63056 87 > [#0][T] 10.10.1.1:1111->10.10.1.2:37083 ESTABLISHED 0 > 63056 88 > [#0][T] 10.10.1.1:1111->10.10.1.2:11431 ESTABLISHED 0 > 61628 411 > [#0][T] 10.10.1.1:1111->10.10.1.2:49626 ESTABLISHED 0 > 61628 412 > [#0][T] 10.10.1.1:1111->10.10.1.2:42865 ESTABLISHED 0 > 61628 413 > [#0][T] 10.10.1.1:1111->10.10.1.2:31260 ESTABLISHED 0 > 61628 414 > [#0][T] 10.10.1.1:1111->10.10.1.2:20171 ESTABLISHED 0 > 61628 415 > [#0][T] 10.10.1.1:1111->10.10.1.2:54702 ESTABLISHED 0 > 61628 416 > [#0][T] 10.10.1.1:1111->10.10.1.2:8501 ESTABLISHED 0 > 61628 417 > [#0][T] 10.10.1.1:1111->10.10.1.2:49424 ESTABLISHED 0 > 61628 418 > > > > From: Yu, Ping > Sent: Thursday, August 23, 2018 4:40 PM > To: Florin Coras <fcoras.li...@gmail.com <mailto:fcoras.li...@gmail.com>> > Cc: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>; Florin Coras (fcoras) > <fco...@cisco.com <mailto:fco...@cisco.com>>; Yu, Ping <ping...@intel.com > <mailto:ping...@intel.com>> > Subject: RE: [vpp-dev] TCP retransmission > > Hi, Florin, > > Attached is the some active session live after echo client timeout. > > > [#0][T] 10.10.1.1:1111->10.10.1.2:61337 ESTABLISHED > flags: Recovery timers: [] > snd_una 4285 snd_nxt 4285 snd_una_max 4285 rcv_nxt 8193 rcv_las 8193 > snd_wnd 65536 rcv_wnd 65536 snd_wl1 8193 snd_wl2 4285 > flight size 0 out space 2856 cc space 2856 rcv_wnd_av 65536 > cong recovery cwnd 2856 ssthresh 2856 rtx_bytes 0 bytes_acked 0 > prev_ssthresh 524288 snd_congestion 8193 dupack 0 limited_transmit 1746905375 > tsecr 1714832397 tsecr_last_ack 1714832397 > rto 200 rto_boff 0 srtt 6 rttvar 2 rtt_ts 0 rtt_seq 2548066206 > tsval_recent 81891189 tsval_recent_age 13028 > scoreboard: sacked_bytes 0 last_sacked_bytes 0 lost_bytes 0 > last_bytes_delivered 0 high_sacked 2548066206 snd_una_adv 0 > cur_rxt_hole 4294967295 high_rxt 0 rescue_rxt 0 > Rx fifo: cursize 0 nitems 65536 has_event 0 > head 8192 tail 8192 segment manager 2 > vpp session 4 thread 0 app session 0 thread 0 > ooo pool 0 active elts newest 4294967295 > Tx fifo: cursize 3908 nitems 65536 has_event 0 > head 4284 tail 8192 segment manager 2 > vpp session 4 thread 0 app session 0 thread 0 > ooo pool 0 active elts newest 4294967295 > [#0][T] 10.10.1.1:1111->10.10.1.2:2210 ESTABLISHED > flags: Recovery timers: [] > snd_una 4285 snd_nxt 4285 snd_una_max 4285 rcv_nxt 8193 rcv_las 8193 > snd_wnd 65536 rcv_wnd 65536 snd_wl1 8193 snd_wl2 4285 > flight size 0 out space 2856 cc space 2856 rcv_wnd_av 65536 > cong recovery cwnd 2856 ssthresh 2856 rtx_bytes 0 bytes_acked 0 > prev_ssthresh 524288 snd_congestion 8193 dupack 0 limited_transmit 1746905375 > tsecr 1714832397 tsecr_last_ack 1714832397 > rto 200 rto_boff 0 srtt 6 rttvar 2 rtt_ts 0 rtt_seq 2548066206 > tsval_recent 81891189 tsval_recent_age 13028 > scoreboard: sacked_bytes 0 last_sacked_bytes 0 lost_bytes 0 > last_bytes_delivered 0 high_sacked 2548066206 snd_una_adv 0 > cur_rxt_hole 4294967295 high_rxt 0 rescue_rxt 0 > Rx fifo: cursize 0 nitems 65536 has_event 0 > head 8192 tail 8192 segment manager 2 > vpp session 5 thread 0 app session 0 thread 0 > ooo pool 0 active elts newest 4294967295 > Tx fifo: cursize 3908 nitems 65536 has_event 0 > head 4284 tail 8192 segment manager 2 > vpp session 5 thread 0 app session 0 thread 0 > ooo pool 0 active elts newest 4294967295 > [#0][T] 10.10.1.1:1111->10.10.1.2:30199 ESTABLISHED > flags: Recovery timers: [] > snd_una 5713 snd_nxt 5713 snd_una_max 5713 rcv_nxt 8193 rcv_las 8193 > snd_wnd 65536 rcv_wnd 65536 snd_wl1 8193 snd_wl2 5713 > flight size 0 out space 2856 cc space 2856 rcv_wnd_av 65536 > cong recovery cwnd 2856 ssthresh 2856 rtx_bytes 0 bytes_acked 0 > prev_ssthresh 524288 snd_congestion 8193 dupack 0 limited_transmit 1748569900 > tsecr 1714832397 tsecr_last_ack 1714832397 > rto 200 rto_boff 0 srtt 6 rttvar 1 rtt_ts 0 rtt_seq 2546405589 > tsval_recent 81891189 tsval_recent_age 13028 > scoreboard: sacked_bytes 0 last_sacked_bytes 0 lost_bytes 0 > last_bytes_delivered 0 high_sacked 2546403109 snd_una_adv 0 > cur_rxt_hole 4294967295 high_rxt 0 rescue_rxt 0 > Rx fifo: cursize 0 nitems 65536 has_event 0 > head 8192 tail 8192 segment manager 2 > vpp session 30 thread 0 app session 0 thread 0 > ooo pool 0 active elts newest 4294967295 > Tx fifo: cursize 2480 nitems 65536 has_event 0 > head 5712 tail 8192 segment manager 2 > vpp session 30 thread 0 app session 0 thread 0 > ooo pool 0 active elts newest 4294967295 > [#0][T] 10.10.1.1:1111->10.10.1.2:48888 ESTABLISHED > flags: Recovery timers: [] > snd_una 5713 snd_nxt 5713 snd_una_max 5713 rcv_nxt 8193 rcv_las 8193 > snd_wnd 65536 rcv_wnd 65536 snd_wl1 8193 snd_wl2 5713 > flight size 0 out space 2856 cc space 2856 rcv_wnd_av 65536 > cong recovery cwnd 2856 ssthresh 2856 rtx_bytes 0 bytes_acked 0 > prev_ssthresh 524288 snd_congestion 8193 dupack 0 limited_transmit 1748569900 > tsecr 1714832397 tsecr_last_ack 1714832397 > rto 200 rto_boff 0 srtt 6 rttvar 1 rtt_ts 0 rtt_seq 2546405589 > tsval_recent 81891189 tsval_recent_age 13028 > scoreboard: sacked_bytes 0 last_sacked_bytes 0 lost_bytes 0 > last_bytes_delivered 0 high_sacked 2546403109 snd_una_adv 0 > cur_rxt_hole 4294967295 high_rxt 0 rescue_rxt 0 > Rx fifo: cursize 0 nitems 65536 has_event 0 > head 8192 tail 8192 segment manager 2 > vpp session 31 thread 0 app session 0 thread 0 > ooo pool 0 active elts newest 4294967295 > Tx fifo: cursize 2480 nitems 65536 has_event 0 > head 5712 tail 8192 segment manager 2 > vpp session 31 thread 0 app session 0 thread 0 > ooo pool 0 active elts newest 4294967295 > > From: Florin Coras [mailto:fcoras.li...@gmail.com > <mailto:fcoras.li...@gmail.com>] > Sent: Tuesday, August 21, 2018 11:18 PM > To: Yu, Ping <ping...@intel.com <mailto:ping...@intel.com>> > Cc: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>; Florin Coras (fcoras) > <fco...@cisco.com <mailto:fco...@cisco.com>> > Subject: Re: [vpp-dev] TCP retransmission > > Hi Ping, > > Just from this, I can’t tell precisely what the issue is. Once the test > hangs, do “show session verbose 2” and you should get the detailed status of > the sessions that froze. I’ll take a look at those once I’m back in the > office. Retransmissions should work fine but one never knows. On the other > hand, the echo apps may be missing some io events. Did vpp report any other > error message? > > Florin > > > On Aug 21, 2018, at 7:14 AM, Yu, Ping <ping...@intel.com > <mailto:ping...@intel.com>> wrote: > > Hello, all > > I am now debugging an issue observed in echo server/client. If nclients == > small number, then everything goes smoothly, but if nclients is big, such as > 500, 1000, echo client will hang without printing final report, and debug > shows that echo client is waiting for response from server as it is expecting > server to send bytes_to_receive to client. The problems occurs in both TCP > and TLS protocol. > > if (sp->bytes_to_receive > 0) > { > delete_session = 0; > } > > My assumption is that due to the big traffic, some packet is missing and > re-transmission does not work well. Below the info for normal case, where > nclients == 100. > > DBGvpp# show error > Count Node Reason > 600 session-queue Packets transmitted > 100 tcp4-rcv-process Pure ACKs received > 100 tcp4-rcv-process FINs received > 600 tcp4-established Packets pushed into rx fifo > 600 tcp4-established Pure ACKs received > 32 ip4-glean address overflow drops > 1 ip4-glean ARP requests sent > 1 arp-input ARP request IP4 source > address learned > > > Server: > DBGvpp# show error > Count Node Reason > 600 session-queue Packets transmitted > 100 tcp4-listen SYNs received > 200 tcp4-rcv-process Pure ACKs received > 600 tcp4-established Packets pushed into rx fifo > 600 tcp4-established Pure ACKs received > 100 tcp4-established FINs received > 1 arp-input ARP replies sent > > > Below is the info when nclients == 500. > > Client: > > DBGvpp# show error > Count Node Reason > 3104 session-queue Packets transmitted > 498 tcp4-rcv-process Pure ACKs received > 498 tcp4-rcv-process FINs received > 2998 tcp4-established Packets pushed into rx fifo > 2892 tcp4-established Pure ACKs received > 106 tcp4-established Duplicate ACK > > Server: > DBGvpp# show error > Count Node Reason > 2998 session-queue Packets transmitted > 500 tcp4-listen SYNs received > 998 tcp4-rcv-process Pure ACKs received > 2996 tcp4-established Packets pushed into rx fifo/ > 2 tcp4-established OOO packets pushed into rx > fifo > 2476 tcp4-established Pure ACKs received > 498 tcp4-established FINs received > > We can see that in clients side, packets transmission is 3104, which is > bigger than normal case 600*5, but in the server side, the recived packet is > only 2996, which is below 3000, which causes only 498 FIN is sent from > client, so 2 live session makes “test echo” hang until timeout. > > > Thanks > Ping > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#10233): https://lists.fd.io/g/vpp-dev/message/10233 > <https://lists.fd.io/g/vpp-dev/message/10233> > Mute This Topic: https://lists.fd.io/mt/24876529/675152 > <https://lists.fd.io/mt/24876529/675152> > Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io> > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub > <https://lists.fd.io/g/vpp-dev/unsub> [fcoras.li...@gmail.com > <mailto:fcoras.li...@gmail.com>] > -=-=-=-=-=-=-=-=-=-=-=- > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#10276): https://lists.fd.io/g/vpp-dev/message/10276 > <https://lists.fd.io/g/vpp-dev/message/10276> > Mute This Topic: https://lists.fd.io/mt/24876529/675152 > <https://lists.fd.io/mt/24876529/675152> > Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io> > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub > <https://lists.fd.io/g/vpp-dev/unsub> [fcoras.li...@gmail.com > <mailto:fcoras.li...@gmail.com>] > -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10277): https://lists.fd.io/g/vpp-dev/message/10277 Mute This Topic: https://lists.fd.io/mt/24876529/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
