Hi Raj, The function used for receiving datagrams is limited to reading at most the length of a datagram from the rx fifo. UDP datagrams are mtu sized, so your reads are probably limited to ~1.5kB. On each epoll rx event try reading from the session handle in a while loop until you get an VPPCOM_EWOULDBLOCK. That might improve performance.
Having said that, udp is lossy so unless you implement your own congestion/flow control algorithms, the data you’ll receive might be full of “holes”. What are the rx/tx error counters on your interfaces (check with “sh int”). Also, with simple tuning like this [1], you should be able to achieve much more than 15Gbps with tcp. Regards, Florin [1] https://wiki.fd.io/view/VPP/HostStack/LDP/iperf > On Jan 19, 2020, at 3:25 PM, Raj Kumar <raj.gauta...@gmail.com> wrote: > > Hi Florin, > By using VCL library in an UDP receiver application, I am able to receive > only 2 Mbps traffic. On increasing the traffic, I see Rx FIFO full error and > application stopped receiving the traffic from the session layer. Whereas, > with TCP I can easily achieve 15Gbps throughput without tuning any DPDK > parameter. UDP tx also looks fine. From an host application I can send > ~5Gbps without any issue. > > I am running VPP( stable/2001 code) on RHEL8 server using Mellanox 100G > (MLNX5) adapters. > Please advise if I can use VCL library to receive high throughput UDP traffic > ( in Gbps). I would be running multiple instances of host application to > receive data ( ~50-60 Gbps). > > I also tried by increasing the Rx FIFO size to 16MB but did not help much. > The host application is just throwing the received packets , it is not doing > any packet processing. > > [root@orc01 vcl_test]# VCL_DEBUG=2 ./udp6_server_vcl > VCL<20201>: configured VCL debug level (2) from VCL_DEBUG! > VCL<20201>: allocated VCL heap = 0x7f39a17ab010, size 268435456 (0x10000000) > VCL<20201>: configured rx_fifo_size 4000000 (0x3d0900) > VCL<20201>: configured tx_fifo_size 4000000 (0x3d0900) > VCL<20201>: configured app_scope_local (1) > VCL<20201>: configured app_scope_global (1) > VCL<20201>: configured api-socket-name (/tmp/vpp-api.sock) > VCL<20201>: completed parsing vppcom config! > vppcom_connect_to_vpp:480: vcl<20201:0>: app (udp6_server) is connected to > VPP! > vppcom_app_create:1104: vcl<20201:0>: sending session enable > vppcom_app_create:1112: vcl<20201:0>: sending app attach > vppcom_app_create:1121: vcl<20201:0>: app_name 'udp6_server', my_client_index > 256 (0x100) > vppcom_epoll_create:2439: vcl<20201:0>: Created vep_idx 0 > vppcom_session_create:1179: vcl<20201:0>: created session 1 > vppcom_session_bind:1317: vcl<20201:0>: session 1 handle 1: binding to local > IPv6 address fd0d:edc4:ffff:2001::203 port 8092, proto UDP > vppcom_session_listen:1349: vcl<20201:0>: session 1: sending vpp listen > request... > vcl_session_bound_handler:604: vcl<20201:0>: session 1 [0x1]: listen > succeeded! > vppcom_epoll_ctl:2541: vcl<20201:0>: EPOLL_CTL_ADD: vep_sh 0, sh 1, events > 0x1, data 0x1! > vppcom_session_create:1179: vcl<20201:0>: created session 2 > vppcom_session_bind:1317: vcl<20201:0>: session 2 handle 2: binding to local > IPv6 address fd0d:edc4:ffff:2001::203 port 8093, proto UDP > vppcom_session_listen:1349: vcl<20201:0>: session 2: sending vpp listen > request... > vcl_session_app_add_segment_handler:765: vcl<20201:0>: mapped new segment > '20190-2' size 134217728 > vcl_session_bound_handler:604: vcl<20201:0>: session 2 [0x2]: listen > succeeded! > vppcom_epoll_ctl:2541: vcl<20201:0>: EPOLL_CTL_ADD: vep_sh 0, sh 2, events > 0x1, data 0x2! > > > vpp# sh session verbose 2 > [#0][U] fd0d:edc4:ffff:2001::203:8092->:::0 > > Rx fifo: cursize 3999125 nitems 3999999 has_event 1 > head 2554045 tail 2553170 segment manager 1 > vpp session 0 thread 0 app session 1 thread 0 > ooo pool 0 active elts newest 4294967295 > Tx fifo: cursize 0 nitems 3999999 has_event 0 > head 0 tail 0 segment manager 1 > vpp session 0 thread 0 app session 1 thread 0 > ooo pool 0 active elts newest 0 > [#0][U] fd0d:edc4:ffff:2001::203:8093->:::0 > > Rx fifo: cursize 0 nitems 3999999 has_event 0 > head 0 tail 0 segment manager 2 > vpp session 1 thread 0 app session 2 thread 0 > ooo pool 0 active elts newest 0 > Tx fifo: cursize 0 nitems 3999999 has_event 0 > head 0 tail 0 segment manager 2 > vpp session 1 thread 0 app session 2 thread 0 > ooo pool 0 active elts newest 0 > Thread 0: active sessions 2 > > [root@orc01 vcl_test]# cat /etc/vpp/vcl.conf > vcl { > rx-fifo-size 4000000 > tx-fifo-size 4000000 > app-scope-local > app-scope-global > api-socket-name /tmp/vpp-api.sock > } > [root@orc01 vcl_test]# > > ------------------- Start of thread 0 vpp_main ------------------- > Packet 1 > > 00:09:53:445025: dpdk-input > HundredGigabitEthernet12/0/0 rx queue 0 > buffer 0x88078: current data 0, length 1516, buffer-pool 0, ref-count 1, > totlen-nifb 0, trace handle 0x0 > ext-hdr-valid > l4-cksum-computed l4-cksum-correct > PKT MBUF: port 0, nb_segs 1, pkt_len 1516 > buf_len 2176, data_len 1516, ol_flags 0x180, data_off 128, phys_addr > 0x75601e80 > packet_type 0x2e1 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 > rss 0x0 fdir.hi 0x0 fdir.lo 0x0 > Packet Offload Flags > PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid > PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid > Packet Types > RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet > RTE_PTYPE_L3_IPV6_EXT_UNKNOWN (0x00e0) IPv6 packet with or without > extension headers > RTE_PTYPE_L4_UDP (0x0200) UDP packet > IP6: b8:83:03:79:9f:e4 -> b8:83:03:79:af:8c 802.1q vlan 2001 > UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203 > tos 0x00, flow label 0x0, hop limit 64, payload length 1458 > UDP: 56944 -> 8092 > length 1458, checksum 0xb22d > 00:09:53:445028: ethernet-input > frame: flags 0x3, hw-if-index 2, sw-if-index 2 > IP6: b8:83:03:79:9f:e4 -> b8:83:03:79:af:8c 802.1q vlan 2001 > 00:09:53:445029: ip6-input > UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203 > tos 0x00, flow label 0x0, hop limit 64, payload length 1458 > UDP: 56944 -> 8092 > length 1458, checksum 0xb22d > 00:09:53:445031: ip6-lookup > fib 0 dpo-idx 6 flow hash: 0x00000000 > UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203 > tos 0x00, flow label 0x0, hop limit 64, payload length 1458 > UDP: 56944 -> 8092 > length 1458, checksum 0xb22d > 00:09:53:445032: ip6-local > UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203 > tos 0x00, flow label 0x0, hop limit 64, payload length 1458 > UDP: 56944 -> 8092 > length 1458, checksum 0xb22d > 00:09:53:445032: ip6-udp-lookup > UDP: src-port 56944 dst-port 8092 > 00:09:53:445033: udp6-input > UDP_INPUT: connection 0, disposition 5, thread 0 > > > thanks, > -Raj > > > On Wed, Jan 15, 2020 at 4:09 PM Raj Kumar via Lists.Fd.Io > <http://lists.fd.io/> <raj.gautam25=gmail....@lists.fd.io > <mailto:gmail....@lists.fd.io>> wrote: > Hi Florin, > Yes, [2] patch resolved the IPv6/UDP receiver issue. > Thanks! for your help. > > thanks, > -Raj > > On Tue, Jan 14, 2020 at 9:35 PM Florin Coras <fcoras.li...@gmail.com > <mailto:fcoras.li...@gmail.com>> wrote: > Hi Raj, > > First of all, with this [1], the vcl test app/client can establish a udpc > connection. Note that udp will most probably lose packets, so large exchanges > with those apps may not work. > > As for the second issue, does [2] solve it? > > Regards, > Florin > > [1] https://gerrit.fd.io/r/c/vpp/+/24332 > <https://gerrit.fd.io/r/c/vpp/+/24332> > [2] https://gerrit.fd.io/r/c/vpp/+/24334 > <https://gerrit.fd.io/r/c/vpp/+/24334> > >> On Jan 14, 2020, at 12:59 PM, Raj Kumar <raj.gauta...@gmail.com >> <mailto:raj.gauta...@gmail.com>> wrote: >> >> Hi Florin, >> Thanks! for the reply. >> >> I realized the issue with the non-connected case. For receiving datagrams, >> I was using recvfrom() with DONOT_WAIT flag because of that >> vppcom_session_recvfrom() api was failing. It expects either 0 or MSG_PEEK >> flag. >> if (flags == 0) >> rv = vppcom_session_read (session_handle, buffer, buflen); >> else if (flags & MSG_PEEK) 0x2 >> rv = vppcom_session_peek (session_handle, buffer, buflen); >> else >> { >> VDBG (0, "Unsupport flags for recvfrom %d", flags); >> return VPPCOM_EAFNOSUPPORT; >> } >> >> I changed the flag to 0 in recvfrom() , after that UDP rx is working fine >> but only for IPv4. >> >> I am facing a different issue with IPv6/UDP receiver. I am getting "no >> listener for dst port" error. >> >> Please let me know if I am doing something wrong. >> Here are the traces : - >> >> [root@orc01 testcode]# VCL_DEBUG=2 LDP_DEBUG=2 >> LD_PRELOAD=/opt/vpp/build-root/install-vpp-native/vpp/lib/libvcl_ldpreload.so >> VCL_CONFIG=/etc/vpp/vcl.cfg ./udp6_rx >> VCL<1164>: configured VCL debug level (2) from VCL_DEBUG! >> VCL<1164>: allocated VCL heap = 0x7ff877439010, size 268435456 (0x10000000) >> VCL<1164>: configured rx_fifo_size 4000000 (0x3d0900) >> VCL<1164>: configured tx_fifo_size 4000000 (0x3d0900) >> VCL<1164>: configured app_scope_local (1) >> VCL<1164>: configured app_scope_global (1) >> VCL<1164>: configured api-socket-name (/tmp/vpp-api.sock) >> VCL<1164>: completed parsing vppcom config! >> vppcom_connect_to_vpp:549: vcl<1164:0>: app (ldp-1164-app) is connected to >> VPP! >> vppcom_app_create:1067: vcl<1164:0>: sending session enable >> vppcom_app_create:1075: vcl<1164:0>: sending app attach >> vppcom_app_create:1084: vcl<1164:0>: app_name 'ldp-1164-app', >> my_client_index 0 (0x0) >> ldp_init:209: ldp<1164>: configured LDP debug level (2) from env var >> LDP_DEBUG! >> ldp_init:282: ldp<1164>: LDP initialization: done! >> ldp_constructor:2490: LDP<1164>: LDP constructor: done! >> socket:974: ldp<1164>: calling vls_create: proto 1 (UDP), is_nonblocking 0 >> vppcom_session_create:1142: vcl<1164:0>: created session 0 >> bind:1086: ldp<1164>: fd 32: calling vls_bind: vlsh 0, addr 0x7fff9a93efe0, >> len 28 >> vppcom_session_bind:1280: vcl<1164:0>: session 0 handle 0: binding to local >> IPv6 address :: port 8092, proto UDP >> vppcom_session_listen:1312: vcl<1164:0>: session 0: sending vpp listen >> request... >> vcl_session_bound_handler:610: vcl<1164:0>: session 0 [0x1]: listen >> succeeded! >> bind:1102: ldp<1164>: fd 32: returning 0 >> >> vpp# sh app server >> Connection App Wrk >> [0:0][CT:U] :::8092->:::0 ldp-1164-app[shm] 0 >> [#0][U] :::8092->:::0 ldp-1164-app[shm] 0 >> >> vpp# sh err >> Count Node Reason >> 7 dpdk-input no error >> 2606 ip6-udp-lookup no listener for dst port >> 8 arp-reply ARP replies sent >> 1 arp-disabled ARP Disabled on this >> interface >> 13 ip6-glean neighbor solicitations sent >> 2606 ip6-input valid ip6 packets >> 4 ip6-local-hop-by-hop Unknown protocol ip6 local >> h-b-h packets dropped >> 2606 ip6-icmp-error destination unreachable >> response sent >> 40 ip6-icmp-input valid packets >> 1 ip6-icmp-input neighbor solicitations >> from source not on link >> 12 ip6-icmp-input neighbor solicitations for >> unknown targets >> 1 ip6-icmp-input neighbor advertisements >> sent >> 1 ip6-icmp-input neighbor advertisements >> received >> 40 ip6-icmp-input router advertisements sent >> 40 ip6-icmp-input router advertisements >> received >> 1 ip4-icmp-input echo replies sent >> 89 lldp-input lldp packets received on >> disabled interfaces >> 1328 llc-input unknown llc ssap/dsap >> vpp# >> >> vpp# show trace >> ------------------- Start of thread 0 vpp_main ------------------- >> Packet 1 >> >> 00:23:39:401354: dpdk-input >> HundredGigabitEthernet12/0/0 rx queue 0 >> buffer 0x8894e: current data 0, length 1516, buffer-pool 0, ref-count 1, >> totlen-nifb 0, trace handle 0x0 >> ext-hdr-valid >> l4-cksum-computed l4-cksum-correct >> PKT MBUF: port 0, nb_segs 1, pkt_len 1516 >> buf_len 2176, data_len 1516, ol_flags 0x180, data_off 128, phys_addr >> 0x75025400 >> packet_type 0x2e1 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 >> rss 0x0 fdir.hi 0x0 fdir.lo 0x0 >> Packet Offload Flags >> PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid >> PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid >> Packet Types >> RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet >> RTE_PTYPE_L3_IPV6_EXT_UNKNOWN (0x00e0) IPv6 packet with or without >> extension headers >> RTE_PTYPE_L4_UDP (0x0200) UDP packet >> IP6: b8:83:03:79:9f:e4 -> b8:83:03:79:af:8c 802.1q vlan 2001 >> UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203 >> tos 0x00, flow label 0x0, hop limit 64, payload length 1458 >> UDP: 60593 -> 8092 >> length 1458, checksum 0x0964 >> 00:23:39:401355: ethernet-input >> frame: flags 0x3, hw-if-index 2, sw-if-index 2 >> IP6: b8:83:03:79:9f:e4 -> b8:83:03:79:af:8c 802.1q vlan 2001 >> 00:23:39:401356: ip6-input >> UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203 >> tos 0x00, flow label 0x0, hop limit 64, payload length 1458 >> UDP: 60593 -> 8092 >> length 1458, checksum 0x0964 >> 00:23:39:401357: ip6-lookup >> fib 0 dpo-idx 5 flow hash: 0x00000000 >> UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203 >> tos 0x00, flow label 0x0, hop limit 64, payload length 1458 >> UDP: 60593 -> 8092 >> length 1458, checksum 0x0964 >> 00:23:39:401361: ip6-local >> UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203 >> tos 0x00, flow label 0x0, hop limit 64, payload length 1458 >> UDP: 60593 -> 8092 >> length 1458, checksum 0x0964 >> 00:23:39:401362: ip6-udp-lookup >> UDP: src-port 60593 dst-port 8092 (no listener) >> 00:23:39:401362: ip6-icmp-error >> UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203 >> tos 0x00, flow label 0x0, hop limit 64, payload length 1458 >> UDP: 60593 -> 8092 >> length 1458, checksum 0x0964 >> 00:23:39:401363: error-drop >> rx:HundredGigabitEthernet12/0/0.2001 >> 00:23:39:401364: drop >> ip6-input: valid ip6 packets >> >> vpp# >> >> >> Thanks, >> -Raj >> >> >> On Tue, Jan 14, 2020 at 1:44 PM Florin Coras <fcoras.li...@gmail.com >> <mailto:fcoras.li...@gmail.com>> wrote: >> Hi Raj, >> >> Session layer does support connection-less transports but udp does not raise >> accept notifications to vcl. UDPC might, but we haven’t tested udpc with vcl >> in a long time so it might not work properly. >> >> What was the problem you were hitting in the non-connected case? >> >> Regards, >> Florin >> >> > On Jan 14, 2020, at 7:13 AM, raj.gauta...@gmail.com >> > <mailto:raj.gauta...@gmail.com> wrote: >> > >> > Hi , >> > I am trying some host application tests ( using LD_PRELOAD) . TCP rx and >> > tx both work fine. UDP tx also works fine. >> > The issue is only with UDP rx . In some discussion it was mentioned that >> > session layer does not support connection-less transports so protocols >> > like udp still need to accept connections and only afterwards read from >> > the fifos. >> > So, I changed the UDP receiver application to use listen() and accept() >> > before read() . But , I am still having issue to make it run. >> > After I started, udp traffic from other server it seems to accept the >> > connection but never returns from the vppcom_session_accept() function. >> > VPP release is 19.08. >> > >> > vpp# sh app server >> > Connection App Wrk >> > [0:0][CT:U] 0.0.0.0:8090->0.0.0.0:0 <http://0.0.0.0:0/> >> > ldp-36646-app[shm] 0 >> > [#0][U] 0.0.0.0:8090->0.0.0.0:0 <http://0.0.0.0:0/> >> > ldp-36646-app[shm] 0 >> > vpp# >> > >> > >> > [root@orc01 testcode]# VCL_DEBUG=2 LDP_DEBUG=2 >> > LD_PRELOAD=/opt/vpp/build-root/install-vpp-native/vpp/lib/libvcl_ldpreload.so >> > VCL_CONFIG=/etc/vpp/vcl.cfg ./udp_rx >> > VCL<36646>: configured VCL debug level (2) from VCL_DEBUG! >> > VCL<36646>: allocated VCL heap = 0x7f77e5309010, size 268435456 >> > (0x10000000) >> > VCL<36646>: configured rx_fifo_size 4000000 (0x3d0900) >> > VCL<36646>: configured tx_fifo_size 4000000 (0x3d0900) >> > VCL<36646>: configured app_scope_local (1) >> > VCL<36646>: configured app_scope_global (1) >> > VCL<36646>: configured api-socket-name (/tmp/vpp-api.sock) >> > VCL<36646>: completed parsing vppcom config! >> > vppcom_connect_to_vpp:549: vcl<36646:0>: app (ldp-36646-app) is connected >> > to VPP! >> > vppcom_app_create:1067: vcl<36646:0>: sending session enable >> > vppcom_app_create:1075: vcl<36646:0>: sending app attach >> > vppcom_app_create:1084: vcl<36646:0>: app_name 'ldp-36646-app', >> > my_client_index 0 (0x0) >> > ldp_init:209: ldp<36646>: configured LDP debug level (2) from env var >> > LDP_DEBUG! >> > ldp_init:282: ldp<36646>: LDP initialization: done! >> > ldp_constructor:2490: LDP<36646>: LDP constructor: done! >> > socket:974: ldp<36646>: calling vls_create: proto 1 (UDP), is_nonblocking 0 >> > vppcom_session_create:1142: vcl<36646:0>: created session 0 >> > Socket successfully created.. >> > bind:1086: ldp<36646>: fd 32: calling vls_bind: vlsh 0, addr >> > 0x7fff3f3c1040, len 16 >> > vppcom_session_bind:1280: vcl<36646:0>: session 0 handle 0: binding to >> > local IPv4 address 0.0.0.0 port 8090, proto UDP >> > vppcom_session_listen:1312: vcl<36646:0>: session 0: sending vpp listen >> > request... >> > vcl_session_bound_handler:610: vcl<36646:0>: session 0 [0x1]: listen >> > succeeded! >> > bind:1102: ldp<36646>: fd 32: returning 0 >> > Socket successfully binded.. >> > listen:2005: ldp<36646>: fd 32: calling vls_listen: vlsh 0, n 5 >> > vppcom_session_listen:1308: vcl<36646:0>: session 0 [0x1]: already in >> > listen state! >> > listen:2020: ldp<36646>: fd 32: returning 0 >> > Server listening.. >> > ldp_accept4:2043: ldp<36646>: listen fd 32: calling vppcom_session_accept: >> > listen sid 0, ep 0x0, flags 0x3f3c0fc0 >> > vppcom_session_accept:1478: vcl<36646:0>: discarded event: 0 >> > >> > >> > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15200): https://lists.fd.io/g/vpp-dev/message/15200 Mute This Topic: https://lists.fd.io/mt/69694900/21656 Mute #vpp-hoststack: https://lists.fd.io/mk?hashtag=vpp-hoststack&subid=1480452 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
