Hi Florin,
I was trying this test with debug binaries, but as soon as I enable the
interface , vpp is crashing.
On the original problem ( multiple listener); if I open multiple sockets
from the same multi threaded application then it works fine. But, if I
start another application then only I see the VPP crash( which I mentioned
in my previous email).
Here is the stack trace with debug binaries which is crashing on startup.
Please let me know what wrong I am doing here ?
#0 0x00007ffff4b748df in raise () from /lib64/libc.so.6
#1 0x00007ffff4b5ecf5 in abort () from /lib64/libc.so.6
#2 0x0000000000407a28 in os_panic () at /opt/vpp/src/vpp/vnet/main.c:366
#3 0x00007ffff5a279af in debugger () at /opt/vpp/src/vppinfra/error.c:84
#4 0x00007ffff5a27d92 in _clib_error (how_to_die=2,
function_name=0x7ffff663d540 <__FUNCTION__.35997>
"vlib_buffer_validate_alloc_free",
line_number=367, fmt=0x7ffff663d0ba "%s %U buffer 0x%x") at
/opt/vpp/src/vppinfra/error.c:143
#5 0x00007ffff65752a3 in vlib_buffer_validate_alloc_free
(vm=0x7ffff6866340 <vlib_global_main>, buffers=0x7fffb586d630, n_buffers=1,
expected_state=VLIB_BUFFER_KNOWN_ALLOCATED) at
/opt/vpp/src/vlib/buffer.c:366
#6 0x00007ffff65675f0 in vlib_buffer_pool_put (vm=0x7ffff6866340
<vlib_global_main>, buffer_pool_index=0 '\000', buffers=0x7fffb586d630,
n_buffers=1) at /opt/vpp/src/vlib/buffer_funcs.h:754
#7 0x00007ffff6567dde in vlib_buffer_free_inline (vm=0x7ffff6866340
<vlib_global_main>, buffers=0x7fffb67e9b14, n_buffers=0, maybe_next=1)
at /opt/vpp/src/vlib/buffer_funcs.h:924
#8 0x00007ffff6567e2e in vlib_buffer_free (vm=0x7ffff6866340
<vlib_global_main>, buffers=0x7fffb67e9b10, n_buffers=1)
at /opt/vpp/src/vlib/buffer_funcs.h:943
#9 0x00007ffff6568cb0 in process_drop_punt (vm=0x7ffff6866340
<vlib_global_main>, node=0x7fffb5475e00, frame=0x7fffb67e9b00,
disposition=ERROR_DISPOSITION_DROP) at /opt/vpp/src/vlib/drop.c:231
#10 0x00007ffff6568db0 in error_drop_node_fn_hsw (vm=0x7ffff6866340
<vlib_global_main>, node=0x7fffb5475e00, frame=0x7fffb67e9b00)
at /opt/vpp/src/vlib/drop.c:247
#11 0x00007ffff65c3447 in dispatch_node (vm=0x7ffff6866340
<vlib_global_main>, node=0x7fffb5475e00, type=VLIB_NODE_TYPE_INTERNAL,
dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x7fffb67e9b00,
last_time_stamp=15480427682338516) at /opt/vpp/src/vlib/main.c:1235
#12 0x00007ffff65c3c02 in dispatch_pending_node (vm=0x7ffff6866340
<vlib_global_main>, pending_frame_index=2,
last_time_stamp=15480427682338516) at /opt/vpp/src/vlib/main.c:1403
#13 0x00007ffff65c58c9 in vlib_main_or_worker_loop (vm=0x7ffff6866340
<vlib_global_main>, is_main=1) at /opt/vpp/src/vlib/main.c:1862
#14 0x00007ffff65c6320 in vlib_main_loop (vm=0x7ffff6866340
<vlib_global_main>) at /opt/vpp/src/vlib/main.c:1990
#15 0x00007ffff65c70e0 in vlib_main (vm=0x7ffff6866340 <vlib_global_main>,
input=0x7fffb586efb0) at /opt/vpp/src/vlib/main.c:2236
#16 0x00007ffff662f311 in thread0 (arg=140737329390400) at
/opt/vpp/src/vlib/unix/main.c:658
#17 0x00007ffff5a465cc in clib_calljmp () at
/opt/vpp/src/vppinfra/longjmp.S:123
#18 0x00007fffffffd0a0 in ?? ()
#19 0x00007ffff662f8a7 in vlib_unix_main (argc=50, argv=0x705f00) at
/opt/vpp/src/vlib/unix/main.c:730
#20 0x0000000000407387 in main (argc=50, argv=0x705f00) at
/opt/vpp/src/vpp/vnet/main.c:291
thanks,
-Raj
On Mon, May 25, 2020 at 5:02 PM Florin Coras <fcoras.li...@gmail.com> wrote:
> Hi Raj,
>
> Okay, so at least with that we have support for bounded listeners (note
> that [2] was merged but to set the connected option you now have to
> use vppcom_session_attr).
>
> As for the trace, something seems off. Why exactly does it crash? It looks
> as if session_get_transport_proto (ls) crashes because of ls being null,
> but prior to that ls is dereferenced and it does’t crash. Could you try
> with debug binaries?
>
> Regards,
> Florin
>
> On May 25, 2020, at 1:43 PM, Raj Kumar <raj.gauta...@gmail.com> wrote:
>
> Hi Florin,
> This works fine with single udp listener. I can see connections going on
> different cores. But, if I run more than one listener then VPP is crashing.
> Here are the VPP stack traces -
>
> (gdb) bt
> #0 0x0000000000000000 in ?? ()
> #1 0x00007ffff7761239 in session_listen (ls=<optimized out>,
> sep=sep@entry=0x7fffb557fd50)
> at /opt/vpp/src/vnet/session/session_types.h:247
> #2 0x00007ffff7788b3f in app_listener_alloc_and_init
> (app=app@entry=0x7fffb76f7d98,
> sep=sep@entry=0x7fffb557fd50,
> listener=listener@entry=0x7fffb557fd28) at
> /opt/vpp/src/vnet/session/application.c:196
> #3 0x00007ffff7788ed8 in vnet_listen (a=a@entry=0x7fffb557fd50) at
> /opt/vpp/src/vnet/session/application.c:1005
> #4 0x00007ffff7779e08 in session_mq_listen_handler (data=0x1300787e9) at
> /opt/vpp/src/vnet/session/session_node.c:65
> #5 session_mq_listen_handler (data=data@entry=0x1300787e9) at
> /opt/vpp/src/vnet/session/session_node.c:36
> #6 0x00007ffff7bbcdb9 in vl_api_rpc_call_t_handler (mp=0x1300787d0) at
> /opt/vpp/src/vlibmemory/vlib_api.c:520
> #7 0x00007ffff7bc5ead in vl_msg_api_handler_with_vm_node
> (am=am@entry=0x7ffff7dd2ea0
> <api_global_main>, vlib_rp=<optimized out>,
> the_msg=0x1300787d0, vm=vm@entry=0x7ffff6d7c200 <vlib_global_main>,
> node=node@entry=0x7fffb553f000, is_private=is_private@entry=0 '\000')
> at /opt/vpp/src/vlibapi/api_shared.c:609
> #8 0x00007ffff7bafee6 in vl_mem_api_handle_rpc (vm=vm@entry=0x7ffff6d7c200
> <vlib_global_main>, node=node@entry=0x7fffb553f000)
> at /opt/vpp/src/vlibmemory/memory_api.c:748
> #9 0x00007ffff7bbd5b3 in vl_api_clnt_process (vm=<optimized out>,
> node=0x7fffb553f000, f=<optimized out>)
> at /opt/vpp/src/vlibmemory/vlib_api.c:326
> #10 0x00007ffff6b1b116 in vlib_process_bootstrap (_a=<optimized out>) at
> /opt/vpp/src/vlib/main.c:1502
> #11 0x00007ffff602fbfc in clib_calljmp () from
> /opt/vpp/build-root/build-vpp-native/vpp/lib/libvppinfra.so.20.05
> #12 0x00007fffb5fa2dd0 in ?? ()
> #13 0x00007ffff6b1e751 in vlib_process_startup (f=0x0, p=0x7fffb553f000,
> vm=0x7ffff6d7c200 <vlib_global_main>)
> at /opt/vpp/src/vppinfra/types.h:133
> #14 dispatch_process (vm=0x7ffff6d7c200 <vlib_global_main>,
> p=0x7fffb553f000, last_time_stamp=14118080390223872, f=0x0)
> at /opt/vpp/src/vlib/main.c:1569
> #15 0x00000000004b84e0 in ?? ()
> #16 0x0000000000000000 in ?? ()
>
> thanks,
> -Raj
>
> On Mon, May 25, 2020 at 2:17 PM Florin Coras <fcoras.li...@gmail.com>
> wrote:
>
>> Hi Raj,
>>
>> Ow, now you’ve hit the untested part of [2]. Could you try this [3]?
>>
>> Regards,
>> Florin
>>
>> [3] https://gerrit.fd.io/r/c/vpp/+/27235
>>
>> On May 25, 2020, at 10:44 AM, Raj Kumar <raj.gauta...@gmail.com> wrote:
>>
>> Hi Florin,
>>
>> I tried the patches[1] & [2] , but still VCL application is crashing.
>> However, session is created in VPP.
>>
>> vpp# sh session verbose 2
>> [0:0][U] 2001:5b0:ffff:700:b883:31f:29e:9880:9978-LISTEN
>> index 0 flags: CONNECTED, OWNS_PORT, LISTEN
>> Thread 0: active sessions 1
>> Thread 1: no sessions
>> Thread 2: no sessions
>> Thread 3: no sessions
>> Thread 4: no sessions
>> vpp# sh app server
>> Connection App Wrk
>> [0:0][U] 2001:5b0:ffff:700:b883:31f:29e:udp6_rx[shm] 1
>> vpp#
>>
>> Here are the VCL application traces. Attached is the updated vppcom.c
>> file.
>>
>> [root@J3SGISNCCRO01 vcl_test]# VCL_DEBUG=2 gdb
>> udp6_server_vcl_threaded_udpc
>> GNU gdb (GDB) Red Hat Enterprise Linux 8.2-6.el8
>> Copyright (C) 2018 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <
>> http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.
>> Type "show copying" and "show warranty" for details.
>> This GDB was configured as "x86_64-redhat-linux-gnu".
>> Type "show configuration" for configuration details.
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>.
>> Find the GDB manual and other documentation resources online at:
>> <http://www.gnu.org/software/gdb/documentation/>.
>>
>> For help, type "help".
>> Type "apropos word" to search for commands related to "word"...
>> Reading symbols from udp6_server_vcl_threaded_udpc...(no debugging
>> symbols found)...done.
>> (gdb) r 2001:5b0:ffff:700:b883:31f:29e:9880 9978
>> Starting program: /home/super/vcl_test/udp6_server_vcl_threaded_udpc
>> 2001:5b0:ffff:700:b883:31f:29e:9880 9978
>> Missing separate debuginfos, use: yum debuginfo-install
>> glibc-2.28-72.el8_1.1.x86_64
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib64/libthread_db.so.1".
>>
>> server addr - 2001:5b0:ffff:700:b883:31f:29e:9880
>> server port 9978,
>> total port = 1
>> VCL<64516>: configured VCL debug level (2) from VCL_DEBUG!
>> VCL<64516>: allocated VCL heap = 0x7fffe6988010, size 268435456
>> (0x10000000)
>> VCL<64516>: configured rx_fifo_size 4000000 (0x3d0900)
>> VCL<64516>: configured tx_fifo_size 4000000 (0x3d0900)
>> VCL<64516>: configured app_scope_local (1)
>> VCL<64516>: configured app_scope_global (1)
>> VCL<64516>: configured api-socket-name (/tmp/vpp-api.sock)
>> VCL<64516>: completed parsing vppcom config!
>> [New Thread 0x7fffe6987700 (LWP 64520)]
>> vppcom_connect_to_vpp:502: vcl<64516:0>: app (udp6_rx) is connected to
>> VPP!
>> vppcom_app_create:1204: vcl<64516:0>: sending session enable
>> vppcom_app_create:1212: vcl<64516:0>: sending app attach
>> vppcom_app_create:1221: vcl<64516:0>: app_name 'udp6_rx', my_client_index
>> 256 (0x100)
>> [New Thread 0x7fffe6186700 (LWP 64521)]
>> [New Thread 0x7fffe5985700 (LWP 64522)]
>> vppcom_connect_to_vpp:502: vcl<64516:1>: app (udp6_rx-wrk-1) is connected
>> to VPP!
>> vl_api_app_worker_add_del_reply_t_handler:235: vcl<0:-1>: worker 1
>> vpp-worker 1 added
>> vcl_worker_register_with_vpp:262: vcl<64516:1>: added worker 1
>> vppcom_epoll_create:2574: vcl<64516:1>: Created vep_idx 0
>> vppcom_session_create:1292: vcl<64516:1>: created session 1
>> vppcom_session_bind:1439: vcl<64516:1>: session 1 handle 16777217:
>> binding to local IPv6 address 2001:5b0:ffff:700:b883:31f:29e:9880 port
>> 9978, proto UDP
>> vppcom_session_listen:1471: vcl<64516:1>: session 16777217: sending vpp
>> listen request...
>>
>> Thread 3 "udp6_server_vcl" received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 0x7fffe6186700 (LWP 64521)]
>> 0x00007ffff7bb0086 in vcl_session_bound_handler (mp=<optimized out>,
>> wrk=0x7fffe698ad00) at /opt/vpp/src/vcl/vppcom.c:604
>> 604 rx_fifo->client_session_index = sid;
>> (gdb) bt
>> #0 0x00007ffff7bb0086 in vcl_session_bound_handler (mp=<optimized out>,
>> wrk=0x7fffe698ad00) at /opt/vpp/src/vcl/vppcom.c:604
>> #1 vcl_handle_mq_event (wrk=wrk@entry=0x7fffe698ad00, e=0x214038ec8) at
>> /opt/vpp/src/vcl/vppcom.c:904
>> #2 0x00007ffff7bb0e64 in vppcom_wait_for_session_state_change
>> (session_index=<optimized out>, state=state@entry=STATE_LISTEN,
>> wait_for_time=<optimized out>) at /opt/vpp/src/vcl/vppcom.c:966
>> #3 0x00007ffff7bb12de in vppcom_session_listen
>> (listen_sh=listen_sh@entry=16777217,
>> q_len=q_len@entry=10) at /opt/vpp/src/vcl/vppcom.c:1477
>> #4 0x00007ffff7bb1671 in vppcom_session_bind (session_handle=16777217,
>> ep=0x7fffe6183b30) at /opt/vpp/src/vcl/vppcom.c:1443
>> #5 0x0000000000401423 in udpRxThread ()
>> #6 0x00007ffff798c2de in start_thread () from /lib64/libpthread.so.0
>> #7 0x00007ffff76bd133 in clone () from /lib64/libc.so.6
>> (gdb)
>>
>> thanks,
>> -Raj
>>
>> On Tue, May 19, 2020 at 12:31 AM Florin Coras <fcoras.li...@gmail.com>
>> wrote:
>>
>>> Hi Raj,
>>>
>>> By the looks of it, something’s not right because in the logs VCL still
>>> reports it’s binding using UDPC. You probably cherry-picked [1] but it
>>> needs [2] as well. More inline.
>>>
>>> [1] https://gerrit.fd.io/r/c/vpp/+/27111
>>> [2] https://gerrit.fd.io/r/c/vpp/+/27106
>>>
>>> On May 18, 2020, at 8:42 PM, Raj Kumar <raj.gauta...@gmail.com> wrote:
>>>
>>>
>>> Hi Florin,
>>> I tried the path [1] , but still VPP is crashing when application is
>>> using listen with UDPC.
>>>
>>> [1] https://gerrit.fd.io/r/c/vpp/+/27111
>>>
>>>
>>>
>>> On a different topic , I have some questions. Could you please provide
>>> your inputs -
>>>
>>> 1) I am using Mellanox NIC. Any idea how can I enable Tx checksum
>>> offload ( for udp). Also, how to change the Tx burst mode and Rx burst
>>> mode to the Vector .
>>>
>>> HundredGigabitEthernet12/0/1 3 up
>>> HundredGigabitEthernet12/0/1
>>> Link speed: 100 Gbps
>>> Ethernet address b8:83:03:9e:98:81
>>> * Mellanox ConnectX-4 Family*
>>> carrier up full duplex mtu 9206
>>> flags: admin-up pmd maybe-multiseg rx-ip4-cksum
>>> rx: queues 4 (max 1024), desc 1024 (min 0 max 65535 align 1)
>>> tx: queues 5 (max 1024), desc 1024 (min 0 max 65535 align 1)
>>> pci: device 15b3:1013 subsystem 1590:00c8 address 0000:12:00.01 numa
>>> 0
>>> switch info: name 0000:12:00.1 domain id 1 port id 65535
>>> max rx packet len: 65536
>>> promiscuous: unicast off all-multicast on
>>> vlan offload: strip off filter off qinq off
>>> rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum
>>> vlan-filter
>>> jumbo-frame scatter timestamp keep-crc rss-hash
>>> rx offload active: ipv4-cksum udp-cksum tcp-cksum jumbo-frame scatter
>>> tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum tcp-tso
>>> outer-ipv4-cksum vxlan-tnl-tso gre-tnl-tso
>>> geneve-tnl-tso
>>> multi-segs udp-tnl-tso ip-tnl-tso
>>> * tx offload active: multi-segs*
>>> rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4
>>> ipv6-tcp-ex
>>> ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
>>> ipv6-ex ipv6 l4-dst-only l4-src-only l3-dst-only
>>> l3-src-only
>>> rss active: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4
>>> ipv6-tcp-ex
>>> ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
>>> ipv6-ex ipv6
>>>
>>> * tx burst mode: No MPW + MULTI + TSO + INLINE + METADATA rx burst
>>> mode: Scalar*
>>>
>>>
>>> FC: Not sure why (might not be supported) but the offloads are not
>>> enabled in dpdk_lib_init for VNET_DPDK_PMD_MLX* nics. You could try
>>> replicating what’s done for the Intel cards and see if that works.
>>> Alternatively, you might want to try the rdma driver, although I don’t know
>>> if it supports csum offloading (cc Ben and Damjan).
>>>
>>>
>>> 2) My application needs to send routing header (SRv6) and Destination
>>> option extension header. On RedHat 8.1 , I was using socket option to add
>>> routing and destination option extension header.
>>> With VPP , I can use SRv6 policy to let VPP add the routing header. But,
>>> I am not sure if there is any option in VPP or HostStack to add the
>>> destination option header.
>>>
>>>
>>> FC: We don’t currently support this.
>>>
>>> Regards,
>>> Florin
>>>
>>>
>>>
>>> Coming back to the original problem, here are the traces-
>>>
>>> VCL<39673>: configured VCL debug level (2) from VCL_DEBUG!
>>> VCL<39673>: using default heapsize 268435456 (0x10000000)
>>> VCL<39673>: allocated VCL heap = 0x7f6b40221010, size 268435456
>>> (0x10000000)
>>> VCL<39673>: using default configuration.
>>> vppcom_connect_to_vpp:487: vcl<39673:0>: app (udp6_rx) connecting to VPP
>>> api (/vpe-api)...
>>> vppcom_connect_to_vpp:502: vcl<39673:0>: app (udp6_rx) is connected to
>>> VPP!
>>> vppcom_app_create:1200: vcl<39673:0>: sending session enable
>>> vppcom_app_create:1208: vcl<39673:0>: sending app attach
>>> vppcom_app_create:1217: vcl<39673:0>: app_name 'udp6_rx',
>>> my_client_index 0 (0x0)
>>> vppcom_connect_to_vpp:487: vcl<39673:1>: app (udp6_rx-wrk-1) connecting
>>> to VPP api (/vpe-api)...
>>> vppcom_connect_to_vpp:502: vcl<39673:1>: app (udp6_rx-wrk-1) is
>>> connected to VPP!
>>> vcl_worker_register_with_vpp:262: vcl<39673:1>: added worker 1
>>> vl_api_app_worker_add_del_reply_t_handler:235: vcl<94:-1>: worker 1
>>> vpp-worker 1 added
>>> vppcom_epoll_create:2558: vcl<39673:1>: Created vep_idx 0
>>> vppcom_session_create:1279: vcl<39673:1>: created session 1
>>> vppcom_session_bind:1426: vcl<39673:1>: session 1 handle 16777217:
>>> binding to local IPv6 address 2001:5b0:ffff:700:b883:31f:29e:9880 port
>>> 6677, proto UDPC
>>> vppcom_session_listen:1458: vcl<39673:1>: session 16777217: sending vpp
>>> listen request...
>>>
>>> #1 0x00007ffff7761259 in session_listen (ls=<optimized out>,
>>> sep=sep@entry=0x7fffb575ad50)
>>> at
>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vnet/session/session_types.h:247
>>> #2 0x00007ffff7788b5f in app_listener_alloc_and_init
>>> (app=app@entry=0x7fffb7273038,
>>> sep=sep@entry=0x7fffb575ad50,
>>> listener=listener@entry=0x7fffb575ad28) at
>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vnet/session/application.c:196
>>> #3 0x00007ffff7788ef8 in vnet_listen (a=a@entry=0x7fffb575ad50)
>>> at
>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vnet/session/application.c:1005
>>> #4 0x00007ffff7779e20 in session_mq_listen_handler (data=0x13007ec01)
>>> at
>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vnet/session/session_node.c:64
>>> #5 session_mq_listen_handler (data=data@entry=0x13007ec01)
>>> at
>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vnet/session/session_node.c:36
>>> #6 0x00007ffff7bbcdd9 in vl_api_rpc_call_t_handler (mp=0x13007ebe8)
>>> at
>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlibmemory/vlib_api.c:520
>>> #7 0x00007ffff7bc5ecd in vl_msg_api_handler_with_vm_node
>>> (am=am@entry=0x7ffff7dd2ea0
>>> <api_global_main>, vlib_rp=<optimized out>,
>>> the_msg=0x13007ebe8, vm=vm@entry=0x7ffff6d7c200 <vlib_global_main>,
>>> node=node@entry=0x7fffb571a000, is_private=is_private@entry=0 '\000')
>>> at
>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlibapi/api_shared.c:609
>>> #8 0x00007ffff7baff06 in vl_mem_api_handle_rpc (vm=vm@entry=0x7ffff6d7c200
>>> <vlib_global_main>, node=node@entry=0x7fffb571a000)
>>> at
>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlibmemory/memory_api.c:748
>>> #9 0x00007ffff7bbd5d3 in vl_api_clnt_process (vm=<optimized out>,
>>> node=0x7fffb571a000, f=<optimized out>)
>>> at
>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlibmemory/vlib_api.c:326
>>> #10 0x00007ffff6b1b136 in vlib_process_bootstrap (_a=<optimized out>)
>>> at
>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlib/main.c:1502
>>> #11 0x00007ffff602fc0c in clib_calljmp () from
>>> /lib64/libvppinfra.so.20.05
>>> #12 0x00007fffb5e34dd0 in ?? ()
>>> #13 0x00007ffff6b1e771 in vlib_process_startup (f=0x0, p=0x7fffb571a000,
>>> vm=0x7ffff6d7c200 <vlib_global_main>)
>>> at
>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vppinfra/types.h:133
>>> #14 dispatch_process (vm=0x7ffff6d7c200 <vlib_global_main>,
>>> p=0x7fffb571a000, last_time_stamp=12611933408198086, f=0x0)
>>> at
>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlib/main.c:1569
>>>
>>> thanks,
>>> -Raj
>>>
>>>
>>>
>>>
>>> On Sat, May 16, 2020 at 8:18 PM Florin Coras <fcoras.li...@gmail.com>
>>> wrote:
>>>
>>>> Hi Raj,
>>>>
>>>> Inline.
>>>>
>>>> On May 16, 2020, at 2:30 PM, Raj Kumar <raj.gauta...@gmail.com> wrote:
>>>>
>>>> Hi Florin,
>>>>
>>>> I am using VPP 20.05 rc0 . Should I upgrade it ?
>>>>
>>>>
>>>> FC: Not necessarily, as long as it’s relatively recent, i.e., it
>>>> includes all of the recent udp updates.
>>>>
>>>>
>>>> Thanks! for providing the patch, i will try it on Monday. Actually, I
>>>> am testing in a controlled environment where I can not change the VPP
>>>> libraries. I will try it on my server.
>>>>
>>>>
>>>> FC: Sounds good. Let me know how it goes!
>>>>
>>>>
>>>> On the UDP connection; yes, the error EINPROGRESS was there because I
>>>> am using non-blocking connection. Now, I am ignoring this error.
>>>> Sometime , VPP crashes when I kills my application ( not gracefully)
>>>> even when there is a single connection .
>>>>
>>>>
>>>> FC: That might have to do with the app dying such that 1) it does not
>>>> detach from vpp (e.g., sigkill and atexit function is not executed) 2) it
>>>> dies with the message queue mutex held and 3) vpp tries to enqueue more
>>>> events before detecting that it crashed (~30s).
>>>>
>>>>
>>>> The good part is that now I am able to move connections on different
>>>> cores by connecting it on receiving the first packet and then re-binding
>>>> the socket to listen.
>>>> Basically, this approach works but I have not tested it thoroughly.
>>>> However , I am still in favor of using the UDPC connection.
>>>>
>>>>
>>>> FC: If you have enough logic in your app to emulate a handshake, i.e.,
>>>> always have the client send a few bytes and wait for a reply from the
>>>> server before opening a new connection, then this approach is probably more
>>>> flexible from core placement perspective.
>>>>
>>>> The patch tries to emulate the old udpc with udp (udpc in vpp was
>>>> confusing for consumers). You might get away with listening from multiple
>>>> vcl workers on the same udp ip:port pair and have vpp load balance accepts
>>>> between them, but I’ve never tested that. You can do this only with udp
>>>> listeners that have been initialized as connected (so only with the patch).
>>>>
>>>>
>>>> Btw, in trace logs I see some ssvm_delete related error when re-binding
>>>> the connection.
>>>>
>>>>
>>>> FC: I think it’s fine. Going over the interactions step by step to see
>>>> if understand what you’re doing (and hopefully help you understand what vpp
>>>> does underneath).
>>>>
>>>>
>>>> vpp# sh session verbose
>>>> Connection State Rx-f
>>>> Tx-f
>>>> [0:0][U] 2001:5b0:ffff:700:b883:31f:29e:9880:6677-LISTEN 0
>>>> 0
>>>> Thread 0: active sessions 1
>>>>
>>>> Connection State Rx-f
>>>> Tx-f
>>>> [1:0][U] 2001:5b0:ffff:700:b883:31f:29e:9880:6677-OPENED 0
>>>> 0
>>>> Thread 1: active sessions 1
>>>>
>>>> Connection State Rx-f
>>>> Tx-f
>>>> [2:0][U] 2001:5b0:ffff:700:b883:31f:29e:9880:6677-OPENED 0
>>>> 0
>>>> Thread 2: active sessions 1
>>>> Thread 3: no sessions
>>>> Thread 4: no sessions
>>>>
>>>> VCL<24434>: configured VCL debug level (2) from VCL_DEBUG!
>>>> VCL<24434>: using default heapsize 268435456 (0x10000000)
>>>> VCL<24434>: allocated VCL heap = 0x7f7f18d1b010, size 268435456
>>>> (0x10000000)
>>>> VCL<24434>: using default configuration.
>>>> vppcom_connect_to_vpp:487: vcl<24434:0>: app (udp6_rx) connecting to
>>>> VPP api (/vpe-api)...
>>>> vppcom_connect_to_vpp:502: vcl<24434:0>: app (udp6_rx) is connected to
>>>> VPP!
>>>> vppcom_app_create:1200: vcl<24434:0>: sending session enable
>>>> vppcom_app_create:1208: vcl<24434:0>: sending app attach
>>>> vppcom_app_create:1217: vcl<24434:0>: app_name 'udp6_rx',
>>>> my_client_index 0 (0x0)
>>>>
>>>>
>>>> FC: Added worker 0
>>>>
>>>> vppcom_connect_to_vpp:487: vcl<24434:1>: app (udp6_rx-wrk-1) connecting
>>>> to VPP api (/vpe-api)...
>>>> vppcom_connect_to_vpp:502: vcl<24434:1>: app (udp6_rx-wrk-1) is
>>>> connected to VPP!
>>>> vcl_worker_register_with_vpp:262: vcl<24434:1>: added worker 1
>>>>
>>>> vl_api_app_worker_add_del_reply_t_handler:235: vcl<94:-1>: worker 1
>>>> vpp-worker 1 added
>>>>
>>>>
>>>> FC: Adding worker 1
>>>>
>>>> vppcom_epoll_create:2558: vcl<24434:1>: Created vep_idx 0
>>>> vppcom_session_create:1279: vcl<24434:1>: created session 1
>>>> vppcom_session_bind:1426: vcl<24434:1>: session 1 handle 16777217:
>>>> binding to local IPv6 address 2001:5b0:ffff:700:b883:31f:29e:9880 port
>>>> 6677, proto UDP
>>>> vppcom_session_listen:1458: vcl<24434:1>: session 16777217: sending vpp
>>>> listen request...
>>>> vcl_session_bound_handler:607: vcl<24434:1>: session 1 [0x0]: listen
>>>> succeeded!
>>>> vppcom_epoll_ctl:2658: vcl<24434:1>: EPOLL_CTL_ADD: vep_sh 16777216, sh
>>>> 16777217, events 0x1, data 0xffffffff!
>>>>
>>>>
>>>> FC: Listened on session 1 and added it to epoll session 0
>>>>
>>>> udpRxThread started!!! ... rx port = 6677
>>>> Waiting for a client to connect on port 6677 ...
>>>> vppcom_session_connect:1742: vcl<24434:1>: session handle 16777217
>>>> (STATE_CLOSED): connecting to peer IPv6 2001:5b0:ffff:700:b883:31f:29e:9886
>>>> port 40300 proto UDP
>>>>
>>>>
>>>> FC: I guess at this point you got data on the listener so you now try
>>>> to connect it to the peer.
>>>>
>>>> vppcom_epoll_ctl:2696: vcl<24434:1>: EPOLL_CTL_MOD: vep_sh 16777216, sh
>>>> 16777217, events 0x2011, data 0x1000001!
>>>>
>>>> vppcom_session_create:1279: vcl<24434:1>: created session 2
>>>> vppcom_session_bind:1426: vcl<24434:1>: session 2 handle 16777218:
>>>> binding to local IPv6 address 2001:5b0:ffff:700:b883:31f:29e:9880 port
>>>> 6677, proto UDP
>>>> vppcom_session_listen:1458: vcl<24434:1>: session 16777218: sending vpp
>>>> listen request...
>>>>
>>>>
>>>> FC: Request to create new listener.
>>>>
>>>> vcl_session_app_add_segment_handler:855: vcl<24434:1>: mapped new
>>>> segment '24177-2' size 134217728
>>>>
>>>>
>>>> FC: This is probably the connects segment manager segment that was just
>>>> created with the first segment in it.
>>>>
>>>> vcl_session_connected_handler:505: vcl<24434:1>: session 1
>>>> [0x100000000] connected! rx_fifo 0x224051c80, refcnt 1, tx_fifo
>>>> 0x224051b80, refcnt 1
>>>>
>>>>
>>>> FC: Connect for previous listener (session 1) succeeded.
>>>>
>>>> vcl_session_app_add_segment_handler:855: vcl<24434:1>: mapped new
>>>> segment '24177-3' size 134217728
>>>>
>>>>
>>>> FC: This is the new listener’s first segment manager segment. So
>>>> session 2 has segment 24177-3 associated to it.
>>>>
>>>> vcl_session_bound_handler:607: vcl<24434:1>: session 2 [0x0]: listen
>>>> succeeded!
>>>>
>>>> vppcom_epoll_ctl:2658: vcl<24434:1>: EPOLL_CTL_ADD: vep_sh 16777216, sh
>>>> 16777218, events 0x1, data 0xffffffff!
>>>>
>>>>
>>>> FC: Listen succeeded on session 2 and it was added to the vep group.
>>>>
>>>> vcl_session_migrated_handler:674: vcl<24434:1>: Migrated 0x100000000 to
>>>> thread 2 0x200000000
>>>>
>>>>
>>>> FC: You got new data on the connected session (session 1) and the
>>>> session was migrated to the rss selected thread in vpp.
>>>>
>>>> new connecton
>>>> vppcom_session_connect:1742: vcl<24434:1>: session handle 16777218
>>>> (STATE_CLOSED): connecting to peer IPv6 2001:5b0:ffff:700:b883:31f:29e:9886
>>>> port 60725 proto UDP
>>>>
>>>>
>>>> FC: Connecting session 2 (the latest listener)
>>>>
>>>> vppcom_epoll_ctl:2696: vcl<24434:1>: EPOLL_CTL_MOD: vep_sh 16777216, sh
>>>> 16777218, events 0x2011, data 0x1000002!
>>>>
>>>> vppcom_session_create:1279: vcl<24434:1>: created session 3
>>>> vppcom_session_bind:1426: vcl<24434:1>: session 3 handle 16777219:
>>>> binding to local IPv6 address 2001:5b0:ffff:700:b883:31f:29e:9880 port
>>>> 6677, proto UDP
>>>> vppcom_session_listen:1458: vcl<24434:1>: session 16777219: sending vpp
>>>> listen request...
>>>>
>>>>
>>>> FC: Trying to listen on a new session (session 3)
>>>>
>>>> *ssvm_delete_shm:205: unlink segment '24177-3': No such file or
>>>> directory (errno 2)*
>>>>
>>>>
>>>> FC: This is okay, I think, because vpp already deleted the shm segment
>>>> (so there’s nothing left to delete).
>>>>
>>>> You might want to consider using memfd segments (although it involves a
>>>> bit of configuration like here [1]).
>>>>
>>>> [1] https://wiki.fd.io/view/VPP/HostStack/LDP/iperf
>>>>
>>>> vcl_segment_detach:467: vcl<24434:1>: detached segment 3 handle 0
>>>> vcl_session_app_del_segment_handler:863: vcl<24434:1>: Unmapped
>>>> segment: 0
>>>>
>>>>
>>>> FC: Because session 2 stopped listening, the underlying segment manager
>>>> (all listeners have a segment manager in vpp) was removed. VPP forced vcl
>>>> to unmap the segment as well.
>>>>
>>>> vcl_session_connected_handler:505: vcl<24434:1>: session 2
>>>> [0x100000000] connected! rx_fifo 0x224051a80, refcnt 1, tx_fifo
>>>> 0x224051980, refcnt 1
>>>> vcl_session_app_add_segment_handler:855: vcl<24434:1>: mapped new
>>>> segment '24177-4' size 134217728
>>>> vcl_session_bound_handler:607: vcl<24434:1>: session 3 [0x0]: listen
>>>> succeeded!
>>>>
>>>> vppcom_epoll_ctl:2658: vcl<24434:1>: EPOLL_CTL_ADD: vep_sh 16777216, sh
>>>> 16777219, events 0x1, data 0xffffffff!
>>>>
>>>>
>>>> FC: New listener (session 3) got new segment (and segment manager) and
>>>> it was added to epoll group.
>>>>
>>>> Regards,
>>>> Florin
>>>>
>>>>
>>>> thanks,
>>>> -Raj
>>>>
>>>>
>>>> On Sat, May 16, 2020 at 2:23 PM Florin Coras <fcoras.li...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Raj,
>>>>>
>>>>> Assuming you are trying to open more than one connected udp session,
>>>>> does this [1] solve the problem (note it's untested)?
>>>>>
>>>>> To reproduce legacy behavior, this allows you to listen on
>>>>> VPPCOM_PROTO_UDPC but that is now converted by vcl into a udp listen that
>>>>> propagates with a “connected” flag to vpp. That should result in a udp
>>>>> listener that behaves like an “old” udpc listener.
>>>>>
>>>>> Regards,
>>>>> Florin
>>>>>
>>>>> [1] https://gerrit.fd.io/r/c/vpp/+/27111
>>>>>
>>>>> On May 16, 2020, at 10:56 AM, Florin Coras via lists.fd.io <
>>>>> fcoras.lists=gmail....@lists.fd.io> wrote:
>>>>>
>>>>> Hi Raj,
>>>>>
>>>>> Are you using master latest/20.05 rc1 or something older? The fact
>>>>> that you’re getting a -115 (EINPROGRESS) suggests you might’ve marked the
>>>>> connection as “non-blocking” although you created it as blocking. If
>>>>> that’s
>>>>> so, the return value is not an error.
>>>>>
>>>>> Also, how if vpp crashing? Are you by chance trying to open a lot of
>>>>> udp connections back to back?
>>>>>
>>>>> Regards,
>>>>> Florin
>>>>>
>>>>> On May 16, 2020, at 10:23 AM, Raj Kumar <raj.gauta...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi Florin,
>>>>> I tried to connect on receiving the first UDP packet . But, it did not
>>>>> work. I am getting error -115 in the application and VPP is crashing.
>>>>>
>>>>> This is something I tried in the code (udp receiver) -
>>>>> sockfd = vppcom_session_create(VPPCOM_PROTO_UDP, 0);
>>>>> rv_vpp = vppcom_session_bind (sockfd, &endpt);
>>>>> if (FD_ISSET(session_idx, &readfds))
>>>>> {
>>>>> n = vppcom_session_recvfrom(sockfd, (char *)buffer, MAXLINE, 0,
>>>>> &client);
>>>>> if(first_pkt)
>>>>> rv_vpp = vppcom_session_connect (sockfd, &client);
>>>>> //Here getting rv_vpp as -115
>>>>> }
>>>>> Please let me know if I am doing something wrong.
>>>>>
>>>>> Here are the traces -
>>>>>
>>>>> VCL<16083>: configured VCL debug level (2) from VCL_DEBUG!
>>>>> VCL<16083>: using default heapsize 268435456 (0x10000000)
>>>>> VCL<16083>: allocated VCL heap = 0x7fd255ed2010, size 268435456
>>>>> (0x10000000)
>>>>> VCL<16083>: using default configuration.
>>>>> vppcom_connect_to_vpp:487: vcl<16083:0>: app (udp6_rx) connecting to
>>>>> VPP api (/vpe-api)...
>>>>> vppcom_connect_to_vpp:502: vcl<16083:0>: app (udp6_rx) is connected to
>>>>> VPP!
>>>>> vppcom_app_create:1200: vcl<16083:0>: sending session enable
>>>>> vppcom_app_create:1208: vcl<16083:0>: sending app attach
>>>>> vppcom_app_create:1217: vcl<16083:0>: app_name 'udp6_rx',
>>>>> my_client_index 0 (0x0)
>>>>>
>>>>> vppcom_connect_to_vpp:487: vcl<16083:1>: app (udp6_rx-wrk-1)
>>>>> connecting to VPP api (/vpe-api)...
>>>>> vppcom_connect_to_vpp:502: vcl<16083:1>: app (udp6_rx-wrk-1) is
>>>>> connected to VPP!
>>>>> vl_api_app_worker_add_del_reply_t_handler:235: vcl<94:-1>: worker 1
>>>>> vpp-worker 1 added
>>>>> vcl_worker_register_with_vpp:262: vcl<16083:1>: added worker 1
>>>>> vppcom_session_create:1279: vcl<16083:1>: created session 0
>>>>> vppcom_session_bind:1426: vcl<16083:1>: session 0 handle 16777216:
>>>>> binding to local IPv6 address 2001:5b0:ffff:700:b883:31f:29e:9880 port
>>>>> 6677, proto UDP
>>>>> vppcom_session_listen:1458: vcl<16083:1>: session 16777216: sending
>>>>> vpp listen request...
>>>>> vcl_session_bound_handler:607: vcl<16083:1>: session 0 [0x0]: listen
>>>>> succeeded!
>>>>> vppcom_session_connect:1742: vcl<16083:1>: session handle 16777216
>>>>> (STATE_CLOSED): connecting to peer IPv6
>>>>> 2001:5b0:ffff:700:b883:31f:29e:9886
>>>>> port 51190 proto UDP
>>>>> udpRxThread started!!! ... rx port = 6677vppcom_session_connect()
>>>>> failed ... -115
>>>>> vcl_session_cleanup:1300: vcl<16083:1>: session 0 [0x0] closing
>>>>> vcl_worker_cleanup_cb:190: vcl<94:-1>: cleaned up worker 1
>>>>> vl_client_disconnect:309: peer unresponsive, give up
>>>>>
>>>>> thanks,
>>>>> -Raj
>>>>>
>>>>>
>>>>> On Fri, May 15, 2020 at 8:10 PM Florin Coras <fcoras.li...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Raj,
>>>>>>
>>>>>> There are no explicit vcl apis that allow a udp listener to be
>>>>>> switched to connected mode. We might decide to do this at one point
>>>>>> through
>>>>>> a new bind api (non-posix like) since we do support this for builtin
>>>>>> applications.
>>>>>>
>>>>>> However, you now have the option of connecting a bound session. That
>>>>>> is, on the first received packet on a udp listener, you can grab the
>>>>>> peer’s
>>>>>> address and connect it. Iperf3 in udp mode, which is part of our make
>>>>>> test
>>>>>> infra, does exactly that. Subsequently, it re-binds the port to accept
>>>>>> more
>>>>>> connections. Would that work for you?
>>>>>>
>>>>>> Regards,
>>>>>> Florin
>>>>>>
>>>>>> On May 15, 2020, at 4:06 PM, Raj Kumar <raj.gauta...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Thanks! Florin,
>>>>>>
>>>>>> OK, I understood that I need to change my application to use UDP
>>>>>> socket and then use vppcom_session_connect().
>>>>>> This is fine for the UDP client ( sender) .
>>>>>>
>>>>>> But ,in UDP Server ( receiver) , I am not sure how to use the
>>>>>> vppcom_session_connect(). .
>>>>>> I am using vppcom_session_listen() to listen on the connections and
>>>>>> then calling vppcom_session_accept() to accept a new connection.
>>>>>>
>>>>>> With UDP*C,* I was able to utilize the RSS ( receiver side scaling)
>>>>>> feature to move the received connections on the different cores /threads.
>>>>>>
>>>>>> Just want to confirm if I can achieve the same with UDP.
>>>>>>
>>>>>> I will change my application and will update you about the result.
>>>>>>
>>>>>> Thanks,
>>>>>> -Raj
>>>>>>
>>>>>>
>>>>>> On Fri, May 15, 2020 at 5:17 PM Florin Coras <fcoras.li...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Raj,
>>>>>>>
>>>>>>> We removed udpc transport in vpp. I’ll push a patch that removes it
>>>>>>> from vcl as well.
>>>>>>>
>>>>>>> Calling connect on a udp connection will give you connected
>>>>>>> semantics now. Let me know if that solves the issue for you.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Florin
>>>>>>>
>>>>>>>
>>>>>>> On May 15, 2020, at 12:15 PM, Raj Kumar <raj.gauta...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>> I am getting segmentation fault in VPP when using VCL
>>>>>>> VPPCOM_PROTO_UDP*C* socket. This issue is observed with both UDP
>>>>>>> sender and UDP receiver application.
>>>>>>>
>>>>>>> However, both UDP sender and receiver works fine
>>>>>>> with VPPCOM_PROTO_UDP.
>>>>>>>
>>>>>>> Here is the stack trace -
>>>>>>>
>>>>>>> (gdb) bt
>>>>>>> #0 0x0000000000000000 in ?? ()
>>>>>>> #1 0x00007ffff775da59 in session_open_vc (app_wrk_index=1,
>>>>>>> rmt=0x7fffb5e34cc0, opaque=0)
>>>>>>> at
>>>>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vnet/session/session.c:1217
>>>>>>> #2 0x00007ffff7779257 in session_mq_connect_handler
>>>>>>> (data=0x7fffb676e7a8)
>>>>>>> at
>>>>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vnet/session/session_node.c:138
>>>>>>> #3 0x00007ffff7780f48 in session_event_dispatch_ctrl
>>>>>>> (elt=0x7fffb643f51c, wrk=0x7fffb650a640)
>>>>>>> at
>>>>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vnet/session/session.h:262
>>>>>>> #4 session_queue_node_fn (vm=<optimized out>, node=<optimized out>,
>>>>>>> frame=<optimized out>)
>>>>>>> at
>>>>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vnet/session/session_node.c:1409
>>>>>>> #5 0x00007ffff6b214c1 in dispatch_node (last_time_stamp=<optimized
>>>>>>> out>, frame=0x0, dispatch_state=VLIB_NODE_STATE_POLLING,
>>>>>>> type=VLIB_NODE_TYPE_INPUT, node=0x7fffb5a9a980,
>>>>>>> vm=0x7ffff6d7c200 <vlib_global_main>)
>>>>>>> at
>>>>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlib/main.c:1235
>>>>>>> #6 vlib_main_or_worker_loop (is_main=1, vm=0x7ffff6d7c200
>>>>>>> <vlib_global_main>)
>>>>>>> at
>>>>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlib/main.c:1815
>>>>>>> #7 vlib_main_loop (vm=0x7ffff6d7c200 <vlib_global_main>) at
>>>>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlib/main.c:1990
>>>>>>> #8 vlib_main (vm=<optimized out>, vm@entry=0x7ffff6d7c200
>>>>>>> <vlib_global_main>, input=input@entry=0x7fffb5e34fa0)
>>>>>>> at
>>>>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlib/main.c:2236
>>>>>>> #9 0x00007ffff6b61756 in thread0 (arg=140737334723072) at
>>>>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlib/unix/main.c:658
>>>>>>> #10 0x00007ffff602fc0c in clib_calljmp () from
>>>>>>> /lib64/libvppinfra.so.20.05
>>>>>>> #11 0x00007fffffffd1e0 in ?? ()
>>>>>>> #12 0x00007ffff6b627ed in vlib_unix_main (argc=<optimized out>,
>>>>>>> argv=<optimized out>)
>>>>>>> at
>>>>>>> /usr/src/debug/vpp-20.05-rc0~748_g83d129837.x86_64/src/vlib/unix/main.c:730
>>>>>>>
>>>>>>> Earlier , I tested this functionality with VPP 20.01 release with
>>>>>>> the following patches and it worked perfectly.
>>>>>>> https://gerrit.fd.io/r/c/vpp/+/24332
>>>>>>> https://gerrit.fd.io/r/c/vpp/+/24334
>>>>>>> https://gerrit.fd.io/r/c/vpp/+/24462
>>>>>>>
>>>>>>> Thanks,
>>>>>>> -Raj
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>> <vppcom.c>
>>
>>
>>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16589): https://lists.fd.io/g/vpp-dev/message/16589
Mute This Topic: https://lists.fd.io/mt/74234856/21656
Mute #vpp-hoststack: https://lists.fd.io/mk?hashtag=vpp-hoststack&subid=1480452
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
