[vpp-dev] VPP crash observed in hash_memory64() while creating loopback interface #vpp #vpp-dev
Hi,
I am using FDIO 20.05 version.
Here we are trying to configure a loopback interface via VAPI.
But in our testing we see that VPP is crashing, the crash is very hard to
reproduce and seen only 2-3 times till now.
Below is the bt
#0 0x7f09134041a2 in hash_memory64 (state=,
n_bytes=, p=)
at /usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vppinfra/hash.c:276
#1 hash_memory (p=0x7f08e8c26ef0, n_bytes=8, state=0) at
/usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vppinfra/hash.c:280
#2 0x7f09134051e5 in key_sum (key=139676241522416, h=0x7f08e93921f0) at
/usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vppinfra/hash.c:341
#3 lookup (v=v@entry=0x7f08e9392330, key=key@entry=139676241522416,
op=op@entry=SET, new_value=0x7f08d372e618, old_value=0x0)
at /usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vppinfra/hash.c:556
#4 0x7f091340599e in _hash_set3 (v=0x7f08e9392330, key=139676241522416,
value=, old_value=old_value@entry=0x0)
at /usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vppinfra/hash.c:848
#5 0x7f0913405b8c in hash_resize_internal (old=old@entry=0x7f08d372e610,
new_size=, free_old=free_old@entry=1)
at /usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vppinfra/hash.c:816
#6 0x7f0913405c3a in hash_resize (old=old@entry=0x7f08d372e610,
new_size=)
at /usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vppinfra/hash.c:830
#7 0x7f09134059c8 in _hash_set3 (v=0x7f08d372e610, key=139675881216528,
value=value@entry=0x7f08d36ddb60, old_value=old_value@entry=0x0)
at /usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vppinfra/hash.c:853
#8 0x7f0913d22baa in register_node () at
/usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vlib/node.c:382
#9 0x7f0913d248b9 in vlib_register_node (vm=vm@entry=0x7f0913f79300
, r=r@entry=0x7f08d36ddc60)
at /usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vlib/node.c:530
#10 0x7f0914ae6107 in vnet_register_interface (vnm=vnm@entry=0x7f09153ad180
, dev_class_index=30, dev_instance=dev_instance@entry=6,
hw_class_index=, hw_instance=9) at
/usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vnet/interface.c:911
#11 0x7f0914b1580b in ethernet_register_interface
(vnm=vnm@entry=0x7f09153ad180 , dev_class_index=,
dev_instance=dev_instance@entry=6,
address=address@entry=0x7f08d36dddea "ޭ",
hw_if_index_return=hw_if_index_return@entry=0x7f08d36ddde4,
flag_change=flag_change@entry=0x0)
at
/usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vnet/ethernet/interface.c:347
#12 0x7f0914b1690b in vnet_create_loopback_interface
(sw_if_indexp=sw_if_indexp@entry=0x7f08d36dde3c,
mac_address=mac_address@entry=0x7f08d36dde42 "",
is_specified=is_specified@entry=0 '\000', user_instance=user_instance@entry=0)
at
/usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vnet/ethernet/interface.c:859
#13 0x7f0914aedaef in vl_api_create_loopback_t_handler (mp=0x7f08e9795bd0)
at
/usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vnet/interface_api.c:1365
#14 0x7f09153f6538 in msg_handler_internal (free_it=0, do_it=1,
trace_it=, the_msg=0x7f08e9795bd0, am=0x7f0915602ea0
)
at
/usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vlibapi/api_shared.c:488
#15 vl_msg_api_handler_no_free (the_msg=0x7f08e9795bd0) at
/usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vlibapi/api_shared.c:675
#16 0x7f09153e8755 in vl_socket_process_api_msg (rp=,
input_v=)
at
/usr/src/debug/vpp-20.05.1-2~gcb5420544_dirty.x86_64/src/vlibmemory/socket_api.c:199
In hash_memory64() function it is crashing in below highlighted line
112 static inline u64
113 hash_memory64 (void *p, word n_bytes, u64 state)
114 {
.
.
133 a = b = 0x9e3779b97f4a7c13LL;
134 c = state;
135 n = n_bytes;
136
137 while (n >= 3 * sizeof (u64))
138 {
139 a += clib_mem_unaligned (q + 0, u64);
140 b += clib_mem_unaligned (q + 1, u64);
*141 c += clib_mem_unaligned (q + 2, u64); < crash is seen in this
line*
142 hash_mix64 (a, b, c);
143 n -= 3 * sizeof (u64);
144 q += 3;
145 }
This happens during initialization when the VPP is coming up freshly, currently
apart from the bt we don't have any information.
Can someone please point out what could go wrong by looking at the bt ? Any
clue will help us to move to correct direction to debug this crash.
Regards,
Sontu
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18889): https://lists.fd.io/g/vpp-dev/message/18889
Mute This Topic: https://lists.fd.io/mt/81198486/21656
Mute #vpp:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp
Mute #vpp-dev:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-dev
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] Performance problem in multithreaded vcl application #vpp
Hello!
I am learning VPP and I need to transfer more than 10G using UDP protocol in
both directions (tx and rx).
I tried to write an UDP server. There are three threads in the server
application. The first thread prints the transmission and reception statistics.
The second thread receives packets in a loop. The third thread activates after
the first received packet and starts transmitting UDP datagrams back in a loop.
If I am using vpp in single threaded mode, I can only transfer 2Gbps per second
and recieve about 8Gbps since the performance of one CPU core is not enough. If
I set the number of workers in the /etc/vpp/startup.conf configuration file in
the cpu section, I get a speed of about 10G for reception and about 100
kilobits per second for transmission (only 55 datagram per second).
Please see my sources and vpp configuration. What should I change to improve
the performance of transmission?
Also, please see the output of my application:
[root@5G-Infra-DU1 vpp_udp_ser_async_example2]# VCL_CONFIG=/etc/vpp/vcl1.conf
./vcl_udp_server
Packets sent PPS 0 (0 bits per second)
Packets recv PPS 0 (0 bits per second)
Packets sent PPS 13344 (27328512 bits per second)
Packets recv PPS 4668681 (9412060896 bits per second)
Packets sent PPS 55 (112640 bits per second)
Packets recv PPS 4916520 (9911704320 bits per second)
Packets sent PPS 55 (112640 bits per second)
Packets recv PPS 5046687 (10174123008 bits per second)
Packets sent PPS 55 (112640 bits per second)
Packets recv PPS 5040438 (10161520992 bits per second)
Thank, you
Makefile
Description: Binary data
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
uint8_t read_flag=0;
uint8_t exit_flag=0;
uint8_t print_flag=0;
std::atomic_uint_fast64_t SentPPS(0);
std::atomic_uint_fast64_t SentBPS(0);
std::atomic_uint_fast64_t RecvPPS(0);
std::atomic_uint_fast64_t RecvBPS(0);
vppcom_endpt_t saved_from;
//int vcl_session_read_ready (vcl_session_t * session);
unsigned char OutFrame[]={
0x31, 0x20, 0x43, 0x44, 0x45, 0x46,
0x47, 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56,
0x57, 0x58, 0x59, 0x5a, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0x4a, 0x4b, 0x4c,
0x4d, 0x4e, 0x4f, 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0x41, 0x42,
0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, 0x50, 0x51, 0x52,
0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48,
0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58,
0x59, 0x5a, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e,
0x4f, 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0x41, 0x42, 0x43, 0x44,
0x45, 0x46, 0x47, 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, 0x50, 0x51, 0x52, 0x53, 0x54,
0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0x4a,
0x4b, 0x4c, 0x4d, 0x4e, 0x4f, 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59, 0x5a,
0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, 0x50,
0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, 0x58, 0x59, 0x5a, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46,
0x47, 0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56,
0x57, 0x58, 0x59, 0x5a, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, 0x48, 0x49, 0x4a, 0x4b, 0x4c,
0x4d, 0x4e, 0x4f, 0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x00
};
static void
signal_handler(int sig_num)
{
if (sig_num == SIGINT) {
printf("\n\nSignal %d received, preparing to exit...\n",
sig_num);
exit_flag = 1;
}
}
void *display_thread_func(void *param)
{
while(!exit_flag)
{
uint_fast64_t SentPPSLocal = SentPPS.exchange(0);
uint_fast64_t SentBPSLocal = SentBPS.exchange(0);
printf("Packets sent PPS %" PRIu64 " (%" PRIu64 " bits per second)\n", SentPPSLocal, SentBPSLocal * 8);
uint_fast64_t RecvPPSLocal = RecvPPS.exchange(0);
uint_fast64_t RecvBPSLocal = RecvBPS.exchange(0);
printf("Packets recv PPS %" PRIu64 " (%" PRIu64 " bits per second)\n", RecvPPSLocal, RecvBPSLocal * 8);
sleep(1);
}
pthread_exit(0);
return NULL;
}
void *recv_thread_func(void *param)
{
int rv, socketfd, num_ev;
vppcom_endpt_t s_addr;
struct sockaddr_in servaddr;
memset (&servaddr, 0, sizeof (&servaddr));
memset (&s_addr, 0, sizeof( &s_addr ));
vppcom_worker_register();
socketfd = vppcom_session_create(VPPCOM_PROTO_UDP, 0);
if( socketfd < 0 )
{
printf("vppcom_session_create error:%d\n", socketfd);
return 0;
}
servaddr.sin_family = AF_INET;
servaddr.sin_addr.s_addr = htonl (INADDR_ANY);
servaddr.sin_port = htons (8080);
s_addr.is_ip4 = 1;
s_addr.ip = (uint8_t *)&s
Re: [vpp-dev] Performance problem in multithreaded vcl application #vpp
I am not a VPP expert, but you should replace any memcpy in your code with clib_memcpy_fast. Hemant From: vpp-dev@lists.fd.io On Behalf Of sokolov.3v...@gmail.com Sent: Tuesday, March 09, 2021 8:17 AM To: vpp-dev@lists.fd.io Subject: [vpp-dev] Performance problem in multithreaded vcl application #vpp Hello! I am learning VPP and I need to transfer more than 10G using UDP protocol in both directions (tx and rx). I tried to write an UDP server. There are three threads in the server application. The first thread prints the transmission and reception statistics. The second thread receives packets in a loop. The third thread activates after the first received packet and starts transmitting UDP datagrams back in a loop. If I am using vpp in single threaded mode, I can only transfer 2Gbps per second and recieve about 8Gbps since the performance of one CPU core is not enough. If I set the number of workers in the /etc/vpp/startup.conf configuration file in the cpu section, I get a speed of about 10G for reception and about 100 kilobits per second for transmission (only 55 datagram per second). Please see my sources and vpp configuration. What should I change to improve the performance of transmission? Also, please see the output of my application: [root@5G-Infra-DU1 vpp_udp_ser_async_example2]# VCL_CONFIG=/etc/vpp/vcl1.conf ./vcl_udp_server Packets sent PPS 0 (0 bits per second) Packets recv PPS 0 (0 bits per second) Packets sent PPS 13344 (27328512 bits per second) Packets recv PPS 4668681 (9412060896 bits per second) Packets sent PPS 55 (112640 bits per second) Packets recv PPS 4916520 (9911704320 bits per second) Packets sent PPS 55 (112640 bits per second) Packets recv PPS 5046687 (10174123008 bits per second) Packets sent PPS 55 (112640 bits per second) Packets recv PPS 5040438 (10161520992 bits per second) Thank, you smime.p7s Description: S/MIME cryptographic signature -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18891): https://lists.fd.io/g/vpp-dev/message/18891 Mute This Topic: https://lists.fd.io/mt/81200436/21656 Mute #vpp:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Performance problem in multithreaded vcl application #vpp
Hi, Is it safe to assume this is vpp master? More inline. > On Mar 9, 2021, at 5:16 AM, sokolov.3v...@gmail.com wrote: > > Hello! > > I am learning VPP and I need to transfer more than 10G using UDP protocol in > both directions (tx and rx). > I tried to write an UDP server. There are three threads in the server > application. The first thread prints the transmission and reception > statistics. The second thread receives packets in a loop. The third thread > activates after the first received packet and starts transmitting UDP > datagrams back in a loop. Took a quick look at the code. Do you expect packets from multiple clients or just one? If just one, you should consider using connected udp (see VPPCOM_ATTR_SET_CONNECTED). These sessions are pinned to a worker and don’t use locks but they can only rx and tx dgrams for a given flow (5-tuple). You’ll probably have to use only one thread for io as the 5 tuple for the 2 connections will be the same. > If I am using vpp in single threaded mode, I can only transfer 2Gbps per > second and recieve about 8Gbps since the performance of one CPU core is not > enough. These do seem a bit low, but it will depend a lot on the speed of your cpu. I’m assuming this is a release (not a debug) image. > If I set the number of workers in the /etc/vpp/startup.conf configuration > file in the cpu section, I get a speed of about 10G for reception and about > 100 kilobits per second for transmission (only 55 datagram per second). Ah, that’s because non-connected udp binds end up with listeners on main thread and by default main thread is not polled. This should somewhat improve if you add “poll-main” to the session stanza in startup.conf. Having said that, for max throughput you should probably use connected udp sessions, if possible. Otherwise, writes are only done from main thread and rx happens on all workers but with a lock. So tx is still limited as main thread is busy with non-forwarding processes and rx is not ideal. Regards, Florin > Please see my sources and vpp configuration. What should I change to improve > the performance of transmission? > Also, please see the output of my application: > > [root@5G-Infra-DU1 vpp_udp_ser_async_example2]# VCL_CONFIG=/etc/vpp/vcl1.conf > ./vcl_udp_server > Packets sent PPS 0 (0 bits per second) > Packets recv PPS 0 (0 bits per second) > Packets sent PPS 13344 (27328512 bits per second) > Packets recv PPS 4668681 (9412060896 bits per second) > Packets sent PPS 55 (112640 bits per second) > Packets recv PPS 4916520 (9911704320 bits per second) > Packets sent PPS 55 (112640 bits per second) > Packets recv PPS 5046687 (10174123008 bits per second) > Packets sent PPS 55 (112640 bits per second) > Packets recv PPS 5040438 (10161520992 bits per second) > > Thank, you > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18892): https://lists.fd.io/g/vpp-dev/message/18892 Mute This Topic: https://lists.fd.io/mt/81200436/21656 Mute #vpp:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] VLAN based routing #vpp-dev
Hi Team, Our architecture is something like this INGRESS>VPP->Plugin->MEMIF(Ethernet mode)->USER Application, In ingress we want to read the packet layer 2. EGRESS ==> User Application --> MEMIF(IP MODE) --> VPP. We want to get the packet routed through VPP. We are seeing the VPP dropping the packets if DST Subnet not matched with any of the interface configuration in VPP. If we add the default route, the VPP sending packets to default router. However we can not make one default route to send all the packets to router/GW. Our requirement is depending on different VLANs we want to forward the packets to different GWs. Can you please provide any pointer? is there any way we can achieve this VLAN based forwarding? //Ravi -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18893): https://lists.fd.io/g/vpp-dev/message/18893 Mute This Topic: https://lists.fd.io/mt/81218380/21656 Mute #vpp-dev:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-dev Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
