Hello all,
I am facing an issue while processing packets in dpdk in which we reach a
situation where packet processing gets stopped. When I attach gdb to
process, I see the following back trace. The data processing thread is
stuck in rte_pause() call inside common_ring_mp_enqueue because of a check
on freepool state. The following is the analysis which I found while
debugging it.
ring <MP_FEDP_MBUF_POOL>@0x7f33b0f0f0c0
flags=0
size=131072
ct=3736608500
ch=3736608500
pt=3736611582
ph=3736612099
used=3082
avail=127989
watermark=0
no statistics available
gdb) bt
#0 0x00007f9fb36ff425 in *common_ring_mp_enqueue (*) from
/usr/lib64/librte_mempool.so.2.1
#1 0x000000000040ad68 in rte_mempool_ops_enqueue_bulk (n=<optimized out>,
obj_table=<optimized out>, mp=<optimized out>)
at
/home/osboxes/Downloads/dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:495
#2 __mempool_generic_put (flags=<optimized out>, cache=0x7f33c3844e40, n=1,
obj_table=0x7f9fab0a9b88, mp=<optimized out>)
at
/home/osboxes/Downloads/dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1071
#3 rte_mempool_generic_put (flags=<optimized out>, cache=0x7f33c3844e40,
n=1, obj_table=0x7f9fab0a9b88, mp=<optimized out>)
at
/home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1110
#4 rte_mempool_put_bulk (n=1, obj_table=0x7f9fab0a9b88, mp=<optimized out>)
at
/home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1173
#5 rte_mempool_put (obj=0x7f33c1634780, mp=<optimized out>)
at
/home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1225
#6 __rte_mbuf_raw_free (m=0x7f33c1634780)
at
/home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:783
#7 rte_pktmbuf_free_seg (m=<optimized out>)
at
/home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:1224
#8 rte_pktmbuf_free (m=0x7f33c1634780)
at
/home/osboxes/Downloads//dpdk/dpdk-latest/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:1245
#9 lcore_main (args=args@entry=0x0) at dp_app.c:565
When we try to use debug APIs given by Dpdk I see that MEMPOOL is giving
error of invalid cache len as shown below.
Auditing mbuf_pool using 'rte_mempool_audit'
MEMPOOL: badness on cache[27]
PANIC in mempool_audit_cache():
*MEMPOOL: invalid cache len-*
When I further dump rte_mempool cache , I saw the following
stqh_last = 0x7f33b0f0f040}}
(gdb) p ((struct rte_mempool *)0x7f33c380e900)->local_cache
$22 = (struct rte_mempool_cache *) 0x7f33c380e9c0
(gdb) p *(((struct rte_mempool *)0x7f33c380e900)->local_cache+26)
$49 = {size = 250, flushthresh = 375, len = 0, objs = {0x0 <repeats 1536
times>}}
*(gdb) p *(((struct rte_mempool *)0x7f33c380e900)->local_cache+27)*
$50 = {size = 250, flushthresh = 375, len = 377, objs = {0x7f33b7f50780,
0x7f33b7f510c0, 0x7f33b7f51a00, 0x7f33b7f52340, 0x7f33b7f52c80,
0x7f33b7f535c0, 0x7f33b7f53f00,
I am really looking forward for help as under what circumstances we can
reach to such condition. Thank you in advance for your help.
-
Thanks
Manish Kumar
