Hi Ben, Chris, thanks for helping debug strace, i was also suspecting the socket return 0 as fd that will cause issue, but root cause for this seemed to be in my application that had logging functionality - tracing the logs from different threads to different file streams. when my application starts, the trace gets initialized first and then application tries to process/apply vpp config. as a test, i just reversed the order of initialization of that and let the other threads (that use VCL Tx/Rx) initialize first. (vpp interface config is before VCL threads). That seemed to solve the problem.
My application has 1 Rx thread for receiving pkts from source, process them, and send out using 'n' TX threads that can run in parallel... the destination ip address of the packet can vary, so each of the Tx threads, have to maintain a 'vppcom_session' for that ip address (connected udp socket). i can try to use vppcom_session_sendto --> for multiple destination ip addresses, and see how it performs/or work fine with N Tx threads. another alternative is to use memif for each Tx threads. -Aniket On Wed, Nov 18, 2020 at 3:38 AM Benoit Ganne (bganne) <bga...@cisco.com> wrote: > Indeed, stdin seems to be closed and vppctl is not written with that in > mind, just check > https://git.fd.io/vpp/tree/src/vpp/app/vppctl.c#n223 or > https://git.fd.io/vpp/tree/src/vpp/app/vppctl.c#n267 for examples... If > you have s->fd == STDIN_FILENO (as it seems to be the case here) things are > going to break. > > If I understand correctly, your issue with API is that you have several > threads which must use the API in parallel? I do not know if/how it > supposed to work. Maybe someone more versed in APIs can jump in? > > Best > ben > > > -----Original Message----- > > From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Chris Luke > > via lists.fd.io > > Sent: mercredi 18 novembre 2020 00:09 > > To: Aniket Pugaonkar <aniket.pugaon...@gmail.com>; Benoit Ganne (bganne) > > <bga...@cisco.com> > > Cc: vpp-dev@lists.fd.io > > Subject: Re: [EXTERNAL] Re: [vpp-dev] vppctl fails from within the > > application (system cmd returns 256) > > > > On this line in the trace: > > > > > > > > [pid 2536742] socket(AF_UNIX, SOCK_STREAM, 0) = 0 > > > > > > > > It says it’s returning file descriptor 0 for the socket. Similarly other > > file opens and subsequent closes are using descriptor 0. 0 is normally > > stdin. Are stdin & out closed? That could cause all kinds of strange > > issues, especially when it tries to print the results to a closed stdout > > descriptor. > > > > > > > > Chris. > > > > > > > > From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Aniket > > Pugaonkar > > Sent: Tuesday, November 17, 2020 17:04 > > To: Benoit Ganne (bganne) <bga...@cisco.com> > > Cc: vpp-dev@lists.fd.io > > Subject: [EXTERNAL] Re: [vpp-dev] vppctl fails from within the > application > > (system cmd returns 256) > > > > > > > > sorry, forgot to reply-all > > > > > > > > Hi Ben, > > > > > > > > When i use the same command in sample test application (just int > main()..) > > they work fine. but when I add them in my multithreaded application, it > > does not work. Please see below Strace output of the application. even > > fully specifying path to vppctl and cli.sock is not working > > > > > > > > since it is multithreaded, the main process creates the app > > (vppcom_app_create). > > > > another thread configures the interfaces using the CLI (vppctl) - which > is > > failing these commands > > > > one thread is using VCL to receive, another thread is using VCL to > > transmit. > > > > before VCL Rx and Tx threads are started, configuration thread tries to > > configure the VPP interface using the CLI. > > > > > > > > We ran into issues while making this application (that uses VCL) work > > with another application that used VAPI to configure VPP, and had to > > revert back the code to vpp cli for other application. > > > > > > > > commands that fail from my multithreaded application: > > > > /usr/bin/vppctl -s /run/vpp/cli.sock create sub-interfaces > > HundredGigabitEthernet12/0/0 701 > > > > /usr/bin/vppctl -s /run/vpp/cli.sock set interface state > > HundredGigabitEthernet12/0/0 up > > > > /usr/bin/vppctl -s /run/vpp/cli.sock set interface state > > HundredGigabitEthernet12/0/0.701up > > > > /usr/bin/vppctl -s /run/vpp/cli.sock set interface ip address > > HundredGigabitEthernet12/0/0.701 fd0d:1:2:cc:a5a5:301f:fe3a:a5a5/64 > > > > > > > > > > > > > > > > [pid 2536742] rt_sigaction(SIGCHLD, {sa_handler=SIG_DFL, sa_mask=[], > > sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7fbced84f790}, > > {sa_handler=0x55ee9d32d180, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, > > sa_restorer=0x7fbced84f790}, 8) = 0 > > [pid 2536742] execve("/usr/bin/vppctl", ["/usr/bin/vppctl", "-s", > > "/run/vpp/cli.sock", "ip", "mroute", "add", > > "ff38:23:2001:5b0:2000::3002/128", "via", "tuntap-0", "Accept"], > > 0x55ee9ead72c0 /* 27 vars */) = 0 > > [pid 2536742] brk(NULL) = 0x55fce62d4000 > > [pid 2536742] arch_prctl(0x3001 /* ARCH_??? */, 0x7ffc94d85710) = -1 > > EINVAL (Invalid argument) > > [pid 2536742] access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such > file > > or directory) > > [pid 2536742] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = > 0 > > [pid 2536742] fstat(0, {st_mode=S_IFREG|0644, st_size=62967, ...}) = 0 > > [pid 2536742] mmap(NULL, 62967, PROT_READ, MAP_PRIVATE, 0, 0) = > > 0x7ffb8ce4f000 > > [pid 2536742] close(0) = 0 > > [pid 2536742] openat(AT_FDCWD, "/lib64/libvppinfra.so.20.05.1", > > O_RDONLY|O_CLOEXEC) = 0 > > [pid 2536742] read(0, > > "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\230\0\0\0\0\0\0"..., > 832) > > = 832 > > [pid 2536742] fstat(0, {st_mode=S_IFREG|0755, st_size=509240, ...}) = 0 > > [pid 2536742] mmap(NULL, 8192, PROT_READ|PROT_WRITE, > > MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ffb8ce4d000 > > [pid 2536742] mmap(NULL, 2603336, PROT_READ|PROT_EXEC, > > MAP_PRIVATE|MAP_DENYWRITE, 0, 0) = 0x7ffb8c9bb000 > > [pid 2536742] mprotect(0x7ffb8ca35000, 2093056, PROT_NONE) = 0 > > [pid 2536742] mmap(0x7ffb8cc34000, 8192, PROT_READ|PROT_WRITE, > > MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 0, 0x79000) = 0x7ffb8cc34000 > > [pid 2536742] mmap(0x7ffb8cc36000, 2376, PROT_READ|PROT_WRITE, > > MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ffb8cc36000 > > [pid 2536742] close(0) = 0 > > [pid 2536742] openat(AT_FDCWD, "/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = > 0 > > [pid 2536742] read(0, > > "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\305\0\0\0\0\0\0"..., > > 832) = 832 > > [pid 2536742] fstat(0, {st_mode=S_IFREG|0755, st_size=2714928, ...}) = 0 > > [pid 2536742] mmap(NULL, 3674432, PROT_READ|PROT_EXEC, > > MAP_PRIVATE|MAP_DENYWRITE, 0, 0) = 0x7ffb8c639000 > > [pid 2536742] mprotect(0x7ffb8c7ba000, 2093056, PROT_NONE) = 0 > > [pid 2536742] mmap(0x7ffb8c9b9000, 8192, PROT_READ|PROT_WRITE, > > MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 0, 0x180000) = 0x7ffb8c9b9000 > > [pid 2536742] close(0) = 0 > > [pid 2536742] openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = > 0 > > [pid 2536742] read(0, > > "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\2607\2\0\0\0\0\0"..., > 832) > > = 832 > > [pid 2536742] fstat(0, {st_mode=S_IFREG|0755, st_size=4176104, ...}) = 0 > > [pid 2536742] mmap(NULL, 3938144, PROT_READ|PROT_EXEC, > > MAP_PRIVATE|MAP_DENYWRITE, 0, 0) = 0x7ffb8c277000 > > [pid 2536742] mprotect(0x7ffb8c430000, 2093056, PROT_NONE) = 0 > > [pid 2536742] mmap(0x7ffb8c62f000, 24576, PROT_READ|PROT_WRITE, > > MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 0, 0x1b8000) = 0x7ffb8c62f000 > > [pid 2536742] mmap(0x7ffb8c635000, 14176, PROT_READ|PROT_WRITE, > > MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ffb8c635000 > > [pid 2536742] close(0) = 0 > > [pid 2536742] mmap(NULL, 12288, PROT_READ|PROT_WRITE, > > MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ffb8ce4a000 > > [pid 2536742] arch_prctl(ARCH_SET_FS, 0x7ffb8ce4a740) = 0 > > [pid 2536742] mprotect(0x7ffb8c62f000, 16384, PROT_READ) = 0 > > [pid 2536742] mprotect(0x7ffb8c9b9000, 4096, PROT_READ) = 0 > > [pid 2536742] mprotect(0x7ffb8cc34000, 4096, PROT_READ) = 0 > > [pid 2536742] mprotect(0x55fce5ec7000, 4096, PROT_READ) = 0 > > [pid 2536742] mprotect(0x7ffb8ce5f000, 4096, PROT_READ) = 0 > > [pid 2536742] munmap(0x7ffb8ce4f000, 62967) = 0 > > [pid 2536742] mmap(NULL, 131072, PROT_READ|PROT_WRITE, > > MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ffb8ce2a000 > > [pid 2536742] socket(AF_UNIX, SOCK_STREAM, 0) = 0 > > [pid 2536742] connect(0, {sa_family=AF_UNIX, > > sun_path="/run/vpp/cli.sock"}, 110) = 0 > > [pid 2536742] ioctl(0, TCGETS, 0x7ffc94d85410) = -1 ENOTTY (Inappropriate > > ioctl for device) > > [pid 2536742] epoll_create1(0) = 9 > > [pid 2536742] epoll_ctl(9, EPOLL_CTL_ADD, 0, {EPOLLIN|EPOLLPRI|EPOLLERR, > > {u32=0, u64=140720308486144}}) = 0 > > [pid 2536742] epoll_ctl(9, EPOLL_CTL_ADD, 0, {EPOLLIN|EPOLLPRI|EPOLLERR, > > {u32=0, u64=140720308486144}}) = -1 EEXIST (File exists) > > [pid 2536742] close(9) = 0 > > [pid 2536742] writev(2, [{iov_base="main: epoll_ctl[0]: File exists\n", > > iov_len=32}], 1main: epoll_ctl[0]: File exists > > ) = 32 > > [pid 2536742] exit_group(1) = ? > > [pid 2536701] <... read resumed>"", 4096) = 0 > > [pid 2536742] +++ exited with 1 +++ > > [pid 2536701] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, > > si_pid=2536742, si_uid=0, si_status=1, si_utime=0, si_stime=0} --- > > [pid 2536701] futex(0x2ab46dc, FUTEX_WAKE_PRIVATE, 1) = 1 > > [pid 2536697] <... futex resumed>) = 0 > > [pid 2536701] close(22 <unfinished ...> > > [pid 2536697] futex(0x2ab4678, FUTEX_WAKE_PRIVATE, 1 <unfinished ...> > > > > > > > > On Tue, Nov 17, 2020 at 11:07 AM Benoit Ganne (bganne) <bga...@cisco.com > > <mailto:bga...@cisco.com> > wrote: > > > > > Could anyone suggest anything else to check > > > > You can try to trace you process to see what is going on, eg. > > ~# strace -f ./a.out > > > > > we had few issues with vpp > > > api/vcl libraries and have not decided to pursue vpp - api for > > now, so > > > vppctl is the only option. > > > > Can you share what prevents you from using the API? This is what is > > supposed to be used by an external controller. I do not think it is a > good > > idea to use vppctl in the long run (nonetheless because errors are hard > to > > diagnose, as you can see tight now). > > > > Best > > ben > > > > > > > > > > > > > > -- > > > > > > Thanks and regards, > > Aniket Pugaonkar > > -- Thanks and regards, Aniket Pugaonkar
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18088): https://lists.fd.io/g/vpp-dev/message/18088 Mute This Topic: https://lists.fd.io/mt/78328795/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
