On Mon, Dec 15, 2025 at 4:12 PM Lange Tang <[email protected]> wrote: > > At 2025-12-15 14:41:42, "Jason Wang" <[email protected]> wrote: > >On Sat, Dec 13, 2025 at 10:41 AM Lange Tang <[email protected]> wrote: > >> > >> At 2025-12-11 16:32:15, "Jason Wang" <[email protected]> wrote: > >> >On Thu, Dec 11, 2025 at 10:52 AM Lange Tang <[email protected]> wrote: > >> >> > >> >> At 2025-12-10 17:04:04, "Michael S. Tsirkin" <[email protected]> wrote: > >> >> >On Thu, Nov 27, 2025 at 11:24:00AM +0800, Longjun Tang wrote: > >> >> >> From: Tang Longjun <[email protected]> > >> >> >> > >> >> >> hi, > >> >> >> virtnet_mon is used to monitor the data packets of the virtio_net > >> >> >> driver > >> >> >> and the related parameters of virtqueue, useful for tracking its > >> >> >> status > >> >> >> and troubleshooting faults. > >> >> >> > >> >> >> pls review. tks > >> >> >> > >> >> >> Best regard. > >> >> > > >> >> >what does this achieve that direct use of tracing would not? > >> >> > >> >> I apologize that my explanation of virtnet_mon was not detailed enough. > >> >> virtnet_mon uses kprobe and buffers to monitor virtio_net. > >> >> To monitor virtio_net, it is necessary to track the member parameters > >> >> of the virtqueue corresponding to each data packet and output them. > >> >> When PPS very high, other tracing techniques, such as ebpf, may not be > >> >> able to handle it, resulting in data loss because they do not have > >> >> sufficiently large buffers to batch export log data. > >> > > >> >Can you expand more about this? For example, in which kind of setup > >> >and what do you want to trace and why ebpf can't handle that. Note > >> >that the most lightweight stuff is the counter, have you tried that? > >> > >> For example, when there is occasional latency in data transmission between > >> the > >> virtual network frontend (virtio_net) and backend (such as vhost_net), > >> we may need to track the time taken for each packet received and sent in > >> the virtio_net driver. > >> Typically, we accomplish this using eBPF, such as bpftrace. The pseudocode > >> might include the following: > >> """ > >> kprobe:skb_recv_done { > >> printf("%ld skb_recv_done Cpu:%d ...\n",...); > >> } > >> kprobe:skb_xmit_done { > >> printf("%ld skb_xmit_done Cpu:%d ...\n",...); > >> } > >> kprobe:virtnet_poll { > >> printf("%ld virtnet_poll Cpu:%d budget:%d ...\n",...); > >> } > >> kprobe:start_xmit { > >> ... > >> printf("%ld start_xmit Cpu:%d type:%s seq:%ld ...\n",...) > >> } > >> kprobe:gro_receive_skb { > >> ... > >> printf("%ld gro_receive_skb Cpu:%d type:%s seq:%ld ...\n",...) > >> } > >> kprobe:receive_buf { > >> ... > >> printf("%ld receive_buf Cpu:%d name:%s avali_idx:%d used_idx:%d > >> ...\n",...); > >> } > >> """ > >> Using the above bpftrace code, we can track the timestamps of the data as > >> it passes through these functions, > >> along with skb and virtqueue information, and output logs via printf for > >> further diagnosis of the causes of the latency. > >> Interestingly, a significant amount of logs were found to be missing when > >> executing these bpftrace codes. > >> Below is the testing environment: > >> VM: 8G8C，virtio_net mq=4, kernel 6.18-rc7, iperf3 -s -p 1314 > >> HOST: iperf3 -c 192.168.122.218 -t 100 -p 1314 -P 4 > >> It was also found that when testing with mq=1, there was no log loss. > >> > >> Compared to mq=1, the reason for log loss at mq=4 is suspected to be due > >> to data being sent or received > >> by different CPUs. Additionally, under the 4-thread iperf testing scenario > >> with PPS > 150,000, > >> the log data is asynchronously output from different CPUs, leading to > >> excessive IO pressure that causes log data loss. > > > >I think what I don't understand is how the things you introduced here > >may help in this case? > > The virtnet_mon module introduced here abandons eBPF and uses kprobe + kfifo. > In the aforementioned cases, all the information that needs to be tracked > first enters kfifo, > then is formatted into logs and cached in a large buffer. > Finally, it is exported to user space in batches through the virtnet_mon_read > function, > thereby reducing IO pressure and preventing log loss.
Well, this "problem" seems not virtio-net specific. Have you tried with BPF ringbuf or perfbuf? Thanks > > Thanks > > > >Thanks > > > >> > >> The above are some of my personal thoughts, and I would love to hear your > >> opinion. > >> Best regard. > >> > >> > > >> >> > >> >> As for the duplicate code, it is only to obtain the layout of the > >> >> relevant structure, and I have not yet thought of a way to avoid > >> >> duplication. I would love to hear your suggestions. > >> > > >> >Thanks > >> > > >> >> > >> >> > > >> >> >> Tang Longjun (7): > >> >> >> tools/virtio/virtnet_mon: create misc driver for virtnet_mon > >> >> >> tools/virtio/virtnet_mon: add kfifo to virtnet_mon > >> >> >> tools/virtio/virtnet_mon: add kprobe start_xmit > >> >> >> tools/virtio/virtnet_mon: add kprobe gro_receive_skb > >> >> >> tools/virtio/virtnet_mon: add kprobe ip_local_deliver > >> >> >> tools/virtio/virtnet_mon: add kprobe skb_xmit_done and > >> >> >> skb_recv_done > >> >> >> tools/virtio/virtnet_mon: add README file for virtnet_moin > >> >> >> > >> >> >> tools/virtio/virtnet_mon/Makefile | 10 + > >> >> >> tools/virtio/virtnet_mon/README | 35 + > >> >> >> tools/virtio/virtnet_mon/virtnet_mon.c | 1048 > >> >> >> ++++++++++++++++++++++++ > >> >> >> 3 files changed, 1093 insertions(+) > >> >> >> create mode 100644 tools/virtio/virtnet_mon/Makefile > >> >> >> create mode 100644 tools/virtio/virtnet_mon/README > >> >> >> create mode 100644 tools/virtio/virtnet_mon/virtnet_mon.c > >> >> >> > >> >> >> -- > >> >> >> 2.43.0
