On Tue, Dec 16, 2025 at 3:48 PM Lange Tang <[email protected]> wrote: > > At 2025-12-16 11:59:14, "Jason Wang" <[email protected]> wrote: > >On Mon, Dec 15, 2025 at 4:12 PM Lange Tang <[email protected]> wrote: > >> > >> At 2025-12-15 14:41:42, "Jason Wang" <[email protected]> wrote: > >> >On Sat, Dec 13, 2025 at 10:41 AM Lange Tang <[email protected]> wrote: > >> >> > >> >> At 2025-12-11 16:32:15, "Jason Wang" <[email protected]> wrote: > >> >> >On Thu, Dec 11, 2025 at 10:52 AM Lange Tang <[email protected]> wrote: > >> >> >> > >> >> >> At 2025-12-10 17:04:04, "Michael S. Tsirkin" <[email protected]> wrote: > >> >> >> >On Thu, Nov 27, 2025 at 11:24:00AM +0800, Longjun Tang wrote: > >> >> >> >> From: Tang Longjun <[email protected]> > >> >> >> >> > >> >> >> >> hi, > >> >> >> >> virtnet_mon is used to monitor the data packets of the virtio_net > >> >> >> >> driver > >> >> >> >> and the related parameters of virtqueue, useful for tracking its > >> >> >> >> status > >> >> >> >> and troubleshooting faults. > >> >> >> >> > >> >> >> >> pls review. tks > >> >> >> >> > >> >> >> >> Best regard. > >> >> >> > > >> >> >> >what does this achieve that direct use of tracing would not? > >> >> >> > >> >> >> I apologize that my explanation of virtnet_mon was not detailed > >> >> >> enough. > >> >> >> virtnet_mon uses kprobe and buffers to monitor virtio_net. > >> >> >> To monitor virtio_net, it is necessary to track the member > >> >> >> parameters of the virtqueue corresponding to each data packet and > >> >> >> output them. > >> >> >> When PPS very high, other tracing techniques, such as ebpf, may not > >> >> >> be able to handle it, resulting in data loss because they do not > >> >> >> have sufficiently large buffers to batch export log data. > >> >> > > >> >> >Can you expand more about this? For example, in which kind of setup > >> >> >and what do you want to trace and why ebpf can't handle that. Note > >> >> >that the most lightweight stuff is the counter, have you tried that? > >> >> > >> >> For example, when there is occasional latency in data transmission > >> >> between the > >> >> virtual network frontend (virtio_net) and backend (such as vhost_net), > >> >> we may need to track the time taken for each packet received and sent > >> >> in the virtio_net driver. > >> >> Typically, we accomplish this using eBPF, such as bpftrace. The > >> >> pseudocode might include the following: > >> >> """ > >> >> kprobe:skb_recv_done { > >> >> printf("%ld skb_recv_done Cpu:%d ...\n",...); > >> >> } > >> >> kprobe:skb_xmit_done { > >> >> printf("%ld skb_xmit_done Cpu:%d ...\n",...); > >> >> } > >> >> kprobe:virtnet_poll { > >> >> printf("%ld virtnet_poll Cpu:%d budget:%d ...\n",...); > >> >> } > >> >> kprobe:start_xmit { > >> >> ... > >> >> printf("%ld start_xmit Cpu:%d type:%s seq:%ld ...\n",...) > >> >> } > >> >> kprobe:gro_receive_skb { > >> >> ... > >> >> printf("%ld gro_receive_skb Cpu:%d type:%s seq:%ld ...\n",...) > >> >> } > >> >> kprobe:receive_buf { > >> >> ... > >> >> printf("%ld receive_buf Cpu:%d name:%s avali_idx:%d used_idx:%d > >> >> ...\n",...); > >> >> } > >> >> """ > >> >> Using the above bpftrace code, we can track the timestamps of the data > >> >> as it passes through these functions, > >> >> along with skb and virtqueue information, and output logs via printf > >> >> for further diagnosis of the causes of the latency. > >> >> Interestingly, a significant amount of logs were found to be missing > >> >> when executing these bpftrace codes. > >> >> Below is the testing environment: > >> >> VM: 8G8C，virtio_net mq=4, kernel 6.18-rc7, iperf3 -s -p 1314 > >> >> HOST: iperf3 -c 192.168.122.218 -t 100 -p 1314 -P 4 > >> >> It was also found that when testing with mq=1, there was no log loss. > >> >> > >> >> Compared to mq=1, the reason for log loss at mq=4 is suspected to be > >> >> due to data being sent or received > >> >> by different CPUs. Additionally, under the 4-thread iperf testing > >> >> scenario with PPS > 150,000, > >> >> the log data is asynchronously output from different CPUs, leading to > >> >> excessive IO pressure that causes log data loss. > >> > > >> >I think what I don't understand is how the things you introduced here > >> >may help in this case? > >> > >> The virtnet_mon module introduced here abandons eBPF and uses kprobe + > >> kfifo. > >> In the aforementioned cases, all the information that needs to be tracked > >> first enters kfifo, > >> then is formatted into logs and cached in a large buffer. > >> Finally, it is exported to user space in batches through the > >> virtnet_mon_read function, > >> thereby reducing IO pressure and preventing log loss. > > > >Well, this "problem" seems not virtio-net specific. Have you tried > >with BPF ringbuf or perfbuf? > > Concerning the ringbuf and perfbuf in bpf, I may need some time to verify > whether this can resolve this "problem". > I will get back to you with the results. > > On the other hand, I did not find any tracepoints in virtio_net that track > the virtqueue, > such as name, num_free, avail.idx, used.idx, last_used_idx, etc. > Could you consider inserting these tracepoints in some key functions? This > would facilitate direct tracking by perf. > For example: > start_xmit > receive_buf > skb_xmit_done > skb_recv_done
I think it can have. I posted a tracepoint support for vhost about 10 years ago (-ENOTIME for a new version though), so perf can track. https://lists.openwall.net/netdev/2012/05/07/59 Thanks > > > > >Thanks > > > >> > >> Thanks > >> > > >> >Thanks > >> > > >> >> > >> >> The above are some of my personal thoughts, and I would love to hear > >> >> your opinion. > >> >> Best regard. > >> >> > >> >> > > >> >> >> > >> >> >> As for the duplicate code, it is only to obtain the layout of the > >> >> >> relevant structure, and I have not yet thought of a way to avoid > >> >> >> duplication. I would love to hear your suggestions. > >> >> > > >> >> >Thanks > >> >> > > >> >> >> > >> >> >> > > >> >> >> >> Tang Longjun (7): > >> >> >> >> tools/virtio/virtnet_mon: create misc driver for virtnet_mon > >> >> >> >> tools/virtio/virtnet_mon: add kfifo to virtnet_mon > >> >> >> >> tools/virtio/virtnet_mon: add kprobe start_xmit > >> >> >> >> tools/virtio/virtnet_mon: add kprobe gro_receive_skb > >> >> >> >> tools/virtio/virtnet_mon: add kprobe ip_local_deliver > >> >> >> >> tools/virtio/virtnet_mon: add kprobe skb_xmit_done and > >> >> >> >> skb_recv_done > >> >> >> >> tools/virtio/virtnet_mon: add README file for virtnet_moin > >> >> >> >> > >> >> >> >> tools/virtio/virtnet_mon/Makefile | 10 + > >> >> >> >> tools/virtio/virtnet_mon/README | 35 + > >> >> >> >> tools/virtio/virtnet_mon/virtnet_mon.c | 1048 > >> >> >> >> ++++++++++++++++++++++++ > >> >> >> >> 3 files changed, 1093 insertions(+) > >> >> >> >> create mode 100644 tools/virtio/virtnet_mon/Makefile > >> >> >> >> create mode 100644 tools/virtio/virtnet_mon/README > >> >> >> >> create mode 100644 tools/virtio/virtnet_mon/virtnet_mon.c > >> >> >> >> > >> >> >> >> -- > >> >> >> >> 2.43.0
