At 2025-12-11 16:32:15, "Jason Wang" <[email protected]> wrote:
>On Thu, Dec 11, 2025 at 10:52 AM Lange Tang <[email protected]> wrote:
>>
>> At 2025-12-10 17:04:04, "Michael S. Tsirkin" <[email protected]> wrote:
>> >On Thu, Nov 27, 2025 at 11:24:00AM +0800, Longjun Tang wrote:
>> >> From: Tang Longjun <[email protected]>
>> >>
>> >> hi,
>> >> virtnet_mon is used to monitor the data packets of the virtio_net driver
>> >> and the related parameters of virtqueue, useful for tracking its status
>> >> and troubleshooting faults.
>> >>
>> >> pls review. tks
>> >>
>> >> Best regard.
>> >
>> >what does this achieve that direct use of tracing would not?
>>
>> I apologize that my explanation of virtnet_mon was not detailed enough.
>> virtnet_mon uses kprobe and buffers to monitor virtio_net.
>> To monitor virtio_net, it is necessary to track the member parameters of the
>> virtqueue corresponding to each data packet and output them.
>> When PPS very high, other tracing techniques, such as ebpf, may not be able
>> to handle it, resulting in data loss because they do not have sufficiently
>> large buffers to batch export log data.
>
>Can you expand more about this? For example, in which kind of setup
>and what do you want to trace and why ebpf can't handle that. Note
>that the most lightweight stuff is the counter, have you tried that?
For example, when there is occasional latency in data transmission between the
virtual network frontend (virtio_net) and backend (such as vhost_net),
we may need to track the time taken for each packet received and sent in the
virtio_net driver.
Typically, we accomplish this using eBPF, such as bpftrace. The pseudocode
might include the following:
"""
kprobe:skb_recv_done {
printf("%ld skb_recv_done Cpu:%d ...\n",...);
}
kprobe:skb_xmit_done {
printf("%ld skb_xmit_done Cpu:%d ...\n",...);
}
kprobe:virtnet_poll {
printf("%ld virtnet_poll Cpu:%d budget:%d ...\n",...);
}
kprobe:start_xmit {
...
printf("%ld start_xmit Cpu:%d type:%s seq:%ld ...\n",...)
}
kprobe:gro_receive_skb {
...
printf("%ld gro_receive_skb Cpu:%d type:%s seq:%ld ...\n",...)
}
kprobe:receive_buf {
...
printf("%ld receive_buf Cpu:%d name:%s avali_idx:%d used_idx:%d ...\n",...);
}
"""
Using the above bpftrace code, we can track the timestamps of the data as it
passes through these functions,
along with skb and virtqueue information, and output logs via printf for
further diagnosis of the causes of the latency.
Interestingly, a significant amount of logs were found to be missing when
executing these bpftrace codes.
Below is the testing environment:
VM: 8G8C，virtio_net mq=4, kernel 6.18-rc7, iperf3 -s -p 1314
HOST: iperf3 -c 192.168.122.218 -t 100 -p 1314 -P 4
It was also found that when testing with mq=1, there was no log loss.
Compared to mq=1, the reason for log loss at mq=4 is suspected to be due to
data being sent or received
by different CPUs. Additionally, under the 4-thread iperf testing scenario with
PPS > 150,000,
the log data is asynchronously output from different CPUs, leading to excessive
IO pressure that causes log data loss.
The above are some of my personal thoughts, and I would love to hear your
opinion.
Best regard.
>
>>
>> As for the duplicate code, it is only to obtain the layout of the relevant
>> structure, and I have not yet thought of a way to avoid duplication. I would
>> love to hear your suggestions.
>
>Thanks
>
>>
>> >
>> >> Tang Longjun (7):
>> >> tools/virtio/virtnet_mon: create misc driver for virtnet_mon
>> >> tools/virtio/virtnet_mon: add kfifo to virtnet_mon
>> >> tools/virtio/virtnet_mon: add kprobe start_xmit
>> >> tools/virtio/virtnet_mon: add kprobe gro_receive_skb
>> >> tools/virtio/virtnet_mon: add kprobe ip_local_deliver
>> >> tools/virtio/virtnet_mon: add kprobe skb_xmit_done and skb_recv_done
>> >> tools/virtio/virtnet_mon: add README file for virtnet_moin
>> >>
>> >> tools/virtio/virtnet_mon/Makefile | 10 +
>> >> tools/virtio/virtnet_mon/README | 35 +
>> >> tools/virtio/virtnet_mon/virtnet_mon.c | 1048 ++++++++++++++++++++++++
>> >> 3 files changed, 1093 insertions(+)
>> >> create mode 100644 tools/virtio/virtnet_mon/Makefile
>> >> create mode 100644 tools/virtio/virtnet_mon/README
>> >> create mode 100644 tools/virtio/virtnet_mon/virtnet_mon.c
>> >>
>> >> --
>> >> 2.43.0
