On Fri, Jan 31, 2025 at 6:04 AM Sahil Siddiq <icegambi...@gmail.com> wrote: > > Hi, > > On 1/24/25 1:04 PM, Eugenio Perez Martin wrote: > > On Fri, Jan 24, 2025 at 6:47 AM Sahil Siddiq <icegambi...@gmail.com> wrote: > >> On 1/21/25 10:07 PM, Eugenio Perez Martin wrote: > >>> On Sun, Jan 19, 2025 at 7:37 AM Sahil Siddiq <icegambi...@gmail.com> > >>> wrote: > >>>> On 1/7/25 1:35 PM, Eugenio Perez Martin wrote: > >>>> [...] > >>>> Apologies for the delay in replying. It took me a while to figure > >>>> this out, but I have now understood why this doesn't work. L1 is > >>>> unable to receive messages from L0 because they get filtered out > >>>> by hw/net/virtio-net.c:receive_filter [1]. There's an issue with > >>>> the MAC addresses. > >>>> > >>>> In L0, I have: > >>>> > >>>> $ ip a show tap0 > >>>> 6: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state > >>>> UNKNOWN group default qlen 1000 > >>>> link/ether d2:6d:b9:61:e1:9a brd ff:ff:ff:ff:ff:ff > >>>> inet 111.1.1.1/24 scope global tap0 > >>>> valid_lft forever preferred_lft forever > >>>> inet6 fe80::d06d:b9ff:fe61:e19a/64 scope link proto kernel_ll > >>>> valid_lft forever preferred_lft forever > >>>> > >>>> In L1: > >>>> > >>>> # ip a show eth0 > >>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state > >>>> UP group default qlen 1000 > >>>> link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff > >>>> altname enp0s2 > >>>> inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic > >>>> noprefixroute eth0 > >>>> valid_lft 83455sec preferred_lft 83455sec > >>>> inet6 fec0::7bd2:265e:3b8e:5acc/64 scope site dynamic > >>>> noprefixroute > >>>> valid_lft 86064sec preferred_lft 14064sec > >>>> inet6 fe80::50e7:5bf6:fff8:a7b0/64 scope link noprefixroute > >>>> valid_lft forever preferred_lft forever > >>>> > >>>> I'll call this L1-eth0. > >>>> > >>>> In L2: > >>>> # ip a show eth0 > >>>> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state > >>>> UP gro0 > >>>> link/ether 52:54:00:12:34:57 brd ff:ff:ff:ff:ff:ff > >>>> altname enp0s7 > >>>> inet 111.1.1.2/24 scope global eth0 > >>>> valid_lft forever preferred_lft forever > >>>> > >>>> I'll call this L2-eth0. > >>>> > >>>> Apart from eth0, lo is the only other device in both L1 and L2. > >>>> > >>>> A frame that L1 receives from L0 has L2-eth0's MAC address (LSB = 57) > >>>> as its destination address. When booting L2 with x-svq=false, the > >>>> value of n->mac in VirtIONet is also L2-eth0. So, L1 accepts > >>>> the frames and passes them on to L2 and pinging works [2]. > >>>> > >>> > >>> So this behavior is interesting by itself. But L1's kernel net system > >>> should not receive anything. As I read it, even if it receives it, it > >>> should not forward the frame to L2 as it is in a different subnet. Are > >>> you able to read it using tcpdump on L1? > >> > >> I ran "tcpdump -i eth0" in L1. It didn't capture any of the packets > >> that were directed at L2 even though L2 was able to receive them. > >> Similarly, it didn't capture any packets that were sent from L2 to > >> L0. This is when L2 is launched with x-svq=false. > >> > > > > That's right. The virtio dataplane goes directly from L0 to L2, you > > should not be able to see any packets in the net of L1. > > I am a little confused here. Since vhost=off is set in L0's QEMU > (which is used to boot L1), I am able to inspect the packets when > tracing/debugging receive_filter in hw/net/virtio-net.c. [1] Does > this mean the dataplane from L0 to L2 passes through L0's QEMU > (so L0 QEMU is aware of what's going on), but bypasses L1 completely > so L1's kernel does not know what packets are being sent/received. >
That's right. We're saving processing power and context switches that way :). > >> With x-svq=true, forcibly setting the LSB of n->mac to 0x57 in > >> receive_filter allows L2 to receive packets from L0. I added > >> the following line just before line 1771 [1] to check this out. > >> > >> n->mac[5] = 0x57; > >> > > > > That's very interesting. Let me answer all the gdb questions below and > > we can debug it deeper :). > > > > Thank you for the primer on using gdb with QEMU. I am able to debug > QEMU now. > > >>> Maybe we can make the scenario clearer by telling which virtio-net > >>> device is which with virtio_net_pci,mac=XX:... ? > >>> > >>>> However, when booting L2 with x-svq=true, n->mac is set to L1-eth0 > >>>> (LSB = 56) in virtio_net_handle_mac() [3]. > >>> > >>> Can you tell with gdb bt if this function is called from net or the > >>> SVQ subsystem? > >> > > It looks like the function is being called from net. > > (gdb) bt > #0 virtio_net_handle_mac (n=0x15622425e, cmd=85 'U', iov=0x555558865980, > iov_cnt=1476792840) at ../hw/net/virtio-net.c:1098 > #1 0x0000555555e5920b in virtio_net_handle_ctrl_iov (vdev=0x555558fdacd0, > in_sg=0x5555580611f8, in_num=1, out_sg=0x555558061208, > out_num=1) at ../hw/net/virtio-net.c:1581 > #2 0x0000555555e593a0 in virtio_net_handle_ctrl (vdev=0x555558fdacd0, > vq=0x555558fe7730) at ../hw/net/virtio-net.c:1610 > #3 0x0000555555e9a7d8 in virtio_queue_notify_vq (vq=0x555558fe7730) at > ../hw/virtio/virtio.c:2484 > #4 0x0000555555e9dffb in virtio_queue_host_notifier_read (n=0x555558fe77a4) > at ../hw/virtio/virtio.c:3869 > #5 0x000055555620329f in aio_dispatch_handler (ctx=0x555557d9f840, > node=0x7fffdca7ba80) at ../util/aio-posix.c:373 > #6 0x000055555620346f in aio_dispatch_handlers (ctx=0x555557d9f840) at > ../util/aio-posix.c:415 > #7 0x00005555562034cb in aio_dispatch (ctx=0x555557d9f840) at > ../util/aio-posix.c:425 > #8 0x00005555562242b5 in aio_ctx_dispatch (source=0x555557d9f840, > callback=0x0, user_data=0x0) at ../util/async.c:361 > #9 0x00007ffff6d86559 in ?? () from /usr/lib/libglib-2.0.so.0 > #10 0x00007ffff6d86858 in g_main_context_dispatch () from > /usr/lib/libglib-2.0.so.0 > #11 0x0000555556225bf9 in glib_pollfds_poll () at ../util/main-loop.c:287 > #12 0x0000555556225c87 in os_host_main_loop_wait (timeout=294672) at > ../util/main-loop.c:310 > #13 0x0000555556225db6 in main_loop_wait (nonblocking=0) at > ../util/main-loop.c:589 > #14 0x0000555555c0c1a3 in qemu_main_loop () at ../system/runstate.c:835 > #15 0x000055555612bd8d in qemu_default_main (opaque=0x0) at > ../system/main.c:48 > #16 0x000055555612be3d in main (argc=23, argv=0x7fffffffe508) at > ../system/main.c:76 > > virtio_queue_notify_vq at hw/virtio/virtio.c:2484 [2] calls > vq->handle_output(vdev, vq). I see "handle_output" is a function > pointer and in this case it seems to be pointing to > virtio_net_handle_ctrl. > > >>>> [...] > >>>> With x-svq=true, I see that n->mac is set by virtio_net_handle_mac() > >>>> [3] when L1 receives VIRTIO_NET_CTRL_MAC_ADDR_SET. With x-svq=false, > >>>> virtio_net_handle_mac() doesn't seem to be getting called. I haven't > >>>> understood how the MAC address is set in VirtIONet when x-svq=false. > >>>> Understanding this might help see why n->mac has different values > >>>> when x-svq is false vs when it is true. > >>> > >>> Ok this makes sense, as x-svq=true is the one that receives the set > >>> mac message. You should see it in L0's QEMU though, both in x-svq=on > >>> and x-svq=off scenarios. Can you check it? > >> > >> L0's QEMU seems to be receiving the "set mac" message only when L1 > >> is launched with x-svq=true. With x-svq=off, I don't see any call > >> to virtio_net_handle_mac with cmd == VIRTIO_NET_CTRL_MAC_ADDR_SET > >> in L0. > >> > > > > Ok this is interesting. Let's disable control virtqueue to start with > > something simpler: > > device virtio-net-pci,netdev=net0,ctrl_vq=off,... > > > > QEMU will start complaining about features that depend on ctrl_vq, > > like ctrl_rx. Let's disable all of them and check this new scenario. > > > > I am still investigating this part. I set ctrl_vq=off and ctrl_rx=off. > I didn't get any errors as such about features that depend on ctrl_vq. > However, I did notice that after booting L2 (x-svq=true as well as > x-svq=false), no eth0 device was created. There was only a "lo" interface > in L2. An eth0 interface is present only when L1 (L0 QEMU) is booted > with ctrl_vq=on and ctrl_rx=on. > Any error messages on the nested guest's dmesg? Is it fixed when you set the same mac address on L0 virtio-net-pci and L1's?
