On 05/04/2018 06:25 PM, Thirunavukarasu Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES LIMITED at Cisco) wrote: > > Hi Laine, > > > > Thanks for taking the time to respond to my question. I think I have > not described my problem clearly. > > > > Let me explain my issue below with the information that you had requested. > > > > My assumption according to the information you gave me is that the > admin MAC and VF MAC are the same in my case. > > I see a PF (GE0-0) interface but I don’t see a vfnetdev interface as > you mentioned in your email. >
If the VF has no separate netdev visible on the host, then it is not being bound to the VF net driver for some reason. This should cause no ill effect - it just means there is no "original" VF mac to restore. > > > * Given this assumption, when the host is booted, the admin MAC and VF > MAC are both*00:00:00:00:00:00. * > Well, if there is no VF netdev on the host, there is no "vf mac" (and the igb driver wouldn't allow it to be 00:00:00:00:00:00). But again, not important > ** > > [root@nfvis ~]# ip link show GE0-0 > "GEO-0"? That's not a normal name for an igb PF. Are you manually selecting this name? > 3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216 > <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen 1000 > <tel:1000> > > link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff > > vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off > > vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off > > [root@nfvis ~]# > > > > * the VF is assigned to a guest, so the admin MAC is set to the > configured value *(52:54:00:29:3c:bf) *as shown below > > [root@nfvis ~]# ip link show GE0-0 > > 3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216 > <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen 1000 > <tel:1000> > > link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff > > vf 0 MAC *52:54:00:29:3c:bf*, spoof checking on, link-state auto, trust on > > vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on > > [root@nfvis ~]# > > > > * Configure bond interface on the guest and added member interface to > the bond > > On doing the above step, bond interface mac*(52:54:00:29:3c:be)* > gets assigned to VF > > [root@nfvis libvirt]# ip link show GE0-0 > > 3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216 > <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen 1000 > <tel:1000> > > link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff > > vf 0 MAC *52:54:00:29:3c:be*, spoof checking on, link-state auto, trust on > > vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on > > [root@nfvis libvirt]# > Okay, this part makes no sense, for 2 reasons: 1) once the admin MAC has been set by the host, a flag is set in the PF driver marking this VF as having an "administratively set" MAC (that's why I call it the "admin mac"), and after that point it should not be possible for a guest to modify the MAC address. If your guest is successfully setting the MAC of the VF, then either you don't have a device that uses the igb driver, or there is a bug in the driver. 2) setting the MAC address of the VF shouldn't update the admin mac for that VF - they are separate entities, and only synched up when the VF driver is reloaded (and in that case it is the *admin mac* that is used as the master copy to set both of them) > > > * The next step would be to restart the VM from within the VM console > (* we are not doing a shut down and start from the host *). > Okay, so that reloads the VF driver in the guest, which would set it to the admin mac (which you have said showed to be ....:be just before the reset. > At this stage, the admin MAC is not reverted to the domain xml > mac*(52:54:00:29:3c:bf)* > > Instead it retains the same bond mac *(52:54:00:29:3c:be)* which > was set before the VM was restarted from the console. ** > > > > [root@nfvis libvirt]# ip link show GE0-0 > > 3: GE0-0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 9216 > <tel:9216> qdisc mq master ovs-system state UP mode DEFAULT qlen 1000 > <tel:1000> > > link/ether a0:23:9f:ce:b1:f8 brd ff:ff:ff:ff:ff:ff > > vf 0 MAC *52:54:00:29:3c:be*, spoof checking on, link-state auto, trust on > > vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust on > > > > So the actual problem we face here, is after the VM is restarted and > after the members of bond interface are released, VF mac is not > getting reverted back to domain xml mac(*52:54:00:29:3c:bf*). > > Instead the VF mac and admin mac are set to the bond mac > “*52:54:00:29:3c:be*” and leading to a duplicate mac issue. > > My expectation was that after the VM is restarted from the console, > libvirt should revert the VF’s MAC to the original MAC, > > which in our case was the domain xml MAC, which is not happening in my > case. > My expectation was that the guest would not be allowed to modify the MAC address of the VF at all, and certainly that even if a change to the guest MAC was allowed, that it wouldn't propagate to the PF's list of admin mac addresses for the VFs. So we've both been disappointed :-) Your VF and PF drivers are behaving strangely. But if you completely shutdown the guest, then restart it, you should get the original "...:bf" back. Even if that works for you, I would not count on the ability of the guest to modify the VF's MAC address - that is not the defined behavior and thus is likely to change in the future. > > > > > Attached is the log file as requested. > > > > *_Interface related contents from domain xml: _* > > <interface type='hostdev' managed='yes'> > > <mac address='52:54:00:29:3c:bf'/> > > <driver name='vfio'/> > > <source> > > <address type='pci' domain='0x0000' bus='0x02' slot='0x10' > function='0x0'/> > > </source> > > <target dev='vnic1'/> > > <model type='virtio'/> > > <alias name='hostdev0'/> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x04' > function='0x0'/> > > </interface> > > > > Thanks > > Thiru. > > *From:*sendmail <justsendmailnothinge...@gmail.com> *On Behalf Of > *Laine Stump > *Sent:* Wednesday, May 2, 2018 10:01 AM > *To:* Thirunavukarasu Sengalvarayan -X (tsengalv - HCL TECHNOLOGIES > LIMITED at Cisco) <tseng...@cisco.com> > *Cc:* Chanda Mendon (cmendon) <cmen...@cisco.com> > *Subject:* Re: VF MAC not reverted to all zero MAC on VM restart > > > > On 04/27/2018 11:08 PM, Thirunavukarasu Sengalvarayan -X (tsengalv - > HCL TECHNOLOGIES LIMITED at Cisco) wrote: > > Hi Laine Stump, > > > > We are running linux based VM on KVM based Hypervisor and facing > issue with respect to VF MAC. > > Our Libvirt version: 3.2.0, host driver IGB version 5.2.16 and VF > driver IGBVF version 2.3.7.1 > > > > *_Description of problem:_* > > When passing a VF to a guest, libvirt sets its MAC according to > the domain xml. > > On shutting down the VM or power off the VM, libvirt attempts to > restore its original MAC(all-zero mac). There is no issue with > this scenario. > > *When we restart the VM, there is no attempt to restore/revert its > original MAC(all zero mac). * > > This problem leads to duplicate MAC issue on the guest (if > configured with portchannel). > > > > Could you please help us in resolving the issue? > > > Questions like this should be sent to libvirt-users@redhat.com > <mailto:libvirt-users@redhat.com>rather than to an individual's email > address. This makes the likelyhood of getting an answer much higher, > and also creates an archive of the problem and eventual solution, > which may help others in the future. I'm Cc'ing this response to that > list so that any further communication will happen there. > > Before reading any of the following, it's useful to know this - for > SRIOV network devices that *properly* support SRIOV (and I consider > those using the igb driver to be in this category) each VF has two MAC > addresses that need to be discussed: > > 1) the VF netdev MAC address (which I will just call the "mac address" > - this is the MAC that is known to the VF net driver, and displayed in > the output of "ip link show dev $vfnetdev". > > 2) the MAC address that is stored in the PF driver for each VF, > displayed in the "VF" lines immediately following the "ip link" info > of the *PF*, and which will be used to initialize the VF's own MAC > address when its driver is re-initialized (I will call this the "admin > mac") > > Some VF drivers initialize the mac to 00:00:00:00:00:00 (e.g. the > Cisco enic driver) and some initialize it to a random number (e.g. igb > and all the other Intel VF net drivers). Likewise, some PF drivers > initialize the admin macs to 00:00:00:00:00:00 and some to random > numbers (as a matter of fact, this behavior changes between different > versions of the same driver in at least one case!). > > Beyond this, some PF and VF drivers allow setting the mac/admin mac to > 00:00:00:00:00:00, and some prohibit one or the other. (in many cases, > a driver that itself initializes the mac/admin mac to > 00:00:00:00:00:00 also prohibits setting it back to that same value > once it has been changed!) > > When libvirt sets up a VF to be used by a guest (either for vfio > device assignment, or for macvtap passthrough), it saves both the mac > and the admin mac with the intent of restoring them later. > > Once these values have been saved, libvirt will set the mac (in the > case of macvtap passthrough) or the admin mac (in the case of vfio > device assignment), and send the device on to qemu to be used by the > guest. > > After the guest is finished running (or the device is detached), > libvirt attempts to restore the settings it had saved at the > beginning. Trouble comes up in 2 ways though - 1) once the admin mac > has been set for a VF (via the PF), it is no longer possible to > directly set the mac (in the VF) (this is done for security reasons, > to prevent the guest from changing its mac address), and 2) as said > above, many drivers don't allow setting 00:00:00:00:00:00, but in many > cases that was the original setting. > > As of libvirt 3.2.0, libvirt uses the following strategies to work > around these problems: > > a) if setting a non-0 mac (to the VF) fails, then libvirt will try > setting the *admin MAC* to that value, then detaching/re-attaching the > VF net driver. This will cause the PF to reinitialize the mac in the > VF driver. (NB: if we do this, then we don't bother trying to later > re-set the admin MAC to its own original value; it's really not > important). > > b) if the failure was in setting the MAC to 00:00:00:00:00:00, then > libvirt will set the mac/admin mac to 02:00:00:00:00:00 (which should > be a legal value for any driver, but also should not conflict with and > "real" mac on the network). > > Okay, enough background. On to your problem. > > > To make sure I understand your situation correctly: > > * When the host is booted, the admin MAC is 00:00:00:00:00:00, and VF > mac is random (call it rr:rr:rr:rr:rr:rr). > > * the VF is assigned to a guest (with <interface type='hostdev'> I > guess?), so the admin MAC is set to the configured value (let's call > it gg:gg:gg:gg:gg:gg) > > * the guest is shutdown, which causes *both* mac and admin mac to be > set to rr:rr:rr:rr:rr:rr > > * the next time the guest is start *its mac is not changed*? > > There must be something I'm not getting. Can you maybe perform this > test and run "ip link show $PF; ip link show $VF" (substituting the > netdev names of the PF and VF for $PF and $VF) after each step to > illustrate the problem? Also, it would be helpful to add this to > /etc/libvirt/libvirtd.conf: > > log_filters="1:util.netdev" > log_outputs="1:file:/var/log/libvirt/libvirtd-netdev.log > <file://var/log/libvirt/libvirtd-netdev.log>" > > then restart libvirtd (systemctl restart libvirtd.service), and post > the contents of libvirtd-netdev.log somewhere and reference it in your > next reply. (don't forget to remove those lines and restart libvirtd > again after your testing is done!) > > Finally, you should include the contents of the <interface ...> > section of your config in your next response. > > (BTW, since you're using libvirt-3.2.0, I assume that you are using > either RHEL 7.4 or CentOS 7.4. If you are using RHEL, you should open > a customer case with Red Hat so that your problem is appropriately > prioritized). >
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
