On Sun, 27 Aug 2017 18:03:25 +0200 Torbjorn Jansson <torbjorn.jans...@mbox200.swipnet.se> wrote:
> On 2017-08-22 19:38, Alex Williamson wrote: > > On Mon, 21 Aug 2017 23:09:55 +0200 > > Torbjorn Jansson <torbjorn.jans...@mbox200.swipnet.se> wrote: > > > >> On 2017-08-21 19:56, Torbjorn Jansson wrote: > >>> On 2017-08-19 07:37, Torbjorn Jansson wrote: > >>>> On 2017-08-19 02:34, taii...@gmx.com wrote: > >>>>> On 08/15/2017 02:12 AM, Torbjorn Jansson wrote: > >>>>> > >>>>>> On 2017-08-15 02:44, taii...@gmx.com wrote: > >>>>>>> On 08/14/2017 12:28 AM, Torbjorn Jansson wrote: > >>>>>>> > >>>>>>>> yes i noticed that part was missing, but the card i have is an > >>>>>>>> I350-T2 > >>>>>>>> and according to: > >>>>>>>> https://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000005722.html > >>>>>>>> > >>>>>>>> > >>>>>>>> the I350-T2 is supposed to have sr-iov support. > >>>>>>>> i guess it is possible they sent me the wrong type of card or > >>>>>>>> something, > >>>>>>>> i'll double check what is listed on the card itself. > >>>>>>> Some OEM's (ex: dell) shut off the SR-IOV ability on their NIC's > >>>>>>> because > >>>>>>> reasons. > >>>>>>> Please provide an lspci -n and lspci -v -v for that device so we can > >>>>>>> see > >>>>>>> what OEM it is. > >>>>>>> > >>>>>>> I would return it and have them send you a real one that isn't nerfed, > >>>>>>> also FYI most NIC's on ebay are counterfeit so be careful what you > >>>>>>> buy. > >>>>>>> https://www.servethehome.com/investigating-fake-intel-i350-network-adapters/ > >>>>>>> (real intel NIC ASIC and SR-IOV but crappy secondary components) > >>>>>>> > >>>>>> > >>>>>> if i look at the back of the card there is a sticker with: I350T2V2 > >>>>>> > >>>>>> here is the output: > >>>>> Weird. > >>>>> I would say you almost definitely got a fake - either way send it back. > >>>>> > >>>> > >>>> yes very strange. > >>>> thing is, i bought it from a well known web shop I've been using forever. > >>>> i have opened a support ticket with them about this, hopefully i can > >>>> return it. > >>>> > >>>> but then question is, how do i find a network card with sr-iov that > >>>> works? > >>>> clearly specs on web shops can't always be trusted. > >>>> > >>>> also strange is that this page: > >>>> http://ark.intel.com/products/84804/Intel-Ethernet-Server-Adapter-I350-T2V2 > >>>> clearly lists sr-iov and this is the model i got. > >>>> > >>> > >>> i did a bit more searching and checking. > >>> found this pdf: > >>> https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/ethernet-controller-i350-datasheet.pdf > >>> > >>> section 6.2.25 looks interesting specifically the IOV Enable bit. > >>> > >>> i used ethtool to dump the eeprom to file and a hex editor to check the > >>> contents and this bit is set to a zero indicating IOV is not exposed. > >>> > >>> but i'm definitely no expert on these things so i might be wrong too. > >>> but i wonder if there is a way to flip this bit and see what happens. > >>> > >> > >> i fixed it :) > >> i figured out how to use ethtool to flip the bit, rebooted and: > >> ------- > >> 03:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network > >> Connection > >> (rev 01) > >> Subsystem: Intel Corporation Ethernet Server Adapter I350-T2 > >> Physical Slot: 6-1 > >> Flags: bus master, fast devsel, latency 0, IRQ 61, NUMA node 0 > >> Memory at fad00000 (32-bit, non-prefetchable) [size=1M] > >> Memory at fae04000 (32-bit, non-prefetchable) [size=16K] > >> Capabilities: [40] Power Management version 3 > >> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ > >> Capabilities: [70] MSI-X: Enable+ Count=10 Masked- > >> Capabilities: [a0] Express Endpoint, MSI 00 > >> Capabilities: [100] Advanced Error Reporting > >> Capabilities: [140] Device Serial Number a0-36-9f-ff-ff-ed-43-b0 > >> Capabilities: [150] Alternative Routing-ID Interpretation (ARI) > >> Capabilities: [160] Single Root I/O Virtualization (SR-IOV) > >> Capabilities: [1a0] Transaction Processing Hints > >> Capabilities: [1c0] Latency Tolerance Reporting > >> Capabilities: [1d0] Access Control Services > >> Kernel driver in use: igb > >> Kernel modules: igb > >> ------- > >> as you can see above sr-iov is now listed in lspci, needed sriov entries > >> is now > >> listed under /sys and i can enable the virtual cards. > >> had to blacklist igbvf and libvirt kept complaining until i added the > >> virtual > >> cards to vfio too. > >> > >> so now i finally got it all working and have a vm with a virtual card. > >> > >> but i think it is really stupid of the manufacture(s) (intel?) to disable > >> sr-iov in the eeprom and not provide any proper tool or documentation on > >> how to > >> enable it. > > > > Would you be so kind as to write-up instructions how you used ethtool > > to resolve this so that others don't need to re-invent your solution? > > Thanks, > > > > yes of course. > > first of all, be careful when you poke around in the cards eeprom. > below is what i did and it worked on my I350-T2V2 card. > theoretically it might work on other I350 cards too but it is not something i > have tested. > > before starting you need to know the device name of your ethernet device, it > is > also useful to check output of lspci -v on your device and check that your > card > is missing this line: > Capabilities: [160] Single Root I/O Virtualization (SR-IOV) > > first make a backup of your eeprom: > ethtool --eeprom-dump DEVNAME raw on >eth-eeprom.bin > > i used this both as backup and to check the contents with a hex editor and to > make sure i understood the documentation correctly regarding endianess and > offsets. > > next we want to check what the iov enable bit is currently set to. > according to documentation i found, this is bit zero of word 0x25 in eeprom. > so this makes byte offset 0x4a (0x25*2 documentation uses 16 bit words so we > need to multiply by 2 to get byte offset) > so: > > ethtool --eeprom-dump DEVNAME offset 0x4a length 1 > > in my case i got 0xf6 back. > so bit zero is set to a zero which means IOV capability is not exposed > (expected based on earlier lspci -v). > there is no point in continuing unless it is a zero. > > take above value you just read out and set bit zero to 1 and then use your > newly calculated value in below command to change it: > > ethtool --change-eeprom DEVNAME magic 0x15218086 offset 0x4a length 1 value > 0xYOUR_VALUE_GOES_HERE > > the magic value to ethtool is i think deliberately not documented to prevent > you from accidentally screwing up your eeprom. > in this case it is the pci vendor and device id (lspci -n). > > once this is done, reboot computer and check output of lspci -v and look for: > Capabilities: [160] Single Root I/O Virtualization (SR-IOV) > if everything worked out it should now show up and you can go ahead with the > rest of sr-iov configuration. > > again, be careful when you do this. > > > > if you need to use this then i would recommend contacting intel support about > this because i think this is not the proper way of doing things. > i consider this more of a "hack" or a workaround than a proper solution. > i would rather have a proper tool or documentation or something from intel > specifying what to expect regarding IOV enable/disable on these cards. > > i found no documentation or tool to enable this or i would not have done this > via ethtool. > > it is not fun to waste weeks on troubleshooting sr-iov only to find out that > it > was disabled in the cards eeprom. > it also doesn't help that when you lookup specs for this card it clearly > lists > that it supports sr-iov, even the place i bought it from and their support > said > it supported sr-iov (incorrectly, kind of) > Thanks for the write-up, this will be useful to have in the archives! Thanks, Alex _______________________________________________ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
