On Tuesday, October 23, 2012 8:40:44 am Harald Schmalzbauer wrote: > schrieb Harald Schmalzbauer am 23.10.2012 11:49 (localtime): > > schrieb Harald Schmalzbauer am 22.10.2012 21:48 (localtime): > >> schrieb Harald Schmalzbauer am 22.10.2012 21:33 (localtime): > >>> Hello, > >>> > >>> when using igb as module, no packet is received. > >>> If I send out anything, I see the packet with tcpdump, also the switch > >>> learns the MAC address, but nothing comes back in - total silenc, no > >>> boradcasts, nothing. > >>> If I unload the module and load it again, everything works as expected! > >>> No matter if I load it by 4th loader, or later, I always have tio unload > >>> first then load it again. > >>> I'ts late here, I'll see tomorrow if things change when compieled into > >>> kernel. > > It doesn't matter if igb is loaded as module or compiled into kernel. > > > >>> Maby somebody has an idea what the source of the problem could be. > >>> Please find atteched some info, the OS is 9-RC2-amd64 on ESXi5.1 and > >>> nics are pci-passthrough. > >> I found one possibly relevant difference: > >> > >> Non-Working state: dev.igb.0.link_irq: 0 > >> Working state: dev.igb.0.link_irq: 2 > > This is only true with msi-x!!! > > If I disable mis-x, the problem itself vanishes. igb just works fine > > from the initial loading (with dev.igb.0.link_irq=0!). > > So dev.igb.0.link_irq is only relevant with msi-x. > > But what makes me curious is why it also works mith mis-x enabled after > > the second kldload!?! > > I think I found the root cause: > When ESXi powers up the guest, the passthru-devices are intialized with: > VMKPCIPassthru: 2565: BDF = 02:00.1 intrType = 2 numVectors: 1 > intrType=2 seems to mean MSI. > I guess, IOMMUIntel is instructed to remap one irq-vector for the > device. But igb uses MSI-X and wants 3 vectors. > If I unload if_igb and reload again, the ESXi-log shows the following: > VMKPCIPassthru: 2565: BDF = 02:00.1 intrType = 4 numVectors: 3 > intrType=4 seems to mean MSI-X. > After that initialization, if_igb works fine and saves 25kIRQ/s! > > I haven't found a way to change the power-up behaviour for the guest > with ESXi. > > Is it possible to re-init a pci device from userland?
The problem is you want the igb driver to retry MSI-X even after a re-init and that basically requires a full detach/attach, so your existing workaround is actually the "best" way to do this. :( Alternatively, you could try forcing igb to not use MSI, only use either MSI-X or INTx. -- John Baldwin _______________________________________________ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
