Dear polite inhabitants of the "netdev" mailing list, this is for a skunkworks project at the fringe of my job... More of a DIY hobby thing :-) I'm tinkering and having fun.
The wizards from linux-ptp have taught me how to use the i210 for precise timestamping, which works fine at all copper speeds and likely also for gigabit fiber (SERDES) which the i210 natively supports (and which links fine). The catch is: I'm trying to get this to work on 100Base-FX :-) which is not a native interface for the i210. It can do SERDES only at 1Gb. But, it can do SGMII. I've managed to find an i210-based NIC from DeLock with an SFP slot and I've managed to find two models of 100Base-FX SGMII SFP's. My assumption is that the SFP's have SGMII instead of SERDES and MDIO instead of SPD I2C - and the manufacturers of the board and of the SFP's don't dispute that. Yet they have both vaguely warned me in advance that it would not work work :-) Well I've given it a try anyway, and I'm stuck at the MDIO level. With one of the SFP's, I know for a fact that it's based on the BCM5461S. The actual marking on the chip goes BCM5461SA... etc. I've managed to modify the EEPROM of the NIC with EEUPDATE. The Device ID wouldn't change, but the "flash config words" that I needed were all willing to accept the change. So I can switch the chip to SGMII with external MDIO, and I can even see the chip generate an MDIO read transaction right after power-up. http://support.fccps.cz/download/adv/frr/ptp/MDIO_oscillograms.zip And this is where I'm stuck: the BCM chip does not respond to the i210 MDIO requests. I've hacked the igb driver a little, adding printks where I needed them, to see what's going on... I believe I've also spotted a minor bug where some PHY detection routine tries to read the status register before the phy.addr is even initialized... This is what the Intel chip does on power up: "read PHY reg. 0 from PHY addr <whatever I specify in the flash> ". (= read PHY_CONTROL) The igb driver does: 1) ret_val = hw->phy.ops.write_reg(hw, 0x1B, 0x8084); ...which succeeds, as apparently there's no "ACK bit" in the write sequence. The MDIO master never gets to know if the slave PHY is alive or not :-) 2) read "PHY control" == read PHY register 0 ... this fails. The PHY just doesn't do its part, and the MDIO controller HW in the MAC doesn't raise the status flag saying "transaction completed". The PHY just doesn't take over. It seems as if the i210 does not even continue sending CLK pulses - as if waiting for an active 0 (ACK?) from the PHY, and the ACK never comes. I have modified the igb driver to just go ahead and try the next step: 3) read PHY device ID == read PHY_ID1 == PHY register 0x02. ...which fails. I did not bother to try other registers. The one thing I'm not certain about is the required PHY addr. Not sure if this is hardwired in the PHY chip, or set by pin straps, or set by soft straps in the SFP's EEPROM or something. (Not that I've seen an EEPROM inside the SFP.) I've noticed some earlier suggested patches by Jonathan Toppins and Alan Liebthal of Cumulus Networks (from mid 2015) who seem to have gotten the "server ATOM's" MAC to work with a BCM5461S. Their code mentioned phy_addr = 5, which was possibly board-specific in their hardware. I've actually tried wrapping the steps 1) 2) 3) above in a FOR loop, iterating over phy_addr from 1 to 31. I can see on an oscilloscope that the i210's MDIO controller keeps trying different PHY addresses in the respective 5 bits of phy_addr, but the SFP just doesn't respond to any read transaction. The MDC (clock) from the i210 is a nice 50:50 rectangle at 2 MHz. I have reasons to believe that the SFP is powered. There's a P-FET in 3.3Vcc driven by SDP3, which opens when SDP3 goes log.0. Which does happen - the required state of SDP3 is encoded in the flash and the driver also writes the CTRL_EXT at runtime just in case. Also, it works in SERDES mode, where the NIC links just fine. And, I believe I can see some erratic responses from the SFP if I turn off external MDIO = when the i210 SGMII tries to run with i2c for management access to the PHY. In i2c mode, the controller runs the clock line at 100 kHz (and obviously the framing/protocol is different) - seems to me on the scope that the PHY starts babbling something on the data line. The read transactions in i2c mode even seem to succeed, but they read garbage. Interestingly regular, garbage. Mii-diag says: Basic registers of MII PHY #1: 7fff ff80 8000 007f 7fff ff80 8000 00ff. And that pattern repeats always the same. The SPF vendor has already replied that, as per spec, the BCM PHY should support up to 2.5 MHz MDC. And, they provided a framing diagram which looks like a perfect IEEE clause 22. Still I'm wondering if I should try to marry the igb driver with the "phylib" (which has some specific support for the BCM5461) and specifically with the bit-banging MDIO driver. Bit-banging might get me around some protocol incompatibility in MDIO (Broadcom vs. Intel) but I would face another problem: the way I understand it, the i210 needs to auto-configure the MAC block based on data it gets via MDIO autonomously from the PHY. (I understand that the high-speed SGMII lane for payload cannot transport PHY management data in-band, is that correct?) So if I bit-bang the MDIO protocol, and even get to talk to the SFP, I probably won't be able to configure the MAC accordingly explicitly... After a few hours of playing with this, I finally stumbled (a little too late) across this FAQ entry in an Intel i210/i211 FAQ PDF: "For external PHY use SerDes SKU and configure to use SGMII with SGMII capable PHY such as a Marvell 88e1112. Be advised, ND has not been able to inter-operate with BRCM SGMII PHY so far." Which does not explain any details. The full PDF is here: https://www.intel.com/content/dam/www/public/us/en/documents/faqs/ethe rnet-controller-i210-i211-faq.pdf Anyway... if you've read this far, thanks for your kind attention - and, any sort of comments are welcome :-) Frank Rysanek
