Alex,
Thanks for the tip about using ethregs to inspect the DCA registers.
However, this has led to another question. Specifically, it seems that
while DCA is enabled for most of the queues, those that land on core 0 seem
to stay disabled for some reason.
Here's a snip from one of my machines with 24 cores (and 24 queues).
001-023 all show enabled, but 000 does not:
[root@sec54 ethregs-1.16.0]# ./ethregs -d 8086:10fb | grep DCA | more
DCA_RXCTRL[000] 00001200
DCA_RXCTRL[001] 1a0002a0
DCA_RXCTRL[002] 190002a0
DCA_RXCTRL[003] 170002a0
DCA_RXCTRL[004] 160002a0
DCA_RXCTRL[005] 150002a0
DCA_RXCTRL[006] 1b0002a0
DCA_RXCTRL[007] 1a0002a0
DCA_RXCTRL[008] 190002a0
DCA_RXCTRL[009] 170002a0
DCA_RXCTRL[010] 160002a0
DCA_RXCTRL[011] 150002a0
DCA_RXCTRL[012] 1b0002a0
DCA_RXCTRL[013] 1a0002a0
DCA_RXCTRL[014] 190002a0
DCA_RXCTRL[015] 170002a0
DCA_RXCTRL[016] 160002a0
DCA_RXCTRL[017] 150002a0
DCA_RXCTRL[018] 1b0002a0
DCA_RXCTRL[019] 1a0002a0
DCA_RXCTRL[020] 190002a0
DCA_RXCTRL[021] 170002a0
DCA_RXCTRL[022] 160002a0
DCA_RXCTRL[023] 150002a0
DCA_RXCTRL[024] 00001200
DCA_RXCTRL[025] 00001200
The TXCTRL shows the same pattern.
I'm not sure what to make of this, and I'm also not sure how to get DCA
running on queue 0.
Thanks,
Scott Silverman | IT | Simplex Investments | 312-360-2444
230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
On Thu, Feb 13, 2014 at 4:18 PM, Alexander Duyck <
[email protected]> wrote:
> Odds are the problem is the order the modules are loading in. In order
> to print the message ioatdma and dca needs to be loaded before ixgbe, if
> the ioatdma driver is loaded after then you won't see the message as it
> is only printed at probe time and not when a DCA provider is registered.
>
> If you are loading dca, ixgbe, and then ioatdma in that order the ixgbe
> won't show as having DCA enabled because the ixgbe_notify_dca call
> doesn't appear to print any message. As a result you would have DCA
> enabled, but it won't have displayed any message stating as such.
>
> The tell-tale sign that you have DCA enabled is to use the ethregs tool
> to dump the registers for the device. The DCA_RXCTRL and DCA_TXCTRL
> registers will have a APIC tag ID in the upper 8 bits. Usually the
> value is something like 1f or 1e. If the value is 00 then it is
> disabled. Below is a snippet of a register dump on a system I have here
> that supports DCA.
>
> DCA Enabled:
> DCA_TXCTRL[0] 1f002220
> DCA_TXCTRL[1] 1f002220
>
> DCA Disabled:
> DCA_TXCTRL[0] 00002220
> DCA_TXCTRL[1] 00002220
>
> Thanks,
>
> Alex
>
> On 02/13/2014 01:38 PM, Scott Silverman wrote:
> > It seems I was making a mistake, just not the one I thought I was.
> > When I looked at 3.18.7, it was after a system boot. When I looked at
> > 3.19.1 it was only after removing and reloading the module.
> >
> > I've attached dmesg output from a system boot just now, showing dca /
> > igb / ixgbe / ioatdma modules all loading. It seems that for whatever
> > reason, during the system boot DCA is not enabled for ixgbe (but it is
> > for igb). If I remove and reload ixgbe, it then enables DCA. (seen at
> > the end of the attached dmesg).
> >
> > Is this the expected behavior? Am I doing something wrong? Do I need
> > to ensure that ioatdma loads before ixgbe? (if so, why doesn't igb
> > seem to care?)
> >
> >
> >
> >
> > Thanks,
> >
> > Scott Silverman | IT | Simplex Investments | 312-360-2444
> > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> >
> >
> > On Thu, Feb 13, 2014 at 3:18 PM, Scott Silverman
> > <[email protected]
> > <mailto:[email protected]>> wrote:
> >
> > Alex/John,
> >
> > Thanks for the clarification with regards to DDIO/DCA.
> >
> > As far as my results with the 3.18.7 driver, I can't duplicate
> > them now, so I'll chalk it up to a mistake on my side. Sorry for
> > the trouble.
> >
> >
> >
> >
> > Thanks,
> >
> > Scott Silverman | IT | Simplex Investments | 312-360-2444
> > <tel:312-360-2444>
> > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> >
> >
> > On Thu, Feb 13, 2014 at 3:03 PM, Duyck, Alexander H
> > <[email protected] <mailto:[email protected]>>
> > wrote:
> >
> > Scott,
> >
> >
> >
> > DDIO does not completely replace DCA. DCA provides
> > functionality for remote socket, while DDIO only functions
> > with the local socket for the device.
> >
> >
> >
> > The ixgbe driver prints that DCA message if DCA was detected
> > when the driver was loaded. Did you try running lsmod to see
> > if ioatdma and dca modules were loaded when you loaded the
> > 3.18.7 driver?
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Alex
> >
> >
> >
> > *From:*Scott Silverman
> > [mailto:[email protected]
> > <mailto:[email protected]>]
> > *Sent:* Thursday, February 13, 2014 12:36 PM
> > *To:* Duyck, Alexander H
> > *Cc:* Ronciak, John; [email protected]
> > <mailto:[email protected]>
> > *Subject:* Re: [E1000-devel] DCA on Sandy Bridge?
> >
> >
> >
> > John,
> >
> >
> >
> > I think you misunderstood. Sandy Bridge ioatdma/dca should be
> > supported in kernel 3.4.41. However, I only see that "DCA"
> > flag in ixgbe/dmesg when I am using ixgbe 3.19.1, not when I
> > am using 3.18.7. I am asking why that might be. Nothing else
> > changed, aside from rmmod ixgbe (3.18.7) and modprobe ixgbe
> > (3.19.1).
> >
> >
> >
> > Alex,
> >
> >
> >
> > There is nothing in dmesg to explain it. In fact, the igb
> > driver always uses DCA (prints "DCA enabled"), so I know that
> > from a platform perspective it is working.
> >
> >
> >
> > To put it simply, I have two questions:
> >
> >
> >
> > 1. On Sandy bridge (and Ivy Bridge, etc), does DDIO replace
> > DCA? If so, what implication does the presence of "DCA" in
> > enabled features on a platform that is meant to have DDIO
> > instead of DCA?
> >
> > 2. What change from 3.18.7 to 3.19.1 would cause the feature
> > to become Enabled on a platform that otherwise supports DCA?
> >
> >
> >
> >
> >
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Scott Silverman | IT | Simplex Investments | 312-360-2444
> > <tel:312-360-2444>
> >
> > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> >
> >
> >
> > On Thu, Feb 13, 2014 at 1:59 PM, Alexander Duyck
> > <[email protected]
> > <mailto:[email protected]>> wrote:
> >
> > Scott,
> >
> > You might try checking the dmesg log on the systems that come
> > up as not
> > supporting DCA. There was a patch submitted a year or so ago
> > that would
> > disable DCA on platforms that had a misconfigured APIC ID tag
> map.
> >
> > It is possible that the platform might have had DCA disabled
> > due to this
> > in the case of the 3.18.7. You might want to go back and
> > recheck the
> > dmesg log to verify if DCA was disabled due to a configuration
> > error in
> > the BIOS.
> >
> > Thanks,
> >
> > Alex
> >
> >
> > On 02/13/2014 08:27 AM, Scott Silverman wrote:
> > > John,
> > >
> > > Thanks for the prompt response, but I still have some
> > questions/concerns.
> > >
> > > -My motherboard (SuperMicro X9DRW) specifically provides a
> > firmware option
> > > to enable "DCA" but you state that the chipset doesn't
> > include it at all?
> > >
> > > -Why is it reported only sometimes?
> > >
> > > Proc, Kern, ixgbe - DCA?
> > > Sandy, 3.4, 3.18.7 - No DCA
> > > Sandy, 3.4, 3.19.1 - DCA
> > > Sandy, 3.12, 3.19.1 - DCA
> > > Ivy, 3.12, 3.18.7 - DCA
> > > Ivy, 3.12, 3.19.1 - DCA
> > >
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Scott Silverman | IT | Simplex Investments | 312-360-2444
> > <tel:312-360-2444>
> > > 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> > >
> > >
> > > On Mon, Feb 10, 2014 at 4:26 PM, Ronciak, John
> > <[email protected] <mailto:[email protected]>>wrote:
> > >
> > >> The driver is just telling you what's possible not that DCA
> > is enabled.
> > >> The newer chipsets do not included it as it's all DDIO.
> > So even though
> > >> the NIC's is capable of support DCA the chipset does not
> > have it so it
> > >> won't be used. That's all this is saying. It's not an
> > issue at all.
> > >>
> > >> Cheers,
> > >> John
> > >>
> > >>
> > >>> -----Original Message-----
> > >>> From: Scott Silverman
> > [mailto:[email protected]
> > <mailto:[email protected]>]
> > >>> Sent: Monday, February 10, 2014 11:59 AM
> > >>> To: [email protected]
> > <mailto:[email protected]>
> > >>> Subject: [E1000-devel] DCA on Sandy Bridge?
> > >>>
> > >>> My understanding is the DDIO completely replaces DCA on
> > sandy bridge
> > >>> and newer hardware (E5+ xeons). I expected this is why I
> > don't see
> > >>> "DCA" listed when I load ixgbe, as I would on my nehalem
> > systems.
> > >>>
> > >>> I see the following when loading ixgbe 3.18.7 on an
> > E5-2670 system
> > >>> (kernel
> > >>> 3.4.41):
> > >>>
> > >>> ixgbe 0000:04:00.1: (PCI Express:5.0GT/s:Width x8)
> > 00:1b:21:5c:66:0d
> > >>> ixgbe 0000:04:00.1: eth3: MAC: 2, PHY: 9, SFP+: 4, PBA No:
> > E68793-002
> > >>> ixgbe 0000:04:00.1: eth3: Enabled Features: RxQ: 32 TxQ:
> > 32 FdirHash
> > >>> RSC ixgbe 0000:04:00.1: eth3: Intel(R) 10 Gigabit Network
> > Connection
> > >>>
> > >>> If I take the same system, unload ixgbe, and load ixgbe
> > 3.19.1, I see
> > >>> this (emphasis mine):
> > >>>
> > >>> ixgbe 0000:04:00.1: PCI Express bandwidth of 32GT/s
> > available ixgbe
> > >>> 0000:04:00.1: (Speed:5.0GT/s, Width: x8, Encoding
> > Loss:20%) ixgbe
> > >>> 0000:04:00.1: eth3: MAC: 2, PHY: 9, SFP+: 4, PBA No:
> > E68793-002 ixgbe
> > >>> 0000:04:00.1: 00:1b:21:5c:66:0d ixgbe 0000:04:00.1: eth3:
> > Enabled
> > >>> Features: RxQ: 32 TxQ: 32 FdirHash *DCA*RSC ixgbe
> > 0000:04:00.1: eth3:
> > >>> Intel(R) 10 Gigabit Network Connection
> > >>>
> > >>> Can anyone explain why I see "DCA" with 3.19.1 and not
> > with 3.18.7.
> > >>> Also, does it matter?
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Scott Silverman | IT | Simplex Investments | 312-360-2444
> > <tel:312-360-2444>
> > >>> 230 S. LaSalle St., Suite 4-100, Chicago, IL 60604
> > >>
> > >
> >
> >
> >
> >
> >
>
>
------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121054471&iu=/4140/ostg.clktrk
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
