[dpdk-dev] Random mbuf corruption

[Paul Barrette]

On 06/20/2014 07:20 AM, Stefan Baranoff wrote:
> All,
>
> We are seeing 'random' memory corruption in mbufs coming from the ixgbe UIO
> driver and I am looking for some pointers on debugging it. Our software was
> running flawlessly for weeks at a time on our old Westmere systems (CentOS
> 6.4) but since moving to a new Sandy Bridge v2 server (also CentOS 6.4) it
> runs for 1-2 minutes and then at least one mbuf is overwritten with
> arbitrary data (pointers/lengths/RSS value/num segs/etc. are all
> ridiculous). Both servers are using the 82599EB chipset (x520) and the DPDK
> version (1.6.0r2) is identical. We recently also tested on a third server
> running RHEL 6.4 with the same hardware as the failing Sandy Bridge based
> system and it is fine (days of runtime no failures).
>
> Running all of this in GDB with 'record' enabled and setting a watchpoint
> on the address which contains the corrupted data and executing a
> 'reverse-continue' never hits the watchpoint [GDB newbie here -- assuming
> 'watch *(uint64_t*)0x7FB.' should work]. My first thought was memory
> corruption but the BIOS memcheck on the ECC RAM shows no issues.
>
> Also looking at mbuf->pkt.data, as an example, the corrupt value was the
> same 6/12 trials but I could not find that value elsewhere in the processes
> memory. This doesn't seem "random" and points to a software bug but I
> cannot for the life of me get GDB to tell me where the program is when that
> memory is written to. Incidentally trying this with the PCAP driver and
> --no-huge to run valgrind shows no memory access errors/uninitialized
> values/etc.
>
> Thoughts? Pointers? Ways to rule in/out hardware other than going 1 by 1
> removing each of the 24 DIMMs?
>
> Thanks so much in advance!
> Stefan
Run memtest to rule out bad ram.

Pb

[dpdk-dev] DMAR fault

[Paul Barrette]

On 08/12/2013 04:19 PM, jinho hwang wrote:
> Hi All,
>
> I am using iommu to receive packets both from hypervisor and from VM. 
> KVM is used for the virtualization. However, after I deliver the 
> kernel options (iommu and pci realloc), I can not receive packets in 
> hypervisor, but VF works fine in VM. When I tried to receive packets 
> in hypervisor, dmesg shows the following:
>
> ixgbe :03:00.1: complete
> ixgbe :03:00.1: PCI INT A disabled
> igb_uio :03:00.1: PCI INT A -> GSI 38 (level, low) -> IRQ 38
> igb_uio :03:00.1: setting latency timer to 64
> igb_uio :03:00.1: irq 87 for MSI/MSI-X
> uio device registered with irq 57
> DRHD: handling fault status reg 2
> DMAR:[DMA Read] Request device [03:00.1] fault addr *b9d0f000*
> DMAR:[fault reason 02] Present bit in context entry is clear
>
> 03:00.1 Ethernet controller: Intel Corporation 82599EB 10 Gigabit Dual 
> Port Backplane Connection (rev 01)
> Subsystem: Intel Corporation Ethernet X520 10GbE Dual Port 
> KX4-KR Mezz
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
> ParErr- Stepping- SERR- FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
> SERR-  Latency: 0, Cache Line Size: 64 bytes
> Interrupt: pin A routed to IRQ 38
> Region 0: Memory at *d940* (64-bit, prefetchable) [size=4M]
> Region 2: I/O ports at ece0 [size=32]
> Region 4: Memory at d9bfc000 (64-bit, prefetchable) [size=16K]
> Expansion ROM at  [disabled]
> Capabilities: 
> Kernel driver in use: igb_uio
> Kernel modules: ixgbe
>
> We can see those addresses are not matched. So the kernel got fault. I 
> am wondering why this happens?
I have seen this happen when VT-d is enabled in the bios.  If you are 
using dpdk 1.4, add "iommu=pt" to your boot line.  Without it, no 
packets are received.

Pb
>
> One suspicion for this is BIOS. I am currently using BIOS version 3.0, 
> but the latest is 6.3.0. Does this affect the matter?
>
> Any help appreciated!
>
> Jinho
>

-- next part --
An HTML attachment was scrubbed...
URL: 

[dpdk-dev] DMAR fault

[Paul Barrette]

On 08/12/2013 06:07 PM, jinho hwang wrote:
>
> On Mon, Aug 12, 2013 at 4:28 PM, Paul Barrette 
> mailto:paul.barrette at windriver.com>> 
> wrote:
>
>
> On 08/12/2013 04:19 PM, jinho hwang wrote:
>> Hi All,
>>
>> I am using iommu to receive packets both from hypervisor and from
>> VM. KVM is used for the virtualization. However, after I deliver
>> the kernel options (iommu and pci realloc), I can not receive
>> packets in hypervisor, but VF works fine in VM. When I tried to
>> receive packets in hypervisor, dmesg shows the following:
>>
>> ixgbe :03:00.1: complete
>> ixgbe :03:00.1: PCI INT A disabled
>> igb_uio :03:00.1: PCI INT A -> GSI 38 (level, low) -> IRQ 38
>> igb_uio :03:00.1: setting latency timer to 64
>> igb_uio :03:00.1: irq 87 for MSI/MSI-X
>> uio device registered with irq 57
>> DRHD: handling fault status reg 2
>> DMAR:[DMA Read] Request device [03:00.1] fault addr *b9d0f000*
>> DMAR:[fault reason 02] Present bit in context entry is clear
>>
>> 03:00.1 Ethernet controller: Intel Corporation 82599EB 10 Gigabit
>> Dual Port Backplane Connection (rev 01)
>> Subsystem: Intel Corporation Ethernet X520 10GbE Dual
>> Port KX4-KR Mezz
>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV-
>> VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast
>> >TAbort- SERR- > Latency: 0, Cache Line Size: 64 bytes
>> Interrupt: pin A routed to IRQ 38
>> Region 0: Memory at *d940* (64-bit, prefetchable)
>> [size=4M]
>> Region 2: I/O ports at ece0 [size=32]
>> Region 4: Memory at d9bfc000 (64-bit, prefetchable)
>> [size=16K]
>> Expansion ROM at  [disabled]
>> Capabilities: 
>> Kernel driver in use: igb_uio
>> Kernel modules: ixgbe
>>
>> We can see those addresses are not matched. So the kernel got
>> fault. I am wondering why this happens?
> I have seen this happen when VT-d is enabled in the bios.  If you
> are using dpdk 1.4, add "iommu=pt" to your boot line.  Without it,
> no packets are received.
>
> Pb
>
>>
>> One suspicion for this is BIOS. I am currently using BIOS version
>> 3.0, but the latest is 6.3.0. Does this affect the matter?
>>
>> Any help appreciated!
>>
>> Jinho
>>
>
>
> Paul,
>
> thanks. I tried your suggestion, but it works like no iommu command in 
> boot line. I passed intel_iommu=pt, and receive packets from 
> hypervisor. However, when I started VM with "-device 
> pci-assign,host=01:00.0", it shows the following message:
>
> qemu-system-x86_64: -device pci-assign,host=03:10.0: No IOMMU found. 
>  Unable to assign device "(null)"
> qemu-system-x86_64: -device pci-assign,host=03:10.0: Device 
> initialization failed.
> qemu-system-x86_64: -device pci-assign,host=03:10.0: Device 
> 'kvm-pci-assign' could not be initialized
>
> The device is detached from kernel, and move to pci-stub. dmesg does 
> not show any DMAR fault message anymore.
>
> Any idea?
>
> Jinho

Jinho,
  you need to specify both

" intel_iommu=on iommu=pt"

Pb
-- next part --
An HTML attachment was scrubbed...
URL: 
<http://dpdk.org/ml/archives/dev/attachments/20130812/2a63772e/attachment.html>

[dpdk-dev] [dpdk-ovs] Problem with OVS

[Paul Barrette]

On 12/12/2013 05:25 PM, Romulo Rosa wrote:
> Hi,
>
> i'm tryin to run OVS and i get a error. In documentation was mentioned a
> possible solution for this problem but it didnt work to me. Someone have
> any idea how to solve this problem?
> The uio module is loaded.
>
> *Command: *./datapath/dpdk/build/ovs_dpdk -c 0xf -n 4 -- -p 0xc
I think you have the wrong port mask.  Try using -p 0x3.

Pb
>   -n 2 -k 2
> --stats=1 --vswitchd=0 --client_switching_core=1 --config="(0,0,2),(1,0,3)"
>
> *Error: *
> EAL: Core 2 is ready (tid=43fd700)
> EAL: Core 3 is ready (tid=3bfc700)
> EAL: Core 1 is ready (tid=4bfe700)
> WARNING: requested port 2 not present - ignoring
> WARNING: requested port 3 not present - ignoring
> config = 16,0,2
> config = 17,0,3
> nb_cfg_params = 2
> Creating mbuf pool 'MProc_pktmbuf_pool' [14336 mbufs] ...
> HASH: Enabling CRC32 instruction for hashing
> APP: memzone address is 2ef33980
> EAL: Error - exiting with code: 1
>Cause: Cannot allocate memory for port tx_q details
>
> Thanks!

[dpdk-dev] unable to compile 1.6.0

[Paul Barrette]

On 04/25/2014 08:36 AM, Vasiliy Tolstov wrote:
> 2014-04-25 16:22 GMT+04:00 Ananyev, Konstantin  intel.com>:
>> Hi Vasiliy,
>> Why just not use ' -cpu host' when starting your kvm?
>
> Because in this case compiled binaries may use extensions that not
> supported on other platforms.
> As i see error happen with MACHINE_CFLAGS = -march=native, but i
> compile all software with match=x86_64 for compat.
>
>
In qemu you can turn off the cpu features that you don't want enabled:
http://www.linux-kvm.org/page/Tuning_KVM
maybe -cpu Nehalem ...

Pb