All, We are seeing 'random' memory corruption in mbufs coming from the ixgbe UIO driver and I am looking for some pointers on debugging it. Our software was running flawlessly for weeks at a time on our old Westmere systems (CentOS 6.4) but since moving to a new Sandy Bridge v2 server (also CentOS 6.4) it runs for 1-2 minutes and then at least one mbuf is overwritten with arbitrary data (pointers/lengths/RSS value/num segs/etc. are all ridiculous). Both servers are using the 82599EB chipset (x520) and the DPDK version (1.6.0r2) is identical. We recently also tested on a third server running RHEL 6.4 with the same hardware as the failing Sandy Bridge based system and it is fine (days of runtime no failures).
Running all of this in GDB with 'record' enabled and setting a watchpoint on the address which contains the corrupted data and executing a 'reverse-continue' never hits the watchpoint [GDB newbie here -- assuming 'watch *(uint64_t*)0x7FB.....' should work]. My first thought was memory corruption but the BIOS memcheck on the ECC RAM shows no issues. Also looking at mbuf->pkt.data, as an example, the corrupt value was the same 6/12 trials but I could not find that value elsewhere in the processes memory. This doesn't seem "random" and points to a software bug but I cannot for the life of me get GDB to tell me where the program is when that memory is written to. Incidentally trying this with the PCAP driver and --no-huge to run valgrind shows no memory access errors/uninitialized values/etc. Thoughts? Pointers? Ways to rule in/out hardware other than going 1 by 1 removing each of the 24 DIMMs? Thanks so much in advance! Stefan
