Hi Anatoly, Since we do not have the driver support of SRIOv with VFIO, we are using IGB_UIO . Basically our application is crashing due to the buffer allocation failure. I believe because it didn't get a contiguous memory location and fails to allocate the memory. Is there any API, I can use to dump before our application dies ? Please let me know.
Thanks, Kamaraj On Mon, Jul 13, 2020 at 2:57 PM Burakov, Anatoly <anatoly.bura...@intel.com> wrote: > On 11-Jul-20 8:51 AM, Kamaraj P wrote: > > Hello Anatoly/Bruce, > > > > We are using the 18_11 version of DPDK and we are using igb_uio. > > The way we observe an issue here is that, after we tried multiple > > iterations of start/stop of container application(which has DPDK), > > we were not able to allocate the memory for port during the init. > > We thought that it could be an issue of not getting continuous > > allocation hence it fails. > > > > Is there an API where I can check if the memory is fragmented before we > > invoke an allocation ? > > Or do we have any such mechanism to defragment the memory allocation > > once we exist from the application ? > > Please advise. > > > > This is unlikely due to fragmentation, because the only way for 18.11 to > be affected my memory fragmentation is 1) if you're using legacy mem > mode, or 2) you're using IOVA as PA mode and you need huge amounts of > contiguous memory. (you are using igb_uio, so you would be in IOVA as PA > mode) > > NICs very rarely, if ever, allocate more than a 2M-page worth of > contiguous memory, because their descriptor rings aren't that big, and > they'll usually get all the IOVA-contiguous space they need even in the > face of heavily fragmented memory. Similarly, while 18.11 mempools will > request IOVA-contiguous memory first, they have a fallback to using > non-contiguous memory and thus too work just fine in the face of high > memory fragmentation. > > The nature of the 18.11 memory subsystem is such that IOVA layout is > decoupled from VA layout, so fragmentation does not affect DPDK as much > as it has for previous versions. The first thing i'd suggest is using > VFIO as opposed to igb_uio, as it's safer to use in a container > environment, and it's less susceptible to memory fragmentation issues > because it can remap memory to appear IOVA-contiguous. > > Could you please provide detailed logs of the init process? You can add > '--log-level=eal,8' to the EAL command-line to enable debug logging in > the EAL. > > > Thanks, > > Kamaraj > > > > > > > > On Fri, Jul 10, 2020 at 9:14 PM Burakov, Anatoly > > <anatoly.bura...@intel.com <mailto:anatoly.bura...@intel.com>> wrote: > > > > On 10-Jul-20 11:28 AM, Bruce Richardson wrote: > > > On Fri, Jul 10, 2020 at 02:52:16PM +0530, Kamaraj P wrote: > > >> Hello All, > > >> > > >> We are running to run DPDK based application in a container mode, > > >> When we do multiple start/stop of our container application, the > > DPDK > > >> initialization seems to be failing. > > >> This is because the hugepage memory fragementated and is not > > able to find > > >> the continuous allocation of the memory to initialize the buffer > > in the > > >> dpdk init. > > >> > > >> As part of the cleanup of the container, we do call > > rte_eal_cleanup() to > > >> cleanup the memory w.r.t our application. However after > > iterations we still > > >> see the memory allocation failure due to the fragmentation issue. > > >> > > >> We also tried to set the "--huge-unlink" as an argument before > > when we > > >> called the rte_eal_init() and it did not help. > > >> > > >> Could you please suggest if there is an option or any existing > > patches > > >> available to clean up the memory to avoid fragmentation issues > > in the > > >> future. > > >> > > >> Please advise. > > >> > > > What version of DPDK are you using, and what kernel driver for NIC > > > interfacing are you using? > > > DPDK versions since 18.05 should be more forgiving of fragmented > > memory, > > > especially if using the vfio-pci kernel driver. > > > > > > > This sounds odd, to be honest. > > > > Unless you're allocating huge chunks of IOVA-contiguous memory, > > fragmentation shouldn't be an issue. How did you determine that this > > was > > in fact due to fragmentation? > > > > > Regards, > > > /Bruce > > > > > > > > > -- > > Thanks, > > Anatoly > > > > > -- > Thanks, > Anatoly >
