> > >-----Original Message----- >From: Burakov, Anatoly <anatoly.bura...@intel.com> >Sent: Tuesday, July 24, 2018 2:33 PM >To: Kumar, Ravi1 <ravi1.ku...@amd.com>; dev@dpdk.org >Subject: Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems > >On 24-Jul-18 9:09 AM, Kumar, Ravi1 wrote: >>> >>> >>> -----Original Message----- >>> From: Burakov, Anatoly <anatoly.bura...@intel.com> >>> Sent: Monday, July 16, 2018 4:05 PM >>> To: Kumar, Ravi1 <ravi1.ku...@amd.com>; dev@dpdk.org >>> Subject: Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs >>> systems >>> >>> On 14-Jul-18 10:44 AM, Kumar, Ravi1 wrote: >>>> >>>> Memory setup with 2M pages works with the default configuration. >>>> With the default configuration and 2M hugepages >>>> >>>> 1. Total amount of memory for each NUMA zone does not >>>> exceed 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE). >>>> >>>> 2. Total number of segment lists per NUMA is limited to >>>> 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE). This constraint is met for >>>> each numa zone. This is the limiting factor for memory per numa >>>> with 2M hugepages and the default configuration. >>>> >>>> 3. The data structures are capable of supporting 64G of >>>> memory for each numa zone (32768 segments * 2M hugepagesize). >>>> >>>> 4. 8 NUMA zones * 64G = 512G. Therefore the total for all >>>> numa zones does not exceed 512G (CONFIG_RTE_MAX_MEM_MB). >>>> >>>> 5. Resources are capable of allocating up to 64G per NUMA >>>> zone. Things will work as long as there are enough 2M hugepages to >>>> cover the memory needs of the DPDK applications AND no memory zone >>>> needs more than 64G. >>>> >>>> With the default configuration and 1G hugepages >>>> >>>> 1. Total amount of memory for each NUMA zone is limited to >>>> 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE). This constraint is hit for >>>> each numa zone. This is the limiting factor for memory per numa. >>>> >>>> 2. Total number of segment lists (128) does not exceed >>>> 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE). There are 128 segments per NUMA. >>>> >>>> 3. The data structures are capable of supporting 128G of >>>> memory for each numa zone (128 segments * 1G hugepagesize). >>>> However, only the first four NUMA zones get initialized before we >>>> hit CONFIG_RTE_MAX_MEM_MB (512G). >>>> >>>> 4. The total for all numa zones is limited to 512G >>>> (CONFIG_RTE_MAX_MEM_MB). This limit is hit after configuring the >>>> first four NUMA zones (4 x 128G = 512G). The rest of the NUMA zones >>>> cannot allocate memory. >>>> >>>> Apparently, it is intended to support max 8 NUMAs by default >>>> (CONFIG_RTE_MAX_NUMA_NODES=8), but when 1G hugepages are use, it can >>>> only support up to 4 NUMAs. >>>> >>>> Possible workarounds when using 1G hugepages: >>>> >>>> 1. Decrease CONFIG_RTE_MAX_MEM_MB_PER_TYPE to 65536 (limit >>>> of 64G per NUMA zone). This is probably the best option unless you >>>> need a lot of memory in any given NUMA. >>>> >>>> 2. Or, increase CONFIG_RTE_MAX_MEM_MB to 1048576. >>> >>> Hi Ravi, >>> >>> OK this makes it much clearer, thanks! >>> >>> I think the first one should be done. I think 64G per NUMA node is >>> still a reasonable amount of memory and it makes the default work (i >>> think we can go as far as reducing this limit to 32G per type!), and >>> whoever has issues with it can change CONFIG_RTE_MAX_MEM_MB_PER_TYPE >>> or CONFIG_RTE_MAX_MEM_MB for their use case. That's what these >>> options are there for :) >>> >>> -- >>> Thanks, >>> Anatoly >>> >> >> Hi Anatoly, >> >> Thanks a lot. Will the next release include this change? >> >> Regards, >> Ravi >> > >No one has submitted a patch for this, so not at this moment. I will do so >now, but i cannot guarantee it getting merged in 18.08 since it's almost RC2 >time, and introducing such a change may be too big a risk. > >-- >Thanks, >Anatoly Thanks Anatoly. I understand.
Regards, Ravi
