> > >-----Original Message----- >From: Burakov, Anatoly <anatoly.bura...@intel.com> >Sent: Monday, July 16, 2018 4:05 PM >To: Kumar, Ravi1 <ravi1.ku...@amd.com>; dev@dpdk.org >Subject: Re: [dpdk-dev] DPDK 18.05 only works with up to 4 NUMAs systems > >On 14-Jul-18 10:44 AM, Kumar, Ravi1 wrote: >> >> Memory setup with 2M pages works with the default configuration. With >> the default configuration and 2M hugepages >> >> 1. Total amount of memory for each NUMA zone does not >> exceed 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE). >> >> 2. Total number of segment lists per NUMA is limited to >> 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE). This constraint is met for >> each numa zone. This is the limiting factor for memory per numa with >> 2M hugepages and the default configuration. >> >> 3. The data structures are capable of supporting 64G of >> memory for each numa zone (32768 segments * 2M hugepagesize). >> >> 4. 8 NUMA zones * 64G = 512G. Therefore the total for all >> numa zones does not exceed 512G (CONFIG_RTE_MAX_MEM_MB). >> >> 5. Resources are capable of allocating up to 64G per NUMA >> zone. Things will work as long as there are enough 2M hugepages to >> cover the memory needs of the DPDK applications AND no memory zone >> needs more than 64G. >> >> With the default configuration and 1G hugepages >> >> 1. Total amount of memory for each NUMA zone is limited to >> 128G (CONFIG_RTE_MAX_MEM_MB_PER_TYPE). This constraint is hit for >> each numa zone. This is the limiting factor for memory per numa. >> >> 2. Total number of segment lists (128) does not exceed >> 32768 (CONFIG_RTE_MAX_MEMSEG_PER_TYPE). There are 128 segments per NUMA. >> >> 3. The data structures are capable of supporting 128G of >> memory for each numa zone (128 segments * 1G hugepagesize). >> However, only the first four NUMA zones get initialized before we hit >> CONFIG_RTE_MAX_MEM_MB (512G). >> >> 4. The total for all numa zones is limited to 512G >> (CONFIG_RTE_MAX_MEM_MB). This limit is hit after configuring the >> first four NUMA zones (4 x 128G = 512G). The rest of the NUMA zones >> cannot allocate memory. >> >> Apparently, it is intended to support max 8 NUMAs by default >> (CONFIG_RTE_MAX_NUMA_NODES=8), but when 1G hugepages are use, it can >> only support up to 4 NUMAs. >> >> Possible workarounds when using 1G hugepages: >> >> 1. Decrease CONFIG_RTE_MAX_MEM_MB_PER_TYPE to 65536 (limit >> of 64G per NUMA zone). This is probably the best option unless you >> need a lot of memory in any given NUMA. >> >> 2. Or, increase CONFIG_RTE_MAX_MEM_MB to 1048576. > >Hi Ravi, > >OK this makes it much clearer, thanks! > >I think the first one should be done. I think 64G per NUMA node is still a >reasonable amount of memory and it makes the default work (i think we can go >as far as reducing this limit to 32G per type!), and whoever has issues with >it can change CONFIG_RTE_MAX_MEM_MB_PER_TYPE or CONFIG_RTE_MAX_MEM_MB for >their use case. That's what these options are there for :) > >-- >Thanks, >Anatoly >
Hi Anatoly, Thanks a lot. Will the next release include this change? Regards, Ravi
