On Fri, Apr 14, 2017 at 09:18:33AM -0700, Ernst, Eric wrote: > On Fri, Apr 14, 2017 at 01:27:53PM +0200, Damjan Marion wrote: > > Eric, > > > > VPP is not allocating hugepages directly. It just passes arguments to DPDK > > EAL which does typical DPDK hugepage cherrypicking game. > > By default VPP requests 256M on each socket. > > > > Issue you are reporting is very old and directly tied to DPDK behavior. > > > > Have you tried to increase vm.max_map_count and kernel.shmmax? > > If not tale a look into src/vpp/conf/80-vpp.conf. > > Damjan, > > Thanks for the quick reply. max_map_count and kernel.shmmax did need > updating, but I'm still seeing same behavior. What I ended up doing is making > edits directly to /etc/sysctl.d/80-vpp.conf in order to increase the huge > pages > as follows: > vm.nr_hugepages=2048 > vm.max_map_count=4096 > kernel.shmmax=4294967296 > > After a reboot, I'm still seeing the same behavior, though can see that 2048 > hugepages > are free. Are there other areas/setting that DPDK will check? > > Through /etc/vpp/startup.conf, I have DPDK grabbing 1M for each socket: > socket-mem 1024,1024
Correction; 1G per socket. > > Thanks, > Eric > > > > > > > That typically helps… > > > > Thanks, > > > > Damjan > > > > > > > On 13 Apr 2017, at 22:53, Ernst, Eric <eric.er...@intel.com> wrote: > > > > > > I’d like to better understand how hugepages are allocated at boot when > > > vpp is not started, as well as what happens what vpp is started (ie, > > > systemctl start vpp). > > > > > > The reason I ask is that I’m running into an issue with hugepage > > > allocation changes causing VPP to fail. Whereas 1024 2MB pages is the > > > default, I find that as I run more vhost-user VMs, I need to back them > > > with hugepages, and need more pages. When doing this and then > > > starting/restarting VPP, I am seeing failures. > > > > > > Any tips? Has anyone else seen this? > > > > > > Thanks, > > > Eric > > > > > > > > > In more detail, > > > > > > Scenario #1: > > > 1. Assuming VPP doesn’t start by default (I had run systemctl > > > disable vpp on a prior boot) > > > 2. Boot system under test > > > 3. Verify via /proc/meminfo that 1024 huge pages have been > > > allocated and 1024 are free > > > 4. Start VPP (systemctl start vpp), see that free pool goes down to > > > 768, as expected and VPP runs without issue > > > > > > Scenario #2: > > > 1. Assuming VPP doesn’t start by default (I had run systemctl > > > disable vpp on a prior boot) > > > 2. Boot system under test > > > 3. Adjust number of hugepages from 1024 to 8192 (sysctl -w > > > vm.nr_hugepages=8192) > > > 4. Verify via /proc/meminfo that 8192 huge pages have been > > > allocated and are free > > > 5. Start VPP (systemctl start vpp), see that VPP fails to start > > > with log shown below at [1], but in summary an EAL failure to allocate > > > memory > > > > > > Scenario #3: > > > 1. Same as scenario #1 to start. After VPP is up and running,…. > > > 2. Adjust number of huge pages from 1024 -> 8192, noting that the > > > number of pages moves to 8192, of which some are still being used by VPP > > > 3. You can add more vhost-user interfaces without issue, and VPP > > > and VMs are functional. > > > 4. Restart VPP (systemctl restart vpp) > > > 5. Note that now VPP fails to start, though there are still _many_ > > > pages free. > > > > > > > > > > > > [1] Snippet of Error log: > > > Apr 13 13:43:41 eernstworkstation systemd[1]: Starting vector packet > > > processing engine... > > > Apr 13 13:43:41 eernstworkstation systemd[1]: Started vector packet > > > processing engine. > > > Apr 13 13:43:41 eernstworkstation vpp[5024]: vlib_plugin_early_init:213: > > > plugin path /usr/lib/vpp_plugins > > > Apr 13 13:43:41 eernstworkstation vpp[5024]: /usr/bin/vpp[5024]: > > > vlib_pci_bind_to_uio: Skipping PCI device 0000:05:00.0 as host interface > > > eth0 is up > > > Apr 13 13:43:41 eernstworkstation /usr/bin/vpp[5024]: > > > vlib_pci_bind_to_uio: Skipping PCI device 0000:05:00.0 as host interface > > > eth0 is up > > > Apr 13 13:43:41 eernstworkstation vpp[5024]: /usr/bin/vpp[5024]: > > > vlib_pci_bind_to_uio: Skipping PCI device 0000:05:00.1 as host interface > > > eth1 is up > > > Apr 13 13:43:41 eernstworkstation /usr/bin/vpp[5024]: > > > vlib_pci_bind_to_uio: Skipping PCI device 0000:05:00.1 as host interface > > > eth1 is up > > > Apr 13 13:43:41 eernstworkstation vpp[5024]: EAL: Detected 32 lcore(s) > > > Apr 13 13:43:41 eernstworkstation vpp[5024]: EAL: No free hugepages > > > reported in hugepages-1048576kB > > > Apr 13 13:43:41 eernstworkstation vpp[5024]: EAL: Probing VFIO support... > > > Apr 13 13:43:41 eernstworkstation vnet[5024]: EAL: Probing VFIO support... > > > Apr 13 13:43:41 eernstworkstation sudo[5038]: eernst : TTY=pts/1 ; > > > PWD=/home/eernst ; USER=root ; COMMAND=/bin/journalctl > > > Apr 13 13:43:41 eernstworkstation sudo[5038]: pam_unix(sudo:session): > > > session opened for user root by eernst(uid=0) > > > Apr 13 13:43:42 eernstworkstation vpp[5024]: EAL: Cannot get a virtual > > > area: Cannot allocate memory > > > Apr 13 13:43:42 eernstworkstation vpp[5024]: EAL: Failed to remap 2 MB > > > pages > > > Apr 13 13:43:42 eernstworkstation vpp[5024]: PANIC in rte_eal_init(): > > > Apr 13 13:43:42 eernstworkstation vpp[5024]: Cannot init memory > > > Apr 13 13:43:42 eernstworkstation vnet[5024]: EAL: Cannot get a virtual > > > area: Cannot allocate memory > > > Apr 13 13:43:42 eernstworkstation vnet[5024]: EAL: Failed to remap 2 MB > > > pages > > > Apr 13 13:43:42 eernstworkstation vnet[5024]: PANIC in rte_eal_init(): > > > Apr 13 13:43:42 eernstworkstation vnet[5024]: Cannot init memory > > > Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Main process > > > exited, code=dumped, status=6/ABRT > > > Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Unit entered > > > failed state. > > > Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Failed with > > > result 'core-dump'. > > > Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Service > > > hold-off time over, scheduling restart. > > > Apr 13 13:43:43 eernstworkstation systemd[1]: Stopped vector packet > > > processing engine. > > > Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Start request > > > repeated too quickly. > > > Apr 13 13:43:43 eernstworkstation systemd[1]: Failed to start vector > > > packet processing engine. > > > > > > _______________________________________________ > > > vpp-dev mailing list > > > vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > > > https://lists.fd.io/mailman/listinfo/vpp-dev > > > <https://lists.fd.io/mailman/listinfo/vpp-dev> > _______________________________________________ > vpp-dev mailing list > vpp-dev@lists.fd.io > https://lists.fd.io/mailman/listinfo/vpp-dev _______________________________________________ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
