On Fri, Apr 14, 2017 at 01:27:53PM +0200, Damjan Marion wrote: > Eric, > > VPP is not allocating hugepages directly. It just passes arguments to DPDK > EAL which does typical DPDK hugepage cherrypicking game. > By default VPP requests 256M on each socket. > > Issue you are reporting is very old and directly tied to DPDK behavior. > > Have you tried to increase vm.max_map_count and kernel.shmmax? > If not tale a look into src/vpp/conf/80-vpp.conf.
Damjan, Thanks for the quick reply. max_map_count and kernel.shmmax did need updating, but I'm still seeing same behavior. What I ended up doing is making edits directly to /etc/sysctl.d/80-vpp.conf in order to increase the huge pages as follows: vm.nr_hugepages=2048 vm.max_map_count=4096 kernel.shmmax=4294967296 After a reboot, I'm still seeing the same behavior, though can see that 2048 hugepages are free. Are there other areas/setting that DPDK will check? Through /etc/vpp/startup.conf, I have DPDK grabbing 1M for each socket: socket-mem 1024,1024 Thanks, Eric > > That typically helps… > > Thanks, > > Damjan > > > > On 13 Apr 2017, at 22:53, Ernst, Eric <eric.er...@intel.com> wrote: > > > > I’d like to better understand how hugepages are allocated at boot when vpp > > is not started, as well as what happens what vpp is started (ie, systemctl > > start vpp). > > > > The reason I ask is that I’m running into an issue with hugepage allocation > > changes causing VPP to fail. Whereas 1024 2MB pages is the default, I find > > that as I run more vhost-user VMs, I need to back them with hugepages, and > > need more pages. When doing this and then starting/restarting VPP, I am > > seeing failures. > > > > Any tips? Has anyone else seen this? > > > > Thanks, > > Eric > > > > > > In more detail, > > > > Scenario #1: > > 1. Assuming VPP doesn’t start by default (I had run systemctl disable > > vpp on a prior boot) > > 2. Boot system under test > > 3. Verify via /proc/meminfo that 1024 huge pages have been allocated > > and 1024 are free > > 4. Start VPP (systemctl start vpp), see that free pool goes down to > > 768, as expected and VPP runs without issue > > > > Scenario #2: > > 1. Assuming VPP doesn’t start by default (I had run systemctl > > disable vpp on a prior boot) > > 2. Boot system under test > > 3. Adjust number of hugepages from 1024 to 8192 (sysctl -w > > vm.nr_hugepages=8192) > > 4. Verify via /proc/meminfo that 8192 huge pages have been allocated > > and are free > > 5. Start VPP (systemctl start vpp), see that VPP fails to start with > > log shown below at [1], but in summary an EAL failure to allocate memory > > > > Scenario #3: > > 1. Same as scenario #1 to start. After VPP is up and running,…. > > 2. Adjust number of huge pages from 1024 -> 8192, noting that the > > number of pages moves to 8192, of which some are still being used by VPP > > 3. You can add more vhost-user interfaces without issue, and VPP and > > VMs are functional. > > 4. Restart VPP (systemctl restart vpp) > > 5. Note that now VPP fails to start, though there are still _many_ > > pages free. > > > > > > > > [1] Snippet of Error log: > > Apr 13 13:43:41 eernstworkstation systemd[1]: Starting vector packet > > processing engine... > > Apr 13 13:43:41 eernstworkstation systemd[1]: Started vector packet > > processing engine. > > Apr 13 13:43:41 eernstworkstation vpp[5024]: vlib_plugin_early_init:213: > > plugin path /usr/lib/vpp_plugins > > Apr 13 13:43:41 eernstworkstation vpp[5024]: /usr/bin/vpp[5024]: > > vlib_pci_bind_to_uio: Skipping PCI device 0000:05:00.0 as host interface > > eth0 is up > > Apr 13 13:43:41 eernstworkstation /usr/bin/vpp[5024]: vlib_pci_bind_to_uio: > > Skipping PCI device 0000:05:00.0 as host interface eth0 is up > > Apr 13 13:43:41 eernstworkstation vpp[5024]: /usr/bin/vpp[5024]: > > vlib_pci_bind_to_uio: Skipping PCI device 0000:05:00.1 as host interface > > eth1 is up > > Apr 13 13:43:41 eernstworkstation /usr/bin/vpp[5024]: vlib_pci_bind_to_uio: > > Skipping PCI device 0000:05:00.1 as host interface eth1 is up > > Apr 13 13:43:41 eernstworkstation vpp[5024]: EAL: Detected 32 lcore(s) > > Apr 13 13:43:41 eernstworkstation vpp[5024]: EAL: No free hugepages > > reported in hugepages-1048576kB > > Apr 13 13:43:41 eernstworkstation vpp[5024]: EAL: Probing VFIO support... > > Apr 13 13:43:41 eernstworkstation vnet[5024]: EAL: Probing VFIO support... > > Apr 13 13:43:41 eernstworkstation sudo[5038]: eernst : TTY=pts/1 ; > > PWD=/home/eernst ; USER=root ; COMMAND=/bin/journalctl > > Apr 13 13:43:41 eernstworkstation sudo[5038]: pam_unix(sudo:session): > > session opened for user root by eernst(uid=0) > > Apr 13 13:43:42 eernstworkstation vpp[5024]: EAL: Cannot get a virtual > > area: Cannot allocate memory > > Apr 13 13:43:42 eernstworkstation vpp[5024]: EAL: Failed to remap 2 MB pages > > Apr 13 13:43:42 eernstworkstation vpp[5024]: PANIC in rte_eal_init(): > > Apr 13 13:43:42 eernstworkstation vpp[5024]: Cannot init memory > > Apr 13 13:43:42 eernstworkstation vnet[5024]: EAL: Cannot get a virtual > > area: Cannot allocate memory > > Apr 13 13:43:42 eernstworkstation vnet[5024]: EAL: Failed to remap 2 MB > > pages > > Apr 13 13:43:42 eernstworkstation vnet[5024]: PANIC in rte_eal_init(): > > Apr 13 13:43:42 eernstworkstation vnet[5024]: Cannot init memory > > Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Main process > > exited, code=dumped, status=6/ABRT > > Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Unit entered > > failed state. > > Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Failed with > > result 'core-dump'. > > Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Service hold-off > > time over, scheduling restart. > > Apr 13 13:43:43 eernstworkstation systemd[1]: Stopped vector packet > > processing engine. > > Apr 13 13:43:43 eernstworkstation systemd[1]: vpp.service: Start request > > repeated too quickly. > > Apr 13 13:43:43 eernstworkstation systemd[1]: Failed to start vector packet > > processing engine. > > > > _______________________________________________ > > vpp-dev mailing list > > vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > > https://lists.fd.io/mailman/listinfo/vpp-dev > > <https://lists.fd.io/mailman/listinfo/vpp-dev> _______________________________________________ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
