> On 01.07.2021., at 16:12, Matthew Smith <mgsm...@netgate.com> wrote:
>
>
>
> On Thu, Jul 1, 2021 at 6:36 AM Damjan Marion <dmar...@me.com> wrote:
>
>
> > On 01.07.2021., at 11:12, Benoit Ganne (bganne) via lists.fd.io
> > <bganne=cisco....@lists.fd.io> wrote:
> >
> >> Yes, allowing dynamic heap growth sounds like it could be better.
> >> Alternatively... if memory allocations could fail and something more
> >> graceful than VPP exiting could occur, that may also be better. E.g. if
> >> I'm adding a route and try to allocate a counter for it and that fails, it
> >> would be better to refuse to add the route than to exit and take the
> >> network down.
> >>
> >> I realize that neither of those options is easy to do btw. I'm just trying
> >> to figure out how to make it easier and more forgiving for users to set up
> >> their configuration without making them learn about various memory
> >> parameters.
> >
> > Understood, but setting a very high default will just make users of smaller
> > config puzzled too 😊 and I think changing all memory allocation callsites
> > to check for NULL would be a big paradigm change in VPP.
> > That's why I think a dynamically growing heap might be better but I do not
> > really know what would be the complexity.
> > That said, you can probably change the default in your own build and that
> > should work.
> >
>
> Fully agree wirth Benoit. We should not increase heap size default value.
>
> Things are actually a bit more complicated. For performance reasons people
> should use
> hugepages whenever they are available, but they are also not default.
> When hugepages are used all pages are immediately backed with physical memory.
>
> So different use cases require different heap configurations and end user
> needs to tune that.
> Same applies for other things like stats segment page size which again may
> impact forwarding
> performance significantly.
>
> If messing with startup.conf is too complicated for end user, some nice
> configuration script may be helpful.
> Or just throwing few startup.confs into extras/startup_configs.
>
> Dynamic heap is possible, but not straight forward, as at some places we use
> offsets
> to the start of the heap, so additional allocation cannot be anywhere.
> Also it will not help in some cases, i.e. when 1G hugepage is used for heap,
> growing up to 2G
> will fail if 2nd 1G page is not pre-allocated.
>
>
> Sorry for not being clear. I was not advocating any change to defaults in VPP
> code in gerrit. I was trying to figure out the impact of changing the default
> value written in startup.conf by the management plane I work on. And also
> have a conversation on whether there are ways that it could be made easier to
> tune memory parameters correctly.
ok, so let me try to answer your original questions:
> It's my understanding that when you set the size of the main heap or the stat
> segment in startup.conf, the size you specify is used to set up virtual
> address space and the system does not actually allocate that full amount of
> memory to VPP. I think when VPP tries to read/write addresses within the
> address space, then memory is requested from the system to back the chunk of
> address space containing the address being accessed. Is my understanding
> correct(ish)?
heap-size parameter defines size of memory mapping created for the heap. With
the normal 4K pages mapping is not backed by physical memory. Instead, first
time you try to access specific page CPU will generate page fault, and kernel
will handle it by allocating 4k chunk of physical memory to back that specific
virtual address and setup MMU mapping for that page.
In VPP we don’t have reverse process, even if all memory allocations which use
specific 4k page are freed, that 4K page will not be returned to kernel, as
kernel simply doesn’t know that specific page is not in use anymore.
Solution would be to somehow track number of memory allocations sharing single
4K page and call madvise() system call when last one is freed...
If you are using hugepages, all virtual memory is immediately backed by
physical memory so VPP with 32G of hugepage heap will use 32G of physical
memory as long as VPP is running.
If you do `show memory main-heap` you will actually see how many physical pages
are allocated:
vpp# show memory main-heap
Thread 0 vpp_main
base 0x7f6f95c9f000, size 1g, locked, unmap-on-destroy, name 'main heap'
page stats: page-size 4K, total 262144, mapped 50702, not-mapped 211442
numa 1: 50702 pages, 198.05m bytes
total: 1023.99M, used: 115.51M, free: 908.49M, trimmable: 905.75M
Out of this you can see that heap is using 4K pages, 262144 total, and 50702
are mapped to physical memory.
All 50702 pages are using memory on numa node 1.
So effectively VPP is using around 198 MB of physical memory for heap while
real heap usage is only 115 MB.
Such a big difference is mainly caused by one place in our code which temporary
allocates ~200M of memory for
temporary vector. That vector is freed but unused pages are never given back to
kernel.
>
> When I add a large number of routes to the FIB (>1M), I have seen VPP crash
> when the main heap or stats segment run out of space. I am wondering if it
> makes sense to just set the heap sizes to some huge value that I am confident
> will not be exceeded. If memory is not allocated unless it's needed, it seems
> like that would be ok to do.
For such a big FIB size, you likely want to use hugepages to avoid TLB misses
which can significantly hurt performance. If you decide to go that way, then
you will be able to set heap size only to value which is smaller than amount of
free physical memory in the system and that memory will not be available for
anything else...
HTH,
—
Damjan
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19687): https://lists.fd.io/g/vpp-dev/message/19687
Mute This Topic: https://lists.fd.io/mt/83856384/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
