Dear Elias,
Let me try to explain a bit underlying mechanics.
Let’s assume your target number of sessions is 10M and we are talking about
16byte key size.
That means each hash entry (KV) is 24 bytes (16 bytes key and 8 bytes value).
In the setup you were mentioning, with 1<<20 buckets, your will need to fit 10
KVs into each bucket.
Initial bihash bucket holds 4 KVs and to accomodate 10 keys (assuming that hash
function gives us equal distribution)
you will need to grow each bucket 2 times. Growing means doubling bucket size.
So at the end you will have 1<<20 buckets where each holds 16 KVs.
Math is:
1<<20 * (16 * 24 /* KV size in bytes */ + 8 /*bucket header size*/)
Which means 392 MB of memory.
If you keep target number of 10M sessions, but you increase number of buckets
to 1 << 22
(which is roughly what formula bellow is trying to do) you end up with the
following math:
Math is:
1<<22 * (4 * 24 /* KV size in bytes */ + 8 /*bucket header size*/)
Which means 416 MB of memory.
So why 2nd one is better. Several reasons:
- in first case you need to grow each bucket twice, that means allocating
memory for the new bucket,
copying existing data from the old bucket and putting old bucket to the free
list. This operation increases
key insertion time and lowers performance
- growing will likely result in significant amount of old buckets sitting in
the free list
and they are effectively wasted memory (bihash tries to reuse that memory but
at some point
there is no demand anymore for smaller buckets)
- performance-wise original bucket (one which first 4 KVs) is collocated with
bucket header.
This is new behaviour Dave introduced earlier this year (and I think it is
present in 20.09).
Bucket collocated with header means that there is no dependant prefetch
needed as both header
and at least part of data sits in the same cacheline. This significantly
improveslookup performance.
So in general, for best performance and optimal memory usage, number of buckets
should be big enough so it
unlikely grow with your target number of KVs. rule of thumb would be rounding
target number of
entries to closer power-of-2 value and then dividing that number with 2.
For example, for 10M entries first lower power-of-2 number is 1<<23 (8M) and
first higher is 1<<24 (16M).
1<<23 is closer, when we divide that by 2 we got 1<<22 (4M) buckets.
Hope this explains….
—
Damjan
> On 26.11.2020., at 17:54, Elias Rudberg <elias.rudb...@bahnhof.net> wrote:
>
> Hello VPP experts,
>
> We are using VPP for NAT44 and are currently looking at how to move
> from VPP 20.05 to 20.09. There are some differences in the way the NAT
> plugin is configured.
>
> One difficulty for us is the maximum number of sessions allowed, we
> need to handle large numbers of sessions so that limit can be
> important for us. For VPP 20.05 we have used "translation hash buckets
> 1048576" and then the maximum number of sessions per thread becomes 10
> times that because of this line in the source code in snat_config():
>
> sm->max_translations = 10 * translation_buckets;
>
> So then we got a limit of about 10 million sessions per thread, which
> we have been happy with so far.
>
> With VPP 20.09 however, things have changed so that the maximum number
> of sessions is now configured explicitly, and the relationship between
> max_translations_per_thread and translation_buckets is no longer a
> factor of 10 but instead given by the nat_calc_bihash_buckets()
> function:
>
> static u32
> nat_calc_bihash_buckets (u32 n_elts)
> {
> return 1 << (max_log2 (n_elts >> 1) + 1);
> }
>
> The above function corresponds to a factor of somewhere between 1 and
> 2 instead of 10. So, if I understood this correctly, for a given
> maximum number of sessions, the corresponding translation_buckets
> value will be something like 5 to 10 times larger in VPP 20.09
> compared to how it was in VPP 20.05, leading to significantly
> increased memory requirement given that we want to have the same
> maximum number of sessions as before.
>
> It seems a little strange that the translation_buckets value would
> change so much between VPP versions, was that change intentional? The
> old relationship "max_translations = 10 * translation_buckets" seems
> to have worked well in practice, at least for our use case.
>
> What could we do to get around this, if we want to switch to VPP 20.09
> but without reducing the maximum number of sessions? If we were to
> simply divide the nat_calc_bihash_buckets() value by 8 or so to make
> it more similar to how it was earlier, would that lead to other
> problems?
>
> Best regards,
> Elias
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18162): https://lists.fd.io/g/vpp-dev/message/18162
Mute This Topic: https://lists.fd.io/mt/78533277/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
