On Mon, Aug 28, 2023 at 01:46:32PM +0200, Gabor LENCSE wrote: > Hi Lyndon, > > Sorry for my late reply. Please see my answers inline. > > On 8/24/2023 11:13 PM, Lyndon Nerenberg (VE7TFX/VE6BBM) wrote: > > Gabor LENCSE writes: > > > > > If you are interested, you can find the results in Tables 18 - 20 of > > > this (open access) paper: https://doi.org/10.1016/j.comcom.2023.08.009 > > Thanks for the pointer -- that's a very interesting paper. > > > > After giving it a quick read through, one thing immediately jumps > > out. The paper mentions (section A.4) a boost in performance after > > increasing the state table size limit. Not having looked at the > > relevant code, so I'm guessing here, but this is a classic indicator > > of a hashing algorithm falling apart when the table gets close to > > full. Could it be that simple? I need to go digging into the pf > > code for a closer look. > > Beware, I wrote it about iptables and not PF! > > As for iptables, it is really so simple. I have done a deeper analysis of > iptables performance as the function of its hash table size. It is > documented in another (open access) paper: > http://doi.org/10.36244/ICJ.2023.1.6 > > However, I am not familiar with the internals of the other two tested > stateful NAT64 implementations, Jool and OpenBSD PF. I have no idea, what > kind of data structures they use for storing the connections.
openbsd uses a red-black tree to look up states. packets are parsed into a key that looks up states by address family, ips, ipproto, ports, etc, to find the relevant state. if a state isnt found, it falls through to ruleset evaluation, which is notionally a linked list, but has been optimised. > > You also describe how the performance degrades over time. This > > exactly matches the behaviour we see. Could the fix be as simple > > as cranking 'set limit states' up to, say, two milltion? There is > > one way to find out ... :-) > > As you could see, the highest number of connections was 40M, and the limit > of the states was set to 1000M. It worked well for me then with the PF of > OpenBSD 7.1. > > It would be interesting to find the root cause of the phenomenon, why the > performance of PF seems to deteriorate with time. E.g., somehow the internal > data structures of PF become "polluted" if many connections are established > and then deleted? my first guess is that you're starting to fight agains the pf state purge processing. pf tries to scan the entire state table every 10 seconds (by default) looking for expired states it can remove. this scan process runs every second, but it tries to cover the whole state table by 10 seconds. the more states you have the more time this takes, and this increases linearly with the number of states you have. until relatively recently (post 7.2), the scan and gc processing effectively stopped the world. at work we run with about 2 million states during business hours, and i was seeing the gc processing take up approx 70ms a second, during which packet processing didnt really happen. now the scan can happen without blocking pf packet processing. it still takes cpu time, so there is a point that processing packets and scanning for states will fight each other for time, but at least they're not fighting each other for locks now. > However, I have deleted the content of the state table after each elementary > measurement step using the "pfctl -F states" command. (I am sorry, this > command is missing from the paper, but it is there in my saved "del-pf" > file!) > > Perhaps PF developers could advise us, if the deletion of the states > generate a fresh state table or not. it marks the states as expired, and then the purge scan is able to take them and actually free them. > Could anyone help us in this question? > > Best regards, > > G??bor > > > > > I use binary search to find the highest lossless rate (throughput). > Especially w > > > > > > --lyndon >
