27.09.2024 10:54, kasak пишет:
23.09.2024 17:48, Stuart Henderson пишет:
On 2024-09-23, kasak <ka...@kasakoff.net> wrote:
23.09.2024 15:22, Brian Conway пишет:
On Mon, Sep 23, 2024, at 6:19 AM, kasak wrote:
Hello, misc!
Could you please share your wisdom about this problem.
On my openbsd firewall, sometimes network become slow and some
daemons
stop working.
/var/log/messages have this messages when slowdown is in place:
Sep 23 13:49:34 gater ntpd[30891]: sendto: Permission denied
Sep 23 13:56:22 gater isakmpd[64631]: sendmsg (14, 0x784ce63ce408,
0):
Permission denied
also nginx have this messages:
connect() to 172.16.0.80:443 failed (13: Permission denied) while
connecting to upstream
also i cannot ping nor nslookup anything also because "permission
denied"
I found workaround by flushing pf states. After pfctl -F states
everything start to work again.
But maybe i should tune something i did not know about?
How can I diagnose this failures?
You may have a full state table. Try:
pfctl -si
pfctl -ss
Do I understand correctly that "current entries" (pfctl -si) is the
number of states?
Yes but just show all of pfctl -si as that may give other clues too.
pfctl -sm and pfctl -st may also be useful.
Alternatively `pfctl -sa` includes all. If you have run out of
available state tracking, I would spot check what is using up all
the state entries and whether it is expected prior to increasing
the limit.
(pfctl -sa will be pretty huge if you have a full state table and you
probably don't want to send that to the list)
Unfortunately, or maybe this is fortunately? I have not experienced
any slowdowns this week.
Nothing surprising here, I have cached this condition only twice for
10+ years of using OpenBSD as a gateway.
But this "twice" happened last week.
But I think the guess was correct. I have been monitoring "current
entries" through all peak hours, and the maximum value was near to 70
000.
Here is output of pfctl, but this is output of normal condition, when
everything is working. Just in case,if I catch any slowdown in future,
so we can compare.
doas pfctl -si
Status: Enabled for 9 days 20:02:33 Debug: err
State Table Total Rate
current entries 61986
half-open tcp 318
searches 13376986308 15742.2/s
inserts 79225635 93.2/s
removals 79163649 93.2/s
Counters
match 97348919 114.6/s
bad-offset 0 0.0/s
fragment 333 0.0/s
short 26083 0.0/s
normalize 223 0.0/s
memory 749907 0.9/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 148233 0.2/s
proto-cksum 16 0.0/s
state-mismatch 642701 0.8/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
translate 0 0.0/s
no-route
kasak@OpenBSD-gater:~$ doas pfctl -sm
states hard limit 100000
src-nodes hard limit 10000
frags hard limit 65536
tables hard limit 1000
table-entries hard limit 200000
pktdelay-pkts hard limit 10000
anchors hard limit 512
doas pfctl -st
tcp.first 120s
tcp.opening 30s
tcp.established 86400s
tcp.closing 900s
tcp.finwait 45s
tcp.closed 90s
tcp.tsdiff 30s
udp.first 60s
udp.single 30s
udp.multiple 60s
icmp.first 20s
icmp.error 10s
other.first 60s
other.single 30s
other.multiple 60s
frag 60s
interval 10s
adaptive.start 60000 states
adaptive.end 120000 states
src.track 0s
a-ha! I caught it again!
And this is indeed low amount of states:
kasak@OpenBSD-gater:~$ doas pfctl -si
Status: Enabled for 30 days 20:20:36 Debug: err
State Table Total Rate
current entries 100000
half-open tcp 20
searches 45458424195 17056.1/s
inserts 242592733 91.0/s
removals 242492733 91.0/s
Counters
match 286966515 107.7/s
bad-offset 0 0.0/s
fragment 81 0.0/s
short 15966 0.0/s
normalize 361 0.0/s
memory 533746 0.2/s
bad-timestamp 0 0.0/s
congestion 12 0.0/s
ip-option 452192 0.2/s
proto-cksum 57 0.0/s
state-mismatch 1744665 0.7/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
translate 0 0.0/s
no-route 0 0.0/s
So I raised limits to 150K and it helped
