Greetings,
Our obsd border router has worked for years with our PF ruleset, but sometime
in the middle of January, we discovered that our webpages were stalling when
viewed 'externally' (from remote Internet clients) but not internally; the
webserver is a box on the 10.0.0.0/24 internal LAN that is accessed with a
'pf' rdr rule. What's more, this only happens if our 'redirected' webserver
is the Solaris 2.6 box, but if we redirect http traffic to an SVR4 box, there
is no problem. Capturing traffic on the internal LAN and on the 'external'
interface of the obsd border router shows 'pf' dropping outgoing traffic
from the webserver after a few data blocks have been sent, and a resulting stall
which never recovers.
I have tried all of the suggestions from archived mailing list posts without
success, including proper 'keep state' and 'flags S/SA' filter rules, adjusting
MTUs and MSSs (one poster reported that his combination of obsd and Speedstream
5861 ADSL router required certain max-mss adjustments -- we use the same
combination), and scrub rule changes didn't help. FWIW, we had _no_ keep state
rules for years when everything worked, but of course state is implicit on
NAT rules. Is there anything to adjust in NAT rules for establishment of state?
Here are 'pf' debug messages and two tcpdump dumps, one for the 'external'
interface and one for the 'internal' interface, for a simple http GET of an
html document (using 'telnet 80' from a remote machine on the 'Net; note that
all dumps are _concurrent_, that is from the single TCP session):
'pf' debug output ===============
# pf: BAD state: TCP 10.0.0.202:80 216.251.177.106:80 66.93.16.53:53859 [lo=3901
572765 high=3901582901 win=5840 modulator=0] [lo=893395808 high=893395856 win=10
136 modulator=0] 4:4 PA seq=893395808 ack=3901572765 len=1448 ackskew=0 pkts=12
dir=out,rev
pf: State failure on: 1 |
pf: BAD state: TCP 10.0.0.202:80 216.251.177.106:80 66.93.16.53:53859 [lo=390157
2765 high=3901582901 win=5840 modulator=0] [lo=893395808 high=893395945 win=1013
6 modulator=0] 4:4 PA seq=893397256 ack=3901572765 len=1254 ackskew=0 pkts=15 di
r=out,rev
pf: State failure on: 1 |
Jan 26 13:13:26 nat1 last message repeated 3 times
Jan 26 13:15:12 nat1 /bsd: pf: BAD state: TCP 10.0.0.202:80 216.251.177.106:80 6
6.93.16.53:53859 [lo=3901572765 high=3901582901 win=5840 modulator=0] [lo=893395
808 high=893395856 win=10136 modulator=0] 4:4 PA seq=893395808 ack=3901572765 le
n=1448 ackskew=0 pkts=12 dir=out,rev
Jan 26 13:13:26 nat1 last message repeated 3 times
Jan 26 13:15:12 nat1 /bsd: pf: BAD state: TCP 10.0.0.202:80 216.251.177.106:80 6
6.93.16.53:53859 [lo=3901572765 high=3901582901 win=5840 modulator=0] [lo=893395
808 high=893395856 win=10136 modulator=0] 4:4 PA seq=893395808 ack=3901572765 le
n=1448 ackskew=0 pkts=12 dir=out,rev
Jan 26 13:15:13 nat1 /bsd: pf: State failure on: 1 |
Jan 26 13:15:13 nat1 /bsd: pf: State failure on: 1 |
Jan 26 13:15:13 nat1 /bsd: pf: BAD state: TCP 10.0.0.202:80 216.251.177.106:80 6
6.93.16.53:53859 [lo=3901572765 high=3901582901 win=5840 modulator=0] [lo=893395
808 high=893395945 win=10136 modulator=0] 4:4 PA seq=893397256 ack=3901572765 le
n=1254 ackskew=0 pkts=15 dir=out,rev
Jan 26 13:15:13 nat1 /bsd: pf: BAD state: TCP 10.0.0.202:80 216.251.177.106:80 6
6.93.16.53:53859 [lo=3901572765 high=3901582901 win=5840 modulator=0] [lo=893395
808 high=893395945 win=10136 modulator=0] 4:4 PA seq=893397256 ack=3901572765 le
n=1254 ackskew=0 pkts=15 dir=out,rev
Jan 26 13:15:13 nat1 /bsd: pf: State failure on: 1 |
Jan 26 13:15:13 nat1 /bsd: pf: State failure on: 1 |
pf: BAD state: TCP 10.0.0.202:80 216.251.177.106:80 66.93.16.53:53859 [lo=390157
2765 high=3901582901 win=5840 modulator=0] [lo=893395808 high=893395945 win=1013
6 modulator=0] 4:4 PA seq=893395808 ack=3901572765 len=1448 ackskew=0 pkts=16 di
r=out,rev
pf: State failure on: 1 |
Jan 26 13:15:16 nat1 /bsd: pf: BAD state: TCP 10.0.0.202:80 216.251.177.106:80 6
6.93.16.53:53859 [lo=3901572765 high=3901582901 win=5840 modulator=0] [lo=893395
808 high=893395945 win=10136 modulator=0] 4:4 PA seq=893395808 ack=3901572765 le
n=1448 ackskew=0 pkts=16 dir=out,rev
Jan 26 13:15:16 nat1 /bsd: pf: BAD state: TCP 10.0.0.202:80 216.251.177.106:80 6
6.93.16.53:53859 [lo=3901572765 high=3901582901 win=5840 modulator=0] [lo=893395
808 high=893395945 win=10136 modulator=0] 4:4 PA seq=893395808 ack=3901572765 le
n=1448 ackskew=0 pkts=16 dir=out,rev
Jan 26 13:15:16 nat1 /bsd: pf: State failure on: 1 |
Jan 26 13:15:16 nat1 /bsd: pf: State failure on: 1 |
pf: BAD state: TCP 10.0.0.202:80 216.251.177.106:80 66.93.16.53:53859 [lo=390157
2765 high=3901582901 win=5840 modulator=0] [lo=893395808 high=893395945 win=1013
6 modulator=0] 4:4 PA seq=893395808 ack=3901572765 len=1448 ackskew=0 pkts=17 di
r=out,rev
pf: State failure on: 1 |
Jan 26 13:15:24 nat1 /bsd: pf: BAD state: TCP 10.0.0.202:80 216.251.177.106:80 6
6.93.16.53:53859 [lo=3901572765 high=3901582901 win=5840 modulator=0] [lo=893395
808 high=893395945 win=10136 modulator=0] 4:4 PA seq=893395808 ack=3901572765 le
n=1448 ackskew=0 pkts=17 dir=out,rev
Jan 26 13:15:24 nat1 /bsd: pf: BAD state: TCP 10.0.0.202:80 216.251.177.106:80 6
6.93.16.53:53859 [lo=3901572765 high=3901582901 win=5840 modulator=0] [lo=893395
808 high=893395945 win=10136 modulator=0] 4:4 PA seq=893395808 ack=3901572765 le
n=1448 ackskew=0 pkts=17 dir=out,rev
Jan 26 13:15:25 nat1 /bsd: pf: State failure on: 1 |
Jan 26 13:15:25 nat1 /bsd: pf: State failure on: 1 |
============== tcpdump of 'internal' interface =============
# tcpdump -vvv -i le0 -lnS host ipx1 and tcp port 80
tcpdump: listening on le0, link-type EN10MB (Ethernet), capture size 96 bytes
13:15:01.529128 IP (tos 0x0, ttl 54, id 56671, offset 0, flags [none], proto
TCP (6), length 60)
66.93.16.53.53859 > 10.0.0.202.80: Flags [S], cksum 0x1e50 (correct), seq 3901572729, win 5840, options [mss 1460,sackOK,TS val
413375270 ecr 0,nop,wscale 7], length 0
13:15:01.533139 IP (tos 0x0, ttl 254, id 13279, offset 0, flags [DF], proto TCP
(6), length 60)
10.0.0.202.80 > 66.93.16.53.53859: Flags [S.], cksum 0x9543 (correct), seq 893390015, ack 3901572730, win 10136, options
[nop,nop,TS val 14235747 ecr 413375270,nop,wscale 0,mss 1460], length 0
13:15:01.642838 IP (tos 0x0, ttl 54, id 56672, offset 0, flags [none], proto
TCP (6), length 52)
66.93.16.53.53859 > 10.0.0.202.80: Flags [.], cksum 0xe854 (correct), seq 3901572730, ack 893390016, win 46, options
[nop,nop,TS val 413375299 ecr 14235747], length 0
13:15:11.635814 IP (tos 0x0, ttl 54, id 56673, offset 0, flags [none], proto
TCP (6), length 87)
66.93.16.53.53859 > 10.0.0.202.80: Flags [P.], seq 3901572730:3901572765, ack 893390016, win 46, options [nop,nop,TS val
413377796 ecr 14235747], length 35
13:15:11.639022 IP (tos 0x0, ttl 254, id 13280, offset 0, flags [DF], proto TCP
(6), length 52)
10.0.0.202.80 > 66.93.16.53.53859: Flags [.], cksum 0xb314 (correct), seq 893390016, ack 3901572765, win 10136, options
[nop,nop,TS val 14236757 ecr 413377796], length 0
13:15:11.709022 IP (tos 0x0, ttl 254, id 13281, offset 0, flags [DF], proto TCP
(6), length 1500)
10.0.0.202.80 > 66.93.16.53.53859: Flags [P.], seq 893390016:893391464, ack 3901572765, win 10136, options [nop,nop,TS val
14236764 ecr 413377796], length 1448
13:15:11.878955 IP (tos 0x0, ttl 54, id 56674, offset 0, flags [none], proto
TCP (6), length 52)
66.93.16.53.53859 > 10.0.0.202.80: Flags [.], cksum 0xd47a (correct), seq 3901572765, ack 893391464, win 69, options
[nop,nop,TS val 413377858 ecr 14236764], length 0
13:15:11.888919 IP (tos 0x0, ttl 254, id 13282, offset 0, flags [DF], proto TCP
(6), length 1500)
10.0.0.202.80 > 66.93.16.53.53859: Flags [.], seq 893391464:893392912, ack 3901572765, win 10136, options [nop,nop,TS val
14236782 ecr 413377858], length 1448
13:15:11.891323 IP (tos 0x0, ttl 254, id 13283, offset 0, flags [DF], proto TCP
(6), length 1500)
10.0.0.202.80 > 66.93.16.53.53859: Flags [P.], seq 893392912:893394360, ack 3901572765, win 10136, options [nop,nop,TS val
14236782 ecr 413377858], length 1448
13:15:12.064821 IP (tos 0x0, ttl 54, id 56675, offset 0, flags [none], proto
TCP (6), length 52)
66.93.16.53.53859 > 10.0.0.202.80: Flags [.], cksum 0xce7b (correct), seq 3901572765, ack 893392912, win 91, options
[nop,nop,TS val 413377905 ecr 14236782], length 0
13:15:12.074694 IP (tos 0x0, ttl 254, id 13284, offset 0, flags [DF], proto TCP
(6), length 1500)
10.0.0.202.80 > 66.93.16.53.53859: Flags [.], seq 893394360:893395808, ack 3901572765, win 10136, options [nop,nop,TS val
14236800 ecr 413377905], length 1448
13:15:12.076960 IP (tos 0x0, ttl 254, id 13285, offset 0, flags [DF], proto TCP
(6), length 1500)
10.0.0.202.80 > 66.93.16.53.53859: Flags [P.], seq 893395808:893397256, ack 3901572765, win 10136, options [nop,nop,TS val
14236800 ecr 413377905], length 1448
13:15:12.378488 IP (tos 0x0, ttl 54, id 56676, offset 0, flags [none], proto
TCP (6), length 52)
66.93.16.53.53859 > 10.0.0.202.80: Flags [.], cksum 0xc8b0 (correct), seq 3901572765, ack 893394360, win 114, options
[nop,nop,TS val 413377917 ecr 14236782], length 0
13:15:12.381233 IP (tos 0x0, ttl 54, id 56677, offset 0, flags [none], proto
TCP (6), length 52)
66.93.16.53.53859 > 10.0.0.202.80: Flags [.], cksum 0xc2be (correct), seq 3901572765, ack 893395808, win 137, options
[nop,nop,TS val 413377950 ecr 14236800], length 0
13:15:12.387658 IP (tos 0x0, ttl 254, id 13286, offset 0, flags [DF], proto TCP
(6), length 1306)
10.0.0.202.80 > 66.93.16.53.53859: Flags [P.], seq 893397256:893398510, ack 3901572765, win 10136, options [nop,nop,TS val
14236832 ecr 413377917], length 1254
13:15:16.245523 IP (tos 0x0, ttl 254, id 13287, offset 0, flags [DF], proto TCP
(6), length 1500)
10.0.0.202.80 > 66.93.16.53.53859: Flags [P.], seq 893395808:893397256, ack 3901572765, win 10136, options [nop,nop,TS val
14237218 ecr 413377950], length 1448
13:15:24.615334 IP (tos 0x0, ttl 254, id 13288, offset 0, flags [DF], proto TCP
(6), length 1500)
10.0.0.202.80 > 66.93.16.53.53859: Flags [P.], seq 893395808:893397256, ack 3901572765, win 10136, options [nop,nop,TS val
14238055 ecr 413377950], length 1448
13:15:30.877705 IP (tos 0x0, ttl 54, id 56678, offset 0, flags [none], proto
TCP (6), length 54)
66.93.16.53.53859 > 10.0.0.202.80: Flags [P.], cksum 0xa378 (correct), seq 3901572765:3901572767, ack 893395808, win 137,
options [nop,nop,TS val 413382608 ecr 14236800], length 2
13:15:31.225716 IP (tos 0x0, ttl 54, id 56679, offset 0, flags [none], proto
TCP (6), length 54)
66.93.16.53.53859 > 10.0.0.202.80: Flags [P.], cksum 0xa321 (correct), seq 3901572765:3901572767, ack 893395808, win 137,
options [nop,nop,TS val 413382695 ecr 14236800], length 2
13:15:31.229699 IP (tos 0x0, ttl 254, id 13289, offset 0, flags [DF], proto TCP
(6), length 52)
10.0.0.202.80 > 66.93.16.53.53859: Flags [.], cksum 0x7770 (correct), seq 893398510, ack 3901572767, win 10136, options
[nop,nop,TS val 14238717 ecr 413382608], length 0
13:15:32.128688 IP (tos 0x0, ttl 54, id 56680, offset 0, flags [none], proto
TCP (6), length 54)
66.93.16.53.53859 > 10.0.0.202.80: Flags [P.], cksum 0xa23d (correct), seq 3901572767:3901572769, ack 893395808, win 137,
options [nop,nop,TS val 413382921 ecr 14236800], length 2
13:15:32.131962 IP (tos 0x0, ttl 254, id 13290, offset 0, flags [DF], proto TCP
(6), length 40)
10.0.0.202.80 > 66.93.16.53.53859: Flags [R], cksum 0xf50a (correct), seq
893398510, win 10136, length 0
22 packets captured
545 packets received by filter
0 packets dropped by kernel
============== tcpdump of 'external' interface ==============
# tcpdump -vvv -i le2 -lnS host waste.org and tcp port 80
tcpdump: listening on le2, link-type EN10MB (Ethernet), capture size 96 bytes
13:15:01.528083 IP (tos 0x0, ttl 55, id 56671, offset 0, flags [DF], proto TCP
(6), length 60)
66.93.16.53.53859 > 216.251.177.106.80: Flags [S], cksum 0x9eb3 (correct), seq 3901572729, win 5840, options [mss
1460,sackOK,TS val 413375270 ecr 0,nop,wscale 7], length 0
13:15:01.533929 IP (tos 0x0, ttl 253, id 13279, offset 0, flags [none], proto
TCP (6), length 60)
216.251.177.106.80 > 66.93.16.53.53859: Flags [S.], cksum 0x15a7 (correct), seq 893390015, ack 3901572730, win 10136, options
[nop,nop,TS val 14235747 ecr 413375270,nop,wscale 0,mss 1460], length 0
13:15:01.642090 IP (tos 0x0, ttl 55, id 56672, offset 0, flags [DF], proto TCP
(6), length 52)
66.93.16.53.53859 > 216.251.177.106.80: Flags [.], cksum 0x68b8 (correct), seq 3901572730, ack 893390016, win 46, options
[nop,nop,TS val 413375299 ecr 14235747], length 0
13:15:11.635044 IP (tos 0x0, ttl 55, id 56673, offset 0, flags [DF], proto TCP
(6), length 87)
66.93.16.53.53859 > 216.251.177.106.80: Flags [P.], seq 3901572730:3901572765, ack 893390016, win 46, options [nop,nop,TS val
413377796 ecr 14235747], length 35
13:15:11.639812 IP (tos 0x0, ttl 253, id 13280, offset 0, flags [none], proto
TCP (6), length 52)
216.251.177.106.80 > 66.93.16.53.53859: Flags [.], cksum 0x3378 (correct), seq 893390016, ack 3901572765, win 10136, options
[nop,nop,TS val 14236757 ecr 413377796], length 0
13:15:11.709874 IP (tos 0x0, ttl 253, id 13281, offset 0, flags [none], proto
TCP (6), length 1500)
216.251.177.106.80 > 66.93.16.53.53859: Flags [P.], seq 893390016:893391464, ack 3901572765, win 10136, options [nop,nop,TS val
14236764 ecr 413377796], length 1448
13:15:11.878078 IP (tos 0x0, ttl 55, id 56674, offset 0, flags [DF], proto TCP
(6), length 52)
66.93.16.53.53859 > 216.251.177.106.80: Flags [.], cksum 0x54de (correct), seq 3901572765, ack 893391464, win 69, options
[nop,nop,TS val 413377858 ecr 14236764], length 0
13:15:11.889730 IP (tos 0x0, ttl 253, id 13282, offset 0, flags [none], proto
TCP (6), length 1500)
216.251.177.106.80 > 66.93.16.53.53859: Flags [.], seq 893391464:893392912, ack 3901572765, win 10136, options [nop,nop,TS val
14236782 ecr 413377858], length 1448
13:15:11.892140 IP (tos 0x0, ttl 253, id 13283, offset 0, flags [none], proto
TCP (6), length 1500)
216.251.177.106.80 > 66.93.16.53.53859: Flags [P.], seq 893392912:893394360, ack 3901572765, win 10136, options [nop,nop,TS val
14236782 ecr 413377858], length 1448
13:15:12.064083 IP (tos 0x0, ttl 55, id 56675, offset 0, flags [DF], proto TCP
(6), length 52)
66.93.16.53.53859 > 216.251.177.106.80: Flags [.], cksum 0x4edf (correct), seq 3901572765, ack 893392912, win 91, options
[nop,nop,TS val 413377905 ecr 14236782], length 0
13:15:12.075506 IP (tos 0x0, ttl 253, id 13284, offset 0, flags [none], proto
TCP (6), length 1500)
216.251.177.106.80 > 66.93.16.53.53859: Flags [.], seq 893394360:893395808, ack 3901572765, win 10136, options [nop,nop,TS val
14236800 ecr 413377905], length 1448
13:15:12.148648 IP (tos 0x0, ttl 55, id 56676, offset 0, flags [DF], proto TCP
(6), length 52)
66.93.16.53.53859 > 216.251.177.106.80: Flags [.], cksum 0x4914 (correct), seq 3901572765, ack 893394360, win 114, options
[nop,nop,TS val 413377917 ecr 14236782], length 0
13:15:12.268389 IP (tos 0x0, ttl 55, id 56677, offset 0, flags [DF], proto TCP
(6), length 52)
66.93.16.53.53859 > 216.251.177.106.80: Flags [.], cksum 0x4322 (correct), seq 3901572765, ack 893395808, win 137, options
[nop,nop,TS val 413377950 ecr 14236800], length 0
13:15:30.876882 IP (tos 0x0, ttl 55, id 56678, offset 0, flags [DF], proto TCP
(6), length 54)
66.93.16.53.53859 > 216.251.177.106.80: Flags [P.], cksum 0x23dc (correct), seq 3901572765:3901572767, ack 893395808, win 137,
options [nop,nop,TS val 413382608 ecr 14236800], length 2
13:15:31.224902 IP (tos 0x0, ttl 55, id 56679, offset 0, flags [DF], proto TCP
(6), length 54)
66.93.16.53.53859 > 216.251.177.106.80: Flags [P.], cksum 0x2385 (correct), seq 3901572765:3901572767, ack 893395808, win 137,
options [nop,nop,TS val 413382695 ecr 14236800], length 2
13:15:31.230472 IP (tos 0x0, ttl 253, id 13289, offset 0, flags [none], proto
TCP (6), length 52)
216.251.177.106.80 > 66.93.16.53.53859: Flags [.], cksum 0xf7d3 (correct), seq 893398510, ack 3901572767, win 10136, options
[nop,nop,TS val 14238717 ecr 413382608], length 0
13:15:32.127868 IP (tos 0x0, ttl 55, id 56680, offset 0, flags [DF], proto TCP
(6), length 54)
66.93.16.53.53859 > 216.251.177.106.80: Flags [P.], cksum 0x22a1 (correct), seq 3901572767:3901572769, ack 893395808, win 137,
options [nop,nop,TS val 413382921 ecr 14236800], length 2
13:15:32.132730 IP (tos 0x0, ttl 253, id 13290, offset 0, flags [none], proto
TCP (6), length 40)
216.251.177.106.80 > 66.93.16.53.53859: Flags [R], cksum 0x756e (correct),
seq 893398510, win 10136, length 0
18 packets captured
202 packets received by filter
0 packets dropped by kernel
I would appreciate any help in analyzing this, even if it is just with the
arithmetic of the window
sizes, scaling, etc.
Thanks much,
Michael
