Hi We are trying to use two firewalls (actually three, but to reproduce the problems we reduced it to the simplest case on our test-machines) using arpbalance and pfsync, and are experiencing stale connections after a while when the packets flowing from server to client pass fw-2 and the packets from client to server pass fw-1.
To reproduce the problem we just SSH from the server (which is arpbalanced to fw-2) to the client (which is arpbalanced to fw-1) and cat a large file. The connection first works and then hangs every time after we send the large file. Is this expected (i.e, is the situation where different firewalls are used for incoming and outgoing packets in the same TCP-session supported)? The following pf.conf was used on both firewalls. pf.conf: ext_if="fxp0" int_if="em1" all_if="{fxp0, em1}" admin_if="rl0" sync_if="em0" set debug misc set skip on lo0 set skip on $admin_if set skip on $int_if pass quick on $sync_if proto pfsync pass quick on $all_if proto carp keep state pass out quick on $ext_if proto tcp from any to any port 22 keep state pass in quick on $ext_if inet proto icmp keep state block log quick all In the log for fw-1 we get lots of messages like below (the last two are repeated): Jun 7 16:38:39 lab6 last message repeated 2 times Jun 7 16:39:43 lab6 /bsd: pfsync: ignoring stale update (4) id: 4486e33b00000004 creatorid: 69a3e6f5 Jun 7 16:39:56 lab6 last message repeated 9 times Jun 7 16:39:58 lab6 /bsd: pf: BAD state: TCP 194.9.94.201:13599 194.9.94.201:13599 194.42.55.190:22 [lo=428695180 high=428763873 win=16384 modulator=0 wscale=0] [lo=777455537 high=777455546 win=33304 modulator=0 wscale=1] 4:2 A seq=777455537 ack=428697265 len=1448 ackskew=-2085 pkts=0:36 dir=in,rev Jun 7 16:39:58 lab6 /bsd: pf: State failure on: 1 | and the state with id 4486e33b00000004 looks like: all tcp 194.9.94.201:13599 -> 194.42.55.190:22 FIN_WAIT_2:ESTABLISHED [428697586 + 66607] wscale 0 [777456985 + 14936] wscale 1 age 00:02:31, expires in 00:13:49, 0:1 pkts, 0:1500 bytes id: 4486e33b00000004 creatorid: 69a3e6f5 In the log for fw-2 we get (repeatedly): Jun 7 16:39:30 lab5 /bsd: pfsync: ignoring stale update (2) id: 4486e33b00000004 creatorid: 69a3e6f5 Jun 7 16:39:30 lab5 /bsd: pfsync: ignoring stale update (3) id: 4486e33b00000004 creatorid: 69a3e6f5 and the state with id 4486e33b00000004 looks like: all tcp 194.9.94.201:13599 -> 194.42.55.190:22 TIME_WAIT:TIME_WAIT [428697586 + 63131] wscale 0 [777439162 + 32759] wscale 1 age 00:02:08, expires in 00:00:42, 52:0 pkts, 5108:0 bytes, rule 3 id: 4486e33b00000004 creatorid: 69a3e6f5 Does anyone have any bright ideas of what to try or what the messages mean? Thanks in advance for any help, it would be really appreciated. Best regards, Jimmy