I must add to this that I have seen a clear pattern now but I can't explain it.
Like I said, round-robin redirection for certain ports to certain hosts. It's very predictable, if netcat worked, next connect won't. But I'm less sure this is because of the LAN machines now, because of this tcpdump output here. lb01 $ tcpdump -nettti pflog0 'tcp' Oct 17 09:58:44.006881 rule 30/(match) pass in on vic2: XX.XX.XX.234.52143 > 10.221.181.22.25: S 3081755618:3081755618(0) win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 421950519 0,[|tcp]> (DF) tcpdump: WARNING: compensating for unaligned libpcap packets Oct 17 09:58:44.006909 rule 38/(match) pass out on vic3: XX.XX.XX.234.52143 > 10.221.181.22.25: S 3081755618:3081755618(0) win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 421950519 0,[|tcp]> (DF) Oct 17 09:58:45.261752 rule 30/(match) pass in on vic2: XX.XX.XX.234.52144 > 10.221.181.21.25: S 523320814:523320814(0) win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 421950532 0,[|tcp]> (DF) Oct 17 09:58:45.261786 rule 38/(match) pass out on vic3: XX.XX.XX.234.52144 > 10.221.181.21.25: S 523320814:523320814(0) win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 421950532 0,[|tcp]> (DF) This is when it works, then CARP makes the packets come in on lb02 the next connection attempt and I get this. lb02 $ tcpdump -nettti pflog0 'tcp' Oct 17 09:59:03.349586 rule 12/(match) block in on vic3: 10.221.181.21.25 > XX.XX.XX.234.52144: S 3675863197:3675863197(0) ack 523320815 win 5792 <mss 1460,sackOK,timestamp 87795141 421950532,nop,wscale 7> (DF) And this always happens on lb02, so after adding multiple variations of pass in on vic3 from 10.221.181.0/24 to any I of course made sure to check the configurations themselves and they're identical. I have this script setup now to regularly check that they're identical but so far they have been without fail. $ cat bin/compare #!/bin/sh ssh lb02 sudo pfctl -sr > /tmp/pf.compare.lb02 sudo pfctl -sr > /tmp/pf.compare.lb01 diff /tmp/pf.compare.lb01 /tmp/pf.compare.lb02 The remote source I'm testing from has been censored by the EPA. ;) 2011/10/17 Stefan Midjich <sweh...@gmail.com>: > I had this gateway with NAT working fine until I added another for > load balancing using carp. So now I've been slowly discovering the ins > and outs of carp in PF rules. Namely that packets seem to be going in > and out of the physical interfaces, but in on the carp interfaces at > the same time. Only a detail I've noted with tcpdump. > > vic0 is management only, works fine. > vic1 is pfsync only, set skip > vic2 is external public ipv4 address, seen here below as XX.XX.XX.59. > vic3 is the first internal network 10.221.181.0/24 > vic4 is the second internal network 10.221.182.0/24 > carp0 is vic2 on both machines, there is no IP-information on the > physical carpdev's, only on the carps. > carp1 is vic3 > carp2 is vic4 > > There are no pfsync update errors in syslog, I can see carp traffic > pass between the carpdevs using proto carp. > > Both loadbalancers are identical in configuration and I use git, > public ssh keys and bash scripts to update the configuration on the > git-server, update it on both loadbalancers and then run pfctl -vf > /etc/pf.conf on both. > > Here is output of pfctl -vvsr, please excuse the mail formatting. I'm > hoping this will shed light on my ruleset. > > @0 block drop on vic0 all > [ Evaluations: 353 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @1 pass in quick on vic0 inet proto tcp from any to 10.220.100.0/24 > port = 1022 flags S/SA keep state label "PassInMGMTSSH" > [ Evaluations: 347 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @2 pass in quick on vic0 inet proto tcp from any to 10.220.100.0/24 > port = ssh flags S/SA keep state label "PassInMGMTSSH" [ Evaluations: > 0 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @3 pass out quick on vic0 inet proto tcp from 10.220.100.0/24 to any > port = 1022 flags S/SA keep state label "PassOutMGMTSSH" > [ Evaluations: 16 Packets: 136 Bytes: 19978 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 1 ] > @4 pass out quick on vic0 inet proto tcp from 10.220.100.0/24 to any > port = ssh flags S/SA keep state label "PassOutMGMTSSH" > [ Evaluations: 15 Packets: 1020 Bytes: 441024 States: 4 ] > [ Inserted: uid 0 pid 3940 State Creations: 30 ] > @5 pass on vic0 proto udp from any to any port = domain keep state > label "PassMGMTDNS" > [ Evaluations: 331 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @6 pass on vic0 inet proto icmp all icmp-type echorep keep state label > "PassMGMTICMP" > [ Evaluations: 331 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @7 pass on vic0 inet proto icmp all icmp-type echoreq keep state label > "PassMGMTICMP" > [ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @8 pass on vic0 inet proto icmp all icmp-type unreach keep state label > "PassMGMTICMP" > [ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @9 pass quick on vic2 proto carp all keep state label "CUST-PassCarp" > [ Evaluations: 337 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @10 pass quick on vic3 proto carp all keep state label "CUST-PassCarp" > [ Evaluations: 332 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @11 pass quick on vic4 proto carp all keep state label "CUST-PassCarp" > [ Evaluations: 331 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @12 block drop log all > [ Evaluations: 337 Packets: 335 Bytes: 36237 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @13 pass log inet proto icmp from any to XX.XX.XX.59 icmp-type echoreq > keep state label "CUST-PingOutPhys" > [ Evaluations: 337 Packets: 1722 Bytes: 144648 States: 1 ] > [ Inserted: uid 0 pid 3940 State Creations: 1 ] > @14 pass log inet proto icmp from any to XX.XX.XX.59 icmp-type echorep > keep state label "CUST-PingOutPhys" > [ Evaluations: 1 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @15 pass log inet proto icmp from any to XX.XX.XX.59 icmp-type unreach > keep state label "CUST-PingOutPhys" > [ Evaluations: 1 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @16 pass log inet proto icmp from any to 10.221.181.10 icmp-type > echoreq keep state label "CUST-PingInternalGW" > [ Evaluations: 2 Packets: 884 Bytes: 74256 States: 1 ] > [ Inserted: uid 0 pid 3940 State Creations: 2 ] > @17 pass log inet proto icmp from any to 10.221.182.10 icmp-type > echoreq keep state label "CUST-PingInternalGW" > [ Evaluations: 2 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @18 pass log inet proto icmp from any to 10.221.181.10 icmp-type > echorep keep state label "CUST-PingInternalGW" > [ Evaluations: 2 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @19 pass log inet proto icmp from any to 10.221.182.10 icmp-type > echorep keep state label "CUST-PingInternalGW" > [ Evaluations: 2 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @20 pass log inet proto icmp from any to 10.221.181.10 icmp-type > unreach keep state label "CUST-PingInternalGW" > [ Evaluations: 2 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @21 pass log inet proto icmp from any to 10.221.182.10 icmp-type > unreach keep state label "CUST-PingInternalGW" > [ Evaluations: 2 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @22 pass log inet proto icmp from 10.221.181.10 to any icmp-type > echoreq keep state label "CUST-PingInternalGW" > [ Evaluations: 2 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @23 pass log inet proto icmp from 10.221.182.10 to any icmp-type > echoreq keep state label "CUST-PingInternalGW" > [ Evaluations: 2 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @24 pass log inet proto icmp from 10.221.181.10 to any icmp-type > echorep keep state label "CUST-PingInternalGW" > [ Evaluations: 2 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @25 pass log inet proto icmp from 10.221.182.10 to any icmp-type > echorep keep state label "CUST-PingInternalGW" > [ Evaluations: 2 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @26 pass log inet proto icmp from 10.221.181.10 to any icmp-type > unreach keep state label "CUST-PingInternalGW" > [ Evaluations: 2 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @27 pass log inet proto icmp from 10.221.182.10 to any icmp-type > unreach keep state label "CUST-PingInternalGW" > [ Evaluations: 2 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @28 match out log on vic2 from ! (egress:2) to any label > "CUST-MatchOutNAT" nat-to (egress:2) round-robin > [ Evaluations: 337 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @29 pass out log inet proto tcp from XX.XX.XX.59 to any port = www > flags S/SA keep state label "CUST-PassOutPorts" > [ Evaluations: 332 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @30 pass out log inet proto tcp from XX.XX.XX.59 to any port = https > flags S/SA keep state label "CUST-PassOutPorts" > [ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @31 pass out log inet proto tcp from XX.XX.XX.59 to any port = smtp > flags S/SA keep state label "CUST-PassOutPorts" > [ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @32 pass out log inet proto tcp from XX.XX.XX.59 to any port = ssh > flags S/SA keep state label "CUST-PassOutPorts" > [ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @33 pass out log inet proto tcp from XX.XX.XX.59 to any port = 1022 > flags S/SA keep state label "CUST-PassOutPorts" > [ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @34 pass in log on vic3 inet proto tcp from 10.221.181.0/24 to any > flags S/SA keep state > [ Evaluations: 337 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @35 pass in log on vic3 inet proto tcp from 10.221.182.0/24 to any > flags S/SA keep state > [ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @36 pass in log on vic4 inet proto tcp from 10.221.181.0/24 to any > flags S/SA keep state > [ Evaluations: 336 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @37 pass in log on vic4 inet proto tcp from 10.221.182.0/24 to any > flags S/SA keep state > [ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @38 pass out log on vic2 inet proto tcp from 10.221.181.0/24 to any > flags S/SA keep state > [ Evaluations: 336 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @39 pass out log on vic2 inet proto tcp from 10.221.182.0/24 to any > flags S/SA keep state > [ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @40 anchor "ftp-proxy/*" all > [ Evaluations: 337 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > @41 pass in log on vic3 inet proto tcp from any to any port = ftp > flags S/SA keep state label "CUST-PassInRDRFTP" rdr-to 127.0.0.1 port > 8021 > [ Evaluations: 337 Packets: 0 Bytes: 0 States: 0 ] > [ Inserted: uid 0 pid 3940 State Creations: 0 ] > > My real issue is with connections going out from the LAN to remote > destinations on the internet. > > From a LAN server I do nc -vv 212.211.132.250 80 # (this is > security.debian.org) and get the following on pflog0. > > Oct 17 07:47:30.793687 rule 34/(match) pass in on vic3: > 10.221.181.21.45667 > 212.211.132.250.80: S 3729652686:3729652686(0) > win 5840 <mss 1460,sackOK,timestamp 85822017 0,nop,wscale 7> (DF) > Oct 17 07:47:30.793786 rule 28/(match) match out on vic2: > XX.XX.XX.59.57433 > 212.211.132.250.80: S 3729652686:3729652686(0) win > 5840 <mss 1460,sackOK,timestamp 85822017 0,nop,wscale 7> (DF) > Oct 17 07:47:30.793800 rule 29/(match) pass out on vic2: > XX.XX.XX.59.57433 > 212.211.132.250.80: S 3729652686:3729652686(0) win > 5840 <mss 1460,sackOK,timestamp 85822017 0,nop,wscale 7> (DF) > > So this tells me that NAT is being done, and it is even sending > packets out on the external carpdev (vic2) to the remote destination. > But nothing comes back and the connection times out. I am not a > networking guy but TCP is a streaming protocol, I know that much, am I > to allow packets coming back through that stream? Or even packets > coming back to establish the stream? > > Even removing 'tcp' from that tcpdump command doesn't give me any new > clues as to what is happening to the rest of the stream. If I do nc > -vv 212.211.132.250 80 from the shell on the gateway it works fine. > Strangely enough the same command from lb02 does not work, it triggers > rule 29 in tcpdump of pflog0 on both systems but on lb02 no connection > is established. > > lb01 $ > carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 > lladdr 01:00:5e:00:01:01 > description: Customer External Public Loadbalancer > priority: 0 > carp: carpdev vic2 advbase 1 balancing ip > state MASTER vhid 1 advskew 0 > state MASTER vhid 2 advskew 100 > groups: carp egress > status: master > inet6 fe80::200:5eff:fe00:101%carp0 prefixlen 64 scopeid 0x8 > inet XX.XX.XX.59 netmask 0xffffff00 broadcast XX.XX.XX.255 > vic2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> > mtu 1500 > lladdr 00:50:56:8e:00:63 > description: Customer External > priority: 0 > media: Ethernet autoselect > status: active > inet6 fe80::250:56ff:fe8e:63%vic2 prefixlen 64 scopeid 0x3 > > lb02 $ > carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 > lladdr 01:00:5e:00:01:01 > description: Customer External Public Loadbalancer > priority: 0 > carp: carpdev vic2 advbase 1 balancing ip > state MASTER vhid 1 advskew 100 > state MASTER vhid 2 advskew 0 > groups: carp egress > status: master > inet6 fe80::200:5eff:fe00:102%carp0 prefixlen 64 scopeid 0x8 > Connection to 10.220.100.53 closed. > inet XX.XX.XX.59 netmask 0xffffff00 broadcast XX.XX.XX.255 > vic2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> > mtu 1500 > lladdr 00:50:56:8e:00:5e > description: Customer External > priority: 0 > media: Ethernet autoselect > status: active > Connection to 10.220.100.53 closed. > inet6 fe80::250:56ff:fe8e:5e%vic2 prefixlen 64 scopeid 0x3 > > Now there is actually another issue here that is of lower priority, > the fact that I can ping the external IP of the gateway without issue > but I can't have my LAN servers ping their gateways, meaning the > IP-addresses of carp1 and carp2. In that case I can see packets going > in on the physical interface behind carp1, but no reply comes back. > > Oct 17 07:45:48.710962 rule 16/(match) pass in on vic3: 10.221.181.21 >> 10.221.181.10: icmp: echo request (DF) > > This is strange to me as the rule to allow ICMP on the external > interface is nearly identical to the ones to allow on the internal > interfaces. > > -- > > > Med vdnliga hdlsningar / With kind regards > > Stefan Midjich > -- Med vdnliga hdlsningar / With kind regards Stefan Midjich
