I have a use case where I have a subnet that is officially routed
to Site1, but I would actually like to have hosted at Site2 (but
cannot route there directly). I've set things up via an IPSec
tunnel between the sites, but Site2 also has redundant firewall/routers
and the iked configuration seems to be interfering with proper
carp operations on Site2.
I'm hoping that someone has some guidance on how to address this.
In the following description, all of FW1, FW2a and FW2b are
OpenBSD 7.6. FW1 is a cloud-hosted VM at Site1, and the latter
two are physical devices at Site2.
ISP1 --- FW1 --- routable Net1 (a.a.a.a/26)
|
IPSec tunnel via upstream ISPs
|
------
| |
| /- |-/--- routable Net2 (b.b.b.b/26)
|/ |/
FW2a FW2b
|\ |\
| \- |-\--- unroutable Net3 (multiple RFC1918 nets
/ / NATed to FW2 carp1, outbound through ISP2)
ISP2 --/----/
All IPs are static:
FW1 external (vio0): c.c.c.c/29
FW1 internal (vio1): a.a.a.1/26
FW1 tunnel (sec0): 192.168.48.1/30
FW2a tunnel (sec0): 192.168.48.2/30
FW2b tunnel (sec0): 192.168.48.2/30 (yes, these are the same)
FW2 internal (carp5): b.b.b.193/26
FW2a internal (vlan5): b.b.b.194/26
FW2b internal (vlan5): b.b.b.195/26
FW2 external (carp1) d.d.d.114/29
FW2a external (em1) d.d.d.115/29
FW2b external (re1) d.d.d.117/29
The desire is that the Internet at large should be able to
reach Net2 via ISP1 and the IPSec tunnel (ISP1 is advertising
route for both Net1 and Net2. The default route for Net2 should
likewise be via the tunnel and out through ISP1).
Net2 should be able to be reached from Net1 via the tunnel.
Net2 should be able to be reached from Net3 directly.
When iked is shut off on FW2a/b or FW1, failover behavior on
the FW2 cluster is behaving as expected.
I was subsequently able to bring up the tunnel and was testing
pings from an offsite host to the three Net2 IPs of the FW2
cluster (there are no other machines yet on Net2). In doing
so, I was sometimes seeing partial packet loss, specifically of the
return packet. On a closer look I found that for carp1 only,
both FW2a and FW2b were in the master state (FW2b was in the
slave state for all other carp interfaces).
My suspicion is that the iked configuration is at issue here; I've
verified that when the tunnel is up that the carp packets for Net2
are ending up on enc0 rather than vlan5. Consequently neither FW2a
nor FW2b are seeing the carp5 packets for the other, and both are
holding onto master.
I believe that the problem is the following line in the FW2 iked.conf:
from b.b.b.192/26 to any
Without that line in there carp behaves properly but the b.b.b.b
packets get routed out to ISP2 instead of through the tunnel,
even if I use reply-to on the inbound ICMP pf rules (see bottom of post).
It also appears that there is no iked.conf syntax that will allow me to say
from b.b.b.192/26 to (anywhere except internal networks,
including b.b.b.192/26 itself)
I contemplated the use of rdomains for the tunnel and Net2 but don't
see how that would solve the carp problem on Net2.
I'm not married to iked if there is a better approach; it just seemed
to be the cleanest option, until now.
Your thoughts are appreciated.
Various config files follow (modified to match the above definitions)
======== FW2a and FW2b /etc/iked.conf:
ikev2 'g65' passive esp \
from 192.168.48.2 to 192.168.48.1 \
from b.b.b.192/26 to any \
local d.d.d.114 peer c.c.c.c \
srcid g65.example.com
======== FW2a and FW2b /etc/hostname.sec0:
inet 192.168.48.2 255.255.255.252 192.168.48.1
up
======== FW2a /etc/hostname.carp5:
inet b.b.b.193 255.255.255.192 NONE carpdev vlan5 vhid 6 pass XXXX
======== FW2b /etc/hostname.carp5:
inet b.b.b.193 255.255.255.192 NONE carpdev vlan5 vhid 6 pass XXXX advskew 100
======== FW2a /etc/hostname.vlan5:
inet b.b.b.194 255.255.255.192 NONE vlandev em0 vnetid 5
======== FW2b /etc/hostname.vlan5:
inet b.b.b.195 255.255.255.192 NONE vlandev re0 vnetid 5
======== FW1 /etc/iked.conf:
ikev2 'g64' active esp \
from 192.168.48.1 to 192.168.48.2 \
from any to b.b.b.192/26 \
local c.c.c.c peer d.d.d.114 \
srcid g64.example.com
======== FW1 /etc/hostname.sec0:
inet 192.168.48.1 255.255.255.252 192.168.48.2
up
!route add -net b.b.b.192/26 192.168.48.2
======== FW2 **partial** /etc/pf.conf:
set block-policy drop
set skip on lo
# Key bits:
# $ext_if --> interface to ISP2
# $net2 --> b.b.b.b/26
# $tunnel_up_net --> c.c.c.c/29
# $tunnel_sec_self --> 192.168.48.2/32
# $ext_carp_ip --> d.d.d.114/32
# <internal_nets> are RFC1918 nets behind FW2
# $net49_if --> vlan for FW2 pfsync
#
include "/etc/pf.defs.conf"
block all
# We should not be sending Net2 over any links but the IPSec tunnel
block out log quick on $ext_if from $net2
block in log quick from urpf-failed
match out on $ext_if from <internal_nets> to any nat-to $ext_carp_ip
pass out
pass out on egress proto { tcp udp icmp } all modulate state
# Don't lock ourselves out
pass in quick on any proto tcp to self port ssh
# Firewall synchronization
pass in quick on $net49_if proto pfsync keep state (no-sync)
pass in quick on $net49_if proto icmp
pass in quick proto carp keep state (no-sync)
# IPSec rules
pass in log quick on $ext_if proto udp from $tunnel_up_net to $ext_carp_ip \
port { isakmp } tag IKED
pass in log quick on $ext_if proto esp from $tunnel_up_net tag IKED
# by default don't let net5 reach the internal networks
pass in on $net2_if
block in on $net2_if from $net2 to <internal_nets>
pass in on $net2_if from $net2 to $net2
# once we have a crossover cable for net49, we should just be
# able to 'set skip' on that interface and use multicast
# in hostname.pfsync0
pass in on $net49_if
block in on $net49_if from $net49 to <internal_nets>
pass in on $net49_if from $net49 to $net49
pass in on enc0 proto icmp to $tunnel_sec_self \
icmp-type { echoreq } \
keep state (if-bound)
pass in on enc0 proto icmp from any to $net2 \
icmp-type { echoreq } \
keep state (if-bound)
