hey,
i know it's been a while since you posted this. hopefully you got
something working, but i have some notes here.
On Wed, Mar 05, 2025 at 04:23:52PM -0700, Devin Reade wrote:
> I have a use case where I have a subnet that is officially routed
> to Site1, but I would actually like to have hosted at Site2 (but
> cannot route there directly). I've set things up via an IPSec
> tunnel between the sites, but Site2 also has redundant firewall/routers
> and the iked configuration seems to be interfering with proper
> carp operations on Site2.
>
> I'm hoping that someone has some guidance on how to address this.
>
> In the following description, all of FW1, FW2a and FW2b are
> OpenBSD 7.6. FW1 is a cloud-hosted VM at Site1, and the latter
> two are physical devices at Site2.
>
> ISP1 --- FW1 --- routable Net1 (a.a.a.a/26)
> |
> IPSec tunnel via upstream ISPs
> |
> ------
> | |
> | /- |-/--- routable Net2 (b.b.b.b/26)
> |/ |/
> FW2a FW2b
> |\ |\
> | \- |-\--- unroutable Net3 (multiple RFC1918 nets
> / / NATed to FW2 carp1, outbound through ISP2)
> ISP2 --/----/
>
> All IPs are static:
>
> FW1 external (vio0): c.c.c.c/29
> FW1 internal (vio1): a.a.a.1/26
> FW1 tunnel (sec0): 192.168.48.1/30
>
> FW2a tunnel (sec0): 192.168.48.2/30
> FW2b tunnel (sec0): 192.168.48.2/30 (yes, these are the same)
> FW2 internal (carp5): b.b.b.193/26
> FW2a internal (vlan5): b.b.b.194/26
> FW2b internal (vlan5): b.b.b.195/26
> FW2 external (carp1) d.d.d.114/29
> FW2a external (em1) d.d.d.115/29
> FW2b external (re1) d.d.d.117/29
>
> The desire is that the Internet at large should be able to
> reach Net2 via ISP1 and the IPSec tunnel (ISP1 is advertising
> route for both Net1 and Net2. The default route for Net2 should
> likewise be via the tunnel and out through ISP1).
ok.
> Net2 should be able to be reached from Net1 via the tunnel.
> Net2 should be able to be reached from Net3 directly.
ok.
> When iked is shut off on FW2a/b or FW1, failover behavior on
> the FW2 cluster is behaving as expected.
im going to skip down to the configs.
>
> I was subsequently able to bring up the tunnel and was testing
> pings from an offsite host to the three Net2 IPs of the FW2
> cluster (there are no other machines yet on Net2). In doing
> so, I was sometimes seeing partial packet loss, specifically of the
> return packet. On a closer look I found that for carp1 only,
> both FW2a and FW2b were in the master state (FW2b was in the
> slave state for all other carp interfaces).
>
> My suspicion is that the iked configuration is at issue here; I've
> verified that when the tunnel is up that the carp packets for Net2
> are ending up on enc0 rather than vlan5. Consequently neither FW2a
> nor FW2b are seeing the carp5 packets for the other, and both are
> holding onto master.
>
> I believe that the problem is the following line in the FW2 iked.conf:
> from b.b.b.192/26 to any
>
> Without that line in there carp behaves properly but the b.b.b.b
> packets get routed out to ISP2 instead of through the tunnel,
> even if I use reply-to on the inbound ICMP pf rules (see bottom of post).
> It also appears that there is no iked.conf syntax that will allow me to say
>
> from b.b.b.192/26 to (anywhere except internal networks,
> including b.b.b.192/26 itself)
>
> I contemplated the use of rdomains for the tunnel and Net2 but don't
> see how that would solve the carp problem on Net2.
>
> I'm not married to iked if there is a better approach; it just seemed
> to be the cleanest option, until now.
>
> Your thoughts are appreciated.
>
> Various config files follow (modified to match the above definitions)
>
> ======== FW2a and FW2b /etc/iked.conf:
> ikev2 'g65' passive esp \
> from 192.168.48.2 to 192.168.48.1 \
> from b.b.b.192/26 to any \
> local d.d.d.114 peer c.c.c.c \
> srcid g65.example.com
in this situation you should set up separate tunnels between FW1 and FW2a,
and FW1 and FW2b. so the FW2a config would be:
ikev2 'FW1' passive esp \
from any to any \
local d.d.d.115 peer c.c.c.c \
srcid fw2a.example.com dstid fw1.example.com \
iface sec0
and FW2b:
ikev2 'FW1' passive esp \
from any to any \
local d.d.d.117 peer c.c.c.c \
srcid fw2b.example.com dstid fw1.example.com \
iface sec0
note that "from XX to XX" doesn't mean anything in iked.conf when
setting up sec(4) tunnels because there's no ipsec policy set up for
them. the whole point of sec(4) is to use routes as the policy for
tunneling packets, not entries in the ipsec policy database in the
kernel.
>
> ======== FW2a and FW2b /etc/hostname.sec0:
> inet 192.168.48.2 255.255.255.252 192.168.48.1
> up
because there are now separate tunnels for FW2a and FW2b, you'll need
different IPs on their links too. sec(4) is a point to point interface,
so you dont need contig IPs on their links.
/etc/hostname.sec0 on FW2a:
inet 192.168.48.21 255.255.255.255 192.168.48.11
up
and /etc/hostname.sec0 on FW2b:
inet 192.168.48.22 255.255.255.255 192.168.48.12
up
>
> ======== FW2a /etc/hostname.carp5:
> inet b.b.b.193 255.255.255.192 NONE carpdev vlan5 vhid 6 pass XXXX
>
> ======== FW2b /etc/hostname.carp5:
> inet b.b.b.193 255.255.255.192 NONE carpdev vlan5 vhid 6 pass XXXX advskew 100
>
> ======== FW2a /etc/hostname.vlan5:
> inet b.b.b.194 255.255.255.192 NONE vlandev em0 vnetid 5
>
> ======== FW2b /etc/hostname.vlan5:
> inet b.b.b.195 255.255.255.192 NONE vlandev re0 vnetid 5
> ======== FW1 /etc/iked.conf:
> ikev2 'g64' active esp \
> from 192.168.48.1 to 192.168.48.2 \
> from any to b.b.b.192/26 \
> local c.c.c.c peer d.d.d.114 \
> srcid g64.example.com
as i said above, you will have separate tunnels on FW1 to each FW2
firewall:
ikev2 'FW2a' active esp \
from any to any \
local c.c.c.c peer d.d.d.115 \
srcid fw1.example.com dstid fw2a.example.com \
iface sec0
ikev2 'FW2b' active esp \
from any to any \
local c.c.c.c peer d.d.d.117 \
srcid fw1.example.com dstid fw2b.example.com \
iface sec1
> ======== FW1 /etc/hostname.sec0:
> inet 192.168.48.1 255.255.255.252 192.168.48.2
> up
> !route add -net b.b.b.192/26 192.168.48.2
and then you'll need separate config for each sec interface:
FW1 /etc/hostname.sec0:
inet 192.168.48.11 255.255.255.255 192.168.48.21
up
FW1 /etc/hostname.sec1:
inet 192.168.48.12 255.255.255.255 192.168.48.22
up
this means FW1 should be able to talk to both FW2 firewalls at the same
time.
the next part of the puzzle is to get FW1 to direct packets to the
FW2 side firewall that is has the active carp role. for that you
should use a routing daemon like bgpd and configure it to depend
on carp state when peering with FW1. the goal is that packets from
Net2 will hit the active carp firewall and get sent over it's sec(4)
link to FW1. When FW1 is sending packets to Net2 it should use sec0
or sec1 depending on which is FW2 firewall is the active carp peer.
the other thing to consider is how you organise routing. this may
be a challenge because routing only uses the destination IP in a
packet to decide where it should go, it does not consider where the
packet comes from at all. this means routing doesn't care if a
packet comes from Net2 or your Net3 IPs, it's going to make a
decision based solely on the destination IP.
if you want to force packets from Net2 and only Net2 to be routed over
the ipsec tunnels to FW1 on their way to the internet, you're going to
have to use either rdomains or route-to/reply-to in pf to force
that behaviour.
personally i find rdomains are a bit more straightforward once i
get them working, but you should play with both. rport(4) might be
useful if you're tinkering with rdomains.
hope this helps.
cheers,
dlg
>
> ======== FW2 **partial** /etc/pf.conf:
>
> set block-policy drop
> set skip on lo
>
> # Key bits:
> # $ext_if --> interface to ISP2
> # $net2 --> b.b.b.b/26
> # $tunnel_up_net --> c.c.c.c/29
> # $tunnel_sec_self --> 192.168.48.2/32
> # $ext_carp_ip --> d.d.d.114/32
> # <internal_nets> are RFC1918 nets behind FW2
> # $net49_if --> vlan for FW2 pfsync
> #
> include "/etc/pf.defs.conf"
>
> block all
>
> # We should not be sending Net2 over any links but the IPSec tunnel
> block out log quick on $ext_if from $net2
>
> block in log quick from urpf-failed
>
> match out on $ext_if from <internal_nets> to any nat-to $ext_carp_ip
>
> pass out
> pass out on egress proto { tcp udp icmp } all modulate state
>
> # Don't lock ourselves out
> pass in quick on any proto tcp to self port ssh
>
> # Firewall synchronization
> pass in quick on $net49_if proto pfsync keep state (no-sync)
> pass in quick on $net49_if proto icmp
> pass in quick proto carp keep state (no-sync)
>
> # IPSec rules
> pass in log quick on $ext_if proto udp from $tunnel_up_net to $ext_carp_ip \
> port { isakmp } tag IKED
> pass in log quick on $ext_if proto esp from $tunnel_up_net tag IKED
>
> # by default don't let net5 reach the internal networks
> pass in on $net2_if
> block in on $net2_if from $net2 to <internal_nets>
> pass in on $net2_if from $net2 to $net2
>
> # once we have a crossover cable for net49, we should just be
> # able to 'set skip' on that interface and use multicast
> # in hostname.pfsync0
> pass in on $net49_if
> block in on $net49_if from $net49 to <internal_nets>
> pass in on $net49_if from $net49 to $net49
>
> pass in on enc0 proto icmp to $tunnel_sec_self \
> icmp-type { echoreq } \
> keep state (if-bound)
>
> pass in on enc0 proto icmp from any to $net2 \
> icmp-type { echoreq } \
> keep state (if-bound)
