On Wed, 2025-03-12 at 18:26 +1000, David Gwynne wrote:
> i know it's been a while since you posted this. hopefully you got
> something working, but i have some notes here.
[...]
> in this situation you should set up separate tunnels between FW1 and
> FW2a, and FW1 and FW2b. [...]
Thanks, David.
Based on your and Stuart's comments, I've altered the configuration
that I posted down-thread (subject: "iked.conf syntax?"), which is
making use of rdomains.
I'm still using gre(4) now instead of sec(4) [just because it was
already in place although I suspect sec(4) would be fine] but I've
taken your advice and set up two distinct tunnels over IPSec,
transport mode:
FW1 <--> FW2a
FW1 <--> FW2b
With the SAs separate, I've disabled sasyncd and instead configured
bgpd on the three (one private AS on FW1, and a second private AS
shared on FW2a and FW2b). Net2 is now advertised by the FW2 BGP
daemons, for use by FW1, with the "depend on carp5" clause.
I also found that I needed to use 'active' rather than 'passive' in
the iked.conf for FW2a and FW2b, otherwise fail-back when rebooting
FW2a resulted in about a 3.5 minute outage (fail-over was fine).
The outage in passive mode would last until new SAs were established
from FW1. Now with active on both sides I see no packet loss as new
SAs are created while the old are timing out.
Works like a charm.
> rport(4) might be useful if you're tinkering with rdomains.
Based on your comment, I successfully tried rport(4) instead of
using 'rtable' in pf.conf. Very cool, thanks for adding it to 7.6,
and impeccible timing.
In retrospect it makes sense, but it did take me a bit to realize that
for traffic traversing from one rdomain to another, I needed to have
essentially duplicate 'in' pf rules in place, both on the FW interface
of the originating rdomain and on the rport on the destination side
of the rport pair. Or maybe the rules on the destination rport are
completely redundant and I can get away with a single "allow anything"
or "skip"? I'm not sure, I don't have a good enough feel for it as
yet, so I'm being conservative.
Hopefully the above bread crumbs are useful to the next person who
goes down this road.
The only downside that I've seen so far on rport(4) is discoverability.
There's a reference to it in the 7.6 change log but (I think) nowhere
else. It could at least use a reference under "SEE ALSO" in the
rdomain(4) man page.
Yes, I could submit a patch for the man page. No, I do not (yet) have
a source tree checked out on a machine running -current. I had such
a machine on the test bench but I needed to prematurely promote it to
production and I'm waiting on replacement hardware to continue that
bench work.
> hope this helps.
It did, very much. Thanks.
Updated diagram:
ISP1 --- FW1 --- routable Net1 (a.a.a.a/26)
| |
gre0 | | gre1 (both over IPSec transport mode)
| |
| /- |-/--- routable Net2 (b.b.b.b/26); carp5
|/ |/
FW2a FW2b
|\ |\
| \- |-\--- unroutable Net3 (multiple RFC1918 nets
/ / NATed to FW2 carp1, outbound through ISP2)
ISP2 --/----/
All IPs are static:
FW1 external (vio0): c.c.c.c/29
FW1 internal (vio1): a.a.a.1/26
FW1 tunnel (gre0): 192.168.48.1
FW1 tunnel (gre1): 192.168.48.3
FW2a tunnel (gre0): 192.168.48.2
FW2b tunnel (gre0): 192.168.48.4
FW2 internal (carp5): b.b.b.193/26
FW2a internal (vlan5): b.b.b.194/26
FW2b internal (vlan5): b.b.b.195/26
FW2 external (carp1) d.d.d.114/29
FW2a external (em1) d.d.d.115/29
FW2b external (re1) d.d.d.117/29
