I have a pair of firewalls running 3.8-stable. Traffic is natted outbound on em0/carp179 (except carp traffic is left alone).
Their interfaces are em0 connection to ISP via unmanaged switch carp179 over em0 fxp0 crossover cable to other firewall em1 internal connection to managed switch(es) vlan (x118) over em1 carp (x118) one over each vlan I have set net.inet.carp.preempt=1 on each machine and run with pfsync enabled on fxp0. My pf rules set skip on fxp0, pass carp on both em and on all vlan. Most of the time, everything runs as we'd expect from the really fine OS that OpenBSD is, and the firewalls have always failed over properly on reboot. Occasionally however, maybe once every several weeks, we find that the external carp179 is in the inverse state from all other interfaces - master on one machine while all the other carp interfaces are backup - and vice versa. This messes up traffic attempting to pass through from inside out. It seems that the preempt setting has failed to cause all carp interfaces to the same state. How should I go about trying to discover why this state inconsistency is happening? (It looks like the sysctl says to do something with advskew which is intended to cause all interfaces to failover in the same direction, but which doesn't actually guarantee the result.) If the cause is not known, not fixed, or not fixable, two possible workarounds appear to be * run ifstated, or * run a routing daemon (which one?) to pass traffic between firewalls when necessary. [helmet on, yes I did google, check with the bug tracker, and check recent mailing list traffic] Any (relevent) suggestions? A lengthy pf.conf and ifconfig are available if needed. -- Christopher Vance
