Re: carp and random disconnects
Bryan Irvine wrote: > On 3/10/06, Steven S <[EMAIL PROTECTED]> wrote: >> Bryan Irvine wrote: >> ... >> ... >>> It happened after we installed the carp firewalls, and seems to be >>> related to ICMP-Redirect coming from the real IP, as opposed to the >>> carp one the request went to. >>> >> ... >> >> Interesting, in my experiments carp interfaces didn't send ICMP >> redirects at all... > > The CARP interface is not. I'm not sure if it's supposed to or not. > I'm guessing because that is the only thing that has changed. With > the exception of the carp and pfsync rules, this is the exact same > ruleset from the old firewall. > > here's what I see on the firewall when I try a traceroute to a remote > network that goes through a different gateway. > > 17:51:50.581658 10.0.0.2 > 10.0.253.236.kent-dhcp.kcjn.com: icmp: > time exceeded in-transit 17:51:50.585106 10.0.0.2 > > 10.0.253.236.kent-dhcp.kcjn.com: icmp: time exceeded in-transit > 17:51:50.585402 10.0.0.2 > 10.0.253.236.kent-dhcp.kcjn.com: icmp: > time exceeded in-transit > > The results of the traceroute: > 1 10.0.0.2 (10.0.0.2) 0.971 ms 0.268 ms 4.880 ms > 2 10.0.0.201 (10.0.0.201) 0.508 ms 0.503 ms 0.359 ms > 3 172.19.1.10 (172.19.1.10) 111.714 ms 111.264 ms 111.691 ms > 4 172.19.4.10 (172.19.4.10) 111.331 ms 113.438 ms 111.278 ms > > > Am I missing something or barking up the wrong tree? > > --Bryan I experienced similar issues. The carp interface does not send an ICMP redirect (I have not had the time to find out why) but instead routes the packet, creating state if you're running PF. My users experienced "slowness" so I ended up adding static routes on the servers (only about 5 of them) for the short-term. There appears to be two things broken, ICMP redirects and routing back through a carp interface. -Steve S.
Re: carp and random disconnects
Bryan Irvine wrote: ... ... > It happened after we installed the carp firewalls, and seems to be > related to ICMP-Redirect coming from the real IP, as opposed to the > carp one the request went to. > ... Interesting, in my experiments carp interfaces didn't send ICMP redirects at all... http://marc.theaimsgroup.com/?l=openbsd-misc&m=113772490126174&w=2 -Steve S.
Strange carp issues
I have two firewalls (FW1 & FW2) with multiple carp interfaces on an external interface (carp1, carp12, carp14, carp15, carp16, carp17, carp18, carp19, carp20). FW1 has all carp interfaces set with advbase 1 advskew 0 and FW2 has all carp interfaces with advbase 1 advskew 180. Frequently FW2 thinks it is the master for some of the carp interfaces. Here is a tcpdump (-ni fxp0 proto carp) from FW2. As you can see, even though FW2 sees the advertisement for carp16, carp17, carp18, carp19 and carp20 from FW1 it sometimes takes over as master for those interfaces and advertises. To find these events look for advskew=180 in the tcpdump below. The event at 19:19:05.023848 seemed to be from lost packets. The event at 19:19:10.013844 is very odd since FW2 saw the carp20 advertisement from FW1 at 19:19:09.07. This should be enough time for a failover, should it? Any pointers would be appreciated (relevant pf rules below.) -Steve S. 19:19:02.290779 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.290807 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.290828 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.290849 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.290869 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.290887 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.290914 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.290936 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.290957 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.890823 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.890849 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.890871 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.890892 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.890912 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.890933 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.890962 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.890986 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:02.891010 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:03.880791 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:03.880818 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:03.880839 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:03.880860 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:03.880881 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:03.880901 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:03.880932 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:03.880955 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:03.880979 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:05.023848 CARPv2-advertise 36: vhid=17 advbase=1 advskew=180 (DF) [tos 0x10] 19:19:05.024936 CARPv2-advertise 36: vhid=18 advbase=1 advskew=180 (DF) [tos 0x10] 19:19:05.026003 CARPv2-advertise 36: vhid=19 advbase=1 advskew=180 (DF) [tos 0x10] 19:19:05.027069 CARPv2-advertise 36: vhid=20 advbase=1 advskew=180 (DF) [tos 0x10] 19:19:05.341023 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:05.341047 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:05.341068 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:05.341088 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:05.341109 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:05.341129 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:05.341154 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:05.341176 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:05.341199 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:06.295736 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:06.295760 CARPv2-advertise 36: vhid=12 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:06.295782 CARPv2-advertise 36: vhid=14 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:06.295802 CARPv2-advertise 36: vhid=15 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:06.295822 CARPv2-advertise 36: vhid=16 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:06.297299 CARPv2-advertise 36: vhid=17 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:06.297318 CARPv2-advertise 36: vhid=18 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:06.297335 CARPv2-advertise 36: vhid=19 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:06.297352 CARPv2-advertise 36: vhid=20 advbase=1 advskew=0 (DF) [tos 0x10] 19:19:06.900831 CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 (DF) [tos 0x
Re: Strange carp issues
Bryan Irvine wrote:
> I don't suppose you are using a quad card of some kind are you?
>
>
...
Three dual cards, dmesg (extracted from /var/log/messages) below:
OpenBSD 3.8-stable (GENERIC.MP) #0: Thu Jan 5 03:55:53 EST 2006
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Intel Pentium III ("GenuineIntel" 686-class) 798 MHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,
FXSR,SSE
real mem = 536436736 (523864K)
avail mem = 482525184 (471216K)
using 4278 buffers containing 26923008 bytes (26292K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 12/31/99, BIOS32 rev. 0 @ 0xf
pcibios0 at bios0: rev 2.1 @ 0xf/0x2000
pcibios0: PCI BIOS has 6 Interrupt Routing table entries
pcibios0: PCI Interrupt Router at 000:15:0 ("ServerWorks ROSB4 SouthBridge"
rev 0x00)
pcibios0: PCI bus #1 is the last bus
bios0: ROM list: 0xc/0x8000 0xc8000/0x4000! 0xe8000/0x6000
0xee000/0x2000!
mainbus0: Intel MP Specification (Version 1.4) (COMPAQ PROLIANT)
cpu0 at mainbus0: apid 3 (boot processor)
cpu0: apic clock running at 132 MHz
cpu1 at mainbus0: apid 0 (application processor)
cpu1: Intel Pentium III ("GenuineIntel" 686-class) 797 MHz
cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,
FXSR,SSE
mainbus0: bus 0 is type PCI
mainbus0: bus 3 is type PCI
mainbus0: bus 9 is type ISA
ioapic0 at mainbus0: apid 8 pa 0xfec0, version 11, 35 pins
ioapic0: misconfigured as apic 0, remapped to apic 8
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "ServerWorks CNB20LE Host" rev 0x05
pchb1 at pci0 dev 0 function 1 "ServerWorks CNB20LE Host" rev 0x05
pci1 at pchb1 bus 3
fxp0 at pci1 dev 4 function 0 "Intel 82557" rev 0x08, i82559: apic 8 int 10
(irq 10), address 00:50:8b:e2:6e:fb
inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4
fxp1 at pci1 dev 5 function 0 "Intel 82557" rev 0x08, i82559: apic 8 int 11
(irq 11), address 00:50:8b:e2:6e:fa
inphy1 at fxp1 phy 1: i82555 10/100 PHY, rev. 4
ppb0 at pci1 dev 6 function 0 "DEC 21154 PCI-PCI" rev 0x05
pci2 at ppb0 bus 4
fxp2 at pci2 dev 4 function 0 "Intel 82557" rev 0x08, i82559: apic 8 int 11
(irq 11), address 00:02:a5:60:58:50
inphy2 at fxp2 phy 1: i82555 10/100 PHY, rev. 4
fxp3 at pci2 dev 5 function 0 "Intel 82557" rev 0x08, i82559: apic 8 int 10
(irq 10), address 00:02:a5:60:58:51
inphy3 at fxp3 phy 1: i82555 10/100 PHY, rev. 4
cac0 at pci0 dev 1 function 0 "Symbios Logic 53c1510" rev 0x02: apic 8 int 3
(irq 3) Compaq Integrated Array
scsibus0 at cac0: 1 targets
sd0 at scsibus0 targ 0 lun 0: SCSI2 0/direct
fixed
sd0: 17359MB, 4357 cyl, 255 head, 32 sec, 512 bytes/sec, 35553120 sec total
vga1 at pci0 dev 3 function 0 "ATI Mach64 GV" rev 0x7a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
"Compaq Netelligent ASMC" rev 0x00 at pci0 dev 4 function 0 not configured
ppb1 at pci0 dev 5 function 0 "IBM 82351 PCI-PCI" rev 0x01
pci3 at ppb1 bus 1
tl0 at pci3 dev 0 function 0 "Compaq DP Netelligent 10/100TX" rev 0x10: apic
8 int 5 (irq 5) address 00:08:c7:a4:84:6d
nsphy0 at tl0 phy 1: DP83840 10/100 PHY, rev. 1
ukphy0 at tl0 phy 31: Generic IEEE 802.3u media interface
ukphy0: OUI 0x100014, model 0x0001, rev. 5
tl1 at pci3 dev 1 function 0 "Compaq DP Netelligent 10/100TX" rev 0x10: apic
8 int 7 (irq 7) address 00:08:c7:a4:84:ed
nsphy1 at tl1 phy 1: DP83840 10/100 PHY, rev. 1
ukphy1 at tl1 phy 31: Generic IEEE 802.3u media interface
ukphy1: OUI 0x100014, model 0x0001, rev. 5
pcib0 at pci0 dev 15 function 0 "ServerWorks ROSB4 SouthBridge" rev 0x4f
pciide0 at pci0 dev 15 function 1 "ServerWorks OSB4 IDE" rev 0x00: DMA
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: SCSI0 5/cdrom
removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0:
spkr0 at pcppi0
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask 0 netmask 0 ttymask 0
pctr: 686-class user-level performance counters enabled
mtrr: Pentium Pro MTRR support
dkcsum: sd0 matches BIOS drive 0x80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
Carp, isakmpd & sasyncd
Are these messages "normal" for a carped pair of firewalls running isakmpd with sasyncd (3.8-stable)? FW1/master - /var/log/message: Mar 16 01:37:40 fw1 isakmpd[32692]: message_recv: invalid cookie(s) 222729dc227c8f28 a0d29ef92ee65243 Mar 16 01:37:40 fw1 isakmpd[32692]: dropped message from x1.x2.x3.178 port 500 due to notification type INVALID_COOKIE Mar 16 01:37:45 fw1 isakmpd[32692]: message_recv: invalid cookie(s) 222729dc227c8f28 a0d29ef92ee65243 Mar 16 01:37:45 fw1 isakmpd[32692]: dropped message from x1.x2.x3.178 port 500 due to notification type INVALID_COOKIE FW2/backup - /var/log/message: Mar 16 01:35:49 fw2 isakmpd[5980]: transport_send_messages: giving up on exchange ISAKMP-peer, no response from peer x1.x2.x3.178:500 Mar 16 01:37:49 fw2 isakmpd[5980]: transport_send_messages: giving up on exchange ISAKMP-peer, no response from peer x1.x2.x3.178:500 -Steve S.
Re: Carp, isakmpd & sasyncd
Simon Slaytor wrote: > > I have two logical external firewalls, each configured as > 3.8-stable HA > pairs using PFSync, CARP, SASync etc. > ... > I have used the traditional isakmpd.conf method of configuring the > VPN's. In both cases the OBSD boxes replaced Checkpoint R55 boxes, > during my extensive testing with a R55 box at one end, non HA > and OBSD > at the other I again saw no such entries. I therefore wonder > if it could > be a R60 thing or a CP HA thing? > > What IPSec device(s) are at the other end of your VPN(s)? ... Theo's e-mail wasn't too encouraging, but I have VPN's with both a Cisco PIX and another OpenBSD 3.8 box. The OpenBSD box is the one I'm getting the most logs for. -Steve S.
Re: Strange carp issues
Anderson Nadal wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Hello. > > I have the same problem. > ... > > Take a look in your date/time, maybe it's the reason of your strange > carp issues. ... I thought of that too. If time changed by a couple seconds on the backup server then the backup might think it hadn't heard from FW1 in the carp time-out, so I stopped ntpd on both servers. I still experienced the problem. Oddly, it's not all carp interfaces on my fxp0. It only seems to affect carp16 - carp20, but inconsistently so. I'm going to try an experiment with a couple lab boxes, multiple carp interfaces, and ifstate (for monitoring). The plan is to see if it is related to the number of carp interfaces. Unless someone has tried this already (hint, hint;-) -Steve S.
Re: Strange carp issues
Bryan Irvine wrote: > I tried before with 2 quad cards to no avail. That was under 3.6 > though IIRC. 1 or 2 if's would fail over within a couple of hours, > but if left to it's own devices, eventually they all would. > > If you do figure something out lemme know, I'd love to go back to the > quad cards. > > ifstated didn't work for me but give it a go. I also had a script on > each machine that would ping the other every 5 seconds for ever > The interfaces seemed to last longer but eventually failed that way > too. > > --Bryan The interfaces that I'm having the most problem with are the built-in interfaces on a Compaq DL360 (I misstated earlier that it was a dual interface nic) although I have two other dual port nics in the machine. I wonder if the built-in looks like a dual port nic? I find it odd that the problem might be related to the multi-port nic. Did you try the same configuration with single port nics and it worked? I'm beginning to think it might be a component of the number of carp interfaces (you would likely have more carp interfaces on a machine with multiport nics.) Ifstated was broken for me on 3.8-stable too. I notice some changes in the 3.9 version so I compiled the 3.9 source under 3.8 and ifstated works *much* better. Perhaps it should be posted for 3.8 as a "reliability patch." -Steve S.
Re: Strange carp issues
Henning Brauer wrote: > * Steven S <[EMAIL PROTECTED]> [2006-03-17 19:10]: >> beginning to think it might be a component of the number of carp >> interfaces > > unlikely. > <[EMAIL PROTECTED]> $ ifconfig | grep '^carp' | wc -l > 15 > and growing. > and yes, that is real-world production use. How do you monitor if a carp interface changes state? And are these on any multi-port NICs? Thanks! -Steve S.
Re: Strange carp issues
Adam D. Morley wrote: ... > Have you checked: > > - carp settings in sysctl? > - carp pass rules (and ordering) in pf.conf (if you have default > deny)? > - that you have advskew set "right" on the backup firewall? > > # grep carp /etc/sysctl.conf > net.inet.carp.allow=1 # allow incoming CARP packets > net.inet.carp.preempt=1 # failover all CARP > interfaces if one fails > > # grep carp /etc/pf.conf > pass quick on $ext_ints proto carp keep state > pass on $int_phys proto carp keep state > pass on $int_vlan proto carp keep state > > # cat /etc/hostname.carp1 > vhid 1 advskew 100 pass > inet XXX 0xff00 Thanks, this is helpful. The settings on the FW's are as above. An incorrect setting (above) would seem to make it not work -- as opposed to what I'm seeing. Sometimes FW2 takes over as MASTER for some interfaces, but FW1 never moves to BACKUP. I do have net.inet.carp.preempt=1 set on FW1, but not FW2. As another experiment I moved advbase on FW2 to '2' for all carps, but the mysterious BACKUP-->MASTER transition still occurred on FW2 (in thinking back, I did this with a reboot of FW2, which re-started ntpd.) Perhaps I'll try again and not starting ntpd. -Steve S.
Re: Strange carp issues
Adam D. Morley wrote: > On Fri, Mar 17, 2006 at 02:35:55PM -0500, Steven S wrote: >> Adam D. Morley wrote: ... >> Thanks, this is helpful. The settings on the FW's are as above. An >> incorrect setting (above) would seem to make it not work -- as >> opposed to > > Ok. But mine works and yours doesn't? > >> what I'm seeing. Sometimes FW2 takes over as MASTER for some >> interfaces, but FW1 never moves to BACKUP. I do have >> net.inet.carp.preempt=1 set on FW1, but not FW2. > > You're supposed to set preempt on both, iirc. With both firewalls set to preempt=1 I had a common DMZ switch get shut-off. Both FW's went to a carp skew of 240. They had a MASTER fight. By setting one with preempt=1 and the other with preempt=0, I avoid this. >> As another experiment I moved advbase on FW2 to '2' for all carps, >> but the > > base is how often. skew is priority. Sort of... 'man ifconfig' Says, "Taken together the advbase and advskew indicate how frequently, in seconds, the host will advertise the fact that it considers itself master of the virtual host. The formula is advbase + (advskew / 256). If the master does not advertise within three times this interval, this host will begin advertising as master." So if I set FW1 with 1/0 and FW2 at 2/180, FW1 advertises every one second. If FW2 hasn't heard a carp advertisement in 2.7*3=8.1 seconds it will take over. When FW1 returns, it will start advertising once/sec. As noted in my OP, this doesn't seem to happen on my FW pair. -Steve S.
Re: Strange carp issues
Joachim Schipper wrote: >> Using NTPDATE in cron (30 minutes), I was able to handle this weird >> behavior. >> >> Take a look in your date/time, maybe it's the reason of your strange >> carp issues. > > As to problems with adjtime(2) and SMP machines, there is a small > diff from tedu@ on tech@ at > http://marc.theaimsgroup.com/?l=openbsd-tech&m=113592306900483&w=2, > which stemmed from the discussion on misc@ around the same time, > involving another SMP machine with severely screwed timekeeping - in > fact, it was so bad that NTPd couldn't keep up. Ted's diff > allows NTP to > keep up with time slew even on very imprecise hosts. > > It's a workaround, but might work for you. I tried the patch, but it didn't apply cleanly against 3.8:-( I tried booting FW2 with the SP kernel, but the problem still persists. It doesn't appear to be ntpd related since ntp updates didn't correlate with carp BACKUP -> MASTER transitions. I'll keep plugging away at it... -Steve S.
Re: Strange carp issues
It would appear my issues are related to timekeeping on these boxes (Compaq
DL360 G1).
If I bump advbase to '3' on each box everything is more stable. Given this,
I now have a roughly 10 second fail-over time, but that is still acceptable.
Since these are production boxes I'll probably wait until my 3.9 arrives to
see if any of the kern_time/kern_clock changes help. I'll let everyone know
more when I do.
Thanks for all the pointers and assistance!
Steve's corollary to Henning's carp theorem ("carp works."): Unless the
system clock is broken:-)
-Steve S.
Re: Strange carp issues
Steven S wrote: > It would appear my issues are related to timekeeping on these boxes > (Compaq DL360 G1). > > If I bump advbase to '3' on each box everything is more stable. > Given this, I now have a roughly 10 second fail-over time, but that > is still acceptable. > > Since these are production boxes I'll probably wait until my > 3.9 arrives to > see if any of the kern_time/kern_clock changes help. I'll let > everyone know more when I do. For the archives... I upgraded the backup firewall to 3.9-stable but it still appeared to have the MASTER-MASTER issue (with primary at 3.8). Based on some other posts in misc I tried using aliases on a single carp interface instead of multiple carp interfaces on the same physical interface. I upgraded the primary to 3.9-stable and things seem to be operating as expected. I have not had any MASTER<-->MASTER issues that weren't self inflicted. I guess I'm not 100% sure if the cure was upgrading or migrating to aliases, but it's working. The self inflicted issue came when I added an alias IP to FW1:carp0 but not yet to FW2:carp0. Both FW1 and FW2 became master for the interface, until I added the alias to FW2. Thanks again for the pointers and the great OS! -Steve S.
Re: PF performance question
'netstat -in' will give you a better indication of duplex mismatches (since it shows errors and collisions.) -Steve S. [EMAIL PROTECTED] wrote: > The ifconfig and brconfig output is as follow:
Re: PF NAT Address Pool Source Interface
[EMAIL PROTECTED] wrote: > On Mon, 5 Dec 2005 10:40:31 -0500 (EST), Brian A. Seklecki wrote: > >> All: >> ... >> Even if other hosts receive a packet and reply to it, they won't be >> able to ARP for it, and if they could, the original OpenBSD box will >> drop the reply with destination host/network unreachable (obviously). >> >> Wouldn't a better behavior to prevent the transmission of the packet >> in the same way the a socket cannot bind to a source port/ip if it >> is not assigned to an interface? >> >> Thoughts? > > Yes! > I'd rather have no change. If somebody uses the capability incorrectly > it would be just another case of shooting-self-in-foot allowed by > having powerful tools. > > My guess is that very few users NAT using an address other > than that of the $ext_if. ... I do, but only because I can;-) I also have a /29 but I do not pay any extra for it. One address is assigned to an interface and I use another addresses for an e-mail server. In my case I use the in-kernel PPPOE and configure a static route to the loopback from the desired address: /etc/rc.local: echo ' Routes'; route add 222.222.222.222 localhost /etc/pf.conf: rdr pass on $ppp_if proto tcp from to $email_addr port smtp \ -> 127.0.0.1 port spamd rdr pass on $ppp_if proto tcp from ! to $email_addr port smtp \ -> 127.0.0.1 port spamd rdr pass on $ppp_if proto tcp from any to $email_addr port smtp \ -> 127.0.0.1 port smtp If I needed the interface to answer an ARP query, I'd simply use a static arp entry. -Steve S.
Re: PF config for exchange
[EMAIL PROTECTED] wrote: ... > All branches have VPN tunnels back to central location and > the firewall rules > have a pass quick over the VPN tunnels > > On the main location I have a > > pass quick log inet from to > keep state > I also have a > pass quick log inet from to > keep state ... > > I have looked over the tcpdumps and I didn't see any blocks > > From within the same location on the Staffsegment off of this > same firewall it > works fine. I would be using the same rules as the remote > branches so it makes > me think its something with the tunnels but not really sure > at this point > > Any direction would be great.. For now, I had to back out and > put junkpoint, I > mean checkpoint in place. Are you logging all blocks (at both locations)? Is traffic leaving the VPN from the remote location through the VPN to the exchange server (as viewed with tcpdump)? Do you have any idea where traffic is being blocked/stopped? Can you ping the exchange servers from the staff segment? Is name resolution working(DNS/WINS) for staff segment? Try "ping exchange" and "nbtstat -a exchange" or whatever the exchange server is called. You might wish to post your sanitized pf.conf and isakmpd.conf. Also, I'm not sure what "From within the same location on the Staffsegment off of this same firewall it works fine." means. But that could be just me. -Steve S.
Generating ICMP Redirects
Greetings, I'm using a pair of 3.8-stable (1/5/06) servers as the firewall and default gw (10.10.0.1/16) for a LAN . VPN users (10.4.0.0/16) come into the LAN from a PIX (10.10.0.254/16) (changing soon to OpenVPN), and when the VPN users hit a server return packets are sent to the default gw. I was expecting the OpenBSD server to generate an ICMP redirect and all would be well. Unfortunately that is not happening. Instead the firewall is sending a host unreachable (yet the fw can ping the VPN host). Any pointers would be appreciated. Here's some relevant info: [EMAIL PROTECTED] tcpdump -nei fxp2 icmp 09:57:26.797397 0:2:a5:60:58:50 0:8:2:ce:99:65 0800 70: 10.10.0.251 > 10.10.0.11: icmp: host 10.4.0.67 unreachable 09:57:28.984736 0:2:a5:60:58:50 0:8:2:ce:99:65 0800 70: 10.10.0.251 > 10.10.0.11: icmp: host 10.4.0.67 unreachable [EMAIL PROTECTED] ping 10.4.0.67 PING 10.4.0.67 (10.4.0.67): 56 data bytes 64 bytes from 10.4.0.67: icmp_seq=0 ttl=128 time=66.969 ms [EMAIL PROTECTED] netstat -rn | grep 10.4 10.4/1610.10.0.254UGS 061208 - fxp2 [EMAIL PROTECTED] ifconfig carp2 carp2: flags=8843 mtu 1500 carp: MASTER carpdev fxp2 vhid 3 advbase 1 advskew 100 groups: carp inet 10.10.0.1 netmask 0x broadcast 10.10.255.255 [EMAIL PROTECTED] ifconfig fxp2 fxp2: flags=8943 mtu 1500 lladdr 00:02:a5:60:58:50 media: Ethernet autoselect (100baseTX full-duplex) status: active inet 10.10.0.251 netmask 0x broadcast 10.10.255.255 inet6 fe80::202:a5ff:fe60:5850%fxp2 prefixlen 64 scopeid 0x3 [EMAIL PROTECTED] pfctl -s rules |grep 10.4 pass in quick on fxp2 inet from 10.10.0.0/16 to 10.4.0.0/16 pass out quick on fxp2 inet from 10.4.0.0/16 to 10.10.0.0/16 [EMAIL PROTECTED] sysctl -a |grep redi net.inet.ip.redirect=1 net.inet.icmp.rediraccept=1 net.inet.icmp.redirtimeout=600 net.inet6.ip6.redirect=1 net.inet6.icmp6.rediraccept=1 net.inet6.icmp6.redirtimeout=600
Re: Generating ICMP Redirects
[EMAIL PROTECTED] wrote: > On Thu, Jan 19, 2006 at 10:32:40AM -0500, Steven S wrote: ... > > What about sysctl net.inet.ip.forwarding? Is it set to 1? > >> wq Claudio Yep. The firewalls are working perfectly aside from this redirect issue. They are even performing ISP load balancing (when the second ISP says up.) FW1 is acting as primary and FW2 is standby (it's off right now.) [EMAIL PROTECTED] sysctl -a |grep forw net.inet.ip.forwarding=1 net.inet6.ip6.forwarding=0 -Steve S.
Re: Generating ICMP Redirects
Stuart Henderson wrote: ... >> [EMAIL PROTECTED] pfctl -s rules |grep 10.4 >> pass in quick on fxp2 inet from 10.10.0.0/16 to 10.4.0.0/16 >> pass out quick on fxp2 inet from 10.4.0.0/16 to 10.10.0.0/16 > > I suspect you will need to allow the packets through in order to get > the redirects sent. Are you allowing the outbound from 10.10 > to 10.4 to > pass in another rule that you didn't include? If not, that's likely to > be the problem. If you're not sure, make sure blocked packets > are logged, > then monitor pflog0. There was nothing in pflog and here are my drop rules. I have 'pass out all keep state' rule at the head of the ruleset (possible issue?). I'll be testing further to find out more later tonight. After some further research I see I'll also need an rdr for the ICMP to source them from the carp interface as opposed to the real ip. [EMAIL PROTECTED] pfctl -s rules | grep block block drop in quick on ! lo inet from 127.0.0.0/8 to any block drop in quick on ! lo inet6 from ::1 to any block drop in quick inet from 127.0.0.1 to any block drop in quick inet6 from ::1 to any block drop in quick on lo0 inet6 from fe80::1 to any block drop in quick on ! fxp2 inet from 10.10.0.0/16 to any block drop in quick inet from 10.10.0.251 to any block drop in quick on fxp2 inet6 from fe80::202:a5ff:fe60:5850 to any block drop in log all block drop in quick inet from any to 255.255.255.255 block drop in quick inet from any to 10.255.255.255 block drop in quick inet from any to 10.10.255.255 block drop in quick on fxp2 proto tcp from any to any port = epmap block drop in quick on fxp2 proto udp from any to any port = epmap block drop in quick on fxp2 proto tcp from any to any port = netbios-ns block drop in quick on fxp2 proto udp from any to any port = netbios-ns block drop in quick on fxp2 proto udp from any to any port = netbios-dgm block drop in quick on fxp2 proto tcp from any to any port = netbios-ssn block drop in quick on fxp2 proto tcp from any to any port = microsoft-ds block drop in quick on fxp2 proto udp from any to any port = ssdp block drop in quick on fxp2 proto udp from any to any port = 5000
Re: Generating ICMP Redirects
... > I know this is not the answer to your question and I'd like > to hear how > you wind up getting the OpenBSD box to send the redirects you are > looking for, but relying on redirects to do your routing for anything > length of time is asking for trouble IMHO. You might just be better > off, temporarily, putting the PIX behind the OpenBSD box if > possible or, > if the servers are few, modifying their local route tables > until the new > VPN solution is in place. We did in fact add static routes to the servers for now (yuck.) I did some more testing on my home fw and it seems that carp interfaces don't like generating ICMP redirects (for me anyhow.) Here is my test, My WS (XP) - 192.168.83.51 My FW (OBSD 3.8)- 192.168.83.1 My server (OBSD 3.8) - 192.168.83.47 My WS normally has a default gw of the FW. My rules to/from the inside LAN to the FW are loose, # pass inquick on $int_if from any to any pass out quick on $int_if from any to any # So I create a route: [EMAIL PROTECTED] sudo route add -net 192.168.80 192.168.83.47 add net 192.168.80: gateway 192.168.83.47 And I pinged 192.168.80.2 from my WS, the FW did the "right thing" [EMAIL PROTECTED] sudo tcpdump -nei fxp1 net 192.168.80 or icmp tcpdump: listening on fxp1, link-type EN10MB 20:54:17.738121 0:11:43:39:e1:59 0:d0:b7:23:c0:e7 0800 74: 192.168.83.51 > 192.168.80.1: icmp: echo request 20:54:17.738340 0:d0:b7:23:c0:e7 0:90:27:42:d4:cc 0800 74: 192.168.83.51 > 192.168.80.1: icmp: echo request 20:54:17.738434 0:d0:b7:23:c0:e7 0:11:43:39:e1:59 0800 70: 192.168.83.1 > 192.168.83.51: icmp: redirect 192.168.80.1 to host 192.168.83.47 Next I created a carp interface on the inside and created a route on my workstation: [EMAIL PROTECTED] sudo ifconfig carp1 create [EMAIL PROTECTED] sudo ifconfig carp1 vhid 1 advskew 100 pass internal 192.168.83.2 netmask 255.255.255.0 [EMAIL PROTECTED] route add 192.168.80.0 mask 255.255.255.0 192.168.83.2 And tried the ping again, [EMAIL PROTECTED] sudo tcpdump -nei fxp1 net 192.168.80 or icmp 21:04:52.711456 0:11:43:39:e1:59 0:0:5e:0:1:1 0800 74: 192.168.83.51 > 192.168.80.2: icmp: echo request 21:04:52.711577 0:d0:b7:23:c0:e7 0:90:27:42:d4:cc 0800 74: 192.168.83.51 > 192.168.80.2: icmp: echo request 21:04:58.043062 0:11:43:39:e1:59 0:0:5e:0:1:1 0800 74: 192.168.83.51 > 192.168.80.2: icmp: echo request 21:04:58.043217 0:d0:b7:23:c0:e7 0:90:27:42:d4:cc 0800 74: 192.168.83.51 > 192.168.80.2: icmp: echo request Odd, since PF allows traffic on fxp1, not carp1. So let's add carp1 to pf... [EMAIL PROTECTED] sudo grep carp /etc/pf.conf pass inquick on carp1 from any to any pass out quick on carp1 from any to any And once again the FW happily routes the packet instead of sending an ICMP redirect. [EMAIL PROTECTED] sudo tcpdump -nei fxp1 net 192.168.80 or icmp tcpdump: listening on fxp1, link-type EN10MB 21:21:21.026831 0:11:43:39:e1:59 0:0:5e:0:1:1 0800 74: 192.168.83.51 > 192.168.80.2: icmp: echo request 21:21:21.026954 0:d0:b7:23:c0:e7 0:90:27:42:d4:cc 0800 74: 192.168.83.51 > 192.168.80.2: icmp: echo request I disabled pf and have the same results. I've hit my knowledge limit so delving into the source would be fruitless and annoying to the rest of you. Should I create a bug report? -Steve S. P.S. I'm not sure why the other box sent "host unreachables" and if I find out more I update the archive.
Ifstated question
Greetings,
I'm trying to use ifstated to determine the state (up or down) of my two ISP
connections. Currently I'm using ping, which I realize is imperfect, but
I'm getting some odd transitions.
For example, ISP2 is very unreliable and ifstated was in the ISP2down state.
>From there, based on the config below, it transitioned to the IPS1down state
even though ISP2 was also down. I would have expected it to transition to
the bothdown state.
Any thoughts or pointers would be appreciated.
-Steve S.
--- /etc/ifstated.conf
init-state bothup
ISP1 = '( "ping -q -c 1 -w 1 10.10.10.1 > /dev/null" every 15)'
ISP2 = '( "ping -q -c 1 -w 2 10.20.20.1 > /dev/null" every 15)'
state bothup {
init {
run "pfctl -a outbound -F rules -f /etc/pf.bothup.conf"
run "mail -s 'FW1 says both ISPs up' root
Re: OpenBSD/i386 3.8 on a Compaq DL380 SMP with GENERIC.MP
[EMAIL PROTECTED] wrote: > On 1/30/06, Bruno Carnazzi <[EMAIL PROTECTED]> wrote: >> Hi all, >> >> Everything seems to work fine but OpenBSD find only one CPU ! :( >> Somebody know why and how can I use the 2 CPUs ? >> ... > > Works fine here on the DL380G4's with -current (worked with release > too if I remember correctly). From your dmesg, this looks like a > first generation box - there's numerous tweaks in the BIOS on these > machines. Until recently I also had a DL360G1 running 3.7 with both > CPUs recognized - this should be more or less the same hardware you > have. > > --Bill I have two dual DL360-G1's in a FW/CARP config running now. I ended up using the Compaq SmartStart CD (or whatever the bootable CD utility is called) and set the OS to "Linux". -Steve S.
Re: inet failover solution
[EMAIL PROTECTED] wrote:
> John R. Shannon wrote:
>> On Monday 06 February 2006 06:46, Nickolay A Burkov wrote:
>>> Hi, All!
>>>
>>> I have a router with two external ethernet links to two different
>>> ISPs. Could someone recommend me a good technique to organize
> failover with
> these
...
>> I use ifstated for that purpose.
>>
>
> I do have a similar situation in my work. We have two ADSL connections
> to two different ISP's. I did an ifstated configuration and some shell
> scripts that basically do the following things:
>
> a) check if any of the internet links in the modems are up, using snmp
> (if your device has support to snmp, the majority of the DSL/ADSL
> routers does)
...
I used ifstated with ping to the other side of the link (as determined by
traceroute). You might need to create a static route or use the route-to pf
command to make sure you're pinging through the correct interface to
determine the state. This shows my ifstated.conf:
http://marc.theaimsgroup.com/?l=openbsd-misc&m=113776959830873&w=2
I ended up moving the ping to, '("ping -q -c 3 -w 2 10.10.10.1 > /dev/null"
every 30)' and using a single "if" statement in the downed states. I also
found moving everything in pf that did a route-to to an anchor was helpful.
Then I reload the anchor as shown in the ifstated.conf in the link. Because
this is an active test I also reserved a little (very little) bandwidth via
altq for this ICMP traffic.
Another approach might be to test to see if there is _any_ traffic coming
into an interface, if not, it is probably down. BTW, I do this with dual
carp'ed firewalls with site-to-site ipsec VPNs and OpenVPN for road
warriors.
Thanks for the great OS!
-Steve S.
Re: inet failover solution
[EMAIL PROTECTED] wrote: > On Mon, 6 Feb 2006 23:54:21 -0500, Steven S wrote: > >> [EMAIL PROTECTED] wrote: >>> John R. Shannon wrote: >>>> On Monday 06 February 2006 06:46, Nickolay A Burkov wrote: >>>>> Hi, All! ... > > I don't see any ping commands of the form: > > ping -I fxp0 .. > > in examples of ifstated use. I would think that forcing the interface > to be used would be useful to prevent misleading results. > > Whilst I'm at it: > Why wouldn't I change the default route by doing a route > delete default > && route add default $SecondChoice type command and the reverse when a > link comes up on $FirstChoice ? > > In general I'd love to see some more configurations with all the > relevant pf.conf bits so that I can study an example or three in > conjunction with the ifstated manpage. > > I think I'm going to have to set up a lab test and see what works well > but some other viewpoints may may choosing a better way easier. I force the interface by creating a static route and not creating any route-to pf rules for the tested IP's. In my case one gw is bridged via wireless to the ISP2 interface, so no route is needed. Never tried the 'ping -I' but it sound easier than the creating routes, so thanks! I'll have to try that against a carp interface for my second ISP since I only have one address and it is assigned to a carp interface. I'm using the round-robin, load balanced route-to command in pf.conf to load share among the available ISPs. So my default gw isn't used much. -Steve S.
Re: carp and kernel pppoe
> -Original Message- > On Behalf Of Christopher Vance > > I have a network being installed with a pair of 3.8 firewalls running > carp for failover. Temporarily, their external connection is via > residential grade router and wireless ADSL modem, with the router > doing pppoe. A real network is on order from the local monopoly ISP. > > I'm happily using kernel pppoe on my 3.9 beta machine at home, so I'm > wondering if I can remove the residential router, get both the > firewalls to run kernel pppoe and still do carp to determine which one > gets the inbound external traffic. > > carp over pppoe over em0? > > pppoe over carp over em0? > > both pppoe and carp over em0? As an experiment I tried to create a carp interface in the same subnet as a pppoe0 interface. It crashed/panicked my 3.8-stable box. I decided carp over a point-to-point protocol didn't make much sense and never pursued it further. -Steve S.
Re: slow network performance
[EMAIL PROTECTED] wrote: > I recently tried to use netperf, but it seemed more to test > my CPU than > the network and thus reporting low througput. benchmarks/netstrain is > much less demanding on the CPU. Of course, one may use ftp to download > large files since the OpenBSD one reports speed as well. > > /Sigfred > > Sebastian Schmitzdorff wrote: >> Hi, >> >> if I understand you correctly your testing consists of up and >> downloads on your openbsd router. This is not the proper way to test >> network performance on a router. I recommend using tools that dont >> involve any i/o operations such as netperf etc. I found iperf to be simple to use, and highlighted the significance of tcp window size selection. Even on a LAN I found a higher (double the default) values for net.inet.tcp.recvspace and net.inet.tcp.sendspace improved (TCP) throughput significantly. -Steve S.
Re: Ifstated question
For the archives, I noticed some commits to ifstated for 3.9-beta so I built the 3.9-beta ifstated on a 3.8-stable box. Ifstated seems to be much more reliable now. Thanks! -Steve S. [EMAIL PROTECTED] wrote: > Greetings, > > I'm trying to use ifstated to determine the state (up or > down) of my two ISP > connections. Currently I'm using ping, which I realize is imperfect, > but I'm getting some odd transitions. > > For example, ISP2 is very unreliable and ifstated was in the ISP2down > state. From there, based on the config below, it transitioned to the > IPS1down state even though ISP2 was also down. I would have expected > it to transition to the bothdown state.
