Hello lists, (I apologize for cross-posting to freebsd-ipfw).
I came across very weird issue with routing today - somehow, FreeBSD box routes
packets it shouldn't be routing.
Here it goes:
One of our customers has VPN connection to one of the remote servers. We are
using OpenVPN-2 tunnel for this purpose because it is relatively easy to setup
and maintain.
The VPN tunnel is used for customer to access internet through remote VPN
server to which he connects, so it basically looks as:
[customer]--->[secure vpn tun via internet]--->[vpn server]--->[internet for
customer]
The whole setup is pretty straightforward:
On customer's end, customer connects to the Net through DSL modem, and has a
small DMZ locally. There is also a FreeBSD box, which is used to setup OpenVPN
tunnel. Both FreeBSD box and customer are in one switch, within one subnet, and
see the Net through the DSL modem (which is plugged into the same switch for
uplink).
On servers' end, there is a standalone FreeBSD box connected to the Net.
Since customer's FreeBSD box is NAT'ed behind DSL modem, OpenVPN tunnel was
setup the way both sides see each other internally (within tunnel) using local
IPs:
192.168.10.3, iface tun0 - the IP remote VPN server has once VPN connection
gets established
192.168.10.4, iface tun0 - the IP customer's FreeBSD has once VPN connection
gets established
The subnet customer uses inside his DMZ is 10.0.0.0/24. The following IPs are
used:
10.0.0.1 - DSL modem
10.0.0.2 - FreeBSD VPN box
10.0.0.3 - customer's computer
So my aim was to route 10.0.0.3 to the Net through remote VPN server. This was
accomplished with the following steps:
1) run natd on remote VPN server, with divert rules as:
divert 8668 ip from 10.0.0.0/24 to any out xmit rl0
2) add routing entry on remote VPN server for 10.0.0.0/24 net:
/sbin/route add 10.0.0.0/24 192.168.10.4
3) turn on interface forwarding on both sides
4) on customer's FreeBSD box setup packet forwarding to remote VPN server as:
ipfw add fwd 192.168.10.3 ip from 10.0.0.3 to any
So the final step remains is to change default gateway on 10.0.0.3 to FreeBSD
VPN box (10.0.0.2) instead of DSL modem. And voila - 10.0.0.3 sees internet
through this VPN tunnel and all works beautifully.
So far all seems great, however now I need to route this customer back through
his DSL modem. To do that, customer simply changes his default gateway back to
DSL modem (10.0.0.1). All works beautifully, but one serious problem just
occured -
customer did not change his route back to 10.0.0.1 (he still had FreeBSD's
10.0.0.2 in his settings), and I removed "fwd" route from ipfw by mistake, and
what we saw was that customer was still able to surf the Net! However, his was
routed through his normal path (not VPN).
- ipfw was flushed
- there's no natd, nor any other nat on FreeBSD box
- no routing or anything
but it still acted as a gateway, routing packets it shouldn't be.
So after such a long thread (which I could probably cut a lot), my question is
- how is it possible that local FreeBSD box passed packets that it shouldn't be
passing?
I bashed my head against the wall for few hours now, and still couldn't figure
- every time I remove "fwd" rule from ipfw, customer goes normal route with no
other changes.
I tweaked all sysctls I could think of, and came to if I disable interface
forwarding - customer gets cut, and he does not see the Net; enable interface
forwarding back and he sees the Net again. I'm totally lost at this point.
Here's what was used to setup VPN tunnel:
- OpenVPN-2.0
- remote VPN server runs FreeBSD-4.10
- local VPN server runs FreeBSD-5.2
- customer's PC runs Windows XP
- 10/100mbit network switch
- DSL modem
local FreeBSD /etc/sysctl & network:
##############################
kern.corefile=/tmp/%N.code
kern.logsigexit=0
net.inet.ip.forwarding=1
net.inet.ip.fastforwarding=1
net.inet.ip.fw.one_pass=0
net.inet.tcp.inflight_enable=1
##############################
$ ifconfig
sk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
inet6 fe80::211:2fff:fe87:919d%sk0 prefixlen 64 scopeid 0x1
ether 00:11:2f:87:91:9d
media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>)
status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
inet6 fe80::211:2fff:fe87:919d%tun0 prefixlen 64 scopeid 0x4
inet 192.168.10.4 --> 192.168.10.3 netmask 0xffffffff
Opened by PID 454
$ netstat -nr
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.0.0.1 UGS 0 77 sk0
10/24 link#1 UC 0 0 sk0
10.0.0.1 00:a0:c5:9a:d7:49 UHLW 1 0 sk0 841
127.0.0.1 127.0.0.1 UH 0 0 lo0
192.168.10.3 192.168.10.4 UH 0 65 tun0
Anything else I need to specify?
I would highly appreciate any help/tips on how to find and remedy this issue.
Sincerely,
pgp48zgv6bHDe.pgp
Description: PGP signature
