Hello Everyone - hoping the right set of eyes looking at this will
find my mistake,
Details below - quick synopsis is that I have a two interface
host running 3.7 (release from CD - no patches or updates) with both
interfaces as members of a bridge. On the "em1" (internal) interface I
have 3 tagged vlan's configured which allow me to segment traffic
between three networks, all on the same subnet, by putting hosts on a
given vlan on the switch connected to the em1 interface. From a bridge
perspective things work great - hosts all communicate fine. When pf is
enabled however, nfs traffic from certain hosts (not all of them) to
other hosts is being dropped, but I can't determine why. These are all
Solaris hosts with the exception of the OpenBSD box.
=== Things I'm generally unsure about ===
1) Is it a good assumption that if I have vlan0, vlan1, vlan2 I should
be able to filter between those vlan interfaces in pf?
2) Are there any known complications doing this when those interfaces
are also members of a bridge?
3) After looking at what I've done below, is there other debugging I
can do to find out where things are being dropped - so far I can't
find it.
=== Problem Details ===
Most of these hosts have some form of NFS mount to another host going
on. Here's a basic diagram:
*10.1.2.13 mounts files from 10.1.2.11 (this has problems)
10.1.2.13 mounts files from 10.1.2.14 (this has no problems - same vlan)
*10.1.2.14 mounts files from 10.1.2.11 (this has problems)
10.1.2.75 mounts files from 10.1.2.11 (this has no problems)
10.1.2.77 mounts files from 10.1.2.11 (this has no problems)
10.1.2.11 mounts files from 10.1.2.12 (no problems - same vlan)
*10.1.2.12 mounts files from 10.1.1.20 (host on public side of
firewall - has problems as well)
When we first implemented the firewall we moved only 10.1.2.13 &
10.1.2.14 onto vlan2 - the remaining hosts were on the public network
(off em0) and everything worked fine. NFS mounts from vlan2 to em0
were no problem and we could filter traffic. Next, we moved 10.1.2.75
& 10.1.2.77 to vlan1, these also had no problems and NFS mounts worked
fine. Last to move were 10.1.2.11 & 10.1.2.12 and they were moved to
vlan0, immediately the above problems happened.
I can only assume that there is some difference between pf enforcement
from vlan <> vlan that does not exist from vlan <> physical interface.
When I enable the firewall and capture traffic on both vlan0 & vlan2 I
see the following traffic appear on vlan2 but NOT on vlan0. Before
enabling the firewall and after disabling the firewall this same
traffic passes fine:
14:31:23.600109 10.1.2.13.751 > 10.1.2.11.2049: xid 0x4944be7e 156
fsstat [|nfs] (DF) (ttl 64, id 22328, len 196)
14:31:23.996376 10.1.2.13.751 > 10.1.2.11.2049: xid 0x4944be7e 156
fsstat [|nfs] (DF) (ttl 64, id 22329, len 196)
14:31:24.796405 10.1.2.13.751 > 10.1.2.11.2049: xid 0x4944be7e 156
fsstat [|nfs] (DF) (ttl 64, id 22330, len 196)
14:31:26.396463 10.1.2.13.751 > 10.1.2.11.2049: xid 0x4944be7e 156
fsstat [|nfs] (DF) (ttl 64, id 22331, len 196)
14:31:29.596454 10.1.2.13.751 > 10.1.2.11.2049: xid 0x4944be7e 156
fsstat [|nfs] (DF) (ttl 64, id 22332, len 196)
I've enabled logging where I can think it would make sense (see
pf.conf below) and still doing the following shows no traffic related
to NFS:
# tcpdump -e -n -ttt -i pflog0
I've set debug loud in the pf.conf as well and see no additional
detail, though I'm not positive I'm looking in the right place
(pflog).
These problems all go away if I perform a 'pfctl -d' and immediately
start upon issuing a 'pfctl -e' so I'm certain the firewall is
blocking the traffic - but I can't tell why.
I have tried the following:
- Commented out 'scrub in' which made no difference
- Added a specific pass rule before the 'block in' rule which should
allow NFS traffic regardless of state
The very interesting thing is that the NFS mounts between { 10.1.2.75
10.1.2.77 } -> 10.1.2.11 work just fine through all of this.
Additionally, NFS mounts from 10.1.1.20 to 10.1.2.11 fail during this
- which go through the firewall but not between vlan's.
I suspect there is some additional debugging/troubleshooting that I
could do so I'm looking for suggestions. I'm stumped at why I cannot
see the traffic being dropped in pflog - I'm sure if I could get that
to happen it would be painfully obvious where I've failed.
=== Topology ===
em0 - Public interface - connects to the rest of the network.
em1 - Private interface - connected to a tagged port on a switch which
is configured for the 3 vlan's used to seperate networks.
- vlan0
- vlan1
- vlan2
(All of the above are added to 'bridge0' except em1) -
/etc/bridgename.bridge0 looks like this:
bash-3.00# cat /etc/bridgename.bridge0
add em0
add vlan0
add vlan1
add vlan2
blocknonip em0
blocknonip vlan0
blocknonip vlan1
blocknonip vlan2
up
vlan 0 has two hosts connected:
10.1.2.11
10.1.2.12
vlan1 has two hosts connected:
10.1.2.75
10.1.2.77
vlan2 has two hosts connected:
10.1.2.13
10.1.2.14
===== PF Config =====
bash-3.00# cat /etc/pf.conf
# $OpenBSD: pf.conf,v 1.28 2004/04/29 21:03:09 frantzen Exp $
#
# See pf.conf(5) and /usr/share/pf for syntax and examples.
# Remember to set net.inet.ip.forwarding=1 and/or net.inet6.ip6.forwarding=1
# in /etc/sysctl.conf if packets are to be forwarded between interfaces.
#*********************
# Variable definitions
#
ext_if="em0"
# If you want to create host-specific rules, please add an alias here so that
# the ruleset is easier to read and understand
althea="10.1.2.12"
atta="10.1.2.77"
europa="10.1.2.14"
flick="10.1.2.75"
miracle="10.1.2.11"
oldmiracle="10.1.2.13"
triton="10.1.2.15"
calypso="10.1.1.32"
#*******
# Tables
# This is a good way of defining multiple networks or hosts, it's very efficient
# from a performance perspective as well if you have many hosts
#
# General table of all hosts behind the firewall
table <protected_hosts> { $miracle $althea $oldmiracle $europa $atta $flick }
# Hosts specific db allowed tables (who is allowed to talk to these db's)
table <db_allow_europa> { $althea $triton }
table <db_allow_oldmiracle> { }
table <db_allow_althea> { $calypso }
table <db_allow_miracle> { }
table <db_allow_flick> { }
table <db_allow_atta> { }
#
# Service specific allow tables (add hosts here to allow given service)
#
# FTP
table <permit_ftp> { $miracle $althea $oldmiracle $europa $atta $flick }
# TELNET
table <permit_telnet> { $miracle $althea $oldmiracle $europa $atta $flick }
# HTTP
table <permit_http> { $althea $europa $flick }
# Netbios (file sharing)
table <permit_nbt> { $miracle $althea $oldmiracle $europa $atta $flick }
#****
# Options
#
set debug loud
#****
# normalize traffic in all directions - prevents malformed packets from passing
#
# Do not normalize NFS traffic from Sun hosts (experimental - seems to
make no difference)
#no scrub in log proto { tcp udp } from <protected_hosts> to
<protected_hosts> port nfs
# Normalize all traffic
scrub in log
#*****************************************************
# Standard blocking first (pass rules below will override)
# - this is also the point where any existing connections
# will be allowed, so if you want to block return traffic
# you need to do so above this or in the outbound rules below
#
# Block HSRP so it doesn't show up in logs
block in proto { tcp udp } to 224.0.0.2 port 1985
# Pass rule for NFS traffic (experimental)
pass quick proto { tcp udp } from <protected_hosts> to
<protected_hosts> port nfs
block in log
pass out keep state
# We want to pass all traffic on local interface
pass quick on { lo }
#***********************************
# Access policies for the firewall itself
#
pass in on $ext_if proto tcp to ($ext_if) port ssh keep state
pass in on $ext_if proto icmp to ($ext_if) keep state
#********************************************
# These are known pass rules which are not logged
#
#
# General pass rules for all hosts
#
# permit ping
pass in quick proto icmp to <protected_hosts> keep state
# permit ssh
pass in quick proto tcp to <protected_hosts> port ssh keep state
# permit telnet
pass in quick proto tcp to <permit_telnet> port telnet keep state
# permit http
pass in quick proto tcp to <permit_http> port 80 keep state
# permit netsaint
pass in quick proto tcp to <protected_hosts> port 1469 keep state
# Permit windows file sharing
pass in quick proto udp to <permit_nbt> port 137 keep state
pass in quick proto udp to <permit_nbt> port 138 keep state
pass in quick proto tcp to <permit_nbt> port 139 keep state
pass in quick proto tcp to <permit_nbt> port 445 keep state
# permit ftp
pass in quick proto tcp to <permit_ftp> port ftp keep state
pass in quick proto tcp to <permit_ftp> port ftp-data keep state
#
# Permit incoming db connections
#
# Permit db connections to Europa
pass in quick on $ext_if proto tcp from <db_allow_europa> to $europa
port >1024 keep state
# Permit db connections to OldMiracle
pass in quick on $ext_if proto tcp from <db_allow_oldmiracle> to
$oldmiracle port >1024 keep state
# Permit db connections to Althea
pass in quick on $ext_if proto tcp from <db_allow_althea> to $althea
port >1024 keep state
# Permit db connections to Miracle
pass in quick on $ext_if proto tcp from <db_allow_miracle> to $miracle
port >1024 keep state
# Permit db connections to Atta
pass in quick on $ext_if proto tcp from <db_allow_atta> to $atta port
>1024 keep state
# Permit db connections to Flick
pass in quick on $ext_if proto tcp from <db_allow_flick> to $flick
port >1024 keep state
#*****************************************************************
# Add restrictions here for now - these are blocked and not logged
#
# Europa db blocking (disabled for experimentation)
#block in quick log on $ext_if proto tcp to $europa port 2526
#block in quick log on $ext_if proto udp to $europa port 2526
#block in quick log on $ext_if proto tcp to $oldmiracle port 2960:2965
#block in quick log on $ext_if proto udp to $oldmiracle port 2960:2965
#********************************************************************************
# Make things wide open for now but log so we can see what we still
need to catch
#
# Allow and log all traffic to <protected_hosts>
pass in log to <protected_hosts> keep state
#**************************************
# Outbound Rules for vlan0 (Production)
#
pass in on vlan0 keep state
#***********************************
# Outbound Rules for vlan1 (Staging)
#
pass in on vlan1 keep state
#***************************************
# Outbound Rules for vlan2 (Development)
#
pass in on vlan2 keep state
==== END PF CONFIG ===
=== ifconfig output ===
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33224
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
address: 00:11:43:e6:4e:0c
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 10.1.2.32 netmask 0xffffff00 broadcast 10.1.2.255
inet6 fe80::211:43ff:fee6:4e0c%em0 prefixlen 64 scopeid 0x1
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
address: 00:11:43:e6:4e:0d
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet6 fe80::211:43ff:fee6:4e0d%em1 prefixlen 64 scopeid 0x2
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33224
pfsync0: flags=0<> mtu 2020
enc0: flags=0<> mtu 1536
vlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
address: 00:11:43:e6:4e:0d
vlan: 1000 parent interface: em1
inet6 fe80::211:43ff:fee6:4e0d%vlan0 prefixlen 64 scopeid 0x7
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
address: 00:11:43:e6:4e:0d
vlan: 1001 parent interface: em1
inet6 fe80::211:43ff:fee6:4e0d%vlan1 prefixlen 64 scopeid 0x8
vlan2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
address: 00:11:43:e6:4e:0d
vlan: 1002 parent interface: em1
inet6 fe80::211:43ff:fee6:4e0d%vlan2 prefixlen 64 scopeid 0x9
bridge0: flags=41<UP,RUNNING> mtu 1500
=== end of ifconfig output ===
