https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=298932
Bug ID: 298932
Summary: pf: `(ifname:0)` is empty on a second point-to-point
interface with the same peer address, NAT fails
silently with map-failed
Product: Base System
Version: 15.1-RELEASE
Hardware: Any
OS: Any
Status: New
Severity: Affects Some People
Priority: ---
Component: kern
Assignee: [email protected]
Reporter: [email protected]
Environment
===========
- FreeBSD 15.1-RELEASE-p3, as shipped in OPNsense 26.7.4_1
(stable/26.7-n283949-083dc7025377). The relevant code is identical in
releng/15.1 and in OPNsense stable/26.7.
- Two PPPoE sessions (mpd5, netgraph ng_iface, renamed to pppoe0 and pppoe1) to
the same ISP. The ISP's BNG presents the same IPCP peer address on both
sessions.
- IP addresses replaced with ones reserved for documentation
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP>
mtu 1492
inet 203.0.113.10 --> 198.51.100.1 netmask 0xffffffff
pppoe1: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP>
mtu 1492
inet 203.0.113.20 --> 198.51.100.1 netmask 0xffffffff
[For transparence: analysis and writeup done by Claude Code, Model Opus 5.5 (1
Mio)]
Description
===========
Outbound NAT on whichever of the two interfaces came up second does not work.
Forwarded IPv4 packets that should leave through it are dropped without a log
entry. IPv6 over the same interface works, and so does traffic the firewall
originates itself.
Typical ruleset (generated by OPNsense):
nat on pppoe1 inet from (SEGMENT_0:network) to any -> (pppoe1:0) port
1024:65535
pass in quick on SEGMENT_0 route-to (pppoe1 198.51.100.1) inet from 10.x.y.52
to any keep state
Observed for a test client, six TCP connection attempts through pppoe1:
- the inbound state is created (2:0 pkts), no outbound state appears;
- pfctl -si: map-failed goes from 144601 to 144613 (+12, one per SYN including
retransmits);
- netstat -s -p ip: packets not forwardable +14;
- tcpdump -i pppoe1 shows nothing; no ICMP error is sent back to the client;
- nothing in pflog.
The same happens without route-to, with a plain host route route add -host
<dst> -interface pppoe1.
Analysis
========
1. ia_getrtprefix() (sys/netinet/in.c, line 988) uses the destination address
as a /32 prefix for point-to-point interfaces. in_addprefix() (line 1092) calls
in_hasrtprefix() (line 1056), finds that the address on pppoe0 already owns the
route to 198.51.100.1 in the same FIB, and returns without installing a route
or setting IFA_ROUTE on the pppoe1 address. This part looks intended.
net.route.multipath=1 does not change it; we tried.
2. pfi_instance_add() (sys/netpfil/pf/pf_if.c, line 731) then skips that
address for (pppoe1:0):
/*
* XXX: For point-to-point interfaces, (ifname:0) and IPv4,
* jump over addresses without a proper route to work
* around a problem with ppp not fully removing the
* address used during IPCP.
*/
if ((ifp->if_flags & IFF_POINTOPOINT) &&
!(ia->ifa_flags & IFA_ROUTE) &&
(flags & PFI_AFLAG_NOALIAS) && (af == AF_INET))
continue;
pppoe1 has exactly one IPv4 address, and it is the correct one. The dynamic
address list for (pppoe1:0) ends up empty.
3. pf_map_addr() (sys/netpfil/pf/pf_lb.c, line 597) finds pfid_acnt4 < 1 for
the NAT pool and returns PFRES_MAPFAILED. The packet is dropped in
pf_test(PF_OUT), ip_output() returns EACCES, and ip_forward() counts it as not
forwardable without sending ICMP.
Whichever interface was up first keeps the route and works. When it goes away,
in_scrubprefix() moves the route and IFA_ROUTE to the other address, so
failover works. When the first interface comes back, it becomes the one without
IFA_ROUTE, and its NAT breaks. For multi-WAN with gateway groups this means
failback fails reliably.
Expected behaviour
==================
(ifname:0) should resolve to the interface's IPv4 address even if the kernel
did not install the peer route for it because another interface already has
one.
Possible fix
============
The workaround's intent is to skip a stale IPCP address when a proper one
exists on the same interface. Narrowing it to that case would keep the ppp
workaround and fix this setup: skip an address without IFA_ROUTE only if the
same interface has another IPv4address that does have IFA_ROUTE. Sketch, not
compiled:
static bool
pfi_ptp_has_routed_addr4(struct ifnet *ifp)
{
struct ifaddr *ia;
CK_STAILQ_FOREACH(ia, &ifp->if_addrhead, ifa_link)
if (ia->ifa_addr != NULL &&
ia->ifa_addr->sa_family == AF_INET &&
(ia->ifa_flags & IFA_ROUTE))
return (true);
return (false);
}
...
if ((ifp->if_flags & IFF_POINTOPOINT) &&
!(ia->ifa_flags & IFA_ROUTE) &&
(flags & PFI_AFLAG_NOALIAS) && (af == AF_INET) &&
pfi_ptp_has_routed_addr4(ifp))
continue;
A caveat: the dynamic address is recomputed on address events. When IFA_ROUTE
moves between interfaces in in_scrubprefix(), (ifname:0) would have to be
refreshed as well.
We have not checked whether that already happens.
How to reproduce without an ISP (untested)
==========================================
ifconfig gif0 create inet 192.0.2.1 198.51.100.1 up
ifconfig gif1 create inet 192.0.2.2 198.51.100.1 up
netstat -rn -f inet | grep 198.51.100.1 # one route, via gif0
# pf.conf:
# nat on gif1 inet from 10.0.0.0/24 to any -> (gif1:0)
# pass out on gif1 from (gif1:0)
pfctl -vvsr | grep gif1 # (gif1:0) resolves to no address
--
You are receiving this mail because:
You are the assignee for the bug.
