On 2020/10/19 19:53, David Gwynne wrote:
> On Mon, Oct 19, 2020 at 09:34:31AM +0100, Stuart Henderson wrote:
> > On 2020/10/19 15:35, David Gwynne wrote:
> > > every few years i try and use route-to in pf, and every time it
> > > goes badly. i tried it again last week in a slightly different
> > > setting, and actually tried to understand the sharp edges i hit
> > > this time instead of giving up. it turns out there are 2 or 3
> > > different things together that have cause me trouble, which is why
> > > the diff below is so big.
> >
> > I used to route-to/reply-to quite a lot at places with poor internet
> > connections to split traffic between lines (mostly those have better
> > connections now so I don't need it as often). It worked as I expected -
> > but I only ever used it with the interface specified.
>
> cool. did it work beyond the first packet in a connection?
It must have done. The webcams would have utterly broken the rest of
traffic if it hadn't :)
> > I mostly used it with pppoe interfaces so the peer address was unknown
> > at ruleset load time. (I was lucky and had static IPs my side, but the
> > ISP side was variable). I relied on the fact that once packets are
> > directed at a point-point interface there's only one place for them to
> > go. I didn't notice that ":peer" might be useful here (and the syntax
> > 'route-to pppoe1:peer@pppoe1' is pretty awkward so I probably wouldn't
> > have come up with it), I had 0.0.0.1@pppoe1, 0.0.0.2@pppoe2 etc
> > (though actually I think it works with $any_random_address@pppoeX).
>
> yes. i was trying to use it with peers over ethernet, and always
> struggled with the syntax.
>
> > > the first and i would argue most fundamental problem is a semantic
> > > problem. if you ask a random person who has some clue about networks
> > > and routing what they would expect the "argument" to route-to or
> > > reply-to to be, they would say "a nexthop address" or "a gateway
> > > address". eg, say i want to force packets to a specific backend
> > > server without using NAT, i would write a rule like this:
> > >
> > > n_servers="192.0.2.128/27"
> > > pass out on $if_internal to $n_servers route-to 192.168.0.1
> > >
> > > pfctl will happily parse this, shove it into the kernel, let you read
> > > the rules back out again with pfctl -sr, and it all looks plausible, but
> > > it turns out that it's using the argument to route-to as an interface
> > > name. because rulesets can refer to interfaces that don't exist yet, pf
> > > just passes the IP address around as a string, hoping i'll plug in an
> > > interface with a driver name that looks like an ip address. i spent
> > > literally a day trying to figure out why a rule like this wasn't
> > > working.
> >
> > I don't think I tried this, but the pf.conf(5) BNF syntax suggests it's
> > supposed to work. So either doc or implementation bug there.
>
> im leaning toward implementation bug.
>
> > route = ( "route-to" | "reply-to" | "dup-to" )
> > ( routehost | "{" routehost-list "}" )
> > [ pooltype ]
> >
> > routehost-list = routehost [ [ "," ] routehost-list ]
> >
> > routehost = host | host "@" interface-name |
> > "(" interface-name [ address [ "/" mask-bits ] ] ")"
> >
> > > the second problem is that the pf_route calls from pfsync don't
> > > have all the information it is supposed to have. more specifically,
> > > an ifp pointer isn't set which leads to a segfault. the ifp pointer
> > > isn't set because pfsync doesnt track which interface a packet is
> > > going out, it assumes the ip layer will get it right again later, or a
> > > rule provided something usable.
> > >
> > > the third problem is that pf_route relies on information from rules to
> > > work correctly. this is a problem in a pfsync environment because you
> > > cannot have the same ruleset on both firewalls 100% of the time, which
> > > means you cannot have route-to/reply-to behave consistently on a pair of
> > > firwalls 100% of the time.
> >
> > I didn't run into this because pppoe(4) and pfsync/carp don't really
> > go well together, but ouch!
> >
> > > all of this together makes things work pretty obviously and smoothly.
> > > in my opinion anyway. route-to now works more like rdr-to, it just
> > > feels like it changes the address used for the route lookup rather
> > > than changing the actual IP address in the packet. it also works
> > > predictably in a pfsync pair, which is great from the point of view of
> > > high availability.
> > >
> > > the main caveat is that it's not backward compatible. if you're already
> > > using route-to, you will need to tweak your rules to have them parse.
> > > however, i doubt anyone is using this stuff because it feels very broken
> > > to me.
> >
> > Do you expect this to work with a bracketed "address" to defer lookup
> > until rule evaluation time? i.e.
> >
> > pass out proto tcp to any port 22 route-to (pppoe1:peer)
>
> in my opinion route-to should be able to take whatever rdr-to accepts.
> however, i just tried it and it doesnt currently work, but i'm sure i
> can figure it out.
>
> > I think that will be all that's needed to allow converting the pppoe
> > use case. I don't have a multiple pppoe setup handy but I can probably
> > hack together some sort of test.
> >
> > I've also used route-to with squid "transparent" proxying (shown in
> > the pkg-readme), I don't do that any more but I can put a squid test
> > together easily enough.
>
> it looks doable.
>
> my changes will break route-to with "no state" rules though. should
> i try and keep those working too?
I don't think I ever used that myself. aja@ added the "no state" in
squid's pkg-readme (to avoid state problems when hairpinning the
intercepted packets back to a machine on a subnet alongside the client,
squid replying to the client directly), no idea if he's still using that,
I would guess probably not.
If I understand the problem correctly I think this case could use "keep
state (sloppy)" instead anyway.
> >
> > > @@ -1842,37 +1833,18 @@ pfrule : action dir logquick interface
> > > decide_address_family($7.src.host, &r.af);
> > > decide_address_family($7.dst.host, &r.af);
> > >
> > > - if ($8.route.rt) {
> > > + if ($8.rt) {
> > > + if ($8.rt != PF_DUPTO && !r.direction) {
> > > + yyerror("direction must be explicit"
> > > + " with rules that specify routing");
> > > + YYERROR;
> > > + }
> > > if (!r.direction) {
> > > yyerror("direction must be explicit "
> > > "with rules that specify routing");
> > > YYERROR;
> > > }
> >
> > this stood out on reading the diff - the added if block doesn't change
> > any outcome, should it have actually been like this?
>
> possibly :)
>
> >
> > - if (!r.direction) {
> > + if ($8.rt != PF_DUPTO && !r.direction) {
> > yyerror("direction must be explicit"
> > " with rules that specify routing");
> > YYERROR;
> > }
> >
>
