I am guessing that the problem lies with flags S/SA. Changing all rules to flags any, and the packets hits the rules, but things go worse: no web navigation... this is driving me mad!
On 3 June 2013 13:09, Raimundo Santos <rait...@gmail.com> wrote: > Hi there! > > I asked, without an answer, something about nat-to and real IPs. Well, I > really need an answer there, so if someone get a clue, I will be glad tho > hear :) > > Now, to the new issue! > > Here in our WiFi ISP we are have contracted a tproxy service from FreeBSD > Brasil. It is somehow working, but I can not figure out exactly how. Here > is a diagram of the desired paths: > > http://devio.us/~raitech/Obsd53PfTproxy.png > > These are my rules by now: > > RFC1918 = "{ 172.16/12, 192.168/16, 10/8, 127/8 }" > table <INT_NET> persist { internal nets, all valid IPs } > > ext_if_1 = "em0" > ext_gw_1 = "187.72.X.X" > ext_ip_1 = "187.72.X.X" > > ext_if_2 = "em1" > ext_gw_2 = "187.72.X.X" > ext_ip_2 = "187.72.X.X" > > ext_if_3 = "alc0" > ext_gw_3 = "187.72.X.X" > ext_ip_3 = "187.72.X.X" > > int_if_1 = "em2" > int_gw_1 = "187.72.X.X" > int_ip_1 = "187.72.X.X" > > squid_master_if = "em3" > squid_master_gw = "187.72.X.X" > squid_master_ip = "187.72.X.X" > > set limit states 6304000 > set limit tables 5000 > set limit src-nodes 200000 > set limit frags 3000 > set optimization aggressive > set state-defaults pflow, no-sync > > set skip on lo > > block in log quick on { \ > $ext_if_1, \ > $ext_if_2, \ > $ext_if_3, \ > $squid_master_if, \ > $int_if_1 } from $RFC1918 label "blocking RFC1918" > > # trying to prioritizing ACKs... > match set prio (3,5) > # ... and all traffic http. https over the others > match proto tcp to port { http, https } set prio (5,6) > match proto tcp from port { http, https } set prio (5,6) > > match proto tcp to port { ssh, 9876 } set prio(5,7) > > pass in on $int_if_1 proto tcp from { <INT_NET>, $int_gw_1 } to port http \ > route-to ($squid_master_if $squid_master_gw) > > pass in on { $ext_if_1, $ext_if_2, $ext_if_3 } proto tcp from port http \ > to { <INT_NET>, $int_gw_1 } \ > route-to ($squid_master_if $squid_master_gw) > > pass in on $squid_master_if proto tcp from { <INT_NET>, $int_gw_1 } to \ > port http no state route-to \ > { \ > ($ext_if_1 $ext_gw_1) , \ > ($ext_if_2 $ext_gw_2) \ > } least-states label "cahce external outbound balancing" > > pass in on $squid_master_if proto tcp from port http \ > to { <INT_NET>, $int_gw_1 } route-to ($int_if_1 $int_gw_1) \ > label "cahce internal outbound routing" > > An here are a pfctl -vsr output: > > block drop in log quick on em0 inet from 172.16.0.0/12 to any label > "blocking RFC1918" > [ Evaluations: 61764339 Packets: 332 Bytes: 32854 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em0 inet from 192.168.0.0/16 to any label > "blocking RFC1918" > [ Evaluations: 5883927 Packets: 114 Bytes: 28621 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em0 inet from 10.0.0.0/8 to any label > "blocking RFC1918" > [ Evaluations: 5883813 Packets: 170 Bytes: 18354 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em0 inet from 127.0.0.0/8 to any label > "blocking RFC1918" > [ Evaluations: 5883643 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em1 inet from 172.16.0.0/12 to any label > "blocking RFC1918" > [ Evaluations: 60684174 Packets: 305 Bytes: 30912 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em1 inet from 192.168.0.0/16 to any label > "blocking RFC1918" > [ Evaluations: 6862827 Packets: 93 Bytes: 9232 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em1 inet from 10.0.0.0/8 to any label > "blocking RFC1918" > [ Evaluations: 6862734 Packets: 196 Bytes: 19396 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em1 inet from 127.0.0.0/8 to any label > "blocking RFC1918" > [ Evaluations: 6862538 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on alc0 inet from 172.16.0.0/12 to any label > "blocking RFC1918" > [ Evaluations: 50726925 Packets: 304 Bytes: 30856 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on alc0 inet from 192.168.0.0/16 to any label > "blocking RFC1918" > [ Evaluations: 1251 Packets: 79 Bytes: 8268 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on alc0 inet from 10.0.0.0/8 to any label > "blocking RFC1918" > [ Evaluations: 1172 Packets: 152 Bytes: 16948 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on alc0 inet from 127.0.0.0/8 to any label > "blocking RFC1918" > [ Evaluations: 1020 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em3 inet from 172.16.0.0/12 to any label > "blocking RFC1918" > [ Evaluations: 50726392 Packets: 304 Bytes: 30856 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em3 inet from 192.168.0.0/16 to any label > "blocking RFC1918" > [ Evaluations: 13589809 Packets: 76 Bytes: 8132 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em3 inet from 10.0.0.0/8 to any label > "blocking RFC1918" > [ Evaluations: 13589733 Packets: 152 Bytes: 16948 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em3 inet from 127.0.0.0/8 to any label > "blocking RFC1918" > [ Evaluations: 13589581 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em2 inet from 172.16.0.0/12 to any label > "blocking RFC1918" > [ Evaluations: 39571927 Packets: 10414 Bytes: 478685 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em2 inet from 192.168.0.0/16 to any label > "blocking RFC1918" > [ Evaluations: 6364466 Packets: 1779 Bytes: 142401 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em2 inet from 10.0.0.0/8 to any label > "blocking RFC1918" > [ Evaluations: 6362687 Packets: 32496 Bytes: 1375238 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > block drop in log quick on em2 inet from 127.0.0.0/8 to any label > "blocking RFC1918" > [ Evaluations: 6330191 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > match all set ( prio(3, 5) ) > [ Evaluations: 61717375 Packets: 13877464 Bytes: 10275632710 States: > 3831 ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > match proto tcp from any to any port = 80 set ( prio(5, 6) ) > [ Evaluations: 61717375 Packets: 13877464 Bytes: 10275632710 States: > 3831 ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > match proto tcp from any to any port = 443 set ( prio(5, 6) ) > [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > match proto tcp from any port = 80 to any set ( prio(5, 6) ) > [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > match proto tcp from any port = 443 to any set ( prio(5, 6) ) > [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > match proto tcp from any to any port = 22 set ( prio(5, 7) ) > [ Evaluations: 51200616 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > match proto tcp from any to any port = 9876 set ( prio(5, 7) ) > [ Evaluations: 51200616 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > pass all no state allow-opts > [ Evaluations: 61717379 Packets: 61549113 Bytes: 41451833770 States: > 0 ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > pass in on em0 inet proto tcp from any port = 80 to <INT_NET> flags S/SA > keep state (no-sync, pflow) route-to 187.72.X.X@em3 > [ Evaluations: 61717379 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > pass in on em1 inet proto tcp from any port = 80 to <INT_NET> flags S/SA > keep state (no-sync, pflow) route-to 187.72.X.X@em3 > [ Evaluations: 55197296 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > pass in on alc0 inet proto tcp from any port = 80 to <INT_NET> flags S/SA > keep state (no-sync, pflow) route-to 187.72.X.X@em3 > [ Evaluations: 38378103 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > pass in on em0 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA > keep state (no-sync, pflow) route-to 187.72.X.X@em3 > [ Evaluations: 48038032 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > pass in on em1 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA > keep state (no-sync, pflow) route-to 187.72.X.X@em3 > [ Evaluations: 44966361 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > pass in on alc0 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA > keep state (no-sync, pflow) route-to 187.72.X.X@em3 > [ Evaluations: 41608198 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > pass in on em2 inet proto tcp from <INT_NET> to any port = 80 flags S/SA > keep state (no-sync, pflow) route-to 187.72.X.X@em3 > [ Evaluations: 48044445 Packets: 1439990 Bytes: 894473590 States: > 435 ] > [ Inserted: uid 0 pid 19584 State Creations: 40060 ] > pass in on em2 inet proto tcp from 187.72.X.X to any port = 80 flags S/SA > keep state (no-sync, pflow) route-to 187.72.X.X@em3 > [ Evaluations: 3694317 Packets: 12437474 Bytes: 9381159120 States: > 3396 ] > [ Inserted: uid 0 pid 19584 State Creations: 128206] > pass in on em3 inet proto tcp from <INT_NET> to any port = 80 no state > label "cahce external outbound balancing" route-to <__automatic_9ca6f8d9_0> > least-states > [ Evaluations: 38420511 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > pass in on em3 inet proto tcp from 187.72.X.X to any port = 80 no state > label "cahce external outbound balancing" route-to <__automatic_9ca6f8d9_1> > least-states > [ Evaluations: 13586403 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > pass in on em3 inet proto tcp from any port = 80 to <INT_NET> flags S/SA > keep state (no-sync, pflow) label "cahce internal outbound routing" > route-to 187.72.X.X@em2 > [ Evaluations: 13731058 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > pass in on em3 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA > keep state (no-sync, pflow) label "cahce internal outbound routing" > route-to 187.72.X.X@em2 > [ Evaluations: 13586403 Packets: 0 Bytes: 0 States: 0 > ] > [ Inserted: uid 0 pid 19584 State Creations: 0 ] > > This is the same behavior with or without multipath routing. What > bahavior? Well, only rules for in on em3 that are destineted to internal > network are working, the others barelly catches a few thousands of packets. > Very strange... > > But, as said before: more strange is the fact that the cache solution is > almost working, just some delays to load a page here, youtube gasps there, > but overall it seems to work! > > Tested without multipath routing, without keep state, and the behavior are > the same. > > Will apreciate any kind of help on this, thank you in advance. > > Raimundo Santos
