Hi there! I asked, without an answer, something about nat-to and real IPs. Well, I really need an answer there, so if someone get a clue, I will be glad tho hear :)
Now, to the new issue! Here in our WiFi ISP we are have contracted a tproxy service from FreeBSD Brasil. It is somehow working, but I can not figure out exactly how. Here is a diagram of the desired paths: http://devio.us/~raitech/Obsd53PfTproxy.png These are my rules by now: RFC1918 = "{ 172.16/12, 192.168/16, 10/8, 127/8 }" table <INT_NET> persist { internal nets, all valid IPs } ext_if_1 = "em0" ext_gw_1 = "187.72.X.X" ext_ip_1 = "187.72.X.X" ext_if_2 = "em1" ext_gw_2 = "187.72.X.X" ext_ip_2 = "187.72.X.X" ext_if_3 = "alc0" ext_gw_3 = "187.72.X.X" ext_ip_3 = "187.72.X.X" int_if_1 = "em2" int_gw_1 = "187.72.X.X" int_ip_1 = "187.72.X.X" squid_master_if = "em3" squid_master_gw = "187.72.X.X" squid_master_ip = "187.72.X.X" set limit states 6304000 set limit tables 5000 set limit src-nodes 200000 set limit frags 3000 set optimization aggressive set state-defaults pflow, no-sync set skip on lo block in log quick on { \ $ext_if_1, \ $ext_if_2, \ $ext_if_3, \ $squid_master_if, \ $int_if_1 } from $RFC1918 label "blocking RFC1918" # trying to prioritizing ACKs... match set prio (3,5) # ... and all traffic http. https over the others match proto tcp to port { http, https } set prio (5,6) match proto tcp from port { http, https } set prio (5,6) match proto tcp to port { ssh, 9876 } set prio(5,7) pass in on $int_if_1 proto tcp from { <INT_NET>, $int_gw_1 } to port http \ route-to ($squid_master_if $squid_master_gw) pass in on { $ext_if_1, $ext_if_2, $ext_if_3 } proto tcp from port http \ to { <INT_NET>, $int_gw_1 } \ route-to ($squid_master_if $squid_master_gw) pass in on $squid_master_if proto tcp from { <INT_NET>, $int_gw_1 } to \ port http no state route-to \ { \ ($ext_if_1 $ext_gw_1) , \ ($ext_if_2 $ext_gw_2) \ } least-states label "cahce external outbound balancing" pass in on $squid_master_if proto tcp from port http \ to { <INT_NET>, $int_gw_1 } route-to ($int_if_1 $int_gw_1) \ label "cahce internal outbound routing" An here are a pfctl -vsr output: block drop in log quick on em0 inet from 172.16.0.0/12 to any label "blocking RFC1918" [ Evaluations: 61764339 Packets: 332 Bytes: 32854 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em0 inet from 192.168.0.0/16 to any label "blocking RFC1918" [ Evaluations: 5883927 Packets: 114 Bytes: 28621 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em0 inet from 10.0.0.0/8 to any label "blocking RFC1918" [ Evaluations: 5883813 Packets: 170 Bytes: 18354 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em0 inet from 127.0.0.0/8 to any label "blocking RFC1918" [ Evaluations: 5883643 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em1 inet from 172.16.0.0/12 to any label "blocking RFC1918" [ Evaluations: 60684174 Packets: 305 Bytes: 30912 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em1 inet from 192.168.0.0/16 to any label "blocking RFC1918" [ Evaluations: 6862827 Packets: 93 Bytes: 9232 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em1 inet from 10.0.0.0/8 to any label "blocking RFC1918" [ Evaluations: 6862734 Packets: 196 Bytes: 19396 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em1 inet from 127.0.0.0/8 to any label "blocking RFC1918" [ Evaluations: 6862538 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on alc0 inet from 172.16.0.0/12 to any label "blocking RFC1918" [ Evaluations: 50726925 Packets: 304 Bytes: 30856 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on alc0 inet from 192.168.0.0/16 to any label "blocking RFC1918" [ Evaluations: 1251 Packets: 79 Bytes: 8268 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on alc0 inet from 10.0.0.0/8 to any label "blocking RFC1918" [ Evaluations: 1172 Packets: 152 Bytes: 16948 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on alc0 inet from 127.0.0.0/8 to any label "blocking RFC1918" [ Evaluations: 1020 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em3 inet from 172.16.0.0/12 to any label "blocking RFC1918" [ Evaluations: 50726392 Packets: 304 Bytes: 30856 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em3 inet from 192.168.0.0/16 to any label "blocking RFC1918" [ Evaluations: 13589809 Packets: 76 Bytes: 8132 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em3 inet from 10.0.0.0/8 to any label "blocking RFC1918" [ Evaluations: 13589733 Packets: 152 Bytes: 16948 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em3 inet from 127.0.0.0/8 to any label "blocking RFC1918" [ Evaluations: 13589581 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em2 inet from 172.16.0.0/12 to any label "blocking RFC1918" [ Evaluations: 39571927 Packets: 10414 Bytes: 478685 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em2 inet from 192.168.0.0/16 to any label "blocking RFC1918" [ Evaluations: 6364466 Packets: 1779 Bytes: 142401 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em2 inet from 10.0.0.0/8 to any label "blocking RFC1918" [ Evaluations: 6362687 Packets: 32496 Bytes: 1375238 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] block drop in log quick on em2 inet from 127.0.0.0/8 to any label "blocking RFC1918" [ Evaluations: 6330191 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] match all set ( prio(3, 5) ) [ Evaluations: 61717375 Packets: 13877464 Bytes: 10275632710 States: 3831 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] match proto tcp from any to any port = 80 set ( prio(5, 6) ) [ Evaluations: 61717375 Packets: 13877464 Bytes: 10275632710 States: 3831 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] match proto tcp from any to any port = 443 set ( prio(5, 6) ) [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] match proto tcp from any port = 80 to any set ( prio(5, 6) ) [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] match proto tcp from any port = 443 to any set ( prio(5, 6) ) [ Evaluations: 51200612 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] match proto tcp from any to any port = 22 set ( prio(5, 7) ) [ Evaluations: 51200616 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] match proto tcp from any to any port = 9876 set ( prio(5, 7) ) [ Evaluations: 51200616 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] pass all no state allow-opts [ Evaluations: 61717379 Packets: 61549113 Bytes: 41451833770 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] pass in on em0 inet proto tcp from any port = 80 to <INT_NET> flags S/SA keep state (no-sync, pflow) route-to 187.72.X.X@em3 [ Evaluations: 61717379 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] pass in on em1 inet proto tcp from any port = 80 to <INT_NET> flags S/SA keep state (no-sync, pflow) route-to 187.72.X.X@em3 [ Evaluations: 55197296 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] pass in on alc0 inet proto tcp from any port = 80 to <INT_NET> flags S/SA keep state (no-sync, pflow) route-to 187.72.X.X@em3 [ Evaluations: 38378103 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] pass in on em0 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA keep state (no-sync, pflow) route-to 187.72.X.X@em3 [ Evaluations: 48038032 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] pass in on em1 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA keep state (no-sync, pflow) route-to 187.72.X.X@em3 [ Evaluations: 44966361 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] pass in on alc0 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA keep state (no-sync, pflow) route-to 187.72.X.X@em3 [ Evaluations: 41608198 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] pass in on em2 inet proto tcp from <INT_NET> to any port = 80 flags S/SA keep state (no-sync, pflow) route-to 187.72.X.X@em3 [ Evaluations: 48044445 Packets: 1439990 Bytes: 894473590 States: 435 ] [ Inserted: uid 0 pid 19584 State Creations: 40060 ] pass in on em2 inet proto tcp from 187.72.X.X to any port = 80 flags S/SA keep state (no-sync, pflow) route-to 187.72.X.X@em3 [ Evaluations: 3694317 Packets: 12437474 Bytes: 9381159120 States: 3396 ] [ Inserted: uid 0 pid 19584 State Creations: 128206] pass in on em3 inet proto tcp from <INT_NET> to any port = 80 no state label "cahce external outbound balancing" route-to <__automatic_9ca6f8d9_0> least-states [ Evaluations: 38420511 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] pass in on em3 inet proto tcp from 187.72.X.X to any port = 80 no state label "cahce external outbound balancing" route-to <__automatic_9ca6f8d9_1> least-states [ Evaluations: 13586403 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] pass in on em3 inet proto tcp from any port = 80 to <INT_NET> flags S/SA keep state (no-sync, pflow) label "cahce internal outbound routing" route-to 187.72.X.X@em2 [ Evaluations: 13731058 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] pass in on em3 inet proto tcp from any port = 80 to 187.72.X.X flags S/SA keep state (no-sync, pflow) label "cahce internal outbound routing" route-to 187.72.X.X@em2 [ Evaluations: 13586403 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 19584 State Creations: 0 ] This is the same behavior with or without multipath routing. What bahavior? Well, only rules for in on em3 that are destineted to internal network are working, the others barelly catches a few thousands of packets. Very strange... But, as said before: more strange is the fact that the cache solution is almost working, just some delays to load a page here, youtube gasps there, but overall it seems to work! Tested without multipath routing, without keep state, and the behavior are the same. Will apreciate any kind of help on this, thank you in advance. Raimundo Santos
