On Wed, 15 Jun 2016 20:18:25 +0100
Andy Lemin <a...@brandwatch.com> wrote:
> Peter is quite right, to add some examples to his suggestion;
>
> tcpdump -nettti pflog0 <- Shows only dropped packets
> tcpdump -nettti em0 <- Shows all packets on the interface, including
> ToS values and VLAN ID etc.
> tcpdump -nettti vlanX <- Shows only packets on the VLAN without the
> extra info.
>
> Sure you can figure out the rest..
>
> There are also a few caveats to writing good PF QoS rules that some
> are not aware off. For example the PRIO value is copied into the VLAN
> header as the CoS value, but if it is an untagged VLAN the frame wont
> have a value as their is no VLAN header to store it in. I.e. PRIO is
> only transitive for connected VLAN subnets, beyond the nexthop you
> cannot control layer 2 CoS, only layer 3 QoS is transitive.
>
> Also PRIO is strictly speaking internal to the firewall, and it works
> for both ingress and egress, whereas queuing/shaping is egress only.
> Best to think of it as a priority picker or scheduler. I.e. packets
> get selected from the buffers for processing based on their priority
> whether they are input or output buffers (I am only 90% sure of this,
> so please correct me if I am wrong).
>
> Also common good practice assumes that you would normally want to use
> two prio values; E.g.
>
> pass quick on { $if_ext, $if_DMZ } proto { tcp, udp } from any to {
> $int_ip_dns0 } port { 53 } queue (wan_web,wan_pri) set prio (2,4)
> The first prio (2) is used for the payload packets in the session
> (ToS not set), and the second prio (4) is used for the control
> packets (ACKs etc because they have the ToS set). This again also
> sets the VLAN CoS correctly for each packet type in the same session.
>
> Another thing to be careful of is setting ToS yourself and using PRIO
> (and if using queues too). For example;
>
> match in proto tcp all scrub (no-df max-mss 1460)
>
> match in proto { udp, icmp } all scrub (no-df max-mss 1472)
>
> match out on { $if_ext } proto { tcp, udp } from any to
> { <ext_voipgateways> } scrub (no-df max-mss 1420) set (tos ef, prio 7)
>
> The first two lines are just housekeeping. But the third line will
> set the ToS value EF on every single packet in the session (payload
> and ACKs) for the VoIP traffic. This means that the later pass rules
> will place all voip traffic into 'second' "queue" and second
> "priority".
>
> And if you didn't spot the clue in the first example, yes, I believe
> state does match returning traffic and does apply the prio to return
> traffic also. But you wont see it with tcpdump unless you are using
> VLANs to inspect the CoS field.
>
>
> In my first example you will also notice I have only one rule that
> matches traffic on both the inside and outside interfaces, so you
> need to make sure you are using the same queue names on both the
> inside and outside interfaces. This is done by adding the "on
> $if_ext" directive to your queues. E.g.
>
> queue ext_root on $if_ext bandwidth 800M
>
> queue qlocal on $if_ext parent ext_root bandwidth 600M
>
> queue local_kern on $if_ext parent qlocal bandwidth 6M min 6M
> burst 10M for 1000ms
>
> queue local_pri on $if_ext parent qlocal bandwidth 60M min 60M
>
> queue local_data on $if_ext parent qlocal bandwidth 510M min
> 100M
>
> queue qwan on $if_ext parent ext_root bandwidth 190M
>
> queue wan_rt on $if_ext parent qwan bandwidth 38M min 19M
> burst 38M for 5000ms
>
> queue wan_int on $if_ext parent qwan bandwidth 19M min 9M
>
> queue wan_pri on $if_ext parent qwan bandwidth 19M min 10M
> burst 25M for 2000ms
>
> queue wan_vpn on $if_ext parent qwan bandwidth 50M min 25M
>
> queue wan_web on $if_ext parent qwan bandwidth 19M min 10M
> burst 19M for 3000ms
>
> queue wan_dflt on $if_ext parent qwan bandwidth 19M min 10M
> burst 19M for 5000ms
>
> queue wan_bulk on $if_ext parent qwan bandwidth 20M max 50M
> default
>
>
> queue trunk_root on $if_trunk bandwidth 4294M
>
> queue qlocal on $if_trunk parent trunk_root bandwidth 4.1G
>
> queue local_kern on $if_trunk parent qlocal bandwidth 8M min
> 8M burst 8M for 1000ms
>
> queue local_pri on $if_trunk parent qlocal bandwidth 150M min
> 150M burst 200M for 2500ms
>
> queue local_data on $if_trunk parent qlocal bandwidth 4G min
> 1G
>
> queue qwan on $if_trunk parent trunk_root bandwidth 190M
>
> queue wan_rt on $if_trunk parent qwan bandwidth 38M min 19M
> burst 38M for 5000ms
>
> queue wan_int on $if_trunk parent qwan bandwidth 19M min 9M
>
> queue wan_pri on $if_trunk parent qwan bandwidth 19M min 10M
> burst 25M for 2000ms
>
> queue wan_vpn on $if_trunk parent qwan bandwidth 50M min 25M
>
> queue wan_web on $if_trunk parent qwan bandwidth 19M min 10M
> burst 19M for 3000ms
>
> queue wan_dflt on $if_trunk parent qwan bandwidth 19M min 10M
> burst 19M for 5000ms
>
> queue wan_bulk on $if_trunk parent qwan bandwidth 20M max 50M
> default
>
>
> Another hint, if you are using VLANs to gain the benefit of using the
> priority field in the VLAN header, to maintain your CoS on your layer
> 2 all the way to the client, you should apply your queues to the
> trunk interface, and not each VLAN.
>
> This way you can share your total download transit bandwidth across
> all internal VLANs instead of having to divide it out across the
> VLANs. Hope that makes sense.
>
>
> Just for some extra reading I have attached a paper that explains
> HFSC (the underlying queuing algorithm).
>
> PF Queue "bandwidth" = Link Sharing Service Curve
>
> PF Queue "min" = Realtime Service Curve
>
> PF Queue "burst" = Short-term realtime service curve for delay target
>
> PF queue "for" = time 't' for burst (delay target to be achieved
> within this initial period).
>
> And I have also attached a PF Packet Flow diagram I made (it's not
> 100% accurate, I need to update it but it should help).
>
>
> And lastly just for fun if you are using CARP with busy layer 2
> networks where you are having problems with heartbeat delay, this
> solves that, again not perfect, just ideas for you;
>
> pass out quick proto carp keep state (no-sync) queue local_kern set
> (prio 7, tos ef)
>
> pass in quick proto carp keep state (no-sync) set (prio 7, tos ef)
>
> pass out quick inet6 proto carp from { fe80::/10 } to { ff00::/8 }
> keep state (no-sync) queue local_kern set (prio 7, tos ef)
>
> pass in quick inet6 proto carp from { fe80::/10 } to { ff00::/8 } keep
> state (no-sync) set (prio 7, tos ef)
>
> pass quick on { $if_pfsync_dev } proto pfsync keep state (no-sync) set
> (prio 7, tos ef)
>
>
> Kind regards, Andy.
Perhaps it would be useful to add that 'set prio' does nothing unless
"hardware is slower at transmitting packets than the thing that
generates these packets to send", as stated here:
[http://marc.info/?l=openbsd-misc&m=145257356119612&w=2]
... thus making it inappropriate for solution of OP's problem.
--
Before enlightenment - chop wood, draw water.
After enlightenment - chop wood, draw water.
Marko Cupać
https://www.mimar.rs/
