On Tuesday 16 November 2010, Robert Lewandowski wrote:
> Hello,
>
> PROBLEM: transfer speed is ONLY HALF if queue is defined in pf.conf
> although queue is 950Mbit (1000Mbit-5%)
> pf disabled: 768 Mbits/sec
> pf enabled, queue 950Mbit: 337 Mbits/sec
>
> ANALYSIS:
>
> - OpenBSD 4.8 default intallation.
> - Test made between OpenBSD 4.8 and Debian Linux.
> (between two Debian systems speed is more than 900Mbit/s)
>
> *********************************************************
> LAN interface: Intel PRO/1000 PT Desktop Adapter (PCIe, model:
> EXPI9300PTBLK)
> DMESG: em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82572EI)" rev
> 0x06: apic 1 int 16 (irq 5), address 00:1b:21:05:1f:39
> *********************************************************
> Default settings of TCP window size:
> net.inet.tcp.recvspace=16384
> net.inet.tcp.sendspace=16384
> *********************************************************
>
> 1a) pf disabled
>
> r...@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.0.0.8 port 27600 connected with 10.0.0.6 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 54.7 MBytes 459 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 1.0- 2.0 sec 54.7 MBytes 458 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 2.0- 3.0 sec 54.7 MBytes 459 Mbits/sec
>
> 1b) pf enabled, no queue
>
> r...@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.0.0.8 port 46912 connected with 10.0.0.6 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 53.9 MBytes 452 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 1.0- 2.0 sec 52.6 MBytes 441 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 2.0- 3.0 sec 54.1 MBytes 454 Mbits/sec
>
> 1c) pf enabled, added queue to default pf.conf:
>
> altq on em0 cbq bandwidth 1Gb queue { q_lan }
> queue q_lan bandwidth 950Mb cbq (default)
>
> r...@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.0.0.8 port 38266 connected with 10.0.0.6 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 33.9 MBytes 284 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 1.0- 2.0 sec 35.0 MBytes 294 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 2.0- 3.0 sec 35.8 MBytes 300 Mbits/sec
>
>
> *********************************************************
> TCP window size changed to 131072.
> net.inet.tcp.recvspace: 16384 -> 131072
> net.inet.tcp.sendspace: 16384 -> 131072
> *********************************************************
>
> 1a) pf disabled
>
> r...@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 128 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.0.0.8 port 32680 connected with 10.0.0.6 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 91.5 MBytes 768 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 1.0- 2.0 sec 92.1 MBytes 773 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 2.0- 3.0 sec 91.2 MBytes 765 Mbits/sec
>
> 1b) pf enabled, no queue
>
> r...@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 128 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.0.0.8 port 41092 connected with 10.0.0.6 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 80.5 MBytes 675 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 1.0- 2.0 sec 80.1 MBytes 672 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 2.0- 3.0 sec 80.2 MBytes 673 Mbits/sec
>
> 1c) pf enabled, added queue to default pf.conf:
>
> altq on em0 cbq bandwidth 1Gb queue { q_lan }
> queue q_lan bandwidth 950Mb cbq (default)
>
> r...@router-test (/root)# iperf -i 1 -t 3 -c 10.0.0.6
> ------------------------------------------------------------
> Client connecting to 10.0.0.6, TCP port 5001
> TCP window size: 128 KByte (default)
> ------------------------------------------------------------
> [ 3] local 10.0.0.8 port 12499 connected with 10.0.0.6 port 5001
> [ ID] Interval Transfer Bandwidth
> [ 3] 0.0- 1.0 sec 40.1 MBytes 337 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 1.0- 2.0 sec 40.1 MBytes 336 Mbits/sec
> [ ID] Interval Transfer Bandwidth
> [ 3] 2.0- 3.0 sec 40.0 MBytes 335 Mbits/sec
>
> *********************************************************
>
> any ideas, suggestions about this situation?
The default length for a queue is 50 packets - this only allows you to queue
around 75,000 bytes and the burstiness of TCP slow-start is likely to well
exceed this in your configuration (due to the BDP). I'd suggest increasing
the queue length - also run 'pfctl -vvs queue' or 'systat queue' and see
what's happening with regards to packets drops.
--
"Stop assuming that systems are secure unless demonstrated insecure;
start assuming that systems are insecure unless designed securely."
- Bruce Schneier
