Dear Olivier,
I tried to use your example for traffic shaping with ipfw and dummynet:
https://bsdrp.net/documentation/examples/fair_traffic_shaping_per_ip_with_ipfw-dummynet
The host is running FreeBSD 12.1-RELEASE-p3 GENERIC amd6.
I am using BSDRP-1.96-full-amd64-serial.img for the test and 4 VMs (three
are actually used in this test) are created with the command:
[root@host]#sh BSDRP-lab-bhyve.sh -n 4 -l 1
In the directory where I start the script I also have the image
BSDRP-1.96-full-amd64-serial.img
The network setup is:
VM1_iperf_server
(vtnet0:192.168.0.11)
|
|
(vtnet0:192.168.0.12)
VM2_router
(vtnet3:192.168.1.12)
|
|
(vtnet4:192.168.1.14)
VM4_iperf_client
Routes for remote networks are added in both VM1 and VM4:
in VM1
[root@VM1]#route add -net 192.168.1.0/24 192.168.0.12
in VM4
[root@VM4]#route add -net 192.168.0.0/24 192.168.1.12
With this configuration I can successfully ping VM1 from VM4 and vice versa.
Below are the settings for ipfw in VM2 (the router) and some statistics
during the iperf session (from all three VMs). I use tmux in the VM's,
hence you will see traces of it in the output.
The problem is that I cannot achieve speeds above 28Mbps from the iperf
client to the iperf server and vice versa. If I set the the 'bw' of the
pipe in ipfw above 30Mbps the throughput drops to ~21Mbps. I have tried
with 'bw' values above 100Mbps as well, with no success. I try only one
iperf session at a time. With two sessions I get about 25Mbps on a 50Mbps
pipe (see 'two sessions output for VM4).
Any ideas?
---===---
VM2 - the router
[root@router]~# systat -ifstat -match vtnet3,vtnet0 -scale mbit 1
/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
Load Average |
Interface Traffic Peak Total
vtnet3 in 22.060 Mb/s 27.505 Mb/s 3.494 GB
out 0.524 Mb/s 0.666 Mb/s 113.552 MB
vtnet0 in 0.526 Mb/s 0.675 Mb/s 113.583 MB
out 21.395 Mb/s 27.277 Mb/s 3.478 GB
────────────────────────────────────────────────────────────────────────────────
[root@router]~# top -CHIPS
last pid: 77493; load averages: 0.11, 0.11, 0.08 up 0+04:36:46
15:08:27
81 threads: 2 running, 57 sleeping, 22 waiting
CPU: 0.0% user, 0.0% nice, 0.4% system, 2.3% interrupt, 97.3% idle
Mem: 2044K Active, 22M Inact, 39M Wired, 13M Buf, 399M Free
Swap:
PID USERNAME PRI NICE SIZE RES STATE TIME CPU COMMAND
10 root 155 ki31 0 16K RUN 274:21 97.44% idle
0 root -92 - 0 368K - 0:50 0.95%
kernel{dummynet}
11 root -92 - 0 352K WAIT 0:11 0.73% intr{irq276:
vir
11 root -92 - 0 352K WAIT 0:07 0.56% intr{irq267:
vir
[VM2] 0:csh* "router.bsdrp.net" 15:10
25-Apr-20
[root@router]~# ipfw show
00100 1466715 106065974 queue 1 ip from any to any xmit vtnet3 out
00200 2693349 4025286410 queue 2 ip from any to any recv vtnet3 in
03000 4148195 4113509760 allow ip from any to any
65535 0 0 deny ip from any to any
[root@router]~#
[VM2] 0:csh* "router.bsdrp.net" 15:10
25-Apr-20
[root@router]~# cat /etc/firewall-bsdrp.rules
#!/bin/sh
fwcmd="/sbin/ipfw"
if ! kldstat -q -m dummynet; then
kldload dummynet
fi
# Flush out the list before we begin.
${fwcmd} -f flush
oif=vtnet3 # Output interface
bwu=50Mbit/s # Maximum upload speed
bwd=100Mbit/s # Maximum download speed
# Declare hard-limit of our links (2 because bidirectional)
${fwcmd} pipe 1 config bw $bwu
${fwcmd} pipe 2 config bw $bwd
# per-ip fair queueing
${fwcmd} queue 1 config pipe 1 mask src-ip 0xffffffff
${fwcmd} queue 2 config pipe 2 mask dst-ip 0xffffffff
# Assing outgoing traffic to upload queue and incoming to download queue
${fwcmd} add queue 1 ip from any to any xmit $oif out
${fwcmd} add queue 2 ip from any to any recv $oif in
# We don't want to block traffic, only shape some
${fwcmd} add 3000 allow ip from any to any
[root@router]~#
---===---
VM4 - iperf client
[root@vm4]#iperf3 -c 192.168.0.11 -t 600 -i 1 -f m -p 9003
.....
.....
[ 5] 37.00-38.00 sec 2.60 MBytes 21.8 Mbits/sec 3 42.5 KBytes
[ 5] 38.00-39.00 sec 2.75 MBytes 23.0 Mbits/sec 1 73.8 KBytes
[ 5] 39.00-40.00 sec 2.44 MBytes 20.4 Mbits/sec 3 59.6 KBytes
[ 5] 40.00-41.00 sec 2.49 MBytes 20.8 Mbits/sec 4 38.2 KBytes
[ 5] 41.00-42.00 sec 2.65 MBytes 22.3 Mbits/sec 1 69.6 KBytes
[ 5] 42.00-43.00 sec 2.48 MBytes 20.8 Mbits/sec 3 55.3 KBytes
[ 5] 43.00-44.00 sec 2.62 MBytes 22.0 Mbits/sec 3 41.0 KBytes
[ 5] 44.00-45.00 sec 2.49 MBytes 20.9 Mbits/sec 2 66.7 KBytes
[ 5] 45.00-46.00 sec 2.50 MBytes 21.0 Mbits/sec 3 52.4 KBytes
[ 5] 46.00-47.00 sec 2.65 MBytes 22.2 Mbits/sec 3 38.2 KBytes
[VM4] 0:iperf3* "router.bsdrp.net" 15:08
25-Apr-20
Two sessions output VM4 (output from tmux)
[ 5] 38.00-39.00 sec 576 KBytes 4.71 Mbits/sec 10 21.3 KBytes
[ 5] 39.00-40.00 sec 939 KBytes 7.69 Mbits/sec 11 21.2 KBytes
[ 5] 40.00-41.00 sec 621 KBytes 5.09 Mbits/sec 14 7.13 KBytes
[ 5] 41.00-42.00 sec 618 KBytes 5.06 Mbits/sec 16 8.50 KBytes
[ 5] 42.00-43.00 sec 527 KBytes 4.32 Mbits/sec 10 7.08 KBytes
[ 5] 43.00-44.00 sec 636 KBytes 5.21 Mbits/sec 14 7.08 KBytes
[ 5] 44.00-45.00 sec 286 KBytes 2.34 Mbits/sec 12 15.7 KBytes
[ 5] 45.00-46.00 sec 628 KBytes 5.14 Mbits/sec 14 15.7 KBytes
[ 5] 46.00-47.00 sec 650 KBytes 5.33 Mbits/sec 9 28.4 KBytes
[ 5] 47.00-48.00 sec 498 KBytes 4.08 Mbits/sec 19 7.08 KBytes
────────────────────────────────────────────────────────────────────────────────
[ 5] 50.00-51.00 sec 2.35 MBytes 19.7 Mbits/sec 3 66.8 KBytes
[ 5] 51.00-52.00 sec 2.10 MBytes 17.6 Mbits/sec 3 63.9 KBytes
[ 5] 52.00-53.00 sec 2.30 MBytes 19.3 Mbits/sec 3 55.3 KBytes
[ 5] 53.00-54.00 sec 2.14 MBytes 17.9 Mbits/sec 4 52.5 KBytes
[ 5] 54.00-55.00 sec 2.32 MBytes 19.4 Mbits/sec 3 56.8 KBytes
[ 5] 55.00-56.00 sec 2.30 MBytes 19.3 Mbits/sec 4 39.6 KBytes
[ 5] 56.00-57.00 sec 2.28 MBytes 19.1 Mbits/sec 3 44.0 KBytes
[ 5] 57.00-58.00 sec 2.41 MBytes 20.2 Mbits/sec 2 69.6 KBytes
[ 5] 58.00-59.00 sec 2.27 MBytes 19.0 Mbits/sec 5 7.08 KBytes
[ 5] 59.00-60.00 sec 2.20 MBytes 18.5 Mbits/sec 3 35.4 KBytes
---===---
VM1 - iperf server
[root@vm1]#iperf3 -s -p 9003
...
...
[ 5] 69.00-70.00 sec 2.50 MBytes 21.0 Mbits/sec
[ 5] 70.00-71.00 sec 2.61 MBytes 21.9 Mbits/sec
[ 5] 71.00-72.00 sec 2.48 MBytes 20.9 Mbits/sec
[ 5] 72.00-73.00 sec 2.55 MBytes 21.4 Mbits/sec
[ 5] 73.00-74.00 sec 2.75 MBytes 23.1 Mbits/sec
[ 5] 74.00-75.00 sec 2.50 MBytes 20.9 Mbits/sec
[ 5] 75.00-76.00 sec 2.52 MBytes 21.1 Mbits/sec
[ 5] 76.00-77.00 sec 2.66 MBytes 22.3 Mbits/sec
[ 5] 77.00-78.00 sec 2.42 MBytes 20.3 Mbits/sec
[ 5] 78.00-79.00 sec 2.56 MBytes 21.5 Mbits/sec
[VM1] 0:iperf3* "router.bsdrp.net" 15:09
25-Apr-20р.-20
---===---
HOST - the host machine
[root@host]#top -CHIPS
Last pid: 74122; load averages: 0.16, 0.40, 0.53 up 0+04:45:22
15:09:18
641 threads: 6 running, 617 sleeping, 17 waiting, 1 lock
CPU 0: 0.8% user, 0.0% nice, 3.9% system, 1.6% interrupt, 93.8% idle
CPU 1: 2.7% user, 0.0% nice, 3.9% system, 0.0% interrupt, 93.4% idle
CPU 2: 0.4% user, 0.0% nice, 2.3% system, 0.0% interrupt, 97.3% idle
CPU 3: 1.2% user, 0.0% nice, 4.3% system, 0.0% interrupt, 94.5% idle
Mem: 436M Active, 424M Inact, 5108K Laundry, 3780M Wired, 2970M Free
ARC: 1112M Total, 232M MFU, 824M MRU, 428K Anon, 6448K Header, 50M Other
793M Compressed, 1356M Uncompressed, 1.71:1 Ratio
Swap: 2048M Total, 2048M Free
PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND
11 root 155 ki31 0 64K RUN 2 264:55 93.95%
idle{idle: c
11 root 155 ki31 0 64K RUN 3 264:55 93.79%
idle{idle: c
11 root 155 ki31 0 64K RUN 0 266:45 93.36%
idle{idle: c
11 root 155 ki31 0 64K RUN 1 268:04 92.94%
idle{idle: c
22270 root 22 0 533M 192M vmidle 3 6:58 5.77%
bhyve{vcpu 0
50150 root 22 0 533M 193M CPU3 3 5:03 3.76%
bhyve{vcpu 0
16970 root 21 0 533M 193M vmidle 0 4:53 3.31%
bhyve{vcpu 0
66317 lyubd 20 0 2658M 399M select 3 0:03 1.87%
firefox{fire
77558 lyubd 20 0 102M 79M select 0 4:21 1.45%
Xorg{Xorg}
22270 root 20 0 533M 192M kqread 2 0:23 1.43%
bhyve{mevent
12 root -88 - 0 288K WAIT 0 2:45 1.36%
intr{irq265:
Best regards,
Lyubo
_______________________________________________
Bsdrp-users mailing list
Bsdrp-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bsdrp-users
