For me this minimal setup let me see the drop off of the network from
the
haproxy server.
2 jails, one with haproxy, one with nginx which is using the following
html
file to be served.
<!DOCTYPE html>
<html>
<head>
<title>Page Title</title>
</head>
<body>
<h1>My First Heading</h1>
<p>My first paragraph.</p>
</body>
</html>
From a remote machine i do a hey -h2 -n 10 -c 10 -z 300s
https://wp.test.nl
Then a ping on the jailhost to the haproxy shows the following
[ /] > ping 10.233.185.20
PING 10.233.185.20 (10.233.185.20): 56 data bytes
64 bytes from 10.233.185.20: icmp_seq=0 ttl=64 time=0.054 ms
64 bytes from 10.233.185.20: icmp_seq=1 ttl=64 time=0.050 ms
64 bytes from 10.233.185.20: icmp_seq=2 ttl=64 time=0.041 ms
<SNIP>
64 bytes from 10.233.185.20: icmp_seq=169 ttl=64 time=0.050 ms
64 bytes from 10.233.185.20: icmp_seq=170 ttl=64 time=0.154 ms
64 bytes from 10.233.185.20: icmp_seq=171 ttl=64 time=0.054 ms
64 bytes from 10.233.185.20: icmp_seq=172 ttl=64 time=0.039 ms
64 bytes from 10.233.185.20: icmp_seq=173 ttl=64 time=0.160 ms
64 bytes from 10.233.185.20: icmp_seq=174 ttl=64 time=0.045 ms
^C
--- 10.233.185.20 ping statistics ---
335 packets transmitted, 175 packets received, 47.8% packet loss
round-trip min/avg/max/stddev = 0.037/0.070/0.251/0.040 ms
ifconfig
vtnet0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0
mtu 1500
options=4c00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
ether 56:16:e9:80:5e:41
inet 87.233.191.146 netmask 0xfffffff0 broadcast 87.233.191.159
inet 87.233.191.156 netmask 0xffffffff broadcast 87.233.191.156
inet 87.233.191.155 netmask 0xffffffff broadcast 87.233.191.155
inet 87.233.191.154 netmask 0xffffffff broadcast 87.233.191.154
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vtnet1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
mtu 1500
options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
ether 56:16:2c:64:32:35
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
mtu
1500
ether 58:9c:fc:10:ff:82
inet 10.233.185.1 netmask 0xffffff00 broadcast 10.233.185.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair20a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000
member: epair18a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 15 priority 128 path cost 2000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
mtu
1500
ether 58:9c:fc:10:d9:1a
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vtnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 2000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
groups: pflog
epair18a: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric
0 mtu 1500
description: jail_web01
options=8<VLAN_MTU>
ether 02:77:ea:19:c7:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair20a: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric
0 mtu 1500
description: jail_haproxy
options=8<VLAN_MTU>
ether 02:9b:93:8c:59:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
jail.conf
# Global settings applied to all jails.
$domain = "test.nl";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.fstab = "/storage/jails/$name.fstab";
exec.system_user = "root";
exec.jail_user = "root";
mount.devfs;
sysvshm="new";
sysvsem="new";
allow.raw_sockets;
allow.set_hostname = 0;
allow.sysvipc;
enforce_statfs = "2";
devfs_ruleset = "11";
path = "/storage/jails/${name}";
host.hostname = "${name}.${domain}";
# Networking
vnet;
vnet.interface = "vnet0";
# Commands to run on host before jail is created
exec.prestart = "ifconfig epair${ip} create up description
jail_${name}";
exec.prestart += "ifconfig epair${ip}a up";
exec.prestart += "ifconfig bridge0 addm epair${ip}a up";
exec.created = "ifconfig epair${ip}b name vnet0";
# Commands to run in jail after it is created
exec.start += "/bin/sh /etc/rc";
# commands to run in jail when jail is stopped
exec.stop = "/bin/sh /etc/rc.shutdown";
# Commands to run on host when jail is stopped
exec.poststop = "ifconfig bridge0 deletem epair${ip}a";
exec.poststop += "ifconfig epair${ip}a destroy";
persist;
web01 {
$ip = 18;
}
haproxy {
$ip = 20;
mount.fstab = "";
path = "/storage/jails/${name}";
}
pf.conf
#######################################################################
ext_if="vtnet0"
table <bruteforcers> persist
table <torlist> persist
table <ssh-trusted> persist file "/usr/local/etc/pf/ssh-trusted"
table <custom-block> persist file "/usr/local/etc/pf/custom-block"
table <jailnetworks> { 10.233.185.0/24, 192.168.10.0/24 }
icmp_types = "echoreq"
junk_ports="{ 135,137,138,139,445,68,67,3222,17500 }"
# Log interface
set loginterface $ext_if
# Set limits
set limit { states 40000, frags 20000, src-nodes 20000 }
scrub on $ext_if all fragment reassemble no-df random-id
# ---- Nat jails to the web
binat on $ext_if from 10.233.185.15/32 to !10.233.185.0/24 ->
87.233.191.156/32 # saltmaste
binat on $ext_if from 10.233.185.20/32 to !10.233.185.0/24 ->
87.233.191.155/32 # haproxy
binat on $ext_if from 10.233.185.22/32 to !10.233.185.0/24 ->
87.233.191.154/32 # web-comb
nat on $ext_if from <jailnetworks> to any -> ($ext_if:0)
# ---- First rule obligatory "Pass all on loopback"
pass quick on lo0 all
pass quick on bridge0 all
pass quick on bridge1 all
# ---- Block TOR exit addresses
block quick proto { tcp, udp } from <torlist> to $ext_if
# ---- Second rule "Block all in and pass all out"
block in log all
pass out all keep state
# IPv6 pass in/out all IPv6 ICMP traffic
pass in quick proto icmp6 all
# Pass all lo0
set skip on lo0
############### FIREWALL
###############################################
# ---- Block custom ip's and logs
block quick proto { tcp, udp } from <custom-block> to $ext_if
# ---- Jail poorten
pass in quick on { $ext_if } proto tcp from any to 10.233.185.22 port
{
smtp 80 443 993 995 1956 } keep state
pass in quick on { $ext_if } proto tcp from any to 10.233.185.20 port
{
smtp 80 443 993 995 1956 } keep state
pass in quick on { $ext_if } proto tcp from any to 10.233.185.15 port
{
4505 4506 } keep state
# ---- Allow ICMP
pass in inet proto icmp all icmp-type $icmp_types keep state
pass out inet proto icmp all icmp-type $icmp_types keep state
pass in quick on $ext_if inet proto tcp from any to $ext_if port { 80,
443
} flags S/SA keep state
pass in quick on $ext_if inet proto tcp from <ssh-trusted> to $ext_if
port
{ 4505 4506 } flags S/SA keep state
block log quick from <bruteforcers>
pass quick proto tcp from <ssh-trusted> to $ext_if port ssh flags S/SA
keep
state
This is as minimal i can get it.
Hope this helps.
regards,
Johan Hendriks
Op za 12 mrt. 2022 om 02:10 schreef Kristof Provost <k...@freebsd.org>:
On 11 Mar 2022, at 18:55, Michael Gmelin wrote:
On 12. Mar 2022, at 01:21, Kristof Provost <k...@freebsd.org> wrote:
On 11 Mar 2022, at 17:44, Johan Hendriks wrote:
On 09/03/2022 20:55, Johan Hendriks wrote:
The problem:
I have a FreeBSD 14 machine and a FreeBSD 13-stable machine, both
running the same jails just to test the workings.
The jails that are running are a salt master, a haproxy jail, 2
webservers, 2 varnish servers, 2 php jails one for php8.0 and one
with 8.1.
All the jails are connected to bridge0 and all the jails use vnet.
I believe this worked on an older 14-HEAD machine, but i did not
do a
lot with it back then, and when i started testing again and after
updating
the OS i noticed that one of the varnish jails lost it's network
connection
after running for a few hours. I thought it was just something on
HEAD so
never really looked at it. But later on when i start using the jails
again
and testing a test wordpress site i noticed that with a simple load
test my
haproxy jail within one minute looses it's network connection. I see
nothing in the logs, on the host and on the jail.
From the jail i can not ping the other jails or the IP adres of
the
bridge. I can however ping the jails own IP adres. From the host i
can also
not ping the haproxy jail IP adres. If i start a tcpdump on the
epaira
interface from the haproxy jail i do see the packets arrive but not
in the
jail.
I used ZFS to send all the jails to a 13-STABLE machine and
copied
over the jail.conf file as well as the pf.conf file and i saw the
same
behavior.
Then i tried to use 13.0-RELEASE-p7 and on that machine i do not
see
this happening. There i can stress test the machine for 10 minutes
without
a problem but on 14-HEAD and 13-STABLE within a minute the jail's
network
connection fails and only a restart of the jail brings it back online
to
exhibit the same behavior if i start a simple load test which it
should
handle nicely.
One of the jail hosts is running under VMWARE and the other is
running under Ubuntu with KVM. The 13.0-RELEASE-p7 jail host is
running
under Ubuntu with KVM
Thank you for your time.
regards
Johan
I did some bisecting and the latest commit that works on FreeBSD
13-Stable is 009a56b2e
Then the commit 2e0bee4c7 if_epair: implement fanout and above is
showing the symptoms described above.
Interestingly I cannot reproduce stalls in simple epair setups.
It would be useful if you could reduce the setup with the problem
into
a minimal configuration so we can figure out what other factors are
involved.
If there are clear instructions on how to reproduce, I’m happy to
help
experimenting (I’m relying heavily on epair at this point).
@Kristof: Did you try on bare metal or on vms?
Both.
Kristof
