Hello, I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5 host having a second NIC and vnt jails attached to that second NIC (basically, the host is a recent Xigmanas with Bastille jails, but the issue also occurs on a vanilla FreeBSD 12.3).
The host is compromised of two NICs, em0 (management only) and igb0 (service/jails). Both, the server and the jails as well as the igb0 interface are residing on the same network, but both NICs are connected to two different ports on a switch, to which we do not have access (part of the campus infrastructure). Both NICs are attached with a IPv4 of the same network, the host is listening on both NICs for services, i.e. port 22 for ssh. No problem to connect to both(!) addresses via ssh. igb0 is member of an if_bridge. The box also hosts a bunch of vnet jails, each jail does have an if_epair created via "jib" and these vnet epairs are members of the bridge, to which ifb0 is also member. Problem: while any service bound to NIC igb0/IPv4 residing on igb0 is accessible flawlessly, accessing an jail is almost impossible. Pinging a jail does work after a while the ping initiating host has been waiting, in ery rare situations someone can access the sshd of the jail, but any access of that kind is highly erratic. From 5 jails, at most two are responding to pings, the other don't and it is non-deterministic which host will respond. Following some advices found on the web, the following sysctl settings are provided to if_bridge: device if_bridge net.link.bridge.ipfw: 0 net.link.bridge.allow_llz_overlap: 0 net.link.bridge.inherit_mac: 0 net.link.bridge.log_stp: 0 net.link.bridge.pfil_local_phys: 0 net.link.bridge.pfil_member: 0 net.link.bridge.ipfw_arp: 0 net.link.bridge.pfil_bridge: 0 net.link.bridge.pfil_onlyip: 0 We do not have access to the switch the box is connected to, so I don't have access to any logs revealing a problem either to a conceptual misunderstanding of networking of mine and so a misconfiguration or a probelm with Layer 2 or the switches themselfes. I'd like to ask whether someone has a similar setup up and running and could report this - or give a hint of the problem I possibly made (igb0 is attached to an IPv4 AND is member of an if_brige on which IPv4 attached vnet jails are residing). We have also already setup another "similar" scenarion with the same FreeBSD 12.3-p5 version and also two NICs, but our "service/jail" NIC is part of a different IPv4 network and the NIC is attached to a different switch (to which we have full access). Thanks in advance, O. Hartmann
