On Tue, 10 May 2022 21:21:29 +0200
FreeBSD User <free...@walstatt-de.de> wrote:
> Hello,
>
> I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5 host having a
> second NIC
> and vnt jails attached to that second NIC (basically, the host is a recent
> Xigmanas with
> Bastille jails, but the issue also occurs on a vanilla FreeBSD 12.3).
>
> The host is compromised of two NICs, em0 (management only) and igb0
> (service/jails).
> Both, the server and the jails as well as the igb0 interface are residing on
> the same
> network, but both NICs are connected to two different ports on a switch, to
> which we do
> not have access (part of the campus infrastructure).
>
> Both NICs are attached with a IPv4 of the same network, the host is listening
> on both
> NICs for services, i.e. port 22 for ssh. No problem to connect to both(!)
> addresses via
> ssh. igb0 is member of an if_bridge. The box also hosts a bunch of vnet
> jails, each jail
> does have an if_epair created via "jib" and these vnet epairs are members of
> the bridge,
> to which ifb0 is also member.
>
> Problem: while any service bound to NIC igb0/IPv4 residing on igb0 is
> accessible
> flawlessly, accessing an jail is almost impossible. Pinging a jail does work
> after a
> while the ping initiating host has been waiting, in ery rare situations
> someone can
> access the sshd of the jail, but any access of that kind is highly erratic.
> From 5
> jails, at most two are responding to pings, the other don't and it is
> non-deterministic
> which host will respond.
>
> Following some advices found on the web, the following sysctl settings are
> provided to
> if_bridge:
>
> device if_bridge
> net.link.bridge.ipfw: 0
> net.link.bridge.allow_llz_overlap: 0
> net.link.bridge.inherit_mac: 0
> net.link.bridge.log_stp: 0
> net.link.bridge.pfil_local_phys: 0
> net.link.bridge.pfil_member: 0
> net.link.bridge.ipfw_arp: 0
> net.link.bridge.pfil_bridge: 0
> net.link.bridge.pfil_onlyip: 0
>
> We do not have access to the switch the box is connected to, so I don't have
> access to
> any logs revealing a problem either to a conceptual misunderstanding of
> networking of
> mine and so a misconfiguration or a probelm with Layer 2 or the switches
> themselfes.
>
> I'd like to ask whether someone has a similar setup up and running and could
> report this
> - or give a hint of the problem I possibly made (igb0 is attached to an IPv4
> AND is
> member of an if_brige on which IPv4 attached vnet jails are residing).
>
> We have also already setup another "similar" scenarion with the same FreeBSD
> 12.3-p5
> version and also two NICs, but our "service/jail" NIC is part of a different
> IPv4
> network and the NIC is attached to a different switch (to which we have full
> access).
>
> Thanks in advance,
>
> O. Hartmann
>
On FreeBSD 12.3-p5, em0 seems to suffer from a bug regarding hardware chesum
support, I
see a lot of :
[...]
Flags [.], cksum 0xe826 (incorrect -> 0x606b), seq 101269476:101270000, ack
5077, win
257, options [nop,nop,TS val 2618589801 ecr 3610923914], length 524
Disabling TXCSUM via "ifconfig em0 -txcsum" renders incorrect -> correct.
em0 is:
em0@pci0:0:25:0: class=0x020000 card=0x20528086 chip=0x153b8086 rev=0x04
hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Connection I217-V'
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xf7d00000, size 131072, enabled
bar [14] = type Memory, range 32, base 0xf7d35000, size 4096, enabled
bar [18] = type I/O Port, range 32, base 0xf080, size 32, enabled
cap 01[c8] = powerspec 2 supports D0 D3 current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 13[e0] = PCI Advanced Features: FLR TP
I remember faintly that there was an issue when I used to use FBSD 12
