netstat -I ix0
Hi All On my FreeBSD 12.3-STABLE r372089 GENERIC amd64 I have a huge amount of RX errors on ix0: = # netstat -n -I ix0 NameMtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll ix01500 a0:36:9f:1d:80:80 2336589698003 774784702245 0 1816532334151 0 0 = = # sysctl dev.ix.0 | grep errs dev.ix.0.mac_stats.checksum_errs: 774729511713 dev.ix.0.mac_stats.rec_len_errs: 31682 dev.ix.0.mac_stats.byte_errs: 14414734 dev.ix.0.mac_stats.ill_errs: 3034419 dev.ix.0.mac_stats.crc_errs: 54705100 dev.ix.0.mac_stats.rx_errs: 774787585085 = Are these errors phy/hw problems e.x. wrong SFP+ hardware/firmware, too high / too low RX levels, broken patch cord? Or may it be some OS errors? Thanks for any advise! -- CU, Victor Gamov
Re: netstat -I ix0
So, it's not a problem but counter only bug? On 24.10.2022 23:29, Cristian Cardoso wrote: Hi After I upgraded my hardware from 12.2 to 12.3-RELEASE, I also started getting error alerts on my FreeBSD monitors via snmp. People asked me to upgrade the version to 13, I did but the problem persisted, I changed SFP modules and fiber cable, even so the error persisted. This was a system change that created more counters from the implementation of a new revision in FreeBSD. I reported my bug at the time: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=262093 From what you could see in the url that Konrad passed, a fix is already being tested. Em seg., 24 de out. de 2022 às 17:21, Konrad Kręciwilk mailto:konrad.kreciw...@korbank.pl>> escreveu: Hello, It seems your issues is related with https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=266048 Regards, KK W dniu 2022-10-24 17:59, Victor Gamov napisał(a): > Hi All > > On my FreeBSD 12.3-STABLE r372089 GENERIC amd64 I have a huge amount > of RX errors on ix0: > > = > # netstat -n -I ix0 > Name Mtu Network Address Ipkts Ierrs Idrop Opkts > Oerrs Coll > ix0 1500 a0:36:9f:1d:80:80 2336589698003 774784702245 > 0 1816532334151 0 0 > = > > = > # sysctl dev.ix.0 | grep errs > dev.ix.0.mac_stats.checksum_errs: 774729511713 > dev.ix.0.mac_stats.rec_len_errs: 31682 > dev.ix.0.mac_stats.byte_errs: 14414734 > dev.ix.0.mac_stats.ill_errs: 3034419 > dev.ix.0.mac_stats.crc_errs: 54705100 > dev.ix.0.mac_stats.rx_errs: 774787585085 > = > > > Are these errors phy/hw problems e.x. wrong SFP+ hardware/firmware, > too high / too low RX levels, broken patch cord? Or may it be some OS > errors? > > > Thanks for any advise! -- CU, Victor Gamov
Re: netstat -I ix0
I check the patch and looks like huge errors counter due huge UDP traffic in my case. I'll try to upgared and check it. Thanks for everyone! On 26.10.2022 14:34, Franco Fichtner wrote: On 26. Oct 2022, at 13:11, Cristian Cardoso wrote: Despite the error increments in the interface, I used the equipment for a month until I migrated my things from the router I had and even with a large error increment I didn't have any problems with packet loss or routing. The cards exhibiting this input error are counting (valid) zero-checksum UDP packets as input errors despite handling them just fine. This is a documented errata. I think the whole "problem" started when Intel opted to push all interface errors into the input error counter in FreeBSD 13 which surfaced this non-issue. -- CU, Victor Gamov
Re: finding optimal ipfw strategy
Hi All Up this thread after few years :-) Now I have following HW/SW setup: - FreeBSD 12.3-STABLE r372089 GENERIC amd64 - Xeon(R) CPU E5-2470 v2 @ 2.40GHz - ix0 hardware - about 10-15 vlans like "vlan: 100 vlanpcp: 0 parent interface: ix0" - and all vlans are bridged via bridgeX - about 200 multicast streams (200K packets / 2G multicast traffic incoming via one vlan100) - ipfw to allow/deny incoming/outgoing traffic on any vlanX: -- net.link.bridge.ipfw=1 -- to enable layer2 filtering (ARP) -- net.link.bridge.ipfw_arp=1 -- to filter ARP -- net.link.bridge.pfil_bridge=0 -- no filtering on bridgeX -- net.link.bridge.pfil_member=1 -- to filter in/out on bridged vlans -- net.link.ether.ipfw=1 -- to filter inter-vlan non IP packets like STP/CDP/etc IPFW optimized strategy based on early messages but still in research :-) Some things are working fine but some I still can't figure out. As documented at ipfw(8) (part "PACKET FLOW") [bdg_forward] at lower layer so ALL packets bridged unconditionaly and I can't drop undesired incoming packets based on incoming vlan (like "deny ip from any to any in recv vlanX") _before_ they bridged ? Then, if packet bridged all packets copied to all bridged vlans? And only in [ip_output] undesired outgoing packets will be dropped ? Is it possible to drop incoming packets _before_ they bridged? -- CU, Victor Gamov
FreeBSD-13 systat -ifstat ix0
Hi All Strange counters on ix0 occured: - FreeBSD 13.1-STABLE #0 stable/13-n252734-56533712694 GENERIC - ix0 has about 10 VLANs and ipfw loaded and some traffic filtered on vlans. systat -ifstat -scale mbit -match ix0 shows me about 86.486Mbit input but real traffic on switch about 3Gbit TX Why this counters so different? -- CU, Victor Gamov
FRR + OSPF/BGP ECMP
Hi All Does FreeBSD support ECMP while using FRR + OSPF/BGP? -- CU Victor Gamov
Re: FRR + OSPF/BGP ECMP
Hi All Olivier, can you explain please why this function turned off by default? I think it's really useful feature in many cases. On 10/12/2022 16:09, Olivier Cochard-Labbé wrote: On Fri, Dec 9, 2022 at 8:56 PM Oleksandr Kryvulia mailto:shur...@shurik.kiev.ua>> wrote: It is enabled by default since eb0b1b33d5af <https://cgit.freebsd.org/src/diff/sys/net/route/route_ctl.c?id=eb0b1b33d5af4e81ee77732dffc77634e57a5879&h=main> I was referring to the MULTIPATH option on the net/frr8 port (not FreeBSD kernel) which is not enabled by default. -- CU Victor Gamov
Re: FRR + OSPF/BGP ECMP
On 10/12/2022 16:09, Olivier Cochard-Labbé wrote: On Fri, Dec 9, 2022 at 8:56 PM Oleksandr Kryvulia mailto:shur...@shurik.kiev.ua>> wrote: It is enabled by default since eb0b1b33d5af <https://cgit.freebsd.org/src/diff/sys/net/route/route_ctl.c?id=eb0b1b33d5af4e81ee77732dffc77634e57a5879&h=main> I was referring to the MULTIPATH option on the net/frr8 port (not FreeBSD kernel) which is not enabled by default. hmm. My installation: - FreeBSD 13.1-STABLE #0 stable/13-n253133-b51ee7ac252c - frr8-8.4 from packages And I have following at FRR: = B>* 172.16.30.82/32 [200/0] via 10.5.7.21, lan, weight 1, 00:00:53 * via 10.5.7.22, lan, weight 1, 00:00:53 = at host: = 172.16.30.82 10.5.7.21 UGH1 18 1500lan 172.16.30.82 10.5.7.22 UGH1 17 1500lan = So, OS-related MULTIPATH is "turned on" _and_ FRR MULTIPATH is "turned on" but freshports says "MULTIPATH=off: Enable multipath function" Or some misunderstanding here? Thanks! -- CU Victor Gamov
Re: FRR + OSPF/BGP ECMP
On 11/12/2022 20:59, Olivier Cochard-Labbé wrote: On Sat, Dec 10, 2022 at 4:44 PM Victor Gamov <mailto:v...@otcnet.ru>> wrote: So, OS-related MULTIPATH is "turned on" _and_ FRR MULTIPATH is "turned on" but freshports says "MULTIPATH=off: Enable multipath function" Or some misunderstanding here? I was misunderstanding :-) I wasn't expecting net/frr8 to support multipath without the --enable-multipath configuration option. But re-reading the FRR documentation, multipath is always enabled (and the default maximum mpath is 16): https://github.com/FRRouting/frr/blob/frr-8.4.1/doc/user/installation.rst So I will fix the port Makefile. Thanks for your question: The result will be an improved net/frr port :-) Thanks Olivier! -- CU Victor Gamov
ECMP, DF-bit and ICMP "Fragmentation needed"
Hi All I have following scheme: - LAN segment 10.5.8.0/24 with router1 (10.5.8.1) and MTU=1500 - two hosts at LAN segment host21 (10.5.8.21) and host22 (10.5.8.22) - host21 and host22 has VIP=172.16.110.30 configured as LAN-interface alias - host21 and host22 ha BGP peering with router1 and announce VIP to router1 - hostX somewhere at intranet - ipsec-tunnel with MTU=1400 ECMP works fine and traffic from other segments to VIP is balanced between host21+host22 by router1. The problem is: when host21 and/or host22 send large packet with DF-bit using VIP as source then ipsec-router sends ICMP "Fragmentation needed" and then this ICMP is _always_ sent to only host22 by router1. I think it may be hard or impossible to find proper VIP-owner to send this ICMP. Is it possible to propagate such ICMP to all VIP-owners in router1 routing-table? Or may some data from ICMP message be used to properly calculate ECMP-hash to find a real VIP-owner which must receive this ICMP? Thanks! -- CU, Victor Gamov
Re: ECMP, DF-bit and ICMP "Fragmentation needed"
On Mon, 27 Feb 2023 at 13:57, Alexander Chernikov wrote: > > > > On 26 Feb 2023, at 12:07, Victor Gamov wrote: > > > > Hi All > > > > I have following scheme: > > - LAN segment 10.5.8.0/24 with router1 (10.5.8.1) and MTU=1500 > > - two hosts at LAN segment host21 (10.5.8.21) and host22 (10.5.8.22) > > - host21 and host22 has VIP=172.16.110.30 configured as LAN-interface > alias > > - host21 and host22 ha BGP peering with router1 and announce VIP to > router1 > > - hostX somewhere at intranet > > - ipsec-tunnel with MTU=1400 > > > > ECMP works fine and traffic from other segments to VIP is balanced > between host21+host22 by router1. > > > > The problem is: > > when host21 and/or host22 send large packet with DF-bit using VIP as > source then ipsec-router sends ICMP "Fragmentation needed" and then this > ICMP is _always_ sent to only host22 by router1. > > > > I think it may be hard or impossible to find proper VIP-owner to send > this ICMP. Is it possible to propagate such ICMP to all VIP-owners in > router1 routing-table? Or may some data from ICMP message be used to > properly calculate ECMP-hash to find a real VIP-owner which must receive > this ICMP? > Generally it’s pretty hard to do. The path may go through the multiple > routers which has it own hash calculation + seed to avoid the traffic > polarisation. Personally I’d suggest doing some sort of ICMP replication on > either the source node or the hosts. > Hi Alexander! Thanks for your reply. In my scheme router1 can replicate such ICMP to all VIP-owners. And only router1 knows about both host21+host22 peers -- for all other network devices this VIP is behind router1. -- CU, Victor Gamov
Packet forwarding stooped when Strongswan install IPsec policy
/show_bug.cgi?id=255678 and https://github.com/strongswan/strongswan/issues/910 and its looks like strongswan/FreeBSD integration issue. I'll appreciate any advice. Thanks! -- CU, Victor Gamov
Re: Packet forwarding stooped when Strongswan install IPsec policy
After more investigation tunnel up and worked:
etc/strongswan.d/charon.conf:
=
install_routes = no
=
This was disabled at first time but lost during configuration experiments.
etc/ipsec.conf:
=
conn pop4-to-pop12-routed
installpolicy = no
=
On Sat, 14 Oct 2023 at 13:25, Victor Gamov wrote:
> Hi All
>
> I have FreeBSD 13.2-STABLE stable/13-n255939-b9da47180fd6 GENERIC amd64
> machine with strongswan-5.9.11_2 installed by pkg.
>
> When routed ipsec is up all outgoing packets forwarded into ipsec-tunnel
> so networking is immediately fails.
>
> FreeBSD config:
> =
> net.fibs=4
> net.inet.ip.forwarding=1
> =
>
>
> ifconfig ipsec10121
> =
> ipsec10121: flags=8050 metric 0 mtu 1400
> description: PoP-12
> tunnel inet 1.1.1.2 --> 2.2.2.2
> inet 172.16.110.129 --> 172.16.110.130 netmask 0xfffc
> groups: ipsec
> reqid: 10121
> nd6 options=29
> =
>
>
> strongswan etc/ipsec.conf:
> =
> conn pop4-to-pop12-routed
> # also = tmpl_route_based
> left = 1.1.1.2
> right = 2.2.2.2
> leftsubnet = 0.0.0.0/0
> rightsubnet = 0.0.0.0/0
> reqid = 10121
> type = tunnel
> authby = psk
> keyexchange = ikev2
> ike = aes256-sha256-modp3072,aes256-sha256-modp3072
> esp = aes256-sha256-modp3072,aes256-sha256-modp3072
> ikelifetime = 28800
> mobike = no
> lifetime = 3600
> dpdaction = restart
> dpddelay = 30s
> auto = start
> =
>
>
> strongswan etc/strongswan.d/charon/kernel-pfkey.conf:
> =
> kernel-pfkey {
> load = yes
> # route_via_internal = no
> }
> =
>
>
> route -n monitor
> =
> got message of size 272 on Sat Oct 14 12:39:39 2023
> RTM_GET: Report Metrics: len 272, pid: 49695, seq 1, errno 0,
> flags:
> locks: inits:
> sockaddrs:
> 0.0.0.0 1.1.1.1 0.0.0.0 vlan200:48.dc.2d.6.4f.f4 1.1.1.2
>
> got message of size 200 on Sat Oct 14 12:39:39 2023
> RTM_GET: Report Metrics: len 200, pid: 49695, seq 2, errno 0,
> flags:
> locks: inits:
> sockaddrs:
> 0.0.0.0 1.1.1.1 0.0.0.0
>
> got message of size 256 on Sat Oct 14 12:39:39 2023
> RTM_ADD: Add Route: len 256, pid: 49695, seq 3, errno 0,
> flags:
> locks: inits:
> sockaddrs:
> 2.2.2.2 1.1.1.1 vlan200:48.dc.2d.6.4f.f4 1.1.1.2
>
> got message of size 272 on Sat Oct 14 12:39:39 2023
> RTM_ADD: Add Route: len 272, pid: 49695, seq 5, errno 0,
> flags:
> locks: inits:
> sockaddrs:
> 128.0.0.0 1.1.1.1 128.0.0.0 vlan200:48.dc.2d.6.4f.f4 1.1.1.2
>
> got message of size 272 on Sat Oct 14 12:39:39 2023
> RTM_ADD: Add Route: len 272, pid: 49695, seq 4, errno 0,
> flags:
> locks: inits:
> sockaddrs:
> 0.0.0.0 1.1.1.1 128.0.0.0 vlan200:48.dc.2d.6.4f.f4 1.1.1.2
> =
>
>
> netstat -r -nW4:
> =
> Routing tables
>
> Internet:
> DestinationGatewayFlags Nhop#Mtu Netif
> Expire
> 0.0.0.0/1 195.34.58.166 US 12 1500vlan200
> default195.34.58.166 UGS 6 1500vlan200
> 10.4.102.128/31link#8 U 8 1500 vlan22
> 10.4.102.129 link#8 UHS 7 16384lo0
> 31.131.95.64/27127.0.0.1 U1B 9 16384lo0
> 46.243.226.103 195.34.58.166 UGHS 10 1500vlan200
> 127.0.0.1 link#5 UHS 1 16384lo0
> 128.0.0.0/1195.34.58.166 US 12 1500vlan200
> 172.16.110.12/31 link#4 U 2 1500 ixl3
> 172.16.110.13 link#4 UHS 3 16384lo0
> 172.16.110.129 link#11UHS11 16384lo0
> 195.34.58.166/31 link#7 U 4 1500vlan200
> 195.34.58.167 link#7 UHS 5 16384lo0
> =
>
>
> netstat -o -nW4
> =
> Nexthop data
>
> Internet:
> Idx Type IFAGateway Flags Use
> Mtu Netif Addrif Refcnt Prepend
> 1 v4/resolve 127.0.0.1 lo0/resolveHS 1366
> 16384lo0 2
> 2 v4/resolve 172.16.110.13 ixl3/resolve 0
> 1500 ixl3 2
> 3 v4/resolve 127.0.0.1 lo0/resolveHS0
> 16384lo0 ixl3 2
> 4 v4/resolve 195.34.58.167 vlan200/resolve 51749
> 1500vlan200 4
> 5 v4/resolve 127.0.0.1 lo0/resolveHS0
> 16384lo0 vlan200 2
> 6v4/gw 195.34.58.167 195.34.58.166 GS37902
> 1500vlan200 2
> 7
freebsd as DMVPN spoke
Hi All I have FreeBSD-based router with 3 uplinks. Everything works fine but new tasks appears. Is it possible to use FreeBSD as spoke for Cisco-based DMVPN hub for hub-and-spoke model? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
multiple if_ipsec
Hi All
I have FreeBSD box (11.1-STABLE FreeBSD 11.1-STABLE #0 r327786) and
simple configuration with two if_ipsec configured like
=
ipsec25: flags=8051 metric 0 mtu 1400
description: -so: Sofy
tunnel inet IP-FreeBSD --> IP-Cisco-RTR-1
inet 10.10.98.6 --> 10.10.98.5 netmask 0xfffc
nd6 options=29
reqid: 25
groups: ipsec
ipsec30: flags=8051 metric 0 mtu 1400
description: -so: Kurskaya
tunnel inet IP-FreeBSD --> IP-Cisco-RTR-2
inet 10.10.98.1 --> 10.10.98.2 netmask 0xfffc
nd6 options=29
reqid: 30
groups: ipsec
=
IPsec started with "flush; spdflush;" only config. FreeBSD setkey -DP
reports (IPv6 skipped)
=
0.0.0.0/0[any] 0.0.0.0/0[any] any
in ipsec
esp/tunnel/IP_Cisco_RTR_1-IP_FreeBSD/unique:25
spid=9 seq=7 pid=94296 scope=ifnet ifname=ipsec25
refcnt=1
0.0.0.0/0[any] 0.0.0.0/0[any] any
in ipsec
esp/tunnel/IP_Cisco_RTR_2-IP_FreeBSD/unique:30
spid=13 seq=5 pid=94296 scope=ifnet ifname=ipsec30
refcnt=1
0.0.0.0/0[any] 0.0.0.0/0[any] any
out ipsec
esp/tunnel/IP_FreeBSD-IP_Cisco_RTR_1-IP/unique:25
spid=10 seq=3 pid=94296 scope=ifnet ifname=ipsec25
refcnt=1
0.0.0.0/0[any] 0.0.0.0/0[any] any
out ipsec
esp/tunnel/IP_FreeBSD-IP_Cisco_RTR_2-IP/unique:30
spid=14 seq=1 pid=94296 scope=ifnet ifname=ipsec30
refcnt=1
=
Then racoon.conf (from security/ipsec-tools-0.8.2_2) configured like
=
remote "kur" {
exchange_mode main;
doi ipsec_doi;
situation identity_only;
my_identifier address IP-FreeBSD;
peers_identifier address IP-Cisco-RTR-2;
verify_identifier on;
nonce_size 16;
lifetime time 240 min; # sec,min,hour
initial_contact on;
#support_mip6 on;
support_proxy on;
proposal_check obey;# obey, strict or claim
proposal {
encryption_algorithm 3des;
hash_algorithm sha1;
authentication_method pre_shared_key ;
dh_group 2 ;
}
}
remote "sofy" {
exchange_mode main;
doi ipsec_doi;
situation identity_only;
my_identifier address IP-FreeBSD;
peers_identifier address IP-Cisco-RTR-1;
verify_identifier on;
nonce_size 16;
lifetime time 240 min; # sec,min,hour
initial_contact on;
#support_mip6 on;
support_proxy on;
proposal_check obey;# obey, strict or claim
proposal {
encryption_algorithm 3des;
hash_algorithm sha1;
authentication_method pre_shared_key ;
dh_group 2 ;
}
}
sainfo anonymous {
pfs_group 2;
lifetime time 24 hour;
encryption_algorithm aes;
authentication_algorithm hmac_sha1, hmac_md5;
compression_algorithm deflate;
}
=
All local SA configured and established and remote side (Cisco routers)
report SA established too.
But traffic goes via only one ipsec-interface.
Can anybody explain where is my problem:
- FreeBSD misconfig
- racoon misconfig
- racoon not support multiple ipsec configuration
- something else
Thanks
--
CU,
Victor Gamov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: multiple if_ipsec
On 20/04/2018 13:04, Andrey V. Elsukov wrote: On 20.04.2018 11:17, Victor Gamov wrote: All local SA configured and established and remote side (Cisco routers) report SA established too. But traffic goes via only one ipsec-interface. If you have all SAs established, you probably need to check your routing configuration. Or at least test that addresses configured on the ipsecXX interfaces are reachable. More correct problem is: last configured ipsec interface tx/rx traffic only. For my example: - ping from 10.10.98.1 to 10.10.98.2 via ipsec30 is OK - ping from 10.10.98.2 to 10.10.98.1 via ipsec30 is OK - ping from 10.10.98.5 (Cisco) to 10.10.98.6 via ipsec25 -- no responses, but I see ESP traffic on external interface and (!!!) ICMP-reply from 10.10.98.5 to 10.10.98.6 on ipsec25 (but no ICMP-request on ipsec25 !!!) - ping from 10.10.98.6 to 10.10.98.5 via ipsec25 -- no responses, I see ICMP-request on ipsec25 but no ESP-traffic on external interface Any suggestion? -- С уважением, Гамов Виктор ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: multiple if_ipsec
On 20/04/2018 19:42, Andrey V. Elsukov wrote: On 20.04.2018 18:48, Victor Gamov wrote: More correct problem is: last configured ipsec interface tx/rx traffic only. For my example: - ping from 10.10.98.1 to 10.10.98.2 via ipsec30 is OK - ping from 10.10.98.2 to 10.10.98.1 via ipsec30 is OK - ping from 10.10.98.5 (Cisco) to 10.10.98.6 via ipsec25 -- no responses, but I see ESP traffic on external interface and (!!!) ICMP-reply from 10.10.98.5 to 10.10.98.6 on ipsec25 (but no ICMP-request on ipsec25 !!!) - ping from 10.10.98.6 to 10.10.98.5 via ipsec25 -- no responses, I see ICMP-request on ipsec25 but no ESP-traffic on external interface This looks like you don't have outbound SA for ipsec25 interface. If you run `netstat -w1 -I ipsec25` and ping 10.10.98.5, there should be output errors. `setkey -D` should have SA: IP-FreeBSD IP-Cisco-RTR-1 esp mode=tunnel spi= reqid=25 .. . state=mature Do you have it? Yes, I have all SA -- two for every ipsec-interface. And no errors at `netstat -w1 -I ipsec25` while ping 10.10.98.5, only output bytes counter show 84 bytes per sec (one for ICMP-request) When I change ipsec-interfaces creation order then only last created interface worked fine again and previously configured interfaces does not work. And very interesting fact: when I ping from remote 10.10.98.5 for example to FreeBSD 10.10.98.6 then no ICMP-request coming over ipsec-interface but ICMP-reply outgoing via this ipsec-interface (but not delivered to 10.10.98.5) Any ideas? -- С уважением, Гамов Виктор ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: multiple if_ipsec
On 23/04/2018 14:13, Andrey V. Elsukov wrote: On 21.04.2018 19:16, Victor Gamov wrote: When I change ipsec-interfaces creation order then only last created interface worked fine again and previously configured interfaces does not work. And very interesting fact: when I ping from remote 10.10.98.5 for example to FreeBSD 10.10.98.6 then no ICMP-request coming over ipsec-interface but ICMP-reply outgoing via this ipsec-interface (but not delivered to 10.10.98.5) Any ideas? I'm lack of any ideas. For further debugging I need to see the output of # sysctl net. | grep ipsec # setkey -DP # setkey -D # ifconfig And probably racoon's logs. Hi Andrey! First of all -- many thanks for your responses! Configs are followed # sysctl net. | grep ipsec = net.inet.ipsec.def_policy: 1 net.inet.ipsec.esp_trans_deflev: 1 net.inet.ipsec.esp_net_deflev: 1 net.inet.ipsec.ah_trans_deflev: 1 net.inet.ipsec.ah_net_deflev: 1 net.inet.ipsec.ah_cleartos: 1 net.inet.ipsec.ah_offsetmask: 0 net.inet.ipsec.dfbit: 0 net.inet.ipsec.ecn: 0 net.inet.ipsec.debug: 0 net.inet.ipsec.filtertunnel: 0 net.inet.ipsec.natt_cksum_policy: 0 net.inet.ipsec.check_policy_history: 0 net.inet.ipsec.crypto_support: 50331648 net.inet6.ipsec6.def_policy: 1 net.inet6.ipsec6.esp_trans_deflev: 1 net.inet6.ipsec6.esp_net_deflev: 1 net.inet6.ipsec6.ah_trans_deflev: 1 net.inet6.ipsec6.ah_net_deflev: 1 net.inet6.ipsec6.ecn: 0 net.inet6.ipsec6.debug: 0 net.inet6.ipsec6.filtertunnel: 0 = # setkey -DP | grep -A 4 '^0' = 0.0.0.0/0[any] 0.0.0.0/0[any] any in ipsec esp/tunnel/__Cisco_30__-__FreeBSD_IP__/unique:30 spid=1 seq=11 pid=99239 scope=ifnet ifname=ipsec30 refcnt=1 0.0.0.0/0[any] 0.0.0.0/0[any] any in ipsec esp/tunnel/__Cisco_26__-__FreeBSD_IP__/unique#16385 spid=5 seq=9 pid=99239 scope=ifnet ifname=ipsec26 refcnt=1 0.0.0.0/0[any] 0.0.0.0/0[any] any in ipsec esp/tunnel/__Cisco_25__-__FreeBSD_IP__/unique:26 spid=9 seq=7 pid=99239 scope=ifnet ifname=ipsec25 refcnt=1 0.0.0.0/0[any] 0.0.0.0/0[any] any out ipsec esp/tunnel/__FreeBSD_IP__-__Cisco_30__/unique:30 spid=2 seq=5 pid=99239 scope=ifnet ifname=ipsec30 refcnt=1 0.0.0.0/0[any] 0.0.0.0/0[any] any out ipsec esp/tunnel/__FreeBSD_IP__-__Cisco_26__/unique#16385 spid=6 seq=3 pid=99239 scope=ifnet ifname=ipsec26 refcnt=1 0.0.0.0/0[any] 0.0.0.0/0[any] any out ipsec esp/tunnel/__FreeBSD_IP__-__Cisco_25__/unique:26 spid=10 seq=1 pid=99239 scope=ifnet ifname=ipsec25 refcnt=1 = # setkey -D = __FreeBSD_IP__ __Cisco_30__ esp mode=tunnel spi=2124688285(0x7ea42b9d) reqid=26(0x001a) E: rijndael-cbc 6ca42c3b c24ce0ec f3f676c8 c9b9e72d fde63423 3f957b0c ee5da59d dce8a66d A: hmac-sha1 2adb7dfb 26d5de00 2fdd9a21 f63701ef 59d95a1a seq=0x replay=4 flags=0x state=mature created: Apr 23 14:02:03 2018 current: Apr 23 14:17:40 2018 diff: 937(s)hard: 3600(s) soft: 2880(s) last: hard: 0(s) soft: 0(s) current: 0(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 0hard: 0 soft: 0 sadb_seq=5 pid=95677 refcnt=1 __FreeBSD_IP__ __Cisco_25__ esp mode=tunnel spi=153891647(0x092c333f) reqid=26(0x001a) E: rijndael-cbc 8f9905fe 6a9cfc76 a0da354b 53a7f901 298dca43 b5feda65 3be012e7 08835553 A: hmac-sha1 aa2ec447 0e6b36e2 23ba9b27 9d0ecc05 4513af70 seq=0x replay=4 flags=0x state=mature created: Apr 23 13:40:24 2018 current: Apr 23 14:17:40 2018 diff: 2236(s) hard: 3600(s) soft: 2880(s) last: hard: 0(s) soft: 0(s) current: 0(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 0hard: 0 soft: 0 sadb_seq=4 pid=95677 refcnt=1 __Cisco_25__ __FreeBSD_IP__ esp mode=tunnel spi=21918183(0x014e71e7) reqid=26(0x001a) E: rijndael-cbc 43e8f54a 0bdda6b5 41a637d5 4469973d 5b3dc8d0 37022187 43c86f0c 34054df8 A: hmac-sha1 cf08a56a beead8b8 e637a14a 5fdbde3d b8c71192 seq=0x replay=4 flags=0x state=mature created: Apr 23 13:40:24 2018 current: Apr 23 14:17:40 2018 diff: 2236(s) hard: 3600(s) soft: 2880(s) last: Apr 23 13:40:25 2018 hard: 0(s) soft: 0(s) current: 46900(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 719 hard: 0 soft: 0 sadb_seq=3 pid=95677 refcnt=1 __FreeBSD_IP__ __Cisco_26__ esp mode=tunnel spi=2471238029(0x934c198d) reqid=26(0x001a) E: rijndael-cbc 01b3235e 0fe554d3 6dbcb505 bb34d511 93f8ee6f b0b15f43 077c411a afdb1b3b A: hmac-sha1 29ab22bd 2c4f0ade e1478e19 0ecf423f ef155ff3 seq=0x replay=4 flags=0x state=mature created: Apr 23 13:42:29 2018 current:
Re: multiple if_ipsec
On 23/04/2018 15:43, Andrey V. Elsukov wrote:
Your security associations doesn't match your security policies.
Probably you did interfaces reconfiguration without clearing old SAs.
I think your configuration will work, if you first will done if_ipsec(4)
configuration, then start racoon and it will generate SAs.
To clear all old/stale configured SAs you can first stop racoon, then
run `setkey -DF` and `setkey -DPF`.
Hi Andrey
Thanks for your advise: I found typo in my rc.conf and now ipsec
interfaces created with properly reqid.
After all ipsec-interfaces created I have many SPD entries configured
like '0.0.0.0/0[any] 0.0.0.0/0[any] any' with properly configured
ifname=ipsec[25|26|30]
But now I'm sure I have racoon misconfiguration: If I use one "sainfo
anonymous" then all created SA binds to last configured ipsec-interface.
So I need sainfo-entry for every remote-entry.
But I still cann't understand how to bind SPD automatically created by
'ifconfig ipsec30 reqid 30 ...' to SA configured like
=
remote __Cisco_IP_30__ {
my_identifier address __FreeBSD_IP__;
peers_identifier address __Cisco_IP_30__;
ph1id 30;
}
sainfo ??? {
remoteid 30;
}
=
If I configure
sainfo address __FreeBSD_IP__ any address __Cisco_IP_30 any {
remoteid 30;
.
}
then I've got following error
=
racoon: DEBUG: getsainfo params: loc='0.0.0.0/0' rmt='0.0.0.0/0'
peer='__Cisco_IP_30__' client='__Cisco_IP_30__' id=30
racoon: DEBUG: evaluating sainfo: loc='__FreeBSD_IP__',
rmt='__Cisco_IP_30__', peer='ANY', id=30
racoon: DEBUG: check and compare ids : value mismatch (IPv4_address)
racoon: DEBUG: cmpid target: '0.0.0.0/0'
racoon: DEBUG: cmpid source: '__FreeBSD_IP__'
racoon: DEBUG: IV freed
=
Can you please explain me how sainfo (or something else) must be
properly configured?
Thanks!
--
CU,
Victor Gamov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: multiple if_ipsec
On 09/05/2018 10:06, peter.b...@bsd4all.org wrote: Andrey, I was planning to move towards Strongswan anyway. The 1st step (with 1 interface worked great) Julian, The idea of having a jail as VPN end-point is going to help me transition step by step and possibly have both racoon and strongswan active. Thx, Peter On 9 May 2018, at 03:08, Julian Elischer <mailto:jul...@freebsd.org>> wrote: On 8/5/18 9:51 pm, Andrey V. Elsukov wrote: On 08.05.2018 14:03, peter.b...@bsd4all.org <mailto:peter.b...@bsd4all.org> wrote: Hi Victor, I’m struggling wit the same issue. My sainfo doesn’t match unless I use anonymous. Hi Andrey, What I don’t understand is why a “catchall” policy is added instead of the policy that matches the inner tunnel. This is because the how IPsec works in BSD network stack. In simple words - outbound traffic is matched by security policy, inbound is matched by security association. When a packet is going to be send from a host, the kernel checks security policies for match. If it is matched, a packet goes into IPsec processing. Then IPsec code using given security policy does lookup for matched security association. And some IPsec transform happens. When a host receives a packet, it handled by network stack first. And if it has corresponding IPsec inner protocol (ESP, AH), it will be handled by IPsec code. A packet has embedded SPI, it is used for security association lookup. If corresponding SA is found, the IPsec code will apply revers IPsec transform to the packet. Then the kernel checks, that there is some security policy for that packet. Now how if_ipsec(4) works. Security policies associated with interface have configured requirements for tunnel mode with configured addresses. Interfaces are designed for route based VPN, and when a packet is going to be send through if_ipsec interface, its "output" routine uses security policy associated with interface and with configured "reqid". If there are no SAs configured with given reqid, the IPsec code will send ACQUIRE message to IKE and it should install SAs, that will be used for IPsec transforms. When a host receives a packet, it handled by network stack, then by IPsec code and when reverse transform is finished, IPsec code checks, if packet was matched by tunnel mode SA it will be checked by if_ipsec input routine. If addresses and reqid from SA matched to if_ipsec configuration, it will be taken by if_ipsec interface. What is supposed to happen here? Is the IKE daemon supposed to update the policy once started. In my understanding IKE is only supposed to install SAs for if_ipsec. It can't change these policies, because they are immutable. I think for proper support of several if_ipsec interfaces racoon needs some patches. But I have not spare time to do this job. I recommend to use strongswan, it has active developers that are responsive and may give some help at least. There was the link with example, but it also uses only one interface: https://genneko.github.io/playing-with-bsd/networking/freebsd-vti-ipsec my answer was to create a jail to act as the endpoint of each vpn using VIMAGE and then allow each jail to run its own raccoon. Hi All I have FreeBSD-11.1-STABLE (r327786) + strongswan-5.6.2_1 Then IKEv1 configured and two ipsec interfaces connected to Cisco-routers works fine at first sight You need both leftsubnet=0.0.0.0/0 and rightsubnet=0.0.0.0/0 configured at strongswan to protocols like OSPF works properly. I'll try to do more tests later. Thanks Andrey! -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
ipfw on bridge connecting vlans
Hi All I have some misunderstanding how ipfw work with VLAN and bridge I have following config bridge2 / | \ / | \ /| \ vlan200 vlan300 vlan400 (igb0)(igb0) (igb1) = net.link.bridge.ipfw: 1 net.link.bridge.allow_llz_overlap: 0 net.link.bridge.inherit_mac: 0 net.link.bridge.log_stp: 0 net.link.bridge.pfil_local_phys: 0 net.link.bridge.pfil_member: 0 net.link.bridge.ipfw_arp: 1 net.link.bridge.pfil_bridge: 0 net.link.bridge.pfil_onlyip: 0 net.link.ether.ipfw=1 = I need to allow some multicast from some vlans, block other multicast and forward allowed multicast into other vlans For example. Allow 239.0.0.10 received via vlan200 but block the same 239.0.0.10 if it comes via other vlan. Then bridge 239.0.0.10 into vlan400 The simplest ipfw rules for this example: = table blockit create type iface table blockit add vlan200 table blockit add vlan300 table blockit add vlan400 1000 allow ip from any to any via igb0 1002 allow ip from any to any via igb2 1100 deny ip from any to any mac-type 0x0806 via bridge2 1102 allow ip from any to any via bridge2 2000 allow ip from any to 239.0.0.10 in via vlan200 4000 allow ip from any to 239.0.0.10 out via vlan400 9000 deny ip from any to any via table(blockit) 65000 allow ip from any to any = My expectations are follows: 1. ethernet packet tagged as VLAN-200 arrives igb0. This packet has igb0 as 'recv'. Packet checked by ipfw now so I need 1000 allow ip from any to any via igb0 1002 allow ip from any to any via igb2 2. ethernet packet untagged and checked by ipfw. This packet has vlan200 as 'recv' Packet pass 2000. If dst-239.0.0.10 comes from vlan300 it blocked by 9000 3. IP-packet comes through if_bridge and checked by ipfw. ARP packet blocked by 1100. Other packets pass via bridge2 by 1102 4. IP multicast packet copied to all bridge members and checked by ipfw on all outgoing interfaces: packet pass 4000 on vlan400, but blocked by 9000 on vlan300. So only one bridge-member has this packet. 5. ethernet packet tagged as VLAN-400 and checked by ipfw. Packet pass by 4000 6. tagged packet out via igb2 and checked by ipfw. packet pass by 1002 Can somebody explain me how tagged multicast packet goes via bridge and passed into IPFW and correct my previous packet path? Thanks! -- CU Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ipfw on bridge connecting vlans
On 27/10/2018 18:44, Eugene Grosbein wrote:
27.10.2018 22:16, Victor Gamov wrote:
Hi All
I have some misunderstanding how ipfw work with VLAN and bridge
I have following config
bridge2
/ | \
/ | \
/| \
vlan200 vlan300 vlan400
(igb0)(igb0) (igb1)
=
net.link.bridge.ipfw: 1
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 0
net.link.bridge.ipfw_arp: 1
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_onlyip: 0
net.link.ether.ipfw=1
=
I need to allow some multicast from some vlans, block other multicast and
forward allowed multicast into other vlans
Your ruleset needs to differentiate packets based on name of incoming bridge
member
but you forgot to enable net.link.bridge.pfil_member=1. Enable it.
Also note that change of net.link.bridge.ipfw from 0 to 1 disables
net.link.bridge.{pfil_member|pfil_onlyip|pfil_bridge} but you are allowed
to enable them after.
net.link.bridge.pfil_member=1 makes frames enter ruleset as incoming from
bridge member, zero disables this pass.
net.link.bridge.ipfw=1 makes frames enter ruleset again as incoming from bridge
interface itself
without distinction of bridge member, and for forwarded frames enter ruleset
one more time as outgoing from the bridge itself.
And frame enters ruleset one MORE time as outgoing from bridge member if
net.link.bridge.pfil_member=1.
Hi Eugene,
Thanks for your quick response!
With current configs I have unexpected behavior when multicast received
via unneeded vlan300 not blocked by rule 9000 and bad traffic bridged
into all bridge members.
So if when net.link.bridge.pfil_member=1 then I can check incoming and
outgoing bridge members as expected? I'll try it later, thanks
Is it possible the following config with net.link.bridge.pfil_member=1 ?
table allow239 create type iface
table allow239 add vlan400
3000 allow ip from any to 239.0.0.10 recv vlan100 xmit table(allow239)
It's still very interesting to know full packet path via all interfaces
(phy, vlan, bridge) and understand where/why ipfw triggered.
For example why i need "allow via igb0"? It's because net.link.ether.ipfw=1?
Can you explain it in details?
Thanks again!
--
CU
Victor Gamov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ipfw on bridge connecting vlans
On 27/10/2018 19:33, Eugene Grosbein wrote: 27.10.2018 23:26, Victor Gamov wrote: [skip] net.link.bridge.pfil_member=1 makes frames enter ruleset as incoming from bridge member, zero disables this pass. net.link.bridge.ipfw=1 makes frames enter ruleset again as incoming from bridge interface itself without distinction of bridge member, and for forwarded frames enter ruleset one more time as outgoing from the bridge itself. And frame enters ruleset one MORE time as outgoing from bridge member if net.link.bridge.pfil_member=1. (*) With current configs I have unexpected behavior when multicast received via unneeded vlan300 not blocked by rule 9000 and bad traffic bridged into all bridge members. So if when net.link.bridge.pfil_member=1 then I can check incoming and outgoing bridge members as expected? I'll try it later, thanks Is it possible the following config with net.link.bridge.pfil_member=1 ? Yes, it should be. table allow239 create type iface table allow239 add vlan400 3000 allow ip from any to 239.0.0.10 recv vlan100 xmit table(allow239) You also should add the "out" keyword to this rule for performance and clearness because only outgoing frames have "xmit" attribute to check, so this rule is not tried for incoming frames (it won't match them anyway). It's still very interesting to know full packet path via all interfaces (phy, vlan, bridge) and understand where/why ipfw triggered. For example why i need "allow via igb0"? It's because net.link.ether.ipfw=1? Can you explain it in details? See above (*). ok, frame path and ipfw as I see it. = 1000 allow ip from any to any via igb0 1002 allow ip from any to any via igb2 1100 deny ip from any to any mac-type 0x0806 via bridge2 1102 allow ip from any to any via bridge2 2000 allow ip from any to 239.0.0.10 in recv vlan200 4000 allow ip from any to 239.0.0.10 out xmit table(allow239) 9000 deny ip from any to any via table(block239) 65000 allow ip from any to any = 1. tagged frame come to physical interface igb0. As I have net.link.ether.ipfw=1 then tagged frame checked by IPFW and I need 1000 allow ip from any to any via igb0 Frame passed via igb0 and untagged. 2. untagged frame appears on vlan100. Now I can get it via `tcpdump -i vlan100` Frame checked by IPFW on vlan100 because net.link.bridge.pfil_member=1 AND vlan100 is a bridge member ( net.link.ether.ipfw=1 is irrelevant? ) So I can pass/block traffic here. 2000 allow ip from any to 239.0.0.10 in recv vlan100 9000 block ip from any to any via table(block239) Frame accepted on VLAN-100 3. frame (untagged but still ethernet frame not IP packet) appears on bridge2 as incoming and checked by IPFW due net.link.bridge.ipfw=1. I block ARP but allow all other frames (1100 + 1102) 4. frame appears on all bridge-member vlans. Frame checked by IPFW on vlan300/vlan400 because net.link.bridge.pfil_member=1 AND vlan300/vlan400 is a bridge member So I can pass/block traffic here. 4000 allow ip from any to 239.0.0.10 out xmit table(allow239) 9000 block ip from any to any via table(block239) Passed frames tagged 5. tagged frame appears on igb2 As I have net.link.ether.ipfw=1 then tagged frame checked by IPFW and I need 1002 allow ip from any to any via igb2 Is this scheme is correct? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ipfw on bridge connecting vlans
On 27/10/2018 21:02, Eugene Grosbein wrote: 28.10.2018 0:48, Victor Gamov wrote: On 27/10/2018 19:33, Eugene Grosbein wrote: 27.10.2018 23:26, Victor Gamov wrote: [skip] net.link.bridge.pfil_member=1 makes frames enter ruleset as incoming from bridge member, zero disables this pass. net.link.bridge.ipfw=1 makes frames enter ruleset again as incoming from bridge interface itself without distinction of bridge member, and for forwarded frames enter ruleset one more time as outgoing from the bridge itself. And frame enters ruleset one MORE time as outgoing from bridge member if net.link.bridge.pfil_member=1. (*) With current configs I have unexpected behavior when multicast received via unneeded vlan300 not blocked by rule 9000 and bad traffic bridged into all bridge members. So if when net.link.bridge.pfil_member=1 then I can check incoming and outgoing bridge members as expected? I'll try it later, thanks Is it possible the following config with net.link.bridge.pfil_member=1 ? Yes, it should be. table allow239 create type iface table allow239 add vlan400 3000 allow ip from any to 239.0.0.10 recv vlan100 xmit table(allow239) You also should add the "out" keyword to this rule for performance and clearness because only outgoing frames have "xmit" attribute to check, so this rule is not tried for incoming frames (it won't match them anyway). It's still very interesting to know full packet path via all interfaces (phy, vlan, bridge) and understand where/why ipfw triggered. For example why i need "allow via igb0"? It's because net.link.ether.ipfw=1? Can you explain it in details? See above (*). ok, frame path and ipfw as I see it. = 1000 allow ip from any to any via igb0 1002 allow ip from any to any via igb2 1100 deny ip from any to any mac-type 0x0806 via bridge2 1102 allow ip from any to any via bridge2 2000 allow ip from any to 239.0.0.10 in recv vlan200 4000 allow ip from any to 239.0.0.10 out xmit table(allow239) 9000 deny ip from any to any via table(block239) 65000 allow ip from any to any = 1. tagged frame come to physical interface igb0. As I have net.link.ether.ipfw=1 then tagged frame checked by IPFW and I need 1000 allow ip from any to any via igb0 Frame passed via igb0 and untagged. 2. untagged frame appears on vlan100. Now I can get it via `tcpdump -i vlan100` Frame checked by IPFW on vlan100 because net.link.bridge.pfil_member=1 AND vlan100 is a bridge member ( net.link.ether.ipfw=1 is irrelevant? ) So I can pass/block traffic here. 2000 allow ip from any to 239.0.0.10 in recv vlan100 9000 block ip from any to any via table(block239) Frame accepted on VLAN-100 3. frame (untagged but still ethernet frame not IP packet) appears on bridge2 as incoming and checked by IPFW due net.link.bridge.ipfw=1. I block ARP but allow all other frames (1100 + 1102) 4. frame appears on all bridge-member vlans. Frame checked by IPFW on vlan300/vlan400 because net.link.bridge.pfil_member=1 AND vlan300/vlan400 is a bridge member So I can pass/block traffic here. 4000 allow ip from any to 239.0.0.10 out xmit table(allow239) 9000 block ip from any to any via table(block239) Passed frames tagged 5. tagged frame appears on igb2 As I have net.link.ether.ipfw=1 then tagged frame checked by IPFW and I need 1002 allow ip from any to any via igb2 Is this scheme is correct? Almost right with one exception: bridge considers ARP (and REVARP) packets special and passed them through the ruleset only when sysctl net.link.bridge.ipfw_arp=1; yes it enabled. Thanks Eugene! -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
how to down interface at startup
Hi All I have configuration where bridge interface need to be down at startup. But "ifconfig_bridge2="down" is not working: bridge always up How I can 'down' bridge interface at startup? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: how to down interface at startup
On 28/07/2019 18:08, Eugene Grosbein wrote: 28.07.2019 21:50, Victor Gamov wrote: I have configuration where bridge interface need to be down at startup. But "ifconfig_bridge2="down" is not working: bridge always up How I can 'down' bridge interface at startup? If you use rc.conf to configure it, please read rc.conf(5) manual page carefully: ... Interfaces that the administrator wishes to store configuration for, but not start at boot should be configured with the "NOAUTO" keyword in their ifconfig_ variables as described below. Eugene Thank you very much! I really need be more carefully while reading docs. Thanks! -- CU Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
finding optimal ipfw strategy
Hi All I have nonstandard network task for my FreeBSD box: many VLANs bridged together via bridge interface and specific multicast traffic must be send from one VLAN to many (but not all) other VLANs. I use ipfw to block traffic on unwanted outgoing interfaces. And my answer: which ipfw rules more optimal 1 or 2 (see 1 and 2 later) when I have about 100 incoming multicast and about 100 vlans? 1 = ipfw table Mcast1_iface_out create type iface ipfw table Mcast1_iface_out add vlan20 ipfw table Mcast1_iface_out add vlan30 ipfw table Mcast1_iface_out add vlan40 ipfw add 25000 allow udp from IP1 to mcast1 out via table(Mcast1_iface_out) ipfw table Mcast2_iface_out create type iface ipfw table Mcast2_iface_out add vlan20 ipfw table Mcast2_iface_out add vlan30 ipfw add 35000 allow udp from IP1 to mcast2 out via table(Mcast2_iface_out) ipfw table All_vlans create type iface ipfw table All_vlans add vlan20 ipfw table All_vlans add vlan30 ipfw table All_vlans add vlan40 ipfw add 5 deny udp from any to any via table(All_vlans) = 2 = ipfw table Mcast_vlan20_out create type addr ipfw table Mcast_vlan20_out add 232.10.20.1/32 ipfw table Mcast_vlan20_out add 232.10.20.2/32 ipfw table Mcast_vlan20_out add 232.10.20.3/32 ipfw add 25000 allow udp from IP1 to table(Mcast_vlan20_out) out via vlan20 ipfw add 25001 deny udp from any to any via vlan20 ipfw table Mcast_vlan30_out create type addr ipfw table Mcast_vlan30_out add 232.10.20.1/32 ipfw table Mcast_vlan30_out add 232.10.20.2/32 ipfw table Mcast_vlan30_out add 232.10.55.5/32 ipfw add 35000 allow udp from IP1 to table(Mcast_vlan30_out) out via vlan30 ipfw add 35001 deny udp from any to any via vlan30 = Thanks for your advise! -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: finding optimal ipfw strategy
Eugene Many thanks for your reply! I need to read more about tablearg and then modify my current production rules step by step. Thank you again! On 24/08/2019 23:11, Eugene Grosbein wrote: 25.08.2019 2:34, Eugene Grosbein wrote: Also, use table arguments and not only table values, do not ignore their existence: ipfw table $Mcast1_iface_out add vlan20 $mcast11 ipfw table $Mcast1_iface_out add vlan20 $mcast12 ipfw table $Mcast1_iface_out add vlan20 $mcast13 ipfw add 25000 allow udp from IP1 to tablearg out xmit "table($Mcast1_iface_out)" Note there is one single checking ipfw rules for all used pairs ($Mcast1_iface_out, $mcastXX) and this time it is not micro-optimization but very important one when you have plenty of mcastXX. I have to correct myself: ipfw table cannot contain multiple values differing with arguments only, so we should rewrite commands this way: first table contains just list of used multicast destination IPs: Mcast_addr_out=1 ipfw table $Mcast_addr_out create type addr ipfw table $Mcast_addr_out add $mcast11 25012 # use range of rules 25012-4 ipfw table $Mcast_addr_out add $mcast12 25014 # increment rule number by 2 ipfw table $Mcast_addr_out add $mcast13 25016 And you have multiple tables for list of interfaces, one table per multicast destination: Mcast1_iface_out=2 ipfw table $Mcast1_iface_out create type iface ipfw table $Mcast1_iface_out add vlan20 ipfw table $Mcast1_iface_out add vlan22 ipfw table $Mcast1_iface_out add vlan39 Then you start filtering by splitting traffic by destination IP that is most efficient: ipfw add 25000 skipto tablearg from $IP1 to "table($Mcast_addr_out)" ipfw add 25010 deny udp from $your_multicast_range to any ipfw add 25011 skipto 5 ip from any to any # past this set of checks Only traffic destined for specific IP hits the rule checking for outgoing interface: ipfw add 25012 allow udp from any to any out xmit "table($Mcast1_iface_out)" ipfw add 25013 deny udp from any to any ipfw add 25014 allow udp from any to any out xmit "table($Mcast2_iface_out)" ipfw add 25015 deny udp from any to any And so on. -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: finding optimal ipfw strategy
On 24/08/2019 22:34, Eugene Grosbein wrote: 25.08.2019 1:13, Victor Gamov wrote: I have nonstandard network task for my FreeBSD box: many VLANs bridged together via bridge interface and specific multicast traffic must be send from one VLAN to many (but not all) other VLANs. It is quite standard filtering bridge :-) Hi All More general question about my current config. I have about 200Mbit input multicasts which bridged and filtered later (about 380 Mbit bridged if trafshow does not lie me :-) ) My FreeBSD box (12.0-STABLE r348449 GENERIC amd64) has one "Intel(R) Xeon(R) CPU E31270 @ 3.40GHz" and 4-ports "Intel(R) PRO/1000 PCI-Express Network Driver". HT disabled and traffic mainly income via igb0 and out both via igb0 and igb2. About 30 VLANs now active some at igb0 and some at igb2. And I have following `top` stat: = CPU 0: 0.0% user, 0.0% nice, 80.5% system, 0.0% interrupt, 19.5% idle CPU 1: 0.0% user, 0.0% nice, 34.1% system, 0.0% interrupt, 65.9% idle CPU 2: 0.0% user, 0.0% nice, 17.1% system, 0.0% interrupt, 82.9% idle CPU 3: 0.0% user, 0.0% nice, 46.3% system, 0.0% interrupt, 53.7% idle = Also `vmstat -i |grep igb`: = irq264: igb0:rxq0 9310734762 5471 irq265: igb0:rxq110186691956 5985 irq266: igb0:rxq2 8190475727 4812 irq267: igb0:rxq310063786697 5913 irq268: igb0:aq 34 0 irq273: igb1:aq1 0 irq274: igb2:rxq011010248236 6469 irq275: igb2:rxq110843712062 6371 irq276: igb2:rxq2 8810194905 5177 irq277: igb2:rxq310975949272 6449 irq278: igb2:aq 10 0 irq283: igb3:aq1 0 = Is it possible to get CPU load about 30% at this config after ipfw optimization? Or may be main bottleneck is not ipfw-specific? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: finding optimal ipfw strategy
On 26/08/2019 20:15, Eugene Grosbein wrote: 26.08.2019 23:25, Victor Gamov wrote: More general question about my current config. I have about 200Mbit input multicasts which bridged and filtered later (about 380 Mbit bridged if trafshow does not lie me :-) ) Don't trust trafshow. Use: systat -ifstat 1 This! systat show me more real picture: 750Mbit in, 650 Mbit out via bridge Is it possible to get CPU load about 30% at this config after ipfw optimization? Or may be main bottleneck is not ipfw-specific? You won't know until you try and nobody can tell. Too many variables. And you better compare it with 11.3 because 12.0 may have some unsolved preformance regressions. I see. I will try 11.3 later. Now after output optimization as Eugene recommended one core load decreased from 88 to 77 percents. But many loads decreased when unused rules removed. Now I'll try to optimize input rules too like = table All_Ifaces create type iface table All_Ifaces add vlan10 20010 table All_Ifaces add vlan20 20020 table All_Ifaces add vlan30 20030 12000 skipto tablearg ip from any to any in recv table(All_Ifaces) 20010 allow udp from src1 to mcast1 in recv via vlan10 20011 deny ip from any to any 20020 allow udp from src2 to mcast2 in recv via vlan20 20021 deny ip from any to any = -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: finding optimal ipfw strategy
On 27/08/2019 21:03, Andrey V. Elsukov wrote: On 26.08.2019 19:25, Victor Gamov wrote: More general question about my current config. I have about 200Mbit input multicasts which bridged and filtered later (about 380 Mbit bridged if trafshow does not lie me :-) ) My FreeBSD box (12.0-STABLE r348449 GENERIC amd64) has one "Intel(R) Xeon(R) CPU E31270 @ 3.40GHz" and 4-ports "Intel(R) PRO/1000 PCI-Express Network Driver". HT disabled and traffic mainly income via igb0 and out both via igb0 and igb2. About 30 VLANs now active some at igb0 and some at igb2. And I have following `top` stat: = CPU 0: 0.0% user, 0.0% nice, 80.5% system, 0.0% interrupt, 19.5% idle CPU 1: 0.0% user, 0.0% nice, 34.1% system, 0.0% interrupt, 65.9% idle CPU 2: 0.0% user, 0.0% nice, 17.1% system, 0.0% interrupt, 82.9% idle CPU 3: 0.0% user, 0.0% nice, 46.3% system, 0.0% interrupt, 53.7% idle = This doesn't look like heavy ipfw load. Andrew, I have 0.0% interrupt but 80.5% system load. As this box hasn't any processes running (besides kernel + ntp + bsnmp) so I think this load produced by ipfw. Also I think this load source may be packets processing by bridge: get one packet, bridge it (copy/malloc?) into many interfaces, drop packets on unnecessary ifaces (free?) E.g. this is top output from slightly loaded firewall (300Mbytes/s ~500kpps): Yes, it's not a problem for 28 cores :-) I have 4 cores only and about 700Mbit in/out via bridge last pid: 58184; load averages: 9.07, 8.98, 8.83 up 72+07:45:55 21:01:36 821 processes: 36 running, 680 sleeping, 105 waiting CPU 0: 0.0% user, 0.0% nice, 0.0% system, 28.1% interrupt, 71.9% idle CPU 27: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle # pmcstat -S instructions -Tw1 PMC: [INSTR_RETIRED_ANY] Samples: 443074 (100.0%) , 0 unresolved Key: q => exiting... %SAMP IMAGE FUNCTION CALLERS 39.2 kernel sched_idletd fork_exit 10.9 ipfw.koipfw_chk ipfw_check_packet 3.6 kernel cpu_search_lowestcpu_search_lowest 2.8 kernel lock_delay _mtx_lock_spin_cookie 2.5 kernel _rm_rlockin6_localip:1.3 pfil_run_hooks:0.6 2.2 kernel rn_match ta_lookup_radix:1.5 fib6_lookup_nh_basic:0.6 As you can see, when ipfw produces high load, interrupt column is more than system. -- С уважением, Гамов Виктор ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: finding optimal ipfw strategy
On 27/08/2019 21:46, Eugene Grosbein wrote: 28.08.2019 1:03, Andrey V. Elsukov wrote: As you can see, when ipfw produces high load, interrupt column is more than system. Interrupt numbers higher than others generally mean that traffic is processed without netisr queueing mostly. That is expected for plain routing. I'm not sure if this would be same in case of bridging. Victor, do you have some non-default tuning in your /boot/loader.conf or /etc/sysctl.conf? If yes, could you show them? Eugene, Nothing special. loader.conf = machdep.hyperthreading_allowed="0" net.inet.ip.fw.default_to_accept=1 = sysctl.conf = net.link.ether.ipfw=1 net.link.bridge.ipfw=1 net.link.bridge.ipfw_arp=1 net.link.bridge.pfil_member=1 net.inet.ip.fw.verbose_limit=100 net.inet.ip.fw.verbose=1 = `sysctl net.isr` = sysctl net.isr net.isr.numthreads: 1 net.isr.maxprot: 16 net.isr.defaultqlimit: 256 net.isr.maxqlimit: 10240 net.isr.bindthreads: 0 net.isr.maxthreads: 1 net.isr.dispatch: direct = I don't know about internals but I think high interrupt load is bad and it because NIC does not support per CPU-queue for example. If not, you should try something like this. For loader.conf: Sorry, it's a production system and I can reboot it at the middle of October only. > #substitute total number of CPU cores in the system here > net.isr.maxthreads=4 > # EOF Is it ok for multicast? It's UDP traffic which must be ordered. I read 'maxthreads=1' used to keep TCP traffic ordered. And if you haven't already seen it, you may find useful my blog post (in Russian) https://dadv.livejournal.com/139170.html It's a bit old but still can give you some light. Yes, I read it already :-) Also some Calomel articles. I'll try to tune system at next reboot. The main question for myself now is "how is this architecture correct" and "how many traffic is possible to process". -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: finding optimal ipfw strategy
On 27/08/2019 21:50, Eugene Grosbein wrote: 28.08.2019 1:46, Eugene Grosbein wrote: 28.08.2019 1:03, Andrey V. Elsukov wrote: As you can see, when ipfw produces high load, interrupt column is more than system. Interrupt numbers higher than others generally mean that traffic is processed without netisr queueing mostly. That is expected for plain routing. I'm not sure if this would be same in case of bridging. Victor, do you have some non-default tuning in your /boot/loader.conf or /etc/sysctl.conf? If yes, could you show them? If not, you should try something like this. For loader.conf: hw.igb.rxd=4096 hw.igb.txd=4096 net.isr.bindthreads=1 net.isr.defaultqlimit=4096 #substitute total number of CPU cores in the system here net.isr.maxthreads=4 # EOF Also, you should monitor interrupt numbers shown by "systat -vm 3" for igb* devices at hours of most load. If they approach 8000 limit but not exceed it, you may be suffering from this and should raise the limit with /boot/loader.conf: hw.igb.max_interrupt_rate=32000 It's about 5000-7000 per rxq -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: finding optimal ipfw strategy
On 27/08/2019 22:59, Eugene Grosbein wrote: 28.08.2019 2:22, Victor Gamov wrote: Also, you should monitor interrupt numbers shown by "systat -vm 3" for igb* devices at hours of most load. If they approach 8000 limit but not exceed it, you may be suffering from this and should raise the limit with /boot/loader.conf: hw.igb.max_interrupt_rate=32000 It's about 5000-7000 per rxq 7000 is quite close considering it is average for quite long period (a second or seconds). It can hit 8000 for some ticks easily in your case. Eugene, can you explain me what max_interrupt_rate means? Calomel explain it as "# maximum number of interrupts per second generated by single igb". So if I increase it then more irq per sec will be generated and more context switch produced. Why I need to increase it? You should raise the limit as soon as possible. I'd advise to double it upto 16000 first but if you cannot afford extra reboot better use 32000 as it's just fine for 1Gbps link. # sysctl hw.igb.max_interrupt_rate sysctl: unknown oid 'hw.igb.max_interrupt_rate' # Is it possible to set it at boot time only? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: finding optimal ipfw strategy
On 27/08/2019 23:30, Eugene Grosbein wrote: 28.08.2019 2:20, Victor Gamov wrote: sysctl.conf = net.link.ether.ipfw=1 net.link.bridge.ipfw=1 net.link.bridge.ipfw_arp=1 net.link.bridge.pfil_member=1 net.inet.ip.fw.verbose_limit=100 net.inet.ip.fw.verbose=1 = You should avoid passing same packet multiple times through the ruleset. Less checks, better performance. Yes, I feel it :-) Do you really use ipfw filtering based on layer2 parameters like MAC addresses? If not, you should disable net.link.ether.ipfw. If yes, you should use "layer2" keyword explicily in rules filtering by ethernet headers and place these rules above others and use "allow ip from any to any layer2" after L2 filtering is done, so L2 packets do not go through other rules extra time. Do you really need to filter each bridged L3 packet twice? Once as "out xmit $bridge" and once as "out xmit $brige_member"? If not, you should disable net.link.bridge.ipfw and keep net.link.bridge.pfil_member=1 only. Packets must be filtered on input VLANs (bridge members) and on output VLANs. So net.link.bridge.pfil_member=1 Perhaps, you are ruining the performance with such settings making same work 3 times without real need. Do you really need filtering ARP? Disable net.link.bridge.ipfw_arp if not. I need to drop ARP moving via bridge. As I use many VLANs all VLAN must be isolated and only multicast must be bridged from one VLAN to others. To block ARP following rule used: deny ip from any to any mac-type 0x0806 via bridge1202 As I understand correctly I need net.link.bridge.ipfw_arp and net.link.bridge.ipfw to do it. I'm not sure about net.link.ether.ipfw `sysctl net.isr` = sysctl net.isr net.isr.numthreads: 1 net.isr.maxprot: 16 net.isr.defaultqlimit: 256 net.isr.maxqlimit: 10240 net.isr.bindthreads: 0 net.isr.maxthreads: 1 net.isr.dispatch: direct = I don't know about internals but I think high interrupt load is bad and it because NIC does >> not support per CPU-queue for example. All decent igb(4) NICs support at least 8 hardware input queues unless disabled by driver/kernel. However, net.isr settings are not about such queues. High interrupt number is definitely better than dropping input frames by NIC chip due to overflow of its internal buffers just because CPU was not notified it's time to get traffic out of these buffers. The driver tries not to overload CPU with interrupts and that's fine but default 8000 limit is not adequate to modern CPU and was not adequate for many years. Raise limit to 32000. I see. Thanks! I'll tune net.isr ASAP. If not, you should try something like this. For loader.conf: Sorry, it's a production system and I can reboot it at the middle of October only. #substitute total number of CPU cores in the system here net.isr.maxthreads=4 # EOF Is it ok for multicast? It's UDP traffic which must be ordered. I read 'maxthreads=1' used to keep TCP traffic ordered. It's a job for uplink to feed your bridge with ordered UDP flows. If you use igb(4) driver, FreeBSD kernel will keep flows ordered automatically. There is no place in the code where they could be reordered if you do not use lagg(4) without LACP. Thanks again. I'll set maxthreads=4 at next reboot. And if you haven't already seen it, you may find useful my blog post (in Russian) https://dadv.livejournal.com/139170.html It's a bit old but still can give you some light. Yes, I read it already :-) Also some Calomel articles. I'll try to tune system at next reboot. The main question for myself now is "how is this architecture correct" and "how many traffic is possible to process". You have read numbers from my posts. ipfw+dummynet+PPPoE+routing+LACP+vlan tagging takes much more CPU power than just ipfw+bridging and my system still processed much more traffic. Make sure you don't pass same packets multiple times through ipfw rules. ipfw has its counters for rules and you should use them to summarize octets carefully and compare with numbers shown by netstat or systat (they both have same in-kernel source) to verify whether packets go through ipfw extra times or not. It's not too easy but I'll try to build test system and check on it. If 'bridge + drop on outgoing' is not a bottleneck I'll tune system and use this approach while it's possible. -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: finding optimal ipfw strategy
On 28/08/2019 24:45, Eugene Grosbein wrote: 28.08.2019 3:59, Victor Gamov wrote: sysctl.conf = net.link.ether.ipfw=1 net.link.bridge.ipfw=1 net.link.bridge.ipfw_arp=1 net.link.bridge.pfil_member=1 net.inet.ip.fw.verbose_limit=100 net.inet.ip.fw.verbose=1 = Do you really use ipfw filtering based on layer2 parameters like MAC addresses? If not, you should disable net.link.ether.ipfw. If yes, you should use "layer2" keyword explicily in rules filtering by ethernet headers and place these rules above others and use "allow ip from any to any layer2" after L2 filtering is done, so L2 packets do not go through other rules extra time. Do you really need to filter each bridged L3 packet twice? Once as "out xmit $bridge" and once as "out xmit $brige_member"? If not, you should disable net.link.bridge.ipfw and keep net.link.bridge.pfil_member=1 only. Packets must be filtered on input VLANs (bridge members) and on output VLANs. So net.link.bridge.pfil_member=1 Perhaps, you are ruining the performance with such settings making same work 3 times without real need. Do you really need filtering ARP? Disable net.link.bridge.ipfw_arp if not. I need to drop ARP moving via bridge. As I use many VLANs all VLAN must be isolated and only multicast must be bridged from one VLAN to others. To block ARP following rule used: deny ip from any to any mac-type 0x0806 via bridge1202 As I understand correctly I need net.link.bridge.ipfw_arp and net.link.bridge.ipfw to do it. I'm not sure about net.link.ether.ipfw Why do you need to filter ARP on bridge? That's unusial. VLANs are isolated by default and by definition, unless you explicitly enable inter-vlan routing and setup your routing table. May be I have some misunderstood here but... If I have many VLANs bridged via bridge interface then ARP received from one VLAN will be send to all bridge members. So it will be send to all unwanted VLANs. Is it correct? Anyway, you can skip entire ipfw pass over a bridge because you filter its members anyway, so just drop ARP coming from any vlan with exception of controlling one: allow ip from any to any layer2 mac-type 0x0806 in recv $controlvlan deny ip from any to any layer2 mac-type 0x0806 in allow ip from any to any layer2 And then disable filtering for bridge itself altogether. Decreasing number of passes over ipfw should be your top priority because that's what can provide you with most benefit. You should even rewrite your ruleset if that is needed to achieve this goal. If I set net.link.bridge.ipfw=0 but net.link.ether.ipfw and net.link.bridge.ipfw still set to 1 is it still possible block unwanted ARP received from one VLAN and bridged to other on outgoing VLAN like deny ip from any to any layer2 mac-type 0x0806 out xmit MAC not $mymac any Is it correct and more effective than net.link.bridge.ipfw=1 if I have "deny mac-type 0x0806 via bridge" at rules top? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: finding optimal ipfw strategy
On 28/08/2019 18:48, Eugene Grosbein wrote:
28.08.2019 17:18, Victor Gamov wrote:
Why do you need to filter ARP on bridge? That's unusial. VLANs are
isolated by default and by definition, unless you explicitly enable
inter-vlan routing and setup your routing table.
May be I have some misunderstood here but...
If I have many VLANs bridged via bridge interface then ARP received
from one VLAN will be send to all bridge members. So it will be send
to all unwanted VLANs. Is it correct?
Yes. So, you really do not want any kind of unicast bridging at all
and use bridge as "poor man's" replacement for inter-vlan multicast routing,
right?
:-) Looks like this
But I start this project as experiment (now in production) to get
"router" which allowed implicitly send multicast whithout needs to
igmp-join from attached VLANs and with my own multicast policy.
In such case you could benefit from small patch that allows you to block ARP
packets unconditionally
as if they were filtered by ipfw without really passing them through the
ruleset.
Use sysctl net.link.bridge.ipfw_arp=-1 with the patch (untested):
I'll try it when make test server. Thanks!
Anyway, you can skip entire ipfw pass over a bridge because you
filter its members anyway, so just drop ARP coming from any vlan with
exception of controlling one:
allow ip from any to any layer2 mac-type 0x0806 in recv $controlvlan deny ip
from any to any layer2 mac-type 0x0806 in allow ip from any
to any layer2
And then disable filtering for bridge itself altogether. Decreasing
number of passes over ipfw should be your top priority because that's
what can provide you with most benefit. You should even rewrite your
ruleset if that is needed to achieve this goal.
If I set net.link.bridge.ipfw=0 but net.link.ether.ipfw and
net.link.bridge.ipfw still set to 1
is it still possible block unwanted ARP received from one VLAN and bridged to
other on outgoing VLAN like
deny ip from any to any layer2 mac-type 0x0806 out xmit MAC not $mymac any
Is it correct and more effective than net.link.bridge.ipfw=1 if I have "deny
mac-type 0x0806 via bridge" at rules top?
Yes. And anything decreasing number of traffic passes over entire ipfw ruleset
is efficient.
ok, will try it some later. Hope no problem to switch on/off this
sysctl on production server :-)
Many thanks, Eugene!
P.S. Two questions about rules syntax optimization. What is more effective:
skipto tablearg udp from any to table(AllMcast_out)
or
skipto tablearg udp from any to table(AllMcast_out) out xmit vlan*
I hope I can place such rule at top of ruleset and only allowed
multicast packets outgoing via VLANs interfaces will hit this rule.
and second:
allow udp from $src1 to { 239.1.2.55 or 239.1.2.56 }
or
allow udp from src1 to 239.1.2.0/24{55,56}
Thanks again!
--
CU,
Victor Gamov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: how to down interface at startup
When I configure vlans like this
=
cloned_interfaces="${cloned_interfaces} vlan221"
ifconfig_vlan221="inet 10.2.2.241/28 vlan 221 vlandev igb0 NOAUTO"
=
then NOAUTO clause has no effect
Small patch to allow NOAUTO for any interface:
===cut here===
*** /etc/network.subr.orig Tue Aug 13 19:24:33 2019
--- /etc/network.subr Tue Aug 20 18:54:22 2019
***
*** 226,232
/etc/rc.d/hostapd start $1
_cfg=0
elif [ ${_cfg} -eq 0 ]; then
! ${IFCONFIG_CMD} $1 up
fi
if dhcpif $1; then
--- 226,236
/etc/rc.d/hostapd start $1
_cfg=0
elif [ ${_cfg} -eq 0 ]; then
! if autoif $1; then
! ${IFCONFIG_CMD} $1 up
! else
! ${IFCONFIG_CMD} $1 down
! fi
fi
if dhcpif $1; then
===/cut here===
On 28/07/2019 18:33, Victor Gamov wrote:
On 28/07/2019 18:08, Eugene Grosbein wrote:
28.07.2019 21:50, Victor Gamov wrote:
I have configuration where bridge interface need to be down at startup.
But "ifconfig_bridge2="down" is not working: bridge always up
How I can 'down' bridge interface at startup?
If you use rc.conf to configure it, please read rc.conf(5) manual page
carefully:
... Interfaces that the
administrator wishes to store configuration for, but
not
start at boot should be configured with the "NOAUTO"
keyword
in their ifconfig_ variables as described
below.
Eugene
Thank you very much!
I really need be more carefully while reading docs.
Thanks!
--
CU
Victor Gamov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
bridged vlan down but traffic still exists
Hi All I have vlan111 and vlan222 bridged into bridge2. Then multicast traffic from vlan111 sending and I get it at vlan222 even if 'ifconfig vlan222 down' issued. Is it bug or known feature? FreeBSD 12.0-STABLE r348449 GENERIC amd64 Thanks! -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
ipsec on multicore VM
Hi All I have FreeBSD 11.2-STABLE #0 r343863 VM with 2 CPU and vxnet3 NIC. This host uses many if_ipsec and strongswan-5.7.2 to make site-to-site ipsec connections. When I use `tcpdump -nn -i src and esp` then I got many reordered IPsec packets. Does tcpdump give me a real picture and I have reordering somewhere "on the wire" or packets may be reordered due more then one CPU read packets from NIC ? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
icmp v4 redirect timeout
Hi All I discover the following problem: FreeBSD host install route recived by ICMP-redirect from default GW and this route is permanents. In my case FreeBSD 192.168.1.10/24 has 192.168.1.254 as default gateway. This network has interconnection with remote network 192.168.88.0/24 via some other gateways -- 192.168.1.199 + 192.168.1.195 for example. All gateways using OSPF to exchange routing info. 192.168.1.10 send packet destined to remote network (192.168.88.0/24) to default gateway 192.168.1.254, receive ICMP-redirect and install route to 192.168.88.0/24 via 192.168.1.199. Then 192.168.1.199 off for some reason but 192.168.1.10 never know about it because route installed via 192.168.1.199 is permanent. I see net.inet6.icmp6.redirtimeout in my FreeBSD 11.2-STABLE #0 r339734 and I think this sysctl set timeout for routes installed via ICMP-redirects (route deletes after this timeout?). Is it possible to get such sysctl for ipv4 ? Thanks! -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
No SNMP ifHCInOctets counters for ipsec interfaces
Hi All In my FreeBSD 11.2-STABLE #0 r339734 I have no SNMP ifHCInOctets counters for ipsec interfaces: = IF-MIB::ifName.1 = STRING: vmx0 IF-MIB::ifName.2 = STRING: vmx1 IF-MIB::ifName.3 = STRING: lo0 IF-MIB::ifName.4 = STRING: vlan102 IF-MIB::ifName.6 = STRING: ipsec1632 IF-MIB::ifName.8 = STRING: enc0 IF-MIB::ifName.9 = STRING: ipsec1614 IF-MIB::ifName.10 = STRING: ipsec20016 IF-MIB::ifInOctets.1 = Counter32: 63580689 IF-MIB::ifInOctets.2 = Counter32: 2466393770 IF-MIB::ifInOctets.3 = Counter32: 338773 IF-MIB::ifInOctets.4 = Counter32: 2273962037 IF-MIB::ifInOctets.6 = Counter32: 839822061 IF-MIB::ifInOctets.8 = Counter32: 3206656133 IF-MIB::ifInOctets.9 = Counter32: 985675403 IF-MIB::ifInOctets.10 = Counter32: 434465084 IF-MIB::ifHCInOctets.1 = Counter64: 21538014781 IF-MIB::ifHCInOctets.2 = Counter64: 19645260518 IF-MIB::ifHCInOctets.4 = Counter64: 19452829789 = Is it bug or known feature? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: AW: No SNMP ifHCInOctets counters for ipsec interfaces
Thanks for your clarification I'll try 12.1 later On 25/10/2019 19:13, hartmut.bra...@dlr.de wrote: Hi, They used to be enabled only when the bit rate reported by the interface was high enough (there is some wording in the corresponding RFC when the HC counters are required and when not). This check was removed at some point - in 12.1 the Hc counters are unconditionally there. You may want to check the bitrate that these interfaces report. harti -Ursprüngliche Nachricht- Von: owner-freebsd-...@freebsd.org [mailto:owner-freebsd-...@freebsd.org] Im Auftrag von Victor Gamov Gesendet: Friday, October 25, 2019 4:45 PM An: freebsd-net@freebsd.org Betreff: No SNMP ifHCInOctets counters for ipsec interfaces Hi All In my FreeBSD 11.2-STABLE #0 r339734 I have no SNMP ifHCInOctets counters for ipsec interfaces: = IF-MIB::ifName.1 = STRING: vmx0 IF-MIB::ifName.2 = STRING: vmx1 IF-MIB::ifName.3 = STRING: lo0 IF-MIB::ifName.4 = STRING: vlan102 IF-MIB::ifName.6 = STRING: ipsec1632 IF-MIB::ifName.8 = STRING: enc0 IF-MIB::ifName.9 = STRING: ipsec1614 IF-MIB::ifName.10 = STRING: ipsec20016 IF-MIB::ifInOctets.1 = Counter32: 63580689 IF-MIB::ifInOctets.2 = Counter32: 2466393770 IF-MIB::ifInOctets.3 = Counter32: 338773 IF-MIB::ifInOctets.4 = Counter32: 2273962037 IF-MIB::ifInOctets.6 = Counter32: 839822061 IF-MIB::ifInOctets.8 = Counter32: 3206656133 IF-MIB::ifInOctets.9 = Counter32: 985675403 IF-MIB::ifInOctets.10 = Counter32: 434465084 IF-MIB::ifHCInOctets.1 = Counter64: 21538014781 IF-MIB::ifHCInOctets.2 = Counter64: 19645260518 IF-MIB::ifHCInOctets.4 = Counter64: 19452829789 = Is it bug or known feature? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: icmp v4 redirect timeout
On 25/10/2019 14:36, Andrey V. Elsukov wrote: On 22.10.2019 17:30, Victor Gamov wrote: Hi All I discover the following problem: FreeBSD host install route recived by ICMP-redirect from default GW and this route is permanents. In my case FreeBSD 192.168.1.10/24 has 192.168.1.254 as default gateway. This network has interconnection with remote network 192.168.88.0/24 via some other gateways -- 192.168.1.199 + 192.168.1.195 for example. All gateways using OSPF to exchange routing info. 192.168.1.10 send packet destined to remote network (192.168.88.0/24) to default gateway 192.168.1.254, receive ICMP-redirect and install route to 192.168.88.0/24 via 192.168.1.199. Then 192.168.1.199 off for some reason but 192.168.1.10 never know about it because route installed via 192.168.1.199 is permanent. I see net.inet6.icmp6.redirtimeout in my FreeBSD 11.2-STABLE #0 r339734 and I think this sysctl set timeout for routes installed via ICMP-redirects (route deletes after this timeout?). Is it possible to get such sysctl for ipv4 ? I think expiring doesn't work for IPv6 too. At least, I didn't find related code from a quick look. So, there are no ways to install route received via ICMP-redirect for specific time not permanent? More general question: is it possible to install temporary route which expired by internal in-kernel timer ? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
FreeBSD as multicast router
Hi All I have (noob) questions about multicast routing under FreeBSD. I have FreeBSD box with two (or more) multicast enabled interfaces (e.x. vlan750 and vlan299). vlan750 connected to multicast source. Then pimd installed and only this two interfaces enabled in pimd config. Multicast routes successfully installed by pimd and listed by `netstat -g -f inet` Then client on vlan299 send IGMP-Join (this Join received by FreeBSD on vlan299) The question is: who will forward muilticast from one interface (vlan750) to another (vlan299)? Is it kernel specific job or I need additional software? Thanks! -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD as multicast router
On 03/11/2019 08:22, Mike Karels wrote: Hi All I have (noob) questions about multicast routing under FreeBSD. I have FreeBSD box with two (or more) multicast enabled interfaces (e.x. vlan750 and vlan299). vlan750 connected to multicast source. Then pimd installed and only this two interfaces enabled in pimd config. Multicast routes successfully installed by pimd and listed by `netstat -g -f inet` Then client on vlan299 send IGMP-Join (this Join received by FreeBSD on vlan299) The question is: who will forward muilticast from one interface (vlan750) to another (vlan299)? Is it kernel specific job or I need additional software? Please read the manpage multicast(4) "man 4 multicast", you should need to build a custom kernel with the "options MROUTING" to enable the multicast forwarding in the kernel. If "netstat -g" shows routes, the kernel must have been built with "options MROUTING". Indeed. The kernel does the forwarding, according to those routing tables installed by pimd or another multicast routing program. Is it not working? It sounds like you are very close. Could it be sysctl net.inet.ip.forwarding? Does that still apply to mroutes? No, they are separate. The test is just whether MROUTING is enabled, and whether a multicast router like pimd is active. One other thing to check would be "netstat -gs" (multicast stats). Oops! = # netstat -f inet -gs No IPv4 MROUTING kernel support. = But I have ip_mroute.ko loaded and netstat -g shows something like = # netstat -f inet -g IPv4 Virtual Interface Table Vif Thresh Local-Address Remote-AddressPkts-In Pkts-Out 0 1 A.A.A.A 0 0 1 1 B.B.B.19 0 0 210 10.199.199.102 0 0 315 10.200.200.677440 0 4 1 A.A.A.A 0 77440 IPv4 Multicast Forwarding Table Origin Group Packets In-Vif Out-Vifs:Ttls 10.200.200.5232.232.8.33184434:1 10.200.200.5232.232.8.171184334:1 10.200.200.5232.232.8.58 460934:1 10.200.200.5232.232.8.154184434:1 10.200.200.5232.232.8.170184434:1 = and = # pimd -r Virtual Interface Table == Vif Local AddressSubnet Thresh Flags Neighbors --- --- -- -- - - 0 A.A.A.AA.A.A.A/251 DR NO-NBR 1 B.B.B.19 B.B.B 1 DR NO-NBR 2 10.199.199.102 10.199.199.100/30 10 DR PIM 10.199.199.101 3 10.200.200.6 10.200.200/29 15 DR NO-NBR 4 A.A.A.Aregister_vif01 Vif SSM GroupSources Multicast Routing Table == --- (S,G) Source GroupRP Address Flags --- --- --- --- 10.200.200.5 232.232.8.33 SSM CACHE SG Joined oifs: j Pruned oifs: . Leaves oifs: . Asserted oifs: . Outgoing oifs: o Incoming : ...I. = A.A.A.A is external IP-address. No multicast trafic must be sended to this interface. 10.200.200.6 -- vlan750, multicast comes from here 10.199.199.102 -- vlan299, multicast must be forfarded here after IGMP-Join received from 10.199.199.101/30 So, kernel with MROUTING options must be configured/installed or ip_mroute.ko is enough? P.S. FreeBSD 11.3-STABLE #0 r351605M -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD as multicast router
)
[gaddr 224.0.0.22 is_ex { }] [gaddr 224.0.0.2 is_ex { }] [gaddr
224.0.0.13 is_ex { }]
00:39:33.091330 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto
IGMP (2), length 40, options (RA))
10.199.199.101 > 224.0.0.22: igmp v3 report, 1 group record(s)
[gaddr 232.232.8.33 to_ex { }]
00:39:35.166091 IP (tos 0xc0, ttl 1, id 0, offset 0, flags [DF], proto
IGMP (2), length 40, options (RA))
10.199.199.101 > 224.0.0.22: igmp v3 report, 1 group record(s)
[gaddr 232.232.8.33 to_ex { }]
=
--
CU,
Victor Gamov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: 10g IPsec ?
On 06/11/2019 01:45, Olivier Cochard-Labbé wrote: On Tue, Nov 5, 2019 at 8:15 PM John-Mark Gurney wrote: AES-GCM can run at over 1GB/sec on a single core, so as long as the traffic can be processed by multiple threads (via multiple queues for example), it should be doable. I didn't bench this setup (10Gb/s IPSec) but I believe we will have the same problem with IPSec as with all VPN setups (like PPPoE or GRE): the IPSec tunnel will generate one IP flow preventing load sharing between all the NIC's RSS queues. I'm not aware of improvement to remove this limitation. Is it possible to make load-sharing based on fmod(ipsec_seq_number / NUM_CPU_CORES) for example? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD as multicast router
On 06/11/2019 05:41, Mike Karels wrote: On 05/11/2019 09:09, Mike Karels wrote: On 03/11/2019 08:22, Mike Karels wrote: Hi All I have (noob) questions about multicast routing under FreeBSD. I have FreeBSD box with two (or more) multicast enabled interfaces (e.x. vlan750 and vlan299). vlan750 connected to multicast source. Then pimd installed and only this two interfaces enabled in pimd config. Multicast routes successfully installed by pimd and listed by `netstat -g -f inet` Then client on vlan299 send IGMP-Join (this Join received by FreeBSD on vlan299) The question is: who will forward muilticast from one interface (vlan750) to another (vlan299)? Is it kernel specific job or I need additional software? Please read the manpage multicast(4) "man 4 multicast", you should need to build a custom kernel with the "options MROUTING" to enable the multicast forwarding in the kernel. If "netstat -g" shows routes, the kernel must have been built with "options MROUTING". Indeed. The kernel does the forwarding, according to those routing tables installed by pimd or another multicast routing program. Is it not working? It sounds like you are very close. Could it be sysctl net.inet.ip.forwarding? Does that still apply to mroutes? No, they are separate. The test is just whether MROUTING is enabled, and whether a multicast router like pimd is active. One other thing to check would be "netstat -gs" (multicast stats). Oops! = # netstat -f inet -gs No IPv4 MROUTING kernel support. = This looks like a bug in netstat; it is doing a test that is wrong for the loadable module. I don't know how much the stats might help, but if you let me know what version you are running, I can build a fixed netstat. Or I can send a source patch. But I have ip_mroute.ko loaded and netstat -g shows something like = # netstat -f inet -g IPv4 Virtual Interface Table Vif Thresh Local-Address Remote-AddressPkts-In Pkts-Out 0 1 A.A.A.A 0 0 1 1 B.B.B.19 0 0 210 10.199.199.102 0 0 315 10.200.200.677440 0 4 1 A.A.A.A 0 77440 IPv4 Multicast Forwarding Table Origin Group Packets In-Vif Out-Vifs:Ttls 10.200.200.5232.232.8.33184434:1 10.200.200.5232.232.8.171184334:1 10.200.200.5232.232.8.58 460934:1 10.200.200.5232.232.8.154184434:1 10.200.200.5232.232.8.170184434:1 I missed this before. Looks like the last column should include 2:1 in each case if pimd saw the join. The multicasts are only being sent to Vif 4, the register-vif (see below); the Pkts-Out for it is the same as the input on 3. I'm not familiar enough with pimd to guess what is wrong. I still have misunderstood here. Pimd installs multicast routes and this routes displayed by `netstat -g`. So, the system knows interface where multicast received. When Join received via interface 2 (vlan299) who must resend multicast from input interface 3 (vlan750) to output interface 2 (vlan299)? I guess it kernel-specific task and kernel must resend multicast without any other helpers. Is it wrong? P.S. I rebuild kernel with MROUTING option but = # netstat -gs -f inet No IPv4 MROUTING kernel support = still here -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD as multicast router
mcast-macaddr 01:00:5e:00:00:01 = and locally started programs cann't read multicast stream while interface not directly specified like udp://vlan750@232.232.8.33: or static route to this group not installed like 'route add 232.232.8.33/32 -iface vlan750' I think problem somewhere here (FreeBSD does not join to multicasts?). P.S. I try to use smcroute but it started with following errors: = SMCRoute version 2.1.0 Adding vlan750 to list of multicast routing interfaces Map iface vlan750 => VIF 0 ifindex 13 flags 0x TTL threshold 20 Failed adding VIF for iface vlan750: Can't assign requested address Adding vlan299 to list of multicast routing interfaces Map iface vlan299 => VIF 1 ifindex 10 flags 0x TTL threshold 20 Failed adding VIF for iface vlan299: Can't assign requested address = and no records reported by 'netstat -f inet -n -g' -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD as multicast router
Hi Eugene! On 08/11/2019 11:22, Eugene Grosbein wrote: 07.11.2019 21:17, Victor Gamov wrote: I still have misunderstood here. Pimd installs multicast routes and this routes displayed by `netstat -g`. So, the system knows interface where multicast received. When Join received via interface 2 (vlan299) who must resend multicast from input interface 3 (vlan750) to output interface 2 (vlan299)? I guess it kernel-specific task and kernel must resend multicast without any other helpers. Is it wrong? I'm not familiar with multicast routing in FreeBSD. Multicast routing has its rules in general, though. For example, Cisco routers never process incoming multicast UDP flows if unicast route to source IP address of UDP packets points to interface that differs from real incoming interface. This is "reverse path filtering" embedded in multicast routing unconditionally. Yes, but FreeBSD can ping source and client in my tests (see my new later at this thread with network scheme) -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD as multicast router
On 08/11/2019 16:47, Eugene Grosbein wrote: 08.11.2019 19:10, Victor Gamov wrote: I'm not familiar with multicast routing in FreeBSD. Multicast routing has its rules in general, though. For example, Cisco routers never process incoming multicast UDP flows if unicast route to source IP address of UDP packets points to interface that differs from real incoming interface. This is "reverse path filtering" embedded in multicast routing unconditionally. Yes, but FreeBSD can ping source and client in my tests (see my new later at this thread with network scheme) It does not matter if source is reachable with unicasts (ping). "Reverse" unicast routes should match incoming interface for multicast UDP. My network scheme is simplest: -- --- | source |-vlan750-| FreeBSD PIM router |-vlan299-| client | |200.5/29| |200.6/29 199.102/30| |199.101/30| -- --- All multicasts comes from 200.5 with 200.5 as source IP. So I hope RPF check passes for FreeBSD. -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD as multicast router
Hi All
Still trying to run FreeBSD-box as multicast router :-)
FreeBSD upgraded to 11.3-STABLE #1 r354778. netstat pacth by Mike Karels
manually applied and netstat -gs looks OK now.
Latest pimd version 3.0beta1 downloaded from git and configured. While
configure it report following:
=
-- Summary --
pimd version 3.0-beta1
Prefix: /usr/local
Sysconfdir: /usr/local/etc
Localstatedir.: /usr/local/var
C Compiler: cc -g -O2
Optional features:
Kernel register encap.: no
Kernel (*,G) support..: no
Kernel MAX VIFs...: 32
Memory save...: no
RSRR (experimental)...: no
Exit on error.: yes
=
What does "Kernel (*,G) support..: no" means?
Then my test multicast network configured (again)
--
-vlan298-| FreeBSD PIM router |-vlan299-| client |
|208.34/29 205.2/29| |205.5/29|
--
Two multicast generated by FreeBSD-router: one (232.232.9.43) sended
from vlan299 and another (232.232.88.173) from vlan298 both with TTL=20
Pimd started with following config:
=
phyint vlan299 enable ttl-threshold 20
phyint vlan298 enable ttl-threshold 20
rp-address 10.200.205.2 232.232.0.0/16
=
Now client is requesting multicast which router is sending from vlan299
and client successfully receiving it. But when client is requests
multicast sending (by router) from vlan298 it doesn't receive it.
My first question: (in theory) is router must send multicast to client
in this situation?
And the second: why :Ttls is 1 at `netstat -f inet -g` output:
=
IPv4 Virtual Interface Table
Vif Thresh Local-Address Remote-AddressPkts-In Pkts-Out
020 10.200.205.20 19247
120 10.200.208.34 0 22249
2 1 10.200.205.20 41496
IPv4 Multicast Forwarding Table
Origin Group Packets In-Vif Out-Vifs:Ttls
10.200.208.34 232.232.88.173 2224912:1
10.200.205.2232.232.9.431924702:1
=
Any suggestion?
--
CU,
Victor Gamov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD as multicast router
On 19/11/2019 03:49, Mike Karels wrote:
Hi All
Still trying to run FreeBSD-box as multicast router :-)
FreeBSD upgraded to 11.3-STABLE #1 r354778. netstat pacth by Mike Karels
manually applied and netstat -gs looks OK now.
Latest pimd version 3.0beta1 downloaded from git and configured. While
configure it report following:
=
-- Summary --
pimd version 3.0-beta1
Prefix: /usr/local
Sysconfdir: /usr/local/etc
Localstatedir.: /usr/local/var
C Compiler: cc -g -O2
Optional features:
Kernel register encap.: no
Kernel (*,G) support..: no
Kernel MAX VIFs...: 32
Memory save...: no
RSRR (experimental)...: no
Exit on error.: yes
=
What does "Kernel (*,G) support..: no" means?
Then my test multicast network configured (again)
--
-vlan298-| FreeBSD PIM router |-vlan299-| client |
|208.34/29 205.2/29| |205.5/29|
--
Two multicast generated by FreeBSD-router: one (232.232.9.43) sended
from vlan299 and another (232.232.88.173) from vlan298 both with TTL=20
Pimd started with following config:
=
phyint vlan299 enable ttl-threshold 20
phyint vlan298 enable ttl-threshold 20
rp-address 10.200.205.2 232.232.0.0/16
=
If the threshold is 20 and the TTL is 20, does that mean that the TTL is
just high enough, or is it at the cutoff? I'd try lowering the threshold
and/or increasing the TTL to see which it is. If the TTL is 20 on the
incoming side, it would be 19 on the outgoing side.
ttl-threshold changed to 10 in pimd.conf. `netstat -g` reports
Thresh=10 now.
Locally FreeBSD-router generated multicast vlan299 comes to receiver
with ttl=20. And it's OK.
Locally FreeBSD-router generated multicast vlan298 does not comes to
receiver.
Multicast generated from another sender on vlan298 comes to router with
TTL=20 but never comes to receiver via vlan299
Now client is requesting multicast which router is sending from vlan299
and client successfully receiving it. But when client is requests
multicast sending (by router) from vlan298 it doesn't receive it.
My first question: (in theory) is router must send multicast to client
in this situation?
In theory yes, modulo TTL and other checks.
I will reconfigure my test network to use dedicated FreeBSD-box as
multicast router with two only multicast interfaces to get more clear
info from `netstat -gs`
Also pimd periodically reports following
=
Kernel busy, retrying (1/3) routing socket read in one sec
=
Is it OK?
And more about pimd. It creates register_vif0 on startup. I assume it
uses this interface (not reported by `ifconfig`) to route all multicast
via. But `netstat -g` reports this interface with threshold=1. Is it OK?
--
CU,
Victor Gamov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD as multicast router
Looks like everything is OK, but multicast routed if S,G specified in JOIN only. Is it a FreBSD-specific limitation? Also `netstat -gs` reports about some errors: = IPv4 multicast forwarding: 973725 multicast forwarding cache lookups 15 multicast forwarding cache misses 498297 upcalls to multicast routing daemon 0 upcall queue overflows (!!!) 16459 upcalls dropped due to full socket buffer 0 cache cleanups 15 datagrams with no route for origin 0 datagrams arrived with bad tunneling 0 datagrams could not be tunneled 475659 datagrams arrived on wrong interface 0 datagrams selectively dropped 0 datagrams dropped due to queue overflow 0 datagrams dropped for being too large = On 19/11/2019 11:05, Victor Gamov wrote: On 19/11/2019 03:49, Mike Karels wrote: Hi All Still trying to run FreeBSD-box as multicast router :-) FreeBSD upgraded to 11.3-STABLE #1 r354778. netstat pacth by Mike Karels manually applied and netstat -gs looks OK now. Latest pimd version 3.0beta1 downloaded from git and configured. While configure it report following: = -- Summary -- pimd version 3.0-beta1 Prefix: /usr/local Sysconfdir: /usr/local/etc Localstatedir.: /usr/local/var C Compiler: cc -g -O2 Optional features: Kernel register encap.: no Kernel (*,G) support..: no Kernel MAX VIFs...: 32 Memory save...: no RSRR (experimental)...: no Exit on error.: yes = What does "Kernel (*,G) support..: no" means? Then my test multicast network configured (again) -- -vlan298-| FreeBSD PIM router |-vlan299-| client | |208.34/29 205.2/29| |205.5/29| -- Two multicast generated by FreeBSD-router: one (232.232.9.43) sended from vlan299 and another (232.232.88.173) from vlan298 both with TTL=20 Pimd started with following config: = phyint vlan299 enable ttl-threshold 20 phyint vlan298 enable ttl-threshold 20 rp-address 10.200.205.2 232.232.0.0/16 = If the threshold is 20 and the TTL is 20, does that mean that the TTL is just high enough, or is it at the cutoff? I'd try lowering the threshold and/or increasing the TTL to see which it is. If the TTL is 20 on the incoming side, it would be 19 on the outgoing side. ttl-threshold changed to 10 in pimd.conf. `netstat -g` reports Thresh=10 now. Locally FreeBSD-router generated multicast vlan299 comes to receiver with ttl=20. And it's OK. Locally FreeBSD-router generated multicast vlan298 does not comes to receiver. Multicast generated from another sender on vlan298 comes to router with TTL=20 but never comes to receiver via vlan299 Now client is requesting multicast which router is sending from vlan299 and client successfully receiving it. But when client is requests multicast sending (by router) from vlan298 it doesn't receive it. My first question: (in theory) is router must send multicast to client in this situation? In theory yes, modulo TTL and other checks. I will reconfigure my test network to use dedicated FreeBSD-box as multicast router with two only multicast interfaces to get more clear info from `netstat -gs` Also pimd periodically reports following = Kernel busy, retrying (1/3) routing socket read in one sec = Is it OK? And more about pimd. It creates register_vif0 on startup. I assume it uses this interface (not reported by `ifconfig`) to route all multicast via. But `netstat -g` reports this interface with threshold=1. Is it OK? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
IGMP on FreeBSD-12.1
Hi All
I have FreeBSD 12.1-STABLE r354850
When I started
ffmpeg -i
'udp://232.232.9.57:3344?localaddr=10.200.207.35&source=10.200.205.2'
then IGMP-Join sended out and ifmcstat reports about 232.232.9.57 on
proper interface.
I kill ffmpeg but ifmcstat still reports about 232.232.9.57 on interface.
Any following ffmpeg start does not generate IGMP-Join as I understand
because kernel think it still joined to this multicast.
Then I start this scenario on FreeBSD 11.3-STABLE #1 r354778 then group
subscription immediately removed from interface at the moment when
ffmpeg killed.
More things. When I request multicast on 11.3 from 12.1 then 11.3
respond to General-query like this:
=
10.200.207.42 > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr
232.232.99.7 is_in { 10.200.208.33 }]
=
When I request multicast on 12.1 from 11.3 then 12.1 respond to
General-query like this:
=
10.200.207.35 > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr
232.232.9.44 to_ex { }]
=
And when ffmpeg started on 12.1 first time as following
ffmpeg -i
'udp://@232.232.9.44:3344?localaddr=10.200.207.35&source=10.200.205.2'
then 12.1 generates IGMP-Join without source:
=
10.200.207.35 > 224.0.0.22: igmp v3 report, 1 group record(s) [gaddr
232.232.9.44 to_ex { }]
=
So, I assume 12.1 have some problem with IGMP
--
CU,
Victor Gamov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
enc0 as netflow exporter
Hi All I have FreeBSD box with many ipsec interfaces. Now I want to export Netflow and trying to use enc0 to export all ipsec traffic but `ngctl mkpeer enc0: netflow lower iface0` failed with: ngctl: send msg: No such file or directory Does enc0 allow to use netgraph? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
'dropped due to full socket buffers' by SNMP
Hi All Can somebody help me to get UDP 'dropped due to full socket buffers' by SNMP? Is it possible? Now I'm getting it with `netstat -n -p udp -f inet -s` but SNMP will be more useful for remote monitoring. Thanks! -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Best way to get per second interface statistic
Hi All I have trunk port with many VLANs attached to FreeBSD 12.2-STABLE box via ix0 interface. What is the best way to get per second traffic statistic for ix0 interface and every VLAN? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: 'dropped due to full socket buffers' by SNMP
And one more question about 'dropped due to full socket buffers': how to avoid it? Which params must be tuned? Thanks. On 30/11/2020 18:33, Victor Gamov wrote: Hi All Can somebody help me to get UDP 'dropped due to full socket buffers' by SNMP? Is it possible? Now I'm getting it with `netstat -n -p udp -f inet -s` but SNMP will be more useful for remote monitoring. Thanks! -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: 'dropped due to full socket buffers' by SNMP
Hi Eugene Thank for your reply On 30/12/2020 02:13, Eugene Grosbein wrote: 29.12.2020 23:36, Victor Gamov wrote: Please do not top-post. Thanks. On 30/11/2020 18:33, Victor Gamov wrote: Hi All Can somebody help me to get UDP 'dropped due to full socket buffers' by SNMP? Is it possible? Now I'm getting it with `netstat -n -p udp -f inet -s` but SNMP will be more useful for remote monitoring. And one more question about 'dropped due to full socket buffers': how to avoid it? Which params must be tuned? Both problems generally mean that outgoing link for that UDP packets is congested, so socket buffers start to fill until they are full. What is "media" for that UDP data - some syntetic tunnel or plain ethernet link or something else? It's 10G Intel card (via ix driver) attached to D-Link switch This Host-A got many multicast UDP streams via many VLANs and resend streams to one VLAN-750 (mainly). Every multicast serviced by its own process. Interface utilization about 1Gbit/s for receive and 800Mbit/s for transmit. You may find useful this old thread dealing with similar problem: http://freebsd.1045724.x6.nabble.com/SNMP-No-Bufferspace-td6333081.html Thanks for this link. 'vmstat -i | grep ix0' output followed: = irq264: ix0:rxq0 90527644527 21675 irq265: ix0:rxq1 95793018351 22936 irq266: ix0:rxq2 96075009541 23004 irq267: ix0:rxq3 93444280619 22374 irq268: ix0:aq 3 0 = 'netstat -idnh -W' show no errs/drops 'netstat -m' = 12525/7470/19995 mbufs in use (current/cache/total) 12484/4766/17250/1005602 mbuf clusters in use (current/cache/total/max) 158/1866 mbuf+clusters out of packet secondary zone in use (current/cache) 0/116/116/502800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/148978 9k jumbo clusters in use (current/cache/total/max) 0/0/0/83800 16k jumbo clusters in use (current/cache/total/max) 28099K/11863K/39962K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 sendfile syscalls 0 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 0 pages were valid at time of a sendfile request 0 pages were valid and substituted to bogus page 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 0 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayed == Currently I'm thinking about ethernet flow control: Host-B connected to VLAN-750 on the third switch has 1G link (via igb driver) and both Host-A and Host-B has fc=3. So when Host-B get microburst it can send PAUSE and Host-A start to fill queue. But FC disabled for all switches and I'm not sure about PAUSE can be propagated from Host-A to Host-B -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: 'dropped due to full socket buffers' by SNMP
On 30/12/2020 12:57, Eugene Grosbein wrote: 30.12.2020 16:44, Victor Gamov wrote: Currently I'm thinking about ethernet flow control: Host-B connected to VLAN-750 on the third switch has 1G link (via igb driver) and both Host-A and Host-B has fc=3. So when Host-B get microburst it can send PAUSE and Host-A start to fill queue. But FC disabled for all switches and I'm not sure about PAUSE can be propagated from Host-A to Host-B AFAIK, pause frames are not forwarded. For short term you should make flow control settings consistent between link partners, As I understand hw.ix.flow_control=3 to allow flow-control for negotiation. Real PAUSE setting will be set during negotiation. So where I can find active flow-control setting for host interface? maybe increase kern.ipc.maxsockbuf and then net.inet.udp.recvspace. Eugene, at first message you suppose Host-A (sender) "outgoing link for that UDP packets is congested" because this host shows non-zero "dropped due to full socket buffers". So is net.inet.udp.recvspace increasing on Host-B (mainly receiver) will be affected for this congestion? Or I need to try to increase both kern.ipc.maxsockbuf and net.inet.udp.recvspace on both hosts? Also how I can check current sockbuf usage? But for long term you should consider adding more links and create LACP aggregate so not input nor output link be close to congestion. I'll migrate to 10G on Host-B at next month. Thanks for your advise! -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: 'dropped due to full socket buffers' by SNMP
Hi Eugene! Thanks for your responces. And Happy New Year for everyone! On 01.01.2021 03:19, Eugene Grosbein wrote: 30.12.2020 23:08, Victor Gamov wrote: As I understand hw.ix.flow_control=3 to allow flow-control for negotiation. Real PAUSE setting will be set during negotiation. At the moment of congestion. As I understand PAUSE feature negotiated during auto-negotiation process. If flow-control disabled on one side (switch for example) then other side (host) will not to use this feature too. Is it right? So where I can find active flow-control setting for host interface? Can't check for ix just now, but for em(4) there is sysctl dev.em.0.fc. It should be similar for ix. I have hw.ix.flow_control=3 (what does is it means ?) and dev.ix.0.fc=3 (and what does is it means?) maybe increase kern.ipc.maxsockbuf and then net.inet.udp.recvspace. Eugene, at first message you suppose Host-A (sender) "outgoing link for that UDP packets is congested" because this host shows non-zero "dropped due to full socket buffers". So is net.inet.udp.recvspace increasing on Host-B (mainly receiver) will be affected for this congestion? Can't tell in details without going deep into your setup :-) You can try it yourself and verify quickly. Or I need to try to increase both kern.ipc.maxsockbuf and net.inet.udp.recvspace on both hosts? Tune one that drops UDP. Also how I can check current sockbuf usage? netstat -xn Unfortunately it never shoes counters about UDP multicast traffic. I'll increase kern.ipc.maxsockbuf and net.inet.udp.recvspace at next week and write about results. Back to my original question: is it possible to monitor `netstat -n -p udp -f inet -s` counters by SNMP? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: 'dropped due to full socket buffers' by SNMP
Hi All On 05.01.2021 12:39, Victor Gamov wrote: Hi Eugene! Thanks for your responces. And Happy New Year for everyone! On 01.01.2021 03:19, Eugene Grosbein wrote: 30.12.2020 23:08, Victor Gamov wrote: As I understand hw.ix.flow_control=3 to allow flow-control for negotiation. Real PAUSE setting will be set during negotiation. At the moment of congestion. As I understand PAUSE feature negotiated during auto-negotiation process. If flow-control disabled on one side (switch for example) then other side (host) will not to use this feature too. Is it right? So where I can find active flow-control setting for host interface? Can't check for ix just now, but for em(4) there is sysctl dev.em.0.fc. It should be similar for ix. I have hw.ix.flow_control=3 (what does is it means ?) and dev.ix.0.fc=3 (and what does is it means?) maybe increase kern.ipc.maxsockbuf and then net.inet.udp.recvspace. Eugene, at first message you suppose Host-A (sender) "outgoing link for that UDP packets is congested" because this host shows non-zero "dropped due to full socket buffers". So is net.inet.udp.recvspace increasing on Host-B (mainly receiver) will be affected for this congestion? Can't tell in details without going deep into your setup :-) You can try it yourself and verify quickly. Or I need to try to increase both kern.ipc.maxsockbuf and net.inet.udp.recvspace on both hosts? Tune one that drops UDP. Also how I can check current sockbuf usage? netstat -xn Unfortunately it never shoes counters about UDP multicast traffic. I'll increase kern.ipc.maxsockbuf and net.inet.udp.recvspace at next week and write about results. I increase kern.ipc.maxsockbuf from 2097152 -> 2597152 -> 3145728 but netstat -sn -p udp | grep 'dropped due to full socket buffers' still show dropped packets. Then I increase net.inet.udp.recvspace 84160 -> 105200 but 'dropped due to full socket buffers' packets still here. Do I need to try to increase something else? -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: 'dropped due to full socket buffers' by SNMP
On 22.01.2021 12:52, Eugene Grosbein wrote: 22.01.2021 16:27, Victor Gamov wrote: I increase kern.ipc.maxsockbuf from 2097152 -> 2597152 -> 3145728 but netstat -sn -p udp | grep 'dropped due to full socket buffers' still show dropped packets. Then I increase net.inet.udp.recvspace 84160 -> 105200 but 'dropped due to full socket buffers' packets still here. Do I need to try to increase something else? Increase of various buffers may help against very short traffic bursts or limited time interface congestion. It won't help in case you have permanent (or nearly permanent) link overload. No link overload: this host attached to network via 10G ix0, many VLANs on this ix0 and some sender on every VLAN sends multicast traffic to this host. Total input traffic about 1Gbit/s -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: 'dropped due to full socket buffers' by SNMP
On 22.01.2021 13:21, Eugene Grosbein wrote: 22.01.2021 17:02, Victor Gamov wrote: No link overload: this host attached to network via 10G ix0, many VLANs on this ix0 and some sender on every VLAN sends multicast traffic to this host. Total input traffic about 1Gbit/s What FreeBSD version do you use currently? Do you use IPv6 for UDP or IPv4 only? FreeBSD 12.2-STABLE r366543 GENERIC amd64 UDP-4 only -- CU, Victor Gamov ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: 'dropped due to full socket buffers' by SNMP
On 22.01.2021 16:09, Eugene Grosbein wrote:
22.01.2021 19:28, Victor Gamov wrote:
On 22.01.2021 13:21, Eugene Grosbein wrote:
22.01.2021 17:02, Victor Gamov wrote:
No link overload: this host attached to network via 10G ix0, many
VLANs on this ix0 and some sender on every VLAN sends multicast
traffic to this host. Total input traffic about 1Gbit/s
What FreeBSD version do you use currently? Do you use IPv6 for UDP or
IPv4 only?
FreeBSD 12.2-STABLE r366543 GENERIC amd64
UDP-4 only
In case of IPv4 UDP the counter "dropped due to full socket buffers"
is increased for incoming packets only. Therefore, the problem is in a code
processing incoming stream(s): either it locks for long time on something,
or it just has no enough raw CPU horsepower to deal with incoming stream.
Look at "top -SHPI" CPU counters, if your CPU cores are loaded evenly;
it's CPU E3-1270 v6 @ 3.80GHz with WCPU about 60% idle + 9%
kernel{if_io_tqg_X) + 5% intr{swi1: netisr X). And many processes about
1% WCPU for multicast receive/resend (one for every multicast)
Also "netstat -na -p udp" shows me zero or very small Recv-Q
if some of cores became overloaded sometimes. You should also draw per-cpu load
graphs.
(f.e. sysctl kern.cp_times)
I have SNMP stats and it show me about 40% load
I have no visible problems with growing "dropped due to full socket
buffers" but I think it's not well when this counter increasing.
--
CU,
Victor Gamov
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
