Hi Joao,
just a short follow up:
I can reproduce the issue with OpenBSD 6.6 and -current (from today).
I could verify that it is related to the additional route with different prio.
As a workaround for your prod environment you could filter this host route
in bgpd.
I'll look into finding a fix for this issue. Of course I can't promise.
Remi
On Fri, Nov 08, 2019 at 01:51:22PM +0100, Joao Alves wrote:
> Hi Remi,
>
> Just tried with 1 FW and static route, yes it's still reproducible.
>
> The previous e-mail outputs were all related to my lab, when you see
> IP's like 192.168.98.x, it's lab.
>
>
> Regarding host1/2 interface configs:
>
> Host1:
>
> host1# more
> /etc/hostname.em0
>
>
>
> up
> inet 192.168.98.202 255.255.255.0 NONE
> !route add default 192.168.98.1
> host1# more
> /etc/hostname.carp1
>
>
>
> up
> inet 10.10.10.10 255.255.255.255 NONE vhid 1 pass 4k5EKt2svYNzSf carpdev
> em0 advskew 0
> host1#
>
>
> Host2:
>
> host2# more
> /etc/hostname.em0
>
>
>
> up
> inet 192.168.98.203 255.255.255.0 NONE
> !route add default 192.168.98.1
> host2# more
> /etc/hostname.carp1
>
>
>
> up
> inet 10.10.10.10 255.255.255.255 NONE vhid 1 pass 4k5EKt2svYNzSf carpdev
> em0 advskew 128
> host2#
>
>
> The interface I kill/restore is the em0. I do it on VMware VSphere by
> disabling it for Host1.
>
>
> So, the steps to reproduce with one FW and static route are the same,
> but I'll detail them again.
>
> Starting and working condition:
>
> fw1# uptime && ospfctl show rib
> 12:10PM up 15 days, 1:40, 1 user, load averages: 0.00, 0.00, 0.00
> Destination Nexthop Path Type Type Cost
> Uptime
> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
> 00:01:03
> 192.168.98.0/24 192.168.98.200 C Intra-Area Network 10
> 00:01:08
>
> fw1# uptime && route -n show | grep 10.10.10
> 12:11PM up 15 days, 1:41, 1 user, load averages: 0.00, 0.00, 0.00
> 10.10.10.10/32 192.168.98.202 UG 0 0 - 32
> em0
> 10.10.10.10/32 192.168.98.204 UGS 0 0 - 40
> em0
> fw1#
>
> Route is set to Host1 without MPATH flag.
>
>
> Now I kill em0 in VSphere for Host1:
>
> fw1#
> fw1# uptime && ospfctl show rib
> 12:13PM up 15 days, 1:43, 1 user, load averages: 0.00, 0.00, 0.00
> Destination Nexthop Path Type Type Cost
> Uptime
> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
> 00:03:57
> 10.10.10.10/32 192.168.98.203 Intra-Area Network 20
> 00:00:23
> 192.168.98.0/24 192.168.98.200 C Intra-Area Network 10
> 00:04:02
>
> fw1# uptime && route -n show | grep 10.10.10
> 12:13PM up 15 days, 1:43, 1 user, load averages: 0.00, 0.00, 0.00
> 10.10.10.10/32 192.168.98.202 UGP 0 6 - 32
> em0
> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
> em0
> 10.10.10.10/32 192.168.98.204 UGS 0 0 - 40
> em0
> fw1#
>
> We are working with standard OSPF timers in LAB, so the dead route stays
> for 40s and then is removed.
>
> The MPATH flag appears because this was true for 40s, although the route
> for host1 results now in a blackhole, we had temporarily 2 routes with
> exact metrics, only with different nexthop.
>
> The old route is then cleared, but MPATH flag stays on the new route.
>
>
> fw1# uptime && ospfctl show rib
> 12:14PM up 15 days, 1:43, 1 user, load averages: 0.00, 0.00, 0.00
> Destination Nexthop Path Type Type Cost
> Uptime
> 10.10.10.10/32 192.168.98.203 Intra-Area Network 20
> 00:00:43
> 192.168.98.0/24 192.168.98.200 C Intra-Area Network 10
> 00:04:22
>
> fw1# uptime && route -n show | grep 10.10.10
> 12:14PM up 15 days, 1:44, 1 user, load averages: 0.00, 0.00, 0.00
> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
> em0
> 10.10.10.10/32 192.168.98.204 UGS 0 0 - 40
> em0
> fw1#
> fw1#
>
> But at this stage we are still with the correct nexthop in the FIB
>
>
>
> We now bring up the Host1 em0 in VSphere:
>
> fw1# uptime && ospfctl show rib
> 12:22PM up 15 days, 1:52, 1 user, load averages: 0.00, 0.00, 0.00
> Destination Nexthop Path Type Type Cost
> Uptime
> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
> 00:00:03
> 192.168.98.0/24 192.168.98.200 C Intra-Area Network 10
> 00:12:28
>
> fw1# uptime && route -n show | grep 10.10.10
> 12:22PM up 15 days, 1:52, 1 user, load averages: 0.00, 0.00, 0.00
> 10.10.10.10/32 192.168.98.203 UGP 0 6 - 32
> em0
> 10.10.10.10/32 192.168.98.204 UGS 0 0 - 40
> em0
> fw1#
> fw1#
>
> You receive the route from Host1, but is not installed on FIB.
>
> fw1# uptime && ospfctl show rib
> 1:39PM up 15 days, 3:09, 1 user, load averages: 0.00, 0.00, 0.00
> Destination Nexthop Path Type Type Cost
> Uptime
> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
> 01:17:11
> 192.168.98.0/24 192.168.98.200 C Intra-Area Network 10
> 01:29:36
>
> fw1# uptime && route -n show | grep 10.10.10
> 1:39PM up 15 days, 3:09, 1 user, load averages: 0.00, 0.00, 0.00
> 10.10.10.10/32 192.168.98.203 UGP 0 7 - 32
> em0
> 10.10.10.10/32 192.168.98.204 UGS 0 0 - 40
> em0
> fw1#
>
>
> This is still true after more then 1h.
>
>
>
> We now reload the FIB:
>
>
> fw1# ospfctl fib reload
> reload request sent.
> fw1#
> fw1# uptime && ospfctl show rib
> 1:40PM up 15 days, 3:10, 1 user, load averages: 0.00, 0.00, 0.00
> Destination Nexthop Path Type Type Cost
> Uptime
> 192.168.98.0/24 0.0.0.0 C Intra-Area Network 10
> 00:00:02
>
> fw1# uptime && route -n show | grep 10.10.10
> 1:40PM up 15 days, 3:10, 1 user, load averages: 0.00, 0.00, 0.00
> 10.10.10.10/32 192.168.98.204 UGS 0 0 - 40
> em0
> fw1#
> fw1#
>
> fw1# uptime && ospfctl show rib
> 1:41PM up 15 days, 3:11, 1 user, load averages: 0.00, 0.00, 0.00
> Destination Nexthop Path Type Type Cost
> Uptime
> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
> 00:00:19
> 192.168.98.0/24 192.168.98.200 C Intra-Area Network 10
> 00:00:24
>
> fw1# uptime && route -n show | grep 10.10.10
> 1:41PM up 15 days, 3:11, 1 user, load averages: 0.00, 0.00, 0.00
> 10.10.10.10/32 192.168.98.202 UG 0 0 - 32
> em0
> 10.10.10.10/32 192.168.98.204 UGS 0 0 - 40
> em0
> fw1#
>
>
> And we get to the good state condition.
>
>
> Hope this helps.
>
>
> Best regards,
>
> João
>
>
>
>
> On 05.11.19 23:19, Remi Locherer wrote:
> > On Thu, Oct 24, 2019 at 02:09:09PM +0200, Joao Alves wrote:
> >> Hi Remi,
> >>
> >> I've installed a lab with OpenBSD6.6 VM's to see if would happen in the
> >> newer version.
> >>
> >> I was able to reproduce it again, but in slightly different manner.
> >>
> >> First of all, you need to have BGP running in FW's also, and have the
> >> same route received through BGP, otherwise the issue is not
> >> reproducible, because the MPATH flag will behave well with OSPF only.
> >> Without the MPATH issue you can't reproduce the rest.
> > Can you also reproduce without BGP but with a static route added like this:
> > route add -net 10.10.10.10 -netmask 255.255.255.255 192.168.98.204
> > -priority 40
> >
> >> So LAB setup(all openbsd6.6 VM's):
> >>
> >> 2x fw/router
> > Can you also reproduce with only 1 fw or are both fw nodes needed to
> > reproduce?
> >
> >> 2x host
> >> 1x bgp router
> >>
> >> fw1:192.168.98.200
> >>
> >> fw2:192.168.98.201
> >>
> >> host1:192.168.98.202
> >>
> >> host2:192.168.98.203
> >>
> >> bgp:192.168.98.204
> >>
> >>
> >> In the hosts I run carp with VIP 10.10.10.10/32, carp configured with
> >> preempt in kernel.
> >>
> >> ospf config for host1/2 is (only router id change):
> >>
> >> host1# more
> >> /etc/ospfd.conf
> >>
> >>
> >>
> >>
> >> # $OpenBSD: ospfd.conf,v 1.2 2018/08/07 07:06:20 claudio Exp $
> >>
> >> # macros
> >> id="192.168.98.202"
> >>
> >> # global configuration
> >> router-id $id
> >> # fib-update no
> >> # stub router no
> >> # spf-delay 1
> >> # spf-holdtime 5
> >>
> >> # auth-key secret
> >> # auth-type simple
> >> # hello-interval 10
> >> metric 10
> >> # retransmit-interval 5
> >> # router-dead-time 40
> >> router-priority 0
> >> # transmit-delay 1
> >>
> >> # rtlabel "DMZ" external-tag 1
> >>
> >> # areas
> >> area 0.0.0.0 {
> >> interface em0 {
> >> auth-type simple
> >> auth-key secret
> >> }
> >>
> >> interface carp1 {
> >> passive
> >> }
> >> }
> >> host1#
> > Please show the configs of em0 and carp1.
> >
> >> For FW1/2 is(router ID and router priority change, in FW2 priority is
> >> 10, so BDR):
> >>
> >> fw1# more
> >> /etc/ospfd.conf
> >>
> >>
> >>
> >>
> >> # $OpenBSD: ospfd.conf,v 1.2 2018/08/07 07:06:20 claudio Exp $
> >>
> >> # macros
> >> id="192.168.98.200"
> >>
> >> # global configuration
> >> router-id $id
> >> # fib-update no
> >> # stub router no
> >> # spf-delay 1
> >> # spf-holdtime 5
> >>
> >> # auth-key secret
> >> # auth-type simple
> >> # hello-interval 10
> >> metric 10
> >> # retransmit-interval 5
> >> # router-dead-time 40
> >> router-priority 100
> >> # transmit-delay 1
> >>
> >> # rtlabel "DMZ" external-tag 1
> >>
> >> # areas
> >> area 0.0.0.0 {
> >> interface em0 {
> >> auth-type simple
> >> auth-key secret
> >> }
> >> }
> >> fw1#
> >>
> >> For BGPD configs:
> >>
> >> FW1/2:
> >>
> >>
> >> fw1# more
> >> /etc/bgpd.conf
> >>
> >>
> >>
> >>
> >> # $OpenBSD: bgpd.conf,v 1.15 2018/11/17 17:22:38 deraadt Exp $
> >> # example bgpd configuration file, see bgpd.conf(5)
> >>
> >> # define our own ASN as a macro
> >> ASN="65123"
> >>
> >> # global configuration
> >> AS $ASN
> >> router-id 192.168.98.200
> >>
> >> # list of networks that may be originated by our ASN
> >> prefix-set mynetworks { \
> >> 192.0.6.0/24 \
> >> 2001:db8:abef::/48 \
> >> }
> >>
> >> # define bogon prefixes which should not be part of the DFZ
> >> prefix-set bogons {
> >> 0.0.0.0/8 or-longer # 'this' network [RFC1122]
> >> 10.0.0.0/8 or-longer # private space [RFC1918]
> >> 100.64.0.0/10 or-longer # CGN Shared [RFC6598]
> >> 127.0.0.0/8 or-longer # localhost [RFC1122]
> >> 169.254.0.0/16 or-longer # link local [RFC3927]
> >> 172.16.0.0/12 or-longer # private space [RFC1918]
> >> 192.0.2.0/24 or-longer # TEST-NET-1 [RFC5737]
> >> 192.88.99.0/24 or-longer # 6to4 anycast relay [RFC7526]
> >> 192.168.0.0/16 or-longer # private space [RFC1918]
> >> 198.18.0.0/15 or-longer # benchmarking [RFC2544]
> >> 198.51.100.0/24 or-longer # TEST-NET-2 [RFC5737]
> >> 203.0.113.0/24 or-longer # TEST-NET-3 [RFC5737]
> >> 224.0.0.0/4 or-longer # multicast
> >> 240.0.0.0/4 or-longer # reserved for future use
> >> ::/8 or-longer # RFC 4291 IPv4-compatible,
> >> loopback, et al
> >> 0100::/64 or-longer # Discard-Only [RFC6666]
> >> 2001:2::/48 or-longer # BMWG [RFC5180]
> >> 2001:10::/28 or-longer # ORCHID [RFC4843]
> >> 2001:db8::/32 or-longer # docu range [RFC3849]
> >> 2002::/16 or-longer # 6to4 anycast relay [RFC7526]
> >> 3ffe::/16 or-longer # old 6bone
> >> fc00::/7 or-longer # unique local unicast
> >> fe80::/10 or-longer # link local unicast
> >> fec0::/10 or-longer # old site local unicast
> >> ff00::/8 or-longer # multicast
> >> }
> >>
> >> # Generate routes for the networks our ASN will originate.
> >> # The communities (read 'tags') are later used to match on what
> >> # is announced to EBGP neighbors
> >> network prefix-set mynetworks set large-community $ASN:1:1
> >>
> >> # assume simple network with 3 routers in IBGP full mesh
> >> group "ibgp mesh v4" {
> >> remote-as $ASN
> >> # use loopback for IBGP sessions, assume its distributed in OSPF
> >> local-address 192.168.98.200
> >> neighbor 192.168.98.204 # router 2 ipv4
> >> # neighbor 192.168.98.201 # router 3 ipv4
> >> }
> >> # define the IPv6 IBGP sessions
> >> group "ibgp mesh v6" {
> >> remote-as $ASN
> >> local-address 2001:db8:abcd::1
> >> neighbor 2001:db8:abcd::2 # router 2 ipv6
> >> neighbor 2001:db8:abcd::3 # router 3 ipv6
> >> }
> >>
> >> # upstream providers
> >> group "upstreams" {
> >> neighbor 203.0.113.1 {
> >> remote-as 65002
> >> descr "IPv4 Transit Provider A"
> >> }
> >> neighbor 198.51.100.0 {
> >> remote-as 65123
> >> descr "IPv4 Transit provider B"
> >> }
> >> neighbor 2001:db8:666::2 {
> >> remote-as 65123
> >> descr "IPv6 Transit provider B"
> >> }
> >> }
> >>
> >> ## rules section
> >>
> >> # uncomment the following two lines to accept a default route from
> >> upstreams
> >> #allow from group upstreams prefix 0.0.0.0/0
> >> #allow from group upstreams prefix ::/0
> >>
> >> ### for simple BGP setups, no editing below this line is required ###
> >>
> >> # Outbound EBGP: only allow self originated networks to ebgp peers
> >> # Don't leak any routes from upstream or peering sessions. This is done
> >> # by checking for routes that are tagged with the large-community $ASN:1:1
> >> allow to ebgp prefix-set mynetworks large-community $ASN:1:1
> >>
> >> # deny more-specifics of our own originated prefixes
> >> deny quick from ebgp prefix-set mynetworks or-longer
> >>
> >> # IBGP: allow all updates to and from our IBGP neighbors
> >> allow from ibgp
> >> allow to ibgp
> >>
> >> # Scrub normal and large communities relevant to our ASN from EBGP
> >> neighbors
> >> # https://tools.ietf.org/html/rfc7454#section-11
> >> match from ebgp set { community delete $ASN:* }
> >> match from ebgp set { large-community delete $ASN:*:* }
> >>
> >> # filter out prefixes longer than 24 or shorter than 8 bits for IPv4
> >> # and longer than 48 or shorter than 16 bits for IPv6.
> >> allow from any inet prefixlen 8 - 24
> >> allow from any inet6 prefixlen 16 - 48
> >>
> >> # Honor requests to gracefully shutdown BGP sessions
> >> # https://tools.ietf.org/html/rfc8326
> >> #match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
> >>
> >> #deny quick from any prefix-set bogons
> >>
> >> # filter bogon AS numbers
> >> # AS_TRANS (23456) is not supposed to show up in any path and indicates a
> >> # missconfiguration. Additionally Private or Reserved ASNs have no place in
> >> # the public DFZ.
> >> http://www.iana.org/assignments/as-numbers/as-numbers.xhtml
> >> #deny quick from any AS 23456
> >> #deny quick from any AS 64496 - 131071
> >> #deny quick from any AS 4200000000 - 4294967295
> >>
> >> # filter out too long paths
> >> deny from any max-as-len 100
> >> fw1#
> >>
> >>
> >> For BGP router:
> >>
> >>
> >> bgp1# more /etc/bgpd.conf
> >> # $OpenBSD: bgpd.conf,v 1.15 2018/11/17 17:22:38 deraadt Exp $
> >> # example bgpd configuration file, see bgpd.conf(5)
> >>
> >> # define our own ASN as a macro
> >> ASN="65123"
> >>
> >> # global configuration
> >> AS $ASN
> >> router-id 192.168.98.204
> >>
> >> # list of networks that may be originated by our ASN
> >> prefix-set mynetworks { \
> >> 192.0.2.0/24 \
> >> 2001:db8:abcd::/48 \
> >> 10.10.10.10/32 \
> >> }
> >>
> >> # define bogon prefixes which should not be part of the DFZ
> >> prefix-set bogons {
> >> 0.0.0.0/8 or-longer # 'this' network [RFC1122]
> >> 10.0.0.0/8 or-longer # private space [RFC1918]
> >> 100.64.0.0/10 or-longer # CGN Shared [RFC6598]
> >> 127.0.0.0/8 or-longer # localhost [RFC1122]
> >> 169.254.0.0/16 or-longer # link local [RFC3927]
> >> 172.16.0.0/12 or-longer # private space [RFC1918]
> >> 192.0.2.0/24 or-longer # TEST-NET-1 [RFC5737]
> >> 192.88.99.0/24 or-longer # 6to4 anycast relay [RFC7526]
> >> 192.168.0.0/16 or-longer # private space [RFC1918]
> >> 198.18.0.0/15 or-longer # benchmarking [RFC2544]
> >> 198.51.100.0/24 or-longer # TEST-NET-2 [RFC5737]
> >> 203.0.113.0/24 or-longer # TEST-NET-3 [RFC5737]
> >> 224.0.0.0/4 or-longer # multicast
> >> 240.0.0.0/4 or-longer # reserved for future use
> >> ::/8 or-longer # RFC 4291 IPv4-compatible,
> >> loopback, et al
> >> 0100::/64 or-longer # Discard-Only [RFC6666]
> >> 2001:2::/48 or-longer # BMWG [RFC5180]
> >> 2001:10::/28 or-longer # ORCHID [RFC4843]
> >> 2001:db8::/32 or-longer # docu range [RFC3849]
> >> 2002::/16 or-longer # 6to4 anycast relay [RFC7526]
> >> 3ffe::/16 or-longer # old 6bone
> >> fc00::/7 or-longer # unique local unicast
> >> fe80::/10 or-longer # link local unicast
> >> fec0::/10 or-longer # old site local unicast
> >> ff00::/8 or-longer # multicast
> >> }
> >>
> >> # Generate routes for the networks our ASN will originate.
> >> # The communities (read 'tags') are later used to match on what
> >> # is announced to EBGP neighbors
> >> network prefix-set mynetworks set large-community $ASN:1:1
> >>
> >> # assume simple network with 3 routers in IBGP full mesh
> >> group "ibgp mesh v4" {
> >> remote-as $ASN
> >> # use loopback for IBGP sessions, assume its distributed in OSPF
> >> local-address 192.168.98.204
> >> neighbor 192.168.98.200 # router 2 ipv4
> >> neighbor 192.168.98.201 # router 3 ipv4
> >> }
> >> # define the IPv6 IBGP sessions
> >> group "ibgp mesh v6" {
> >> remote-as $ASN
> >> local-address 2001:db8:abcd::1
> >> neighbor 2001:db8:abcd::2 # router 2 ipv6
> >> neighbor 2001:db8:abcd::3 # router 3 ipv6
> >> }
> >>
> >> # upstream providers
> >> group "upstreams" {
> >> neighbor 203.0.113.1 {
> >> remote-as 65002
> >> descr "IPv4 Transit Provider A"
> >> }
> >> neighbor 198.51.100.0 {
> >> remote-as 65123
> >> descr "IPv4 Transit provider B"
> >> }
> >> neighbor 2001:db8:666::2 {
> >> remote-as 65123
> >> descr "IPv6 Transit provider B"
> >> }
> >> }
> >>
> >> ## rules section
> >>
> >> # uncomment the following two lines to accept a default route from
> >> upstreams
> >> #allow from group upstreams prefix 0.0.0.0/0
> >> #allow from group upstreams prefix ::/0
> >>
> >> ### for simple BGP setups, no editing below this line is required ###
> >>
> >> # Outbound EBGP: only allow self originated networks to ebgp peers
> >> # Don't leak any routes from upstream or peering sessions. This is done
> >> # by checking for routes that are tagged with the large-community $ASN:1:1
> >> allow to ebgp prefix-set mynetworks large-community $ASN:1:1
> >>
> >> # deny more-specifics of our own originated prefixes
> >> deny quick from ebgp prefix-set mynetworks or-longer
> >>
> >> # IBGP: allow all updates to and from our IBGP neighbors
> >> allow from ibgp
> >> allow to ibgp
> >>
> >> # Scrub normal and large communities relevant to our ASN from EBGP
> >> neighbors
> >> # https://tools.ietf.org/html/rfc7454#section-11
> >> match from ebgp set { community delete $ASN:* }
> >> match from ebgp set { large-community delete $ASN:*:* }
> >>
> >> # filter out prefixes longer than 24 or shorter than 8 bits for IPv4
> >> # and longer than 48 or shorter than 16 bits for IPv6.
> >> allow from any inet prefixlen 8 - 24
> >> allow from any inet6 prefixlen 16 - 48
> >>
> >> # Honor requests to gracefully shutdown BGP sessions
> >> # https://tools.ietf.org/html/rfc8326
> >> #match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
> >>
> >> #deny quick from any prefix-set bogons
> >>
> >> # filter bogon AS numbers
> >> # AS_TRANS (23456) is not supposed to show up in any path and indicates a
> >> # missconfiguration. Additionally Private or Reserved ASNs have no place in
> >> # the public DFZ.
> >> http://www.iana.org/assignments/as-numbers/as-numbers.xhtml
> >> #deny quick from any AS 23456
> >> #deny quick from any AS 64496 - 131071
> >> #deny quick from any AS 4200000000 - 4294967295
> >>
> >> # filter out too long paths
> >> deny from any max-as-len 100
> >> bgp1#
> >>
> >>
> >>
> >> Reproduce the issue:
> >>
> >>
> >> The problem in this case is faced in the recovery of the failed host
> >> instead.
> >>
> >> start condition:
> >>
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UG 0 0 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1#
> >>
> >> We now force the MPATH flag by temporarly advertise the network from
> >> host 2. For this we do "ifconfig carp1 state master" on the backup host2:
> >>
> >>
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UG 0 3 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UG 0 3 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UGP 0 3 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UGP 0 3 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UGP 0 3 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UGP 0 3 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1#
> >>
> >>
> >> We now have the MPATH flag active, even having only one OSPF route.
> >>
> >>
> >> So, we now kill the interface of host1:
> > Which interface? How do you kill it?
> >
> >>
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UGP 0 3 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UGP 0 6 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# ospfctl show rib | grep 10.10.10
> >> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
> >> 00:58:16
> >> 10.10.10.10/32 192.168.98.203 Intra-Area Network 20
> >> 00:00:07
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UGP 0 9 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# ospfctl show rib | grep 10.10.10
> >> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
> >> 00:58:23
> >> 10.10.10.10/32 192.168.98.203 Intra-Area Network 20
> >> 00:00:14
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UGP 0 9 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UGP 0 9 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UGP 0 9 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# ospfctl show rib | grep 10.10.10
> >> 10.10.10.10/32 192.168.98.203 Intra-Area Network 20
> >> 00:00:24
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1#
> >>
> >>
> >> In this output we see that temporarily we have both OSPF routes, and
> >> then the first one timesout living the one related to host2.
> >> This behaviour is different from what we have in production, so in this
> >> case, at this stage we still have service because route ended up with
> >> correct next-hop for host2.
> >>
> >> But again, MPATH flag was kept active when it shouldn't.
> >>
> >> so current status is:
> >>
> >>
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# ospfctl show rib | grep 10.10.10
> >> 10.10.10.10/32 192.168.98.203 Intra-Area Network 20
> >> 00:06:11
> >> fw1#
> >>
> >>
> >> Now we raise interface on host1 and get service down:
> > Again: which interface and how?
> >
> >>
> >>
> >> fw1# ospfctl show rib | grep 10.10.10
> >> 10.10.10.10/32 192.168.98.203 Intra-Area Network 65545
> >> 00:07:19
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.203 UGP 0 6 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# ospfctl show rib | grep 10.10.10
> >> 10.10.10.10/32 192.168.98.203 Intra-Area Network 65545
> >> 00:07:24
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.203 UGP 0 6 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# ospfctl show rib | grep 10.10.10
> >> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
> >> 00:00:04
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.203 UGP 0 6 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# ospfctl show rib | grep 10.10.10
> >> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
> >> 00:00:10
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.203 UGP 0 6 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# ospfctl show rib | grep 10.10.10
> >> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
> >> 00:00:15
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.203 UGP 0 6 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# ospfctl show rib | grep 10.10.10
> >> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
> >> 00:00:19
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.203 UGP 0 6 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1#
> > This output (and following similar ones) is a bit hard to read. I'd prefere
> > to see the output for the 1st state and then the output for the 2nd state.
> > Maybe with a comment about the elapsed time in between.
> >
> >>
> >>
> >> Notice that the FIB next-hop wasn't updated to 192.168.98.202.
> >>
> >>
> >>
> >> Now the fix (commands issued repeatedly):
> >>
> >>
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.203 UGP 0 9 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# ospfctl fib
> >> reload
> >>
> >>
> >>
> >>
> >> reload request sent.
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.203 UG 0 0 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.203 UG 0 0 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UG 0 0 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1# route -n show | grep 10.10.10.10
> >> 10.10.10.10/32 192.168.98.202 UG 0 0 - 32
> >> em0
> >> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
> >> em0
> >> fw1#
> >>
> >>
> >> We still see temporarily the wrong next-hop, but it then converges to
> >> the correct one, and we now don't have the MPATH flag anymore.
> > Is this about your prod or your lab setup?
> >
> >>
> >> Hope this helps in your lab setup, as my real scenario is a bit
> >> different behaviour, prbably because I'm using quagga to advertise the
> >> prefix with network command, and the prefix is configured on same
> >> physical interface.
> >>
> >> In this lab scenario, the interface with the VIP is diferent from the
> >> main OSPF interface, and because of that I was able to announce it with
> >> passive command.
> > In ospfd carp interfaces are always announced as passive.
> >
> >> For the quagga's, there are constantly hello's being sent for the
> >> prefix, even then no adjacency is formed on that prefix subnet.
> >> In this LAB scenario you see updates being trade about the prefix.
> >>
> >> This is the captures in the lab environment when I issue the ospfctl fib
> >> reload (captures being taken on fw2):
> >>
> >>
> >> 13:46:36.739839 192.168.98.200 > 224.0.0.5: OSPFv2-hello 44: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 34942, len 64)
> >> 13:46:36.740057 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 10 dead 40 dr 192.168.98.201 nbrs 192.168.98.200 192.168.98.202
> >> 192.168.98.203 [tos 0xc0] [ttl 1] (id 20542, len 76)
> >> 13:46:36.740182 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> >> 192.168.98.203 [tos 0xc0] [ttl 1] (id 47644, len 76)
> >> 13:46:36.740269 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> >> 192.168.98.203 [tos 0xc0] [ttl 1] (id 30044, len 76)
> >> 13:46:36.740550 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd 100: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
> >> 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs 192.168.98.202
> >> 192.168.98.203 192.168.98.201 } { E S 8000000D age 1 rtr 192.168.98.201
> >> } [|ospf] [tos 0xc0] [ttl 1] (id 26274, len 120)
> >> 13:46:36.741005 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> >> 192.168.98.202 [tos 0xc0] [ttl 1] (id 59174, len 76)
> >> 13:46:36.741692 192.168.98.200 > 224.0.0.5: OSPFv2-hello 44: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 48099, len 64)
> >> 13:46:38.743078 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack 64: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
> >> 192.168.98.201 if 192.168.98.201 } { E S 8000000D age 1 rtr
> >> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 55607, len 84)
> >> 13:46:38.744309 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack 64: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
> >> 192.168.98.201 if 192.168.98.201 } { E S 8000000D age 1 rtr
> >> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 31253, len 84)
> >> 13:46:41.752541 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd 76: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" { E S 80000021 age 6 rtr
> >> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> >> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> >> [ttl 1] (id 9379, len 96)
> >> 13:46:41.752841 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd 76: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" { E S 80000021 age 7 rtr
> >> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> >> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> >> [ttl 1] (id 52657, len 96)
> >> 13:46:41.753134 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd 76: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" { E S 80000017 age 6 rtr
> >> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> >> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> >> [ttl 1] (id 6608, len 96)
> >> 13:46:41.753335 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd 76: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" { E S 80000017 age 7 rtr
> >> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> >> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> >> [ttl 1] (id 22390, len 96)
> >> 13:46:43.747722 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd 76:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000017 age 9 rtr
> >> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> >> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> >> [ttl 1] (id 29312, len 96)
> >> 13:46:43.747795 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd 76:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000021 age 9 rtr
> >> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> >> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> >> [ttl 1] (id 42301, len 96)
> >> 13:46:43.748098 192.168.98.203 > 192.168.98.201: OSPFv2-ls_ack 44:
> >> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 80000017 age 9 rtr
> >> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 56865, len 64)
> >> 13:46:43.748136 192.168.98.202 > 192.168.98.201: OSPFv2-ls_ack 44:
> >> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000021 age 9 rtr
> >> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 43582, len 64)
> >> 13:46:43.762045 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack 44: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" { E S 80000017 age 7 rtr
> >> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 1486, len 64)
> >> 13:46:43.763373 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack 44: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" { E S 80000021 age 7 rtr
> >> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 2479, len 64)
> >> 13:46:46.742883 192.168.98.200 > 224.0.0.5: OSPFv2-hello 44: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 41604, len 64)
> >> 13:46:46.743098 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> >> 192.168.98.203 [tos 0xc0] [ttl 1] (id 732, len 76)
> >> 13:46:46.743099 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
> >> 192.168.98.202 [tos 0xc0] [ttl 1] (id 28204, len 76)
> >> 13:46:46.743141 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 10 dead 40 dr 192.168.98.201 nbrs 192.168.98.200 192.168.98.202
> >> 192.168.98.203 [tos 0xc0] [ttl 1] (id 35443, len 76)
> >> 13:46:46.744985 192.168.98.200 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 100 dead 40 dr 192.168.98.200 nbrs 192.168.98.203 192.168.98.201
> >> 192.168.98.202 [tos 0xc0] [ttl 1] (id 33192, len 76)
> >> 13:46:46.745048 192.168.98.200 > 192.168.98.201: OSPFv2-dd 32: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF222A
> >> [tos 0xc0] [ttl 1] (id 6154, len 52)
> >> 13:46:46.745344 192.168.98.201 > 192.168.98.200: OSPFv2-dd 32: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF222B
> >> [tos 0xc0] [ttl 1] (id 38910, len 52)
> >> 13:46:46.745345 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> >> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17052, len 76)
> >> 13:46:46.745399 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.200 nbrs 192.168.98.201 192.168.98.200
> >> 192.168.98.203 [tos 0xc0] [ttl 1] (id 30271, len 76)
> >> 13:46:46.745399 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.200 nbrs 192.168.98.201 192.168.98.200
> >> 192.168.98.202 [tos 0xc0] [ttl 1] (id 49111, len 76)
> >> 13:46:46.745530 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd 100: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
> >> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
> >> { E S 80000001 age 1:00:00 net dr 192.168.98.201 if 192.168.98.201 }
> >> [|ospf] [tos 0xc0] [ttl 1] (id 42835, len 120)
> >> 13:46:46.745579 192.168.98.203 > 192.168.98.201: OSPFv2-dd 32: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DEE91C
> >> [tos 0xc0] [ttl 1] (id 39759, len 52)
> >> 13:46:46.745580 192.168.98.202 > 192.168.98.201: OSPFv2-dd 32: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 361091BE
> >> [tos 0xc0] [ttl 1] (id 15662, len 52)
> >> 13:46:46.745654 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> >> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 41333, len 76)
> >> 13:46:46.745657 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> >> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 35399, len 76)
> >> 13:46:46.745657 192.168.98.200 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> >> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 5018, len 76)
> >> 13:46:46.745783 192.168.98.201 > 192.168.98.203: OSPFv2-dd 32: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DF7C38
> >> [tos 0xc0] [ttl 1] (id 19213, len 52)
> >> 13:46:46.745930 192.168.98.200 > 192.168.98.201: OSPFv2-dd 132: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E M mtu 1500 S 99BF222B { E S
> >> 80000013 age 10 rtr 192.168.98.200 } { E S 8000000C age 13:17 rtr
> >> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 9643, len 152)
> >> 13:46:46.745954 192.168.98.201 > 192.168.98.202: OSPFv2-dd 32: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 3610C5FF
> >> [tos 0xc0] [ttl 1] (id 48123, len 52)
> >> 13:46:46.746257 192.168.98.201 > 192.168.98.200: OSPFv2-dd 132: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E M/MS mtu 1500 S 99BF222C { E
> >> S 80000012 age 13:21 rtr 192.168.98.200 } { E S 8000000E age 0 rtr
> >> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 60171, len 152)
> >> 13:46:46.746524 192.168.98.200 > 192.168.98.201: OSPFv2-dd 32: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF222C [tos
> >> 0xc0] [ttl 1] (id 20148, len 52)
> >> 13:46:46.746593 192.168.98.201 > 192.168.98.200: OSPFv2-dd 32: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E MS mtu 1500 S 99BF222D [tos
> >> 0xc0] [ttl 1] (id 29634, len 52)
> >> 13:46:46.746857 192.168.98.200 > 192.168.98.201: OSPFv2-dd 32: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF222D [tos
> >> 0xc0] [ttl 1] (id 15253, len 52)
> >> 13:46:46.746858 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req 60:
> >> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 } {
> >> rtr 192.168.98.202 } { rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 8075,
> >> len 80)
> >> 13:46:46.747077 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req 48:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
> >> net dr 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 29745,
> >> len 68)
> >> 13:46:46.747152 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd 160:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
> >> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
> >> { E S 80000017 age 12 rtr 192.168.98.202 } [|ospf] [tos 0xc0] [ttl 1]
> >> (id 21137, len 180)
> >> 13:46:46.747581 192.168.98.200 > 192.168.98.201: OSPFv2-ls_ack 44:
> >> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000017 age 12
> >> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 7102, len 64)
> >> 13:46:46.747666 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd 112: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
> >> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> >> } { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } } { E [|ospf]
> >> [tos 0xc0] [ttl 1] (id 40964, len 132)
> >> 13:46:46.747696 192.168.98.200 > 192.168.98.201: OSPFv2-dd 32: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BFEEF2
> >> [tos 0xc0] [ttl 1] (id 16937, len 52)
> >> 13:46:46.747813 192.168.98.201 > 192.168.98.200: OSPFv2-dd 32: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF5FD9
> >> [tos 0xc0] [ttl 1] (id 3855, len 52)
> >> 13:46:46.747848 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd 96: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 1 rtr
> >> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
> >> { E S 8000000C age 1 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
> >> [tos 0xc0] [ttl 1] (id 45786, len 116)
> >> 13:46:46.748155 192.168.98.200 > 192.168.98.201: OSPFv2-dd 152: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E M mtu 1500 S 99BF5FD9 { E S
> >> 80000014 age 0 rtr 192.168.98.200 } { E S 8000000D age 12 rtr
> >> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 24835, len 172)
> >> 13:46:46.748551 192.168.98.201 > 192.168.98.200: OSPFv2-dd 132: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E M/MS mtu 1500 S 99BF5FDA { E
> >> S 80000012 age 13:21 rtr 192.168.98.200 } { E S 8000000E age 0 rtr
> >> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 47662, len 152)
> >> 13:46:46.748693 192.168.98.200 > 192.168.98.201: OSPFv2-dd 32: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF5FDA [tos
> >> 0xc0] [ttl 1] (id 21631, len 52)
> >> 13:46:46.748799 192.168.98.201 > 192.168.98.200: OSPFv2-dd 32: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E MS mtu 1500 S 99BF5FDB [tos
> >> 0xc0] [ttl 1] (id 48058, len 52)
> >> 13:46:46.748938 192.168.98.200 > 192.168.98.201: OSPFv2-dd 32: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF5FDB [tos
> >> 0xc0] [ttl 1] (id 16434, len 52)
> >> 13:46:46.748962 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req 36:
> >> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> >> [tos 0xc0] [ttl 1] (id 28004, len 56)
> >> 13:46:46.749045 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req 60:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
> >> net dr 192.168.98.200 if 192.168.98.200 } { net dr 192.168.98.201 if
> >> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 33841, len 80)
> >> 13:46:46.749209 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd 64:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
> >> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
> >> [tos 0xc0] [ttl 1] (id 5235, len 84)
> >> 13:46:48.752528 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack 104: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" { E S 8000000D age 12 rtr
> >> 192.168.98.201 } { E S 80000017 age 13 rtr 192.168.98.202 } { E S
> >> 80000022 age 1 rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 50978, len 124)
> >> 13:46:48.752670 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack 64: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
> >> 192.168.98.203 } { E S 80000001 age 13 net dr 192.168.98.201 if
> >> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 17804, len 84)
> >> 13:46:48.752696 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack 44: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
> >> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 60705, len 64)
> >> 13:46:48.752887 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack 64: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" { E S 80000013 age 11 rtr
> >> 192.168.98.200 } { E S 8000000B age 11 net dr 192.168.98.200 if
> >> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 30728, len 84)
> >> 13:46:51.756275 192.168.98.201 > 192.168.98.203: OSPFv2-dd 32: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DF7C38
> >> [tos 0xc0] [ttl 1] (id 13078, len 52)
> >> 13:46:51.756285 192.168.98.201 > 192.168.98.202: OSPFv2-dd 32: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 3610C5FF
> >> [tos 0xc0] [ttl 1] (id 53410, len 52)
> >> 13:46:51.756310 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req 60:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
> >> net dr 192.168.98.200 if 192.168.98.200 } { net dr 192.168.98.201 if
> >> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 33659, len 80)
> >> 13:46:51.756826 192.168.98.202 > 192.168.98.201: OSPFv2-dd 32: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 361091BE
> >> [tos 0xc0] [ttl 1] (id 64021, len 52)
> >> 13:46:51.756827 192.168.98.203 > 192.168.98.201: OSPFv2-dd 32: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DEE91C
> >> [tos 0xc0] [ttl 1] (id 12508, len 52)
> >> 13:46:51.756827 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd 96:
> >> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> >> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
> >> { E S 8000000C age 6 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
> >> [tos 0xc0] [ttl 1] (id 13626, len 116)
> >> 13:46:51.756828 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req 36:
> >> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> >> [tos 0xc0] [ttl 1] (id 41975, len 56)
> >> 13:46:51.756828 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd 132:
> >> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> >> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
> >> { E S 8000000C age 6 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
> >> [tos 0xc0] [ttl 1] (id 52527, len 152)
> >> 13:46:51.757543 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd 76: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" { E S 80000018 age 7 rtr
> >> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> >> { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } } [tos 0xc0]
> >> [ttl 1] (id 2414, len 96)
> >> 13:46:51.757553 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack 44:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> >> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 49941, len 64)
> >> 13:46:51.757553 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack 44:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000C age 6 net
> >> dr 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 54649, len 64)
> >> 13:46:51.757554 192.168.98.201 > 192.168.98.202: OSPFv2-dd 132: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E M mtu 1500 S 361091BE { E S
> >> 80000012 age 13:26 rtr 192.168.98.200 } { E S 8000000E age 5 rtr
> >> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 35843, len 152)
> >> 13:46:51.757555 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd 64:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> >> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
> >> [tos 0xc0] [ttl 1] (id 14024, len 84)
> >> 13:46:51.757556 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd 64: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age 1:00:00 net
> >> dr 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs
> >> 192.168.98.202 192.168.98.203 192.168.98.201 } [tos 0xc0] [ttl 1] (id
> >> 22555, len 84)
> >> 13:46:51.757583 192.168.98.201 > 192.168.98.203: OSPFv2-dd 132: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E M mtu 1500 S E1DEE91C { E S
> >> 80000012 age 13:26 rtr 192.168.98.200 } { E S 8000000E age 5 rtr
> >> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 33297, len 152)
> >> 13:46:51.757954 192.168.98.202 > 192.168.98.201: OSPFv2-dd 152: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" E M/MS mtu 1500 S 361091BF { E
> >> S 80000014 age 6 rtr 192.168.98.200 } { E S 8000000D age 16 rtr
> >> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 9420, len 172)
> >> 13:46:51.758049 192.168.98.201 > 192.168.98.202: OSPFv2-dd 32: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S 361091BF [tos
> >> 0xc0] [ttl 1] (id 26856, len 52)
> >> 13:46:51.758204 192.168.98.202 > 192.168.98.201: OSPFv2-dd 32: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" E MS mtu 1500 S 361091C0 [tos
> >> 0xc0] [ttl 1] (id 29314, len 52)
> >> 13:46:51.758286 192.168.98.203 > 192.168.98.201: OSPFv2-dd 152: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" E M/MS mtu 1500 S E1DEE91D { E
> >> S 80000014 age 6 rtr 192.168.98.200 } { E S 8000000D age 16 rtr
> >> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 24179, len 172)
> >> 13:46:51.758304 192.168.98.201 > 192.168.98.202: OSPFv2-dd 32: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S 361091C0 [tos
> >> 0xc0] [ttl 1] (id 18366, len 52)
> >> 13:46:51.758353 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req 36:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
> >> [tos 0xc0] [ttl 1] (id 39434, len 56)
> >> 13:46:51.758392 192.168.98.201 > 192.168.98.203: OSPFv2-dd 32: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S E1DEE91D [tos
> >> 0xc0] [ttl 1] (id 19583, len 52)
> >> 13:46:51.758574 192.168.98.203 > 192.168.98.201: OSPFv2-dd 32: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" E MS mtu 1500 S E1DEE91E [tos
> >> 0xc0] [ttl 1] (id 46477, len 52)
> >> 13:46:51.758655 192.168.98.201 > 192.168.98.203: OSPFv2-dd 32: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S E1DEE91E [tos
> >> 0xc0] [ttl 1] (id 10148, len 52)
> >> 13:46:51.758695 192.168.98.201 > 192.168.98.203: OSPFv2-ls_req 36:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.203 }
> >> [tos 0xc0] [ttl 1] (id 20293, len 56)
> >> 13:46:51.758725 192.168.98.202 > 192.168.98.201: OSPFv2-ls_req 36:
> >> rtrid 192.168.98.202 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> >> [tos 0xc0] [ttl 1] (id 8976, len 56)
> >> 13:46:51.758725 192.168.98.202 > 192.168.98.201: OSPFv2-ls_upd 76:
> >> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000018 age 1 rtr
> >> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> >> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> >> [ttl 1] (id 38525, len 96)
> >> 13:46:51.758990 192.168.98.203 > 192.168.98.201: OSPFv2-ls_req 36:
> >> rtrid 192.168.98.203 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> >> [tos 0xc0] [ttl 1] (id 3058, len 56)
> >> 13:46:51.759005 192.168.98.203 > 192.168.98.201: OSPFv2-ls_upd 76:
> >> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 80000023 age 1 rtr
> >> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> >> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> >> [ttl 1] (id 6586, len 96)
> >> 13:46:53.760281 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack 84: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> >> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
> >> 192.168.98.200 } { E S 80000018 age 7 rtr 192.168.98.202 } [tos 0xc0]
> >> [ttl 1] (id 44630, len 104)
> >> 13:46:53.760438 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack 104: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> >> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
> >> 192.168.98.200 } { E S 80000001 age 18 net dr 192.168.98.201 if
> >> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 60174, len 124)
> >> 13:46:53.760534 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack 104: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" { E S 80000018 age 6 rtr
> >> 192.168.98.202 } { E S 8000000E age 6 rtr 192.168.98.201 } { E S
> >> 80000001 age 1:00:00 net dr 192.168.98.201 if 192.168.98.201 } [tos
> >> 0xc0] [ttl 1] (id 59994, len 124)
> >> 13:46:53.760642 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack 64: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
> >> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
> >> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 11941, len 84)
> >> 13:46:56.751086 192.168.98.200 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> >> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 17219, len 76)
> >> 13:46:56.751111 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> >> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 54762, len 76)
> >> 13:46:56.751228 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> >> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17270, len 76)
> >> 13:46:56.751383 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> >> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 10624, len 76)
> >> 13:46:56.765328 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd 64:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> >> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> >> } } [tos 0xc0] [ttl 1] (id 29796, len 84)
> >> 13:46:56.765380 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd 64:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> >> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> >> } } [tos 0xc0] [ttl 1] (id 5472, len 84)
> >> 13:46:56.765418 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req 36:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
> >> [tos 0xc0] [ttl 1] (id 56664, len 56)
> >> 13:46:56.765519 192.168.98.203 > 192.168.98.201: OSPFv2-ls_req 36:
> >> rtrid 192.168.98.203 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> >> [tos 0xc0] [ttl 1] (id 20557, len 56)
> >> 13:46:56.765519 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd 76: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" { E S 80000023 age 6 rtr
> >> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> >> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
> >> [ttl 1] (id 55930, len 96)
> >> 13:46:56.765597 192.168.98.202 > 192.168.98.201: OSPFv2-ls_req 36:
> >> rtrid 192.168.98.202 backbone auth "secret^@^@" { rtr 192.168.98.201 }
> >> [tos 0xc0] [ttl 1] (id 60479, len 56)
> >> 13:46:56.765737 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack 44:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 6 rtr
> >> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 10666, len 64)
> >> 13:46:56.765796 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd 64: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 1 rtr
> >> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> >> } } [tos 0xc0] [ttl 1] (id 13632, len 84)
> >> 13:46:56.765819 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd 116: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" { E S 80000023 age 7 rtr
> >> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> >> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } { E [|ospf]
> >> [tos 0xc0] [ttl 1] (id 18213, len 136)
> >> 13:46:58.769384 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack 64: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> >> 192.168.98.201 } { E S 8000000D age 1 net dr 192.168.98.200 if
> >> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 16783, len 84)
> >> 13:46:58.769522 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack 44: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000D age 1 net dr
> >> 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 18538, len 64)
> >> 13:46:58.769542 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack 44: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" { E S 8000000F age 1 rtr
> >> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 29090, len 64)
> >> 13:46:58.769738 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack 124: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
> >> 192.168.98.201 } { E S 80000001 age 1:00:00 net dr 192.168.98.201 if
> >> 192.168.98.201 } { E S 80000023 age 7 rtr 192.168.98.203 } [tos 0xc0]
> >> [ttl 1] (id 11688, len 144)
> >> 13:47:01.770218 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd 76:
> >> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000023 age 12
> >> rtr 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric
> >> 65535 } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos
> >> 0xc0] [ttl 1] (id 39965, len 96)
> >> 13:47:01.770242 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd 76: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
> >> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> >> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> >> [ttl 1] (id 32673, len 96)
> >> 13:47:01.770449 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req 36:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
> >> [tos 0xc0] [ttl 1] (id 8167, len 56)
> >> 13:47:01.770476 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd 64:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
> >> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> >> } } [tos 0xc0] [ttl 1] (id 4781, len 84)
> >> 13:47:01.770490 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd 64: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000E age 12 rtr
> >> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> >> } } [tos 0xc0] [ttl 1] (id 42500, len 84)
> >> 13:47:01.770518 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd 76: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" { E S 80000019 age 7 rtr
> >> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> >> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> >> [ttl 1] (id 15481, len 96)
> >> 13:47:01.770628 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd 100:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age
> >> 1:00:00 net dr 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs
> >> 192.168.98.202 192.168.98.203 192.168.98.201 } { E S 8000000F age 6 rtr
> >> 192.168.98.201 } [|ospf] [tos 0xc0] [ttl 1] (id 18229, len 120)
> >> 13:47:01.770739 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack 44:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 12
> >> rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 23688, len 64)
> >> 13:47:01.770839 192.168.98.202 > 192.168.98.201: OSPFv2-ls_upd 76:
> >> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
> >> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
> >> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
> >> [ttl 1] (id 4824, len 96)
> >> 13:47:01.771012 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack 44:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
> >> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 48592, len 64)
> >> 13:47:01.771070 192.168.98.203 > 192.168.98.201: OSPFv2-ls_ack 44:
> >> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
> >> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 50511, len 64)
> >> 13:47:03.778476 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack 104: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" { E S 80000001 age 1:00:00 net
> >> dr 192.168.98.201 if 192.168.98.201 } { E S 8000000F age 7 rtr
> >> 192.168.98.201 } { E S 80000001 age 1:00:00 net dr 192.168.98.201 if
> >> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 63751, len 124)
> >> 13:47:03.778985 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack 44: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
> >> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 34324, len 64)
> >> 13:47:06.759348 192.168.98.200 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> >> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 28952, len 76)
> >> 13:47:06.759349 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> >> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 25646, len 76)
> >> 13:47:06.759524 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> >> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 3310, len 76)
> >> 13:47:06.759532 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> >> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 23021, len 76)
> >> 13:47:06.788050 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd 112: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000F age 13 rtr
> >> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
> >> } } { E S 80000019 age 13 rtr 192.168.98.202 } [|ospf] [tos 0xc0] [ttl
> >> 1] (id 12815, len 132)
> >> 13:47:06.788051 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd 76:
> >> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000019 age 12
> >> rtr 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric
> >> 10 } { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos
> >> 0xc0] [ttl 1] (id 31226, len 96)
> >> 13:47:06.788317 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack 44:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 13
> >> rtr 192.168.98.201 } [tos 0xc0] [ttl 1] (id 26313, len 64)
> >> 13:47:06.788367 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack 44:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 13
> >> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 15304, len 64)
> >> 13:47:06.788414 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack 44:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 12
> >> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 22491, len 64)
> >> 13:47:06.789275 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd 112: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" { E S 80000023 age 18 rtr
> >> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
> >> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } { E [|ospf]
> >> [tos 0xc0] [ttl 1] (id 22696, len 132)
> >> 13:47:06.789496 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack 44:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 18
> >> rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 777, len 64)
> >> 13:47:06.789497 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack 44:
> >> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age
> >> 1:00:00 net dr 192.168.98.201 if 192.168.98.201 } [tos 0xc0] [ttl 1] (id
> >> 26560, len 64)
> >> 13:47:16.771753 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> >> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 16260, len 76)
> >> 13:47:16.772049 192.168.98.200 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> >> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 9947, len 76)
> >> 13:47:16.772050 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> >> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 54548, len 76)
> >> 13:47:16.772050 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> >> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 21814, len 76)
> >> 13:47:26.774278 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> >> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 50490, len 76)
> >> 13:47:26.774423 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> >> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17263, len 76)
> >> 13:47:26.774518 192.168.98.200 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> >> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 24212, len 76)
> >> 13:47:26.774518 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> >> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 49902, len 76)
> >> 13:47:36.778146 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
> >> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 27885, len 76)
> >> 13:47:36.778382 192.168.98.200 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
> >> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 9857, len 76)
> >> 13:47:36.778618 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> >> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 57404, len 76)
> >> 13:47:36.778618 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
> >> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
> >> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
> >> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 23172, len 76)
> >> ^C
> >> 3693 packets received by filter
> >> 0 packets dropped by kernel
> >> fw2#
> >>
> >>
> >>
> >> This post is very extensive already, let me know in case you need any
> >> specific info.
> >>
> >> I'll keep lab running.
> >>
> >> Thank you.
> >>
