Not that it's a "Fix" but have you tried rebooting the box? If this is a bug in the forwarding plane that might clear/rebuild it. And maybe it works correctly after that.
Friend saw something similar on a Juniper MX with DPC cards that had run out of FIB space. It would show correctly in all places, but was failing to install in FIB (Router cut a error about it in the log). So the actual traffic didn't follow the same path. I saw you specifically referenced "forwarding" in one of your copy pastes. And it looked right. But I don't know enough about Cisco to say if that's really what is in FIB. Or just what it thinks is in FIB. On Tue, Apr 4, 2023 at 5:47 PM Mike Hammett <na...@ics-il.net> wrote: > Via all mechanisms I could find in the router, it thinks the best path is > the direct path, the packets just don't go that way. > > The in traffic isn't a concern at this time, just the out (from my > perspective). > > > > ----- > Mike Hammett > Intelligent Computing Solutions > http://www.ics-il.com > > Midwest-IX > http://www.midwest-ix.com > > ------------------------------ > *From: *"David Bass" <davidbass...@gmail.com> > *To: *"Mike Hammett" <na...@ics-il.net> > *Cc: *"Matthew Huff" <mh...@ox.com>, "NANOG" <nanog@nanog.org> > *Sent: *Tuesday, April 4, 2023 11:39:26 AM > *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging > > If you are both connected to the same upstream, but the customer wants > traffic destined to the upstream to go through you (in and out), then they > need to do something on their devices to try and affect the inbound path to > their AS. From the upstream carrier in question they’ll take the best path > to a prefix, which direct connection is generally going to be preferred > over a transit AS (basic BGP best path algorithm stuff) unless there is > some manipulation of the prefix advertisement happening. > > To confirm the path being taken you should be able to do a few trace > routes from various locations as well as use looking glasses. > > Now the sflow data is an entirely different thing to analyze. > > On Tue, Apr 4, 2023 at 7:45 AM Mike Hammett <na...@ics-il.net> wrote: > >> >> sh ip bgp neighbor advertised-routes shows the only routes being >> advertised to Y are the routes that should be advertised to them. I checked >> a variety of other peers and have the expected results. >> >> >> >> From my perspective: >> >> Packets come in on port A, supposed to leave on port X, but they leave on >> port Y. All of the troubleshooting steps I've done on my own (or suggested >> by mailing lists) say the packets should be leaving on the desired port X. >> >> >> From the customer's perspective, they're supposed to be coming from me on >> port X, but they're arriving on port Y, another network. >> >> Port X in both scenarios is our direct connection, while port Y is a >> mutual upstream provider. >> >> >> Without knowing more about the specific platform, it seems to me like a >> bug in the platform. If all indicators (not just configurations, but show >> commands as well) say the packet should be leaving on X and it leaves on Y, >> then I'm not sure what else it could be, besides a bug. >> >> >> >> ----- >> Mike Hammett >> Intelligent Computing Solutions >> http://www.ics-il.com >> >> Midwest-IX >> http://www.midwest-ix.com >> >> ------------------------------ >> *From: *"David Bass" <davidbass...@gmail.com> >> *To: *"Mike Hammett" <na...@ics-il.net> >> *Cc: *"Matthew Huff" <mh...@ox.com>, "NANOG" <nanog@nanog.org> >> *Sent: *Monday, April 3, 2023 9:12:52 PM >> >> *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging >> >> You said that they are seeing traffic from another upstream…are you >> advertising the prefix to them? Are you advertising their prefix to your >> upstream? >> >> Looks like the route maps are involved in some dual redistribution…might >> want to make sure everything is matching correctly, and being advertised >> like you want. >> >> On Mon, Apr 3, 2023 at 4:20 PM Mike Hammett <na...@ics-il.net> wrote: >> >>> I don't see any route-maps applied to interfaces, so there must not be >>> any PBR going on. I only see ACLs, setting communities, setting local pref, >>> etc. in the route maps that are applied to neighbors. >>> >>> >>> >>> ----- >>> Mike Hammett >>> Intelligent Computing Solutions >>> http://www.ics-il.com >>> >>> Midwest-IX >>> http://www.midwest-ix.com >>> >>> ------------------------------ >>> *From: *"Mike Hammett" <na...@ics-il.net> >>> *To: *"Matthew Huff" <mh...@ox.com> >>> *Cc: *"NANOG" <nanog@nanog.org> >>> *Sent: *Monday, April 3, 2023 8:26:30 AM >>> >>> *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding >>> Debugging >>> >>> Only two VRFs, default and manangement. IIRC, everything I saw before >>> mentioned the default VRF. >>> >>> I do see a ton of route-maps. It's mostly Greek to me, so I'll have to >>> dig through this a bit to see what's going on. >>> >>> >>> >>> ----- >>> Mike Hammett >>> Intelligent Computing Solutions >>> http://www.ics-il.com >>> >>> Midwest-IX >>> http://www.midwest-ix.com >>> >>> ------------------------------ >>> *From: *"Matthew Huff" <mh...@ox.com> >>> *To: *"Mike Hammett" <na...@ics-il.net> >>> *Cc: *"NANOG" <nanog@nanog.org> >>> *Sent: *Monday, April 3, 2023 8:06:51 AM >>> *Subject: *RE: Cisco Nexus 3k Route Selection\Packet Forwarding >>> Debugging >>> >>> What about VRFs and/or policy based routing? >>> >>> switch-core1# show vrf >>> VRF-Name VRF-ID State Reason >>> >>> default 1 Up -- >>> >>> management 2 Up -- >>> >>> >>> switch-core1# show route-map >>> route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 >>> Match clauses: >>> interface: Ethernet1/33 >>> route-type: internal >>> Set clauses: >>> metric 40000000 10 255 1 1500 >>> route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 >>> Match clauses: >>> interface: Ethernet1/34 >>> route-type: internal >>> Set clauses: >>> metric 40000000 30 255 1 1500 >>> route-map rmap_static_to_eigrp, permit, sequence 10 >>> Match clauses: >>> ip address prefix-lists: prefix_static_to_eigrp >>> Set clauses: >>> route-map rmap_static_to_eigrp_v6, permit, sequence 10 >>> Match clauses: >>> ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp >>> Set clauses: >>> >>> >>> >>> From: Mike Hammett <na...@ics-il.net> >>> Sent: Monday, April 3, 2023 9:00 AM >>> To: Matthew Huff <mh...@ox.com> >>> Cc: NANOG <nanog@nanog.org> >>> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging >>> >>> It could be an sFlow bug, but I come at this from a reported problem and >>> gathering data on that problem as opposed to looking at data for problems. >>> >>> The snmp if index reported by the Nexus matches the if index in >>> ElastiFlow. >>> >>> >>> >>> >>> ----- >>> Mike Hammett >>> Intelligent Computing Solutions >>> http://www.ics-il.com >>> >>> Midwest-IX >>> http://www.midwest-ix.com >>> >>> ________________________________________ >>> From: "Matthew Huff" <mailto:mh...@ox.com> >>> To: "Mike Hammett" <mailto:na...@ics-il.net> >>> Cc: "NANOG" <mailto:nanog@nanog.org> >>> Sent: Monday, April 3, 2023 7:50:08 AM >>> Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging >>> SFlow misconfiguration or bug on either the nexus or the sflow monitor? >>> On the monitor, can you verify that the snmp interfaces are mapped to the >>> correct ones on the nexus? >>> >>> >>> >>> >>> >>> From: Mike Hammett <mailto:na...@ics-il.net> >>> Sent: Monday, April 3, 2023 8:47 AM >>> To: Matthew Huff <mailto:mh...@ox.com> >>> Cc: NANOG <mailto:nanog@nanog.org> >>> Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging >>> >>> It shows the desired result. >>> >>> >>> ----- >>> Mike Hammett >>> Intelligent Computing Solutions >>> http://www.ics-il.com >>> >>> Midwest-IX >>> http://www.midwest-ix.com >>> >>> ________________________________________ >>> From: "Matthew Huff" <mailto:mh...@ox.com> >>> To: "Mike Hammett" <mailto:na...@ics-il.net>, "NANOG" <mailto: >>> nanog@nanog.org> >>> Sent: Monday, April 3, 2023 5:38:23 AM >>> Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging >>> >>> switch-core1# sh forwarding route x.x.x.x >>> >>> slot 1 >>> ======= >>> >>> >>> IPv4 routes for table default/base >>> >>> >>> ------------------+-----------------------------------------+----------------------+-----------------+----------------- >>> Prefix | Next-hop | Interface >>> | Labels | Partial Install >>> >>> ------------------+-----------------------------------------+----------------------+-----------------+----------------- >>> x.x.x.x/24 x.x.x.250 Ethernet1/29 >>> >>> >>> switch-core1# show routing hash x.x.x.x y.y.y.y >>> Load-share parameters used for software forwarding: >>> load-share mode: address source-destination port source-destination >>> Hash for VRF "default" >>> Hashing to path *y.y.y.y Eth1/29 >>> For route: >>> y.y.y.0/24, ubest/mbest: 1/0 >>> *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal >>> >>> >>> >>> >>> From: NANOG <mailto:nanog-bounces+mhuff=ox....@nanog.org> On Behalf Of >>> Mike Hammett >>> Sent: Monday, April 3, 2023 1:21 AM >>> To: NANOG <mailto:nanog@nanog.org> >>> Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging >>> >>> We have a Nexus 3064 that is setup with partial BGP tables and is >>> routing based on that. >>> >>> >>> I've done a show ip bgp for an IP of interest and it has an expected >>> next hop IP. I show ip arp on that next hop IP and it has the expected >>> interface. >>> >>> >>> However, sFlows show the packets leaving on a different interface, the >>> one that would carry the default route for routes not otherwise known. >>> >>> >>> If the next hop IP is expected and the ARP of that next hop IP is >>> expected, why are packets leaving out an unexpected interface? >>> >>> >>> ----- >>> Mike Hammett >>> Intelligent Computing Solutions >>> http://www.ics-il.com >>> >>> Midwest-IX >>> http://www.midwest-ix.com >>> >>> >>> >>> >> >
