Bryan, Great stuff - look forward to seeing what you can find out.
I was randomly playing with a CSC lab a while ago and came across this but put it down to a quirk in Dynamips and forgot about it and moved on. I had to lab it up again when I saw your original email as the behavior sounded familiar! Jo 2009/9/4 Bryan Bartik <[email protected]> > Jo, > > That's what it appears. Because it is in a VRF, it creates that VPN label > as it would to advertise to a regular PE. It advertises that same label when > the LDP peering comes up to R1. No split-horizon in LDP! R9 doesn't know any > better...but I think R1 should! I am still going to dig for some more > information, maybe query TAC. I'll let you know if I find anything else. > > > On Fri, Sep 4, 2009 at 11:49 AM, Jo Knight <[email protected]> wrote: > >> I am seeing the same thing on this IOS: >> >> Cisco IOS Software, 7200 Software (C7200-K91P-M), Version 12.2(25)S9, >> RELEASE SOFTWARE (fc1) >> >> 00:27:36: %LDP-5-NBRCHG: LDP Neighbor 150.1.1.1:0 is DOWN >> 00:27:43: %LDP-5-NBRCHG: LDP Neighbor 150.1.1.1:0 is UP >> 00:30:43: %LDP-5-NBRCHG: LDP Neighbor 150.1.1.1:0 is DOWN >> 00:30:50: %LDP-5-NBRCHG: LDP Neighbor 150.1.1.1:0 is UP >> 00:33:50: %LDP-5-NBRCHG: LDP Neighbor 150.1.1.1:0 is DOWN >> 00:34:00: %LDP-5-NBRCHG: LDP Neighbor 150.1.1.1:0 is UP >> 00:37:00: %LDP-5-NBRCHG: LDP Neighbor 150.1.1.1:0 is DOWN >> 00:37:11: %LDP-5-NBRCHG: LDP Neighbor 150.1.1.1:0 is UP >> 00:40:11: %LDP-5-NBRCHG: LDP Neighbor 150.1.1.1:0 is DOWN >> >> >> I implemented the workaround and it seems stable now. >> >> Is it happening due to ldp being configured on an interface within a VRF >> in this case? >> >> Good spot though - thanks. >> >> Jo >> >> 2009/9/4 Bryan Bartik <[email protected]> >> >> Well I think Antoine was on to something...and I found another workaround >>> :-) >>> >>> I did a packet capture in Dynamips and just before the session drops, the >>> TCP/LDP keepalive between R1 and R9 loop between each other. I get about 254 >>> packets in under a sec and I can see the TTL decrement on each one! This >>> happens every minute and eventually the timeout expires because these >>> keepalive are not received and processed locally by R9 (thus every 3 minutes >>> it drops). R1 is sending the keepalive as a labeled packet even though R9 is >>> directly connected, I think it should be an imp-null label. According to the >>> capture R1 is putting label 20 on the keepalive. Here I verified it: >>> >>> R1#sho mpls ldp bindings >>> lib entry: 123.1.9.0/24, rev 2 >>> local binding: label: imp-null >>> remote binding: lsr: 123.1.9.9:0, label: 20 >>> >>> R9#sho mpls forwarding-table labels 20 >>> Local Outgoing Prefix Bytes Label Outgoing Next >>> Hop >>> Label Label or VC or Tunnel Id Switched >>> interface >>> 20 Pop Label 123.1.9.0/24[V] <http://123.1.9.0/24%5BV%5D> >>> 3358266 Se1/2 point2point >>> R9# >>> >>> R1 is learning the label 20 and using that to send packets onto the >>> directly connected network. R9 receives it, pops it and sends it back out of >>> the interface instead of receiving it locally. (I am using the interface as >>> ldp transport address for this example). It's hard to tell who is at fault >>> here! >>> >>> In order for the session to actually come up again, all the learned >>> labels have to get flushed so the TCP exchange can happen. Once it does, TCP >>> handshake and label exchange occurs without any label encapsulation. R9 then >>> advertises the vrf subnet (connected to R1) in a label mapping message with >>> a label of 20 (verified with capture). This label is now used by R1 and any >>> further TCP exchange uses that label and packets loop. It's funny to see it >>> happen, because as soon as R9 sends that label mapping, R1 uses the label in >>> the ACK! However this does not cause any problems at this point, because R1 >>> also sends an unlabeled ACK as well, completeing the exchange (weird). >>> >>> To fix this, I have configured the following on R1: >>> >>> access-list 1 deny 123.1.9.0 0.0.0.255 >>> access-list 1 permit any >>> mpls ldp neighbor 123.1.9.9 labels accept 1 >>> >>> On R3 I did similar: >>> >>> access-list 1 deny 123.3.8.0 0.0.0.255 >>> access-list 1 permit any >>> mpls ldp neighbor 123.3.8.8 labels accept 1 >>> >>> It's been up for more than 3 minutes: >>> >>> R1#sho mpls ldp neighbor >>> Peer LDP Ident: 123.1.9.9:0; Local LDP Ident 123.123.123.1:0 >>> TCP connection: 123.1.9.9.11032 - 123.1.9.1.646 >>> State: Oper; Msgs sent/rcvd: 13/13; Downstream >>> Up time: 00:04:55 >>> LDP discovery sources: >>> Serial1/0, Src IP addr: 123.1.9.9 >>> Addresses bound to peer LDP Ident: >>> 123.1.9.9 >>> >>> R3#sho mpls ldp neighbor >>> Peer LDP Ident: 123.3.8.8:0; Local LDP Ident 123.123.123.3:0 >>> TCP connection: 123.3.8.8.11036 - 123.3.8.3.646 >>> State: Oper; Msgs sent/rcvd: 15/15; Downstream >>> Up time: 00:06:13 >>> LDP discovery sources: >>> Serial1/0, Src IP addr: 123.3.8.8 >>> Addresses bound to peer LDP Ident: >>> 123.3.8.8 >>> >>> Can't believe that is they way it should be designed (I still think bug), >>> but for now it is stable :-) >>> >>> thanks all, >>> >>> >>> On Fri, Sep 4, 2009 at 8:47 AM, Rick Mur <[email protected]> wrote: >>> >>>> True, the CSC code can be buggy sometimes. As this 3 minute marker is >>>> definitely a bug. I recall I also had some issues where it worked only some >>>> period of time, so maybe I ran into that same issue. Still it's sometimes >>>> buggy and I don't think that will happen on your lab. >>>> >>>> -- >>>> Regards, >>>> >>>> Rick Mur >>>> CCIE2 #21946 (R&S / Service Provider) >>>> Sr. Support Engineer – IPexpert, Inc. >>>> URL: http://www.IPexpert.com >>>> >>>> On 4 sep 2009, at 14:11, Bryan Bartik wrote: >>>> >>>> Thanks Guys, I am using 7200s in dynamips with 12.2S code. In fact, this >>>> is a mock set up of vol 2 lab 4, I just took everything else out of the >>>> equation. I started from scratch just doing the MPLS VPN scenarios. Even R2 >>>> is not doing anything (ospf or ldp) right now. I am also just using >>>> physical >>>> interfaces on my frame relay cloud :) I don't see the client tagging the >>>> packets but I could look deeper into this. It's just amazing that is every >>>> 3 >>>> minutes pretty much on the dot! I wonder if it's some type of CSC bug >>>> having >>>> to do with MPLS on the VRF interface...I will do some more testing and let >>>> you know. >>>> >>>> On Fri, Sep 4, 2009 at 3:22 AM, Antonie Henning - MWEB < >>>> [email protected]> wrote: >>>> >>>>> Had a similar issue. I narrowed it down to point to point frame-relay >>>>> subinterface with csc igp (ospf) and ldp enabled. >>>>> >>>>> What I saw was a loop. The client would tag packets to the carrier for >>>>> the directly connected subnet. I was expecting to see it pop the label: >>>>> >>>>> R2(config-subif)#do sh ip cef 123.2.6.6 >>>>> 123.2.6.0/24 >>>>> attached to Serial4/0.206 label 614 >>>>> >>>>> The 614 label then sends the packet back to the client and a loop >>>>> forms: >>>>> >>>>> R2(config)#do trace 123.2.6.6 >>>>> >>>>> Type escape sequence to abort. >>>>> Tracing the route to 123.2.6.6 >>>>> >>>>> 1 123.2.6.6 [MPLS: Label 614 Exp 0] 8 msec 32 msec 12 msec >>>>> 2 123.2.6.2 20 msec 40 msec 40 msec >>>>> 3 123.2.6.6 [MPLS: Label 614 Exp 0] 44 msec 24 msec 44 msec >>>>> 4 123.2.6.2 20 msec 48 msec 64 msec >>>>> 5 123.2.6.6 [MPLS: Label 614 Exp 0] 40 msec 24 msec 40 msec >>>>> 6 123.2.6.2 48 msec 72 msec 60 msec >>>>> 7 123.2.6.6 [MPLS: Label 614 Exp 0] 60 msec 36 msec 64 msec >>>>> 8 123.2.6.2 56 msec 60 msec 88 msec >>>>> 9 123.2.6.6 [MPLS: Label 614 Exp 0] 60 msec 108 msec 60 msec >>>>> >>>>> Changing the frame-relay config to use the main interface solved the >>>>> problem. >>>>> >>>>> Hth >>>>> 21500.net >>>>> >>>>> -----Original Message----- >>>>> From: [email protected] [mailto:[email protected]] On Behalf >>>>> Of Francisco Baena >>>>> Sent: 04 September 2009 08:11 AM >>>>> To: 'Bryan Bartik'; [email protected]; >>>>> [email protected] >>>>> Subject: RE: Carrier Supporting Carrier (CSC) - LDP peering between SPs >>>>> drops every 3 minutes >>>>> >>>>> Make it two. I had the same problem with vol II - Lab 4 (I think), but >>>>> from >>>>> R2 to R6. >>>>> >>>>> At that point I blamed on dynamips being buggy as all the routing/MPLS >>>>> tables seem fine. >>>>> >>>>> I look forward to a resolution on this. The interesting thing is that >>>>> when I >>>>> shut down the connection from R1 to R2, the problem went away, so it >>>>> sounds >>>>> like an IGP issue. However even making a Sham link between r6 and r9 >>>>> and >>>>> increasing the OSPF cost from R1 to R2 (to ensure AS200 was the exit >>>>> point), >>>>> made no difference. >>>>> >>>>> I would say that a possible workaround could be to make the R1-R9 LDP >>>>> sessions targeted, but obviously what we all want to know is what the >>>>> heck >>>>> happened there in the first place. >>>>> >>>>> During my testing I disabled MPLS TE too in AS200, just in case, but no >>>>> cigar.... >>>>> >>>>> Cheers, >>>>> Francisco >>>>> http://www.linkedin.com/in/fbaena >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: [email protected] [mailto:[email protected]] On Behalf >>>>> Of >>>>> Bryan Bartik >>>>> Sent: 04 September 2009 05:14 >>>>> To: [email protected]; [email protected] >>>>> Subject: Carrier Supporting Carrier (CSC) - LDP peering between SPs >>>>> drops >>>>> every 3 minutes >>>>> >>>>> Hello, >>>>> >>>>> I am running an inter-as + CSC scenario with OSPF and MPLS between the >>>>> SPs. >>>>> The LDP peering is dropping exactly every 3 minutes and coming back up. >>>>> Connectivity is fine throughout the VPN when the session is up. I don't >>>>> see >>>>> any route flapping in the debugs. >>>>> >>>>> AS65123[R3]---------[R8]AS100---------AS200[R9]---------[R1]AS65123 >>>>> >>>>> AS100 and AS200 have an inter-as VPN supporting the 2nd level carrier >>>>> AS65123. >>>>> R9 and R8 have vrf interfaces connected to R1 and R3 respectively. >>>>> R1 has an LDP/OSPF peering with R9 (in the vrf) in AS100 >>>>> R3 has an LDP/OSPF peering with R8 (in the vrf) in AS200 >>>>> >>>>> Both LDP sessions are bouncing on queue every 3 minutes! Here is R8 for >>>>> example: >>>>> >>>>> R8# >>>>> *Sep 3 21:50:49.575: %LDP-5-NBRCHG: LDP Neighbor 123.123.123.3:0 is >>>>> DOWN >>>>> *Sep 3 21:50:59.475: %LDP-5-NBRCHG: LDP Neighbor 123.123.123.3:0 is >>>>> UP >>>>> *Sep 3 21:53:59.491: %LDP-5-NBRCHG: LDP Neighbor 123.123.123.3:0 is >>>>> DOWN >>>>> *Sep 3 21:54:09.291: %LDP-5-NBRCHG: LDP Neighbor 123.123.123.3:0 is >>>>> UP >>>>> *Sep 3 21:57:09.311: %LDP-5-NBRCHG: LDP Neighbor 123.123.123.3:0 is >>>>> DOWN >>>>> *Sep 3 21:57:19.331: %LDP-5-NBRCHG: LDP Neighbor 123.123.123.3:0 is >>>>> UP >>>>> *Sep 3 22:00:19.351: %LDP-5-NBRCHG: LDP Neighbor 123.123.123.3:0 is >>>>> DOWN >>>>> *Sep 3 22:00:29.239: %LDP-5-NBRCHG: LDP Neighbor 123.123.123.3:0 is >>>>> UP >>>>> >>>>> Now just before the session dies I get the following from "debug mpls >>>>> ldp >>>>> transport events interface x/x". This is from R9: >>>>> >>>>> R9# >>>>> *Sep 3 22:09:27.295: ldp: Send ldp hello; Serial1/2, src/dst >>>>> 123.1.9.9/224.0.0.2, inst_id 0 >>>>> *Sep 3 22:09:28.787: ldp: Rcvd ldp hello; Serial1/2, from 123.1.9.1 ( >>>>> 123.123.123.1:0), intf_id 0, opt 0xC >>>>> *Sep 3 22:09:32.127: ldp: Send ldp hello; Serial1/2, src/dst >>>>> 123.1.9.9/224.0.0.2, inst_id 0 >>>>> *Sep 3 22:09:33.239: ldp: Rcvd ldp hello; Serial1/2, from 123.1.9.1 ( >>>>> 123.123.123.1:0), intf_id 0, opt 0xC >>>>> *Sep 3 22:09:36.111: ldp: Send ldp hello; Serial1/2, src/dst >>>>> 123.1.9.9/224.0.0.2, inst_id 0 >>>>> *Sep 3 22:09:36.907: tagcon: Session KeepAlive timer expired, peer >>>>> 123.123.123.1:0 (pp 0x64027BD8) >>>>> *Sep 3 22:09:36.911: ldp: Close LDP transport conn for adj 0x63486648 >>>>> *Sep 3 22:09:36.915: ldp: Closing ldp conn 123.1.9.9:646 <-> >>>>> 123.123.123.1:11015, adj 0x63486648 >>>>> *Sep 3 22:09:36.919: ldp: Adj 0x63486648; state set to closed >>>>> *Sep 3 22:09:36.923: %LDP-5-NBRCHG: LDP Neighbor 123.123.123.1:0 is >>>>> DOWN >>>>> >>>>> Even though I am sending and receiving hellos, I am getting the session >>>>> keepalive timer expired. Any ideas? >>>>> >>>>> Thanks! >>>>> -- >>>>> Bryan Bartik >>>>> CCIE #23707 (R&S), CCNP >>>>> Sr. Support Engineer - IPexpert, Inc. >>>>> URL: http://www.IPexpert.com >>>>> >>>>> _____________________________________________________________________ >>>>> Subscription information: http://www.groupstudy.com/list/comserv.html >>>>> >>>>> No virus found in this incoming message. >>>>> Checked by AVG - www.avg.com >>>>> Version: 8.5.409 / Virus Database: 270.13.76/2343 - Release Date: >>>>> 09/03/09 >>>>> 05:50:00 >>>>> >>>>> _____________________________________________________________________ >>>>> Subscription information: http://www.groupstudy.com/list/comserv.html >>>>> Connect with South Africa’s leading Internet Service Provider and >>>>> discover the magic of the Internet and all its possibilities. >>>>> Call 08600 32000 or click here(http://www.mweb.co.za/productsservices/) >>>>> for more. >>>>> >>>>> MWEB :-) CONNECT AND YOU CAN. >>>>> >>>>> This electronic communication and the attached file(s) are subject to a >>>>> disclaimer which can be accessed on the following link: >>>>> Disclaimer - or copy the following URL into your browser - >>>>> http://www.mweb.co.za/disclaimer. >>>>> If you are unable to view the disclaimer, please contact >>>>> [email protected] for a copy. >>>>> >>>> >>>> >>>> >>>> -- >>>> Bryan Bartik >>>> CCIE #23707 (R&S), CCNP >>>> Sr. Support Engineer - IPexpert, Inc. >>>> URL: http://www.IPexpert.com >>>> _______________________________________________ >>>> For more information regarding industry leading CCIE Lab training, >>>> please visit www.ipexpert.com >>>> >>>> >>>> >>> >>> >>> -- >>> Bryan Bartik >>> CCIE #23707 (R&S), CCNP >>> Sr. Support Engineer - IPexpert, Inc. >>> URL: http://www.IPexpert.com >>> >>> _______________________________________________ >>> For more information regarding industry leading CCIE Lab training, please >>> visit www.ipexpert.com >>> >>> >> > > > -- > Bryan Bartik > CCIE #23707 (R&S), CCNP > Sr. Support Engineer - IPexpert, Inc. > URL: http://www.IPexpert.com >
_______________________________________________ For more information regarding industry leading CCIE Lab training, please visit www.ipexpert.com
