Good afternoon everyone,
I had an interesting issue creep up today while I was studying Multicast
routing using ProctorLabs this morning. I was working on IPExpert Lab
Preparation Workbook Vol. 1 Lab 25 for the v4.0 blueprint (the workbook is
actually version 11.0). I skipped my normal pattern and didn't verify
network connectivity before starting because I had done this lab before and
I knew there were no sneaky layer 2 or layer 3 problems hidden in waiting. I
was almost through the lab when I noticed that my frame-relay interfaces
between R2, R4 and R6 were not up. I was on Task 25.5b when I noticed that
MSDP wasn't peering up between R2 and R6. I did a quick examination and
noticed that the serial interfaces themselves were down. I went ahead and
finished the lab (I only had one more task left) and then set about
troubleshooting the frame-relay connection. I am pretty comfortable with
frame-relay, but what I found stumped me. After troubleshooting a bit, I
determined that it looked like a hardware issue. I was a little nervous
about this, as every time I have heard of someone saying they found a
hardware issue when it comes to CCIE labs they have turned out wrong. So, I
backed up my configs, reverted my entire pod and reloaded the Vol. 1 Lab 25
initial configs over again. Well, the problem was still there. At this
point, I opened up a ticket with after hours support, and they assured me it
was a configuration issue. I did some more troubleshooting and with a little
trepidation I went back to them again. They said they would escalate the
problem to Tier 2 on Monday, but for now, I was stuck. They suggested I post
my problem to the OSL and see what you guys think. Any help on this is much
appreciated. I'll post the data from one of my routers and see what we can
find. Here is the troubleshooting information I gathered for R2, although
the problem existed on R4 and R6 identically.
On R2, my Serial0/1/0 interface was configured as follows (straight from the
initial configs):
interface Serial0/1/0
no ip address
encapsulation frame-relay
no arp frame-relay
no frame-relay inverse-arp
frame-relay lmi-type cisco
!
interface Serial0/1/0.24 point-to-point
ip address 150.50.24.2 255.255.255.252
frame-relay interface-dlci 204
!
interface Serial0/1/0.26 point-to-point
ip address 150.50.26.2 255.255.255.252
frame-relay interface-dlci 206
!
If you would like the whole config, you can look at either the initial or
final configs from Lab 25 as I had the same symptoms under both.
Here's the output from "show interface s0/1/0"
Serial0/1/0 is down, line protocol is down
Hardware is GT96K Serial
MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation FRAME-RELAY, loopback not set
Keepalive set (10 sec)
CRC checking enabled
LMI enq sent 0, LMI stat recvd 0, LMI upd recvd 0, DTE LMI down
LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
LMI DLCI 1023 LMI type is CISCO frame relay DTE
FR SVC disabled, LAPF state down
Broadcast queue 0/64, broadcasts sent/dropped 0/0, interface broadcasts 0
Last input never, output never, output hang never
Last clearing of "show interface" counters 01:16:33
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/0/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 1158 kilobits/sec
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 4 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
DCD=down DSR=down DTR=up RTS=up CTS=down
You'll notice that the line is down/down. Now, even if my config was
completely wrong and I had done something silly like configured the
interface for PPP it should still show as up/down right? This was what got
me thinking that this was a hardware issue. You'll also notice "DTE LMI
down" which isn't a good thing. This meant that I wasn't even communicating
with the frame-relay switch. So, I fired up debug and took a look at what
was going on. I turned on debugging for frame-relay packet, frame-relay lmi
and frame-relay detailed. I think shutdown the s0/1/0 interface and brought
it back up again. Here is what I got:
Feb 26 17:30:45.959: Serial0/1/0.24: encaps failed--line down
Feb 26 17:30:45.959: Serial0/1/0.26: encaps failed--line down
That's it. Just those two lines. Well, you can't see a frame-relay switch
over an interface that is down so I double checked R4 and R6 and they were
identical. The odds of having all three routers with a serial port failure
was too unlikely, so I started looking to the frame-relay switch side of
things. The next thing I looked at was "show frame-relay lmi" whose output
follows:
LMI Statistics for interface Serial0/1/0 (Frame Relay DTE) LMI TYPE = CISCO
Invalid Unnumbered info 0 Invalid Prot Disc 0
Invalid dummy Call Ref 0 Invalid Msg Type 0
Invalid Status Message 0 Invalid Lock Shift 0
Invalid Information ID 0 Invalid Report IE Len 0
Invalid Report Request 0 Invalid Keep IE Len 0
Num Status Enq. Sent 0 Num Status msgs Rcvd 0
Num Update Status Rcvd 0 Num Status Timeouts 0
Last Full Status Req never Last Full Status Rcvd never
That definitely didn't look good. 0 messages of any sort had been sent, good
or bad. This is what I would expect to see on a router that had absolutely
nothing plugged into the serial port. So, that's what I checked next with
"show controller s0/1/0"
Interface Serial0/1/0
Hardware is GT96K
DTE V.35 clocks stopped.
idb at 0x686DC3CC, driver data structure at 0x686DD908
wic_info 0x686DDF2C
Physical Port 3, SCC Num 3
<output omitted>
I cut this one short for the e-mail because the relevant part is right up
top: DTE V.35 clocks stopped. That's when I decided this had to be an error
with the frame-relay switch. I could see that a cable was present, but I
wasn't receiving any clock signaling from the switch. I used the web
interface for ProctorLabs to power cycle the frame-relay switch, but that
didn't make any difference. I couldn't even tell whether or not it had
rebooted since my routers couldn't even detect the frame-relay switch at
layer 1. That's all I could do since we don't have access to the frame-relay
switch for troubleshooting. So I double checked everything again, and then
opened up an after hours support ticket, which is what led me to here.
I think I've done my due diligence on this one, but do you guys see anything
I missed? Can you think of something that might cause this? If I had to
guess, I would say that the frame-relay switch was actually powered off,
however support assures me it is on and operational. Any thoughts, input,
ideas, or criticism on my troubleshooting are appreciated. Problems like
these are simply another opportunity to learn.
Thanks everyone,
Don Pezet
_______________________________________________
For more information regarding industry leading CCIE Lab training, please visit
www.ipexpert.com
