Jon, I think we can safely conclude from the information provided that you're looking at some sort of a misconfigured traffic mirroring or [un]lawful intercept.
Sadly, as neither Sprint nor your loop provider will fess up, I don't think you're going to get much further on here. Probably best to order a new loop and cancel the existing one. Drive Slow, Paul - Original message - I just went ahead and "re-broke" the circuit ... On 8/17/08, Jon Lewis <[EMAIL PROTECTED]> wrote: > On Tue, 12 Aug 2008, Jon Lewis wrote: > >>> What would happen if you pinged the Ocala router such that the TTL was 1 >>> when travelling over the DS3? From your traceroute it seems it travelled >>> two IP hops that did not send ICMP error messages, but it might just be >>> that the ICMP errors from the Ocala router are arriving first. >> >> Based on where the dupes are coming from, I assume pinging across the DS3 >> with TTL tuned to expire at the Ocala side would result in TTL exceeded >> messages from both Ocala and the Sprint router where the packets are >> injected >> into Sprint's network. It doesn't look as if IOS gives the option to set >> TTL >> on ping...so I'd try this from a Linux machine in our data center. > > I just went ahead and "re-broke" the circuit for a bit by turning it back > to hdlc to see if the issue is still there and to run some additional > tests. Someone is still cross connecting our Orlando->Ocala traffic over > to Sprint. > > I did your suggested ping with short TTL and the result was close to what > I expected. > > $ traceroute ocalflxa-br-1 > traceroute to ocalflxa-br-1.atlantic.net (209.208.6.229), 30 hops max, 38 > byte packets > 1 209.208.25.165 (209.208.25.165) 0.539 ms 0.426 ms 0.388 ms > 2 69.28.72.162 (69.28.72.162) 0.246 ms 0.351 ms 0.223 ms > 3 andc-br-3-f2-0 (209.208.9.138) 0.559 ms 0.435 ms 0.471 ms > 4 ocalflxa-br-1-s1-0 (209.208.112.98) 2.735 ms * 2.656 ms > > So, I need a TTL of 4 to get there from this machine. > > $ ping -t4 ocalflxa-br-1 > PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data. > 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 > time=2.68 ms > 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 > time=2.72 ms > 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=2 ttl=252 > time=2.88 ms > > Decrease ttl by one, and I get the expected ttl exceeded from the Orlando > side of the circuit. > > $ ping -t 3 ocalflxa-br-1 > PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data. > >From andc-br-3-f2-0.atlantic.net (209.208.9.138) icmp_seq=0 Time to live > exceeded > > Now, here's a mild surprise. You'll notice that in the above -t4 trace, I > didn't hear back from Sprint. > > $ ping -t 5 ocalflxa-br-1 > PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data. > 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 > time=2.89 ms > 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 > time=3.10 ms > 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=2 ttl=252 > time=2.97 ms > hmm...still no ttl exceeded from Sprint? > > $ ping -t 6 ocalflxa-br-1 > PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data. > 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 > time=2.95 ms > >From sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) icmp_seq=0 Time to > live exceeded > 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 > time=2.78 ms > >From sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) icmp_seq=1 Time to > live exceeded > > $ ping -t 7 ocalflxa-br-1 > PING ocalflxa-br-1.atlantic.net (209.208.6.229) 56(84) bytes of data. > 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=0 ttl=252 > time=2.88 ms > >From sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) icmp_seq=0 Time to > live exceeded > 64 bytes from ocalflxa-br-1.atlantic.net (209.208.6.229): icmp_seq=1 ttl=252 > time=2.84 ms > >From sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) icmp_seq=1 Time to > live exceeded > > Is it just coincidence that there are 2 private IP hops in some > traceroutes between us and Sprint? i.e. Look at this trace from cogent: > > Tracing the route to 209.208.33.1 > > 1 fa0-8.na01.b005944-0.dca01.atlas.cogentco.com (66.250.56.189) 0 msec 4 > msec 4 msec > 2 gi3-9.3507.core01.dca01.atlas.cogentco.com (66.28.67.225) 160 msec 4 > msec 8 msec > 3 te3-1.ccr02.dca01.atlas.cogentco.com (154.54.3.158) 0 msec 0 msec 4 > msec > 4 vl3493.mpd01.dca02.atlas.cogentco.com (154.54.7.230) 28 msec 4 msec > te4-1.mpd01.dca02.atlas.cogentco.com (154.54.2.182) 52 msec > 5 vl3494.mpd01.iad01.atlas.cogentco.com (154.54.5.42) 4 msec 4 msec > vl3497.mpd01.iad01.atlas.cogentco.com (154.54.5.66) 4 msec > 6 timewarner.iad01.atlas.cogentco.com (154.54.13.250) 4 msec > peer-01-ge-3-1-2-13.asbn.twtelecom.net (66.192.252.217) 4 msec 12 msec > 7 66-194-200-202.static.twtelecom.net (66.194.200.202) 28 msec 28 msec 32 > msec > 8 66-194-200-202.static.twtelecom.net (66.194.200.202) 32 msec 32 msec 28 > msec > 9 andc-br-3-f2-0.atlantic.net (209.208.9.138) 32 msec 32 msec 32 msec > 10 172.22.122.1 32 msec 32 msec 32 msec > 11 10.247.28.205 32 msec 32 msec 32 msec > 12 sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) 32 msec 32 msec 28 > msec > 13 sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) 28 msec 32 msec 32 > msec > 14 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 32 msec 32 msec 28 > msec > 15 vlan79.csw2.Washington1.Level3.net (4.68.17.126) 28 msec > vlan89.csw3.Washington1.Level3.net (4.68.17.190) 32 msec > vlan79.csw2.Washington1.Level3.net (4.68.17.126) 40 msec > 16 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 28 msec > ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 28 msec > ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 32 msec > 17 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 48 msec 48 msec 56 msec > 18 ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 44 msec 48 msec > ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 52 msec > 19 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 56 msec 104 msec 56 msec > 20 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 52 msec 52 msec 56 msec > 21 unknown.Level3.net (63.209.98.66) 52 msec 52 msec 56 msec > 22 andc-br-3-f2-0.atlantic.net (209.208.9.138) 52 msec 52 msec 56 msec > 23 172.22.122.1 52 msec 56 msec 52 msec > 24 10.247.28.205 52 msec 52 msec 56 msec > 25 sl-crs2-dc-0-5-3-0.sprintlink.net (144.232.19.93) 52 msec 56 msec 52 > msec > 26 sl-st20-ash-9-0-0.sprintlink.net (144.232.18.228) 56 msec 56 msec 56 > msec > 27 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 52 msec 52 msec 52 > msec > 28 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 52 msec > vlan69.csw1.Washington1.Level3.net (4.68.17.62) 56 msec > vlan89.csw3.Washington1.Level3.net (4.68.17.190) 56 msec > 29 ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 64 msec > ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 52 msec 56 msec > 30 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 76 msec 72 msec 72 msec > > I've seen the 172.22.122.1 & 10.247.28.205 hops before. They occasionally > show up in traces when the traffic is jumping over to Sprint. Sometimes > they don't show up though. i.e. Tracing from my house: > > traceroute to 209.208.33.1 (209.208.33.1), 30 hops max, 40 byte packets > 1 172.31.0.1 (172.31.0.1) 0.336 ms 0.272 ms 0.268 ms > 2 10.210.160.1 (10.210.160.1) 10.109 ms 11.719 ms 14.265 ms > 3 gig7-0-4-101.orldflaabv-rtr1.cfl.rr.com (24.95.232.100) 15.302 ms > 15.324 ms 16.687 ms > 4 198.228.95.24.cfl.res.rr.com (24.95.228.198) 16.688 ms 18.812 ms > 18.816 ms > 5 te-3-3.car1.Orlando1.Level3.net (4.79.116.145) 20.084 ms 19.946 ms > te-3-1.car1.Orlando1.Level3.net (4.79.116.137) 21.328 ms > 6 unknown.Level3.net (63.209.98.66) 19.900 ms 14.714 ms 14.689 ms > 7 andc-br-3-f2-0.atlantic.net (209.208.9.138) 104.058 ms 11.932 ms > 13.584 ms > 8 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 15.872 ms 15.886 ms > 17.238 ms > 9 * * * > 10 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 41.277 ms 41.964 ms > 41.955 ms > 11 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 43.360 ms 44.578 ms > 35.635 ms > 12 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 37.035 ms 37.062 > ms 33.185 ms > 13 vlan89.csw3.Washington1.Level3.net (4.68.17.190) 44.060 ms 44.057 ms > vlan99.csw4.Washington1.Level3.net (4.68.17.254) 39.603 ms > 14 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 38.123 ms > ae-91-91.ebr1.Washington1.Level3.net (4.69.134.141) 39.546 ms > ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) 38.115 ms > 15 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 46.284 ms 46.275 ms > 46.274 ms > 16 ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 52.523 ms > ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 53.338 ms > ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 53.299 ms > 17 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 34.964 ms 39.582 ms > 38.088 ms > 18 ae-6-6.car1.Orlando1.Level3.net (4.69.133.77) 36.701 ms 38.144 ms > 36.949 ms > 19 unknown.Level3.net (63.209.98.66) 36.902 ms 37.750 ms 37.717 ms > 20 andc-br-3-f2-0.atlantic.net (209.208.9.138) 37.729 ms 35.812 ms > 35.048 ms > 21 ocalflxa-br-1-s1-0.atlantic.net (209.208.112.98) 37.485 ms 37.601 ms > 36.495 ms > 22 * * * > 23 sl-bb20-dc-6-0-0.sprintlink.net (144.232.8.174) 56.459 ms 56.449 ms > 57.709 ms > 24 sl-st20-ash-10-0.sprintlink.net (144.232.20.152) 57.694 ms 57.692 ms > 60.243 ms > 25 te-10-1-0.edge2.Washington4.level3.net (4.68.63.209) 103.257 ms > 100.829 ms 82.571 ms > 26 vlan99.csw4.Washington1.Level3.net (4.68.17.254) 70.401 ms > vlan89.csw3.Washington1.Level3.net (4.68.17.190) 69.262 ms > vlan99.csw4.Washington1.Level3.net (4.68.17.254) 82.700 ms > 27 ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 74.132 ms > ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) 74.135 ms > ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) 75.540 ms > 28 ae-2.ebr3.Atlanta2.Level3.net (4.69.132.85) 58.656 ms 60.838 ms > 54.346 ms > 29 ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 59.323 ms > ae-61-60.ebr1.Atlanta2.Level3.net (4.69.138.2) 59.336 ms > ae-71-70.ebr1.Atlanta2.Level3.net (4.69.138.18) 63.323 ms > 30 ae-1-8.bar1.Orlando1.Level3.net (4.69.137.149) 127.652 ms 57.884 ms > 57.851 ms > > >From the traces I've seen, it seems if the first Sprint hop is sl-bb20-dc, > the private IP hops don't show up. If the first Sprint hop is sl-crs2-dc, > then the private IP hops are there. I wonder if anyone from Sprint can > shed some light on that? > > Unfortunately, the Sprint engineer I intitially made contact with who was > helpful and seemed curious about the issue seems to have vanished and > isn't returning my calls or emails. Anyone else from Sprintlink care to > play? > > ---------------------------------------------------------------------- > Jon Lewis | I route > Senior Network Engineer | therefore you are > Atlantic Net | > _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________ > > -- Sent from Gmail for mobile | mobile.google.com
