1) Burn media converters no matter what-- they are EVIL. I've used many from 3 different manufacturers, and they are, as a whole, horrible. They aren't just regularly poor performers, but their reliability is crap.
2) Your issue (prima facie) sounds like a classic duplex mismatch issue between the copper side of one the IMC's and its connecting device... most likely the provider side (since you don't have control over it.) Short Frames, CRC, and alignment errors are the classic signs of a duplex mismatch between devices. If you don't control the provider's IMC, then that's probably where the problem lies. You stated that you performed a lot of testing over the circuit and you have no testing ability on the provider side, If you could test the 'circuit' without the BGP session (with something like Iperf or MTR (MTR xxx.xxx.xxx.xxx -i .03 for rapid packet rate) I think you could confirm the duplex mismatch issue with loads of errors on these two tests, prior to accepting the fiber circuit... but I would still take the fiber circuit. One day one of those IMC's will burn out or flake, and you'll have a outage. Dan Farrell Applied Innovations [EMAIL PROTECTED] > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of > Per Engelbrecht > Sent: Wednesday, September 20, 2006 11:06 AM > To: misc@openbsd.org > Subject: Re: BGP router now running desperately low on memory [epilogue] > > Hi all, > > Just to make sure nobody's sitting and wondering what happened with this > thread, then here's a final mail with a short description of what's > cooking right now and what was boiling back then. > > Below you'll find: > - case > - situation > - conclusion > - physical connection > - hardware > - a few tips > > > ###### > Case: # > ###### > When I added another bgp peer to my router the overall network/routing > performance on the server was brought to an almost staggering halt until > I downed the bgp session again. > > > ########## > Situation: # > ######### > At first I had warp-speed on the wire and all tests on the connection > (*) seemed okay. > Trivialities like speed-, duplex-, mtu settings etc. was agreed upon > before the connections was established. > The time elapsed from initiating the BGP session to severe performance > degradation was <2 minutes and if I did not down the BGP session within > the next minute (literally) then routing and network performance would > drop like a piano out of the sky. In short I was using all mbuf (Kbytes > allocated to network >97%). > Raising kern.maxclusters stepwise gave me a short lived break until I > reached a given point (see tips below). Above that I gained nothing and > stopped raising it any further. > > The new carrier had a lot of alignment errors (CRC/FCS) and packet size > problems (Jabbers/rxOversizedPkts) in their log / on "their" side. We > both had heavy packelosses after these few minutes. > 'tcpdump' did not reveal any significant signs of a sick connection on > my side. > A lot of testing has been done since. The connections however, is still > not running but adjustments on the peers side and replacements on the > connection itself has raised the "panic-threshold" from <2min. to around > 18min. before disaster strikes. > > > ############ > Conclusion: # > ########### > I'll receive a fiber directly to my front door from the new peer shortly > i.e. we'll bypass the copper-fiber-copper connection. I don't like not > being able to pinpoint the problem before moving on, but I have no way > of seeing what's going on on the "other side". I have an idea that the > Cisco box and the converters do not like each other, but again it's only > a guess. > > What I do know is that an error-prone connection combined with a well > connected BGP peer, can jeopardize an entire bgp routers performance. > BGP can not "see" how well the connection is runing - it can only see > link and link = traffic = congestion. > > I can not claim to have found the 'holy grale' in BGP troubleshooting > but I can rightfully claim that I've eliminated my OpenBGPD as source of > error (both as i386 and amd64) and I can also rightfully claim to have > found a few settings that actually makes a difference. > If the carrier find the problem and inform me, I will of course inform > all of you as well. > > > > ################## > Physical connection: # > ################## > We are terminating with this carrier in a FE port but due to the > distance between them and us at the datacenter location, a FDDI > connection was placed in between like: > > [our > router]----[100baseTX]----[IMC**]----//..fiber..//----[IMC**]---- > [100baseTX]----[switch > integrated in a Cisco 7200 iron]----[Cisco iron itself/router] > > * Attenuation on the FDDI part was 1.2db respectively 1.3db which is not > brilliant, but okay. More importantly it's within the specifications of > the IMC's. > > ** (IMC = MOXA Industrial Media Converter 101 a.k.a. IMC-101 for both > Single- and Multi mode / SC connectors. We even replaced these with MOXA > EDS-208-M-SC (larger model) as well). > > All Cat6 STP cables has been replaced more than once and the fiber once. > > > ########## > Hardware: # > ########## > My OpenBGPD setup is plain-vanilla with 4 BGP peers, one eBGP peer and > two public networks on the inside (700+ servers). > The BGP box I have (OpenBSD 3.9 -stable / amd64 / bsd.mp) is a > "serverworks" based box with 2GB of ram per cpu, Intel PRO/1000MT dual > and quard server nic's, U320 SCSI etc., etc. -> i.e. this is not about > exhaustion due to inferior or inadequate hardware. > My network performance related sysctl settings: > net.inet.ip.ifq.maxlen=250 > kern.maxclusters=32768 (this has been tested stepwise (~6500 at a > time) from the std. setting [6144] and up) > > Note_0: normally I run this on a i386 Xeon based box with 4GB of ram, > but the box is down for upgrade/maintenance, hence the temporary amd64 > arch. > > Note_1: the new boxes I'm building has a 64-bit Xeon cpu, 2GB of ram, > Syskonnect nics and i386 as arch. > > > ########### > A few tips: # > ########## > The tips I've put below are all "confirmed successes" and a mixture of > experience, what I've been told by Henning/Claudio and what I've seen on > this list (some of the sysctl settings). > The important thing is that they actually work. > > 0 - run busy BGP routers on i386 compared to amd64 > > 1 - run busy BGP routers on [serverworks based] single cpu systems. > > 2 - run busy BGP routers on 2GB of memory at the most. > On a healthy box going from 4 GB of ram to 2GB gives a drop on almost > 20% in 'Kbytes allocated to network'. > >2GB of ram is in fact counterproductive ..! > > 3 - work carefully with kern.maxclusters - don't just raise it. > The effect actually differs between arch. > > > > > Hope you can use some of it. > The best to your all, > > /per > [EMAIL PROTECTED]
