Re: wet-behind-the-ears whippersnapper seeking advice on building a nationwide network
*This message was transferred with a trial version of CommuniGate(r) Pro* >Does anybody actually *have* a functional 7 track drive? The folks restoring at least one IBM 1401 probably have several. http://ibm-1401.info/ Other than replacing a lot of older tab shop hardware, a primary function for may 1401s was to do card reading and printing for jobs submitted on 7 track tape to 7094s.
Re: Slightly OT: Calculating HVAC requirements for server rooms
Below are a few snippets from recent posts, in no particular order, that had me saying to myself "does not anyone remember an interesting alternative I thought had come up on NANOG a few years ago?" Well maybe it was some other list, but it is not really worth going back and looking. It isn't quite true, or totally wise, but you can EASILY ignore HOT and COLD isle systems, and you could even have adjacent cabinets in any row blowing opposite directions and randoml;y facing across the isle to a cabinet in the next row that could be facing EITHER WAY, and you need not care! (don't really do it, BUT YOU SAFELY COULD) No raised floors needed, unless you need to for cables, and relatively low ceilings and ladder racks / cable trays with massive wads of cables that would block normally required air flow being no problem at all. And NO it isn't chilled water to leak and destroy your equipment Oh, and at maybe only 30KW capacity per cabinet (with an extra very REAL 50% reserve capacity) - is that enough for you...?? And no problem with random cabinets or multiple whole rows with no equipment yet or even just turned off until they pay their bill. Nothing freezes, and nothing roasts. Just works. All the pieces I snipped below are about trying to get the heat from the cabinets to the CRAC and the cold air back with as little mixing and possible. The more you mix, the more total air you have to circulate and the lower your efficiency goes. *** start of snips *** (And the elephant in the room no one has mentioned is "air flow". Cooling capacity is only half the equation. Air flow *volume* is just as important.) You shouldn't even *have* a drop ceiling in a modern computer room. You want the room to be as tall as practical so that the air from the hot aisles has somewhere to go on its way back to the HVAC, other than back through and around the cabinets. I love my 30' ceiling. Even with all the things that are wrong with our HVAC setup, the servers survive due to that ceiling. ... versus HVAC professionals that understand the holistic picture including hot aisles, cold aisles, humidity control and flow. I wouldn't want to call in a professional without first understanding the problem well enough to assess whether I was getting a competent answer. The return feed is below the units pulling ambient air, and the cold air is injected 15+ feet above the isle behind the servers, intermixing with the hot air as it rises up the wall. At least it works, but it could be better and changes will need to be made before I can reach 50% capacity in the racks. *** end of snips *** A few simplifying ground rules. An existing "dumb-ass" grade CRAC system can be used to control humidity and to do any required fresh air changes, etc. With cabinets of electronics, we are ONLY talking about a sensible load with the stuff below. We DO NOT WANT or need to deal with humidity with the system I'm referring to. It is designed to JUST remove sensible heat VERY VERY efficiently. Its refrigerant lines, both supply and return, are almost at room temperature. No sweating, no dripping, no insulation really needed! Piping from the mechanical room is designed for low pressure drop for efficiency, but this refrigerant, R744, does a lot of cooling with very small pipes. The regular refrigeration systems we are used to have oil circulating with the refrigerant and special care is needed to ensure its return to the compressor. Also, the traditional system needs a thermostatic expansion valve (TXV) so the evaporator has liquid refrigerant until almost the end of the coil, but no liquid is returned to damage the compressor. A typical TXV may have a MOPD (minimum operating pressure differential) of 100PSI. In some cool outside weather conditions, you artificially throttle fans or even bypass some of the condenser coils to keep your head pressure up to keep that 100PSI MOPD so you get adequate liquid flow through the TXV and don't starve the evaporator (which cuts its capacity and can lead to icing that can progress across the face of some coil designs and then blocks air flow). But keeping that 100 PSI in mild weather is inefficient, too. Anyway, with NO oil circulation, and NO compressor in the loop at all and NO TXV needed, you have something that is very close to a two pipe steam heating system with a condensate return pump for the boiler. Typical of many buildings and even some larger homes especially some years ago. Here, however, the "BOILER" is a finned coil on the back of each of your cabinets. This coil is fed from a local manifold via ball valves, excess flow safety shutoff valves, and flexible metal hoses. The finned coil equipped rear door also has its own fans. It gets raw undiluted hot air exiting your equipment, passes it over coils loaded with a liquid refrigerant just below room temperature and AT a system pressure that any additional heat added will just boil off some of the liquid which will be entrained as b
Re: DSX cross-connect solution
Use the ADC 84 port DSX blocks with six 32 pair CHAMP connectors on the rear. You will stay saner if use the ones with three sections - A B C - on the front each with 1-14 jacks on the top row and 15-28 on the bottom. The alternate that looks the same from a distance simply has a top row of 1-42 and the bottom row is 43-84. We use and LIKE the ones that have 1/2 the wirewrap pins on the left and the other half on the right - ON THE FRONT. ADC has dropped hundreds of oddball custom DSX they used to make for every primadonna boonies telco and have standardised on the higher volume items at LOWER prices. Ten years ago, I could save a couple of hundred $s by ordering a particular DSX pannel from North Supply and yet their price would suck compared to Anixter's on some other panel and Alltel (now Windstream) would beat them all on yet another one. It all boiled down to ITEM VOLUME PRICING and not product family volume pricing from ADC. You had to use the supply house that had LOTS of other customers that used the panel you wanted. The plus there was that they would also be in stock and you didn't have to wait a couple of weeks for the truck from Mexico to arrive. In ALL cases, wire the 5th green "TL" (tag light) wire when you crossconnect, and so when you jack intio the monitor port, you are also grounding the TL wire and the tag led at each end of the xconnect flashes for the 1st minute and then stays solidly on. Use REAL ADC xconnect wire - two pairs and the green TL wire. It is TINNED, and you really WANT tinned wire if you are wrapping. Trivial to learn. Get any of several styles of cut-and-strip-to-wrap-length tools and a good squeeze the long trigger bar manual wrap gun. Don't get an manual twisty pencil size one - that is for the tech's emergency tool bag. An electric wrap gun or even a pneumatic one is crazy for your volume. The UNWRAP tool can be the manual pencil size one. ALL 5 wires can be stripped at once. DO NOT play with cut-strip-wrap bits - they are for assembly line work with HIGH TORQUE tools and where incoming QA makes sure THE ONE WIRE TYPE being used meets the special standards needed for such tools. I LIKE single guage bits, but reality is that maybe 22-24 is good if you have much 22GA ABAM you need to wrap, but more likely you want a 24-26ga combo bit and can use it for everything else. Extra long bits and sleeves may be a plus, too, depending on your panel. OK tool in NY makes all the pieces you would need. It is pure nonsense in a small shop where you are NOT feeding copper T1s out to the boonies on your own cable plant to be even thinking of 22GA. In our COLOs where the typical customers are all ISPs in racks less than 100 feet away, we use a lot of 24GA and now 26GA cable and it is rats-ass Cat-3. NOT ABAM, and there are no problems. We always will have O and I in seperate cables, but that is more how panels come than necessity. The big EVIL ABAM shields avoids is NEXT and there simply isn't any Near End Cross Talk when everything is very close and all signal levels are about the same. If there is any chance it will be wrapped ever, get TINNED cable. If I'm ordering 28 pair cable with 32 pair connectors on both ends, I still get tinned. It may be cut into two blunt tailed shorter cables that get that end wrapped. It costs almost nothing more if your suppliers are well stocked and typically are using the same cable for many other customers. DO NOT think of wrapping cat-5! you will hate straightening out each wire so you can stick it in the wrap bit. And you won't be able to strip more than one pair at a time and even then you may want to do a wire at a time. If I have 30 M13s per rack, the wad of cables going up to ladder rack between racks is 60 cables. Using 26 ga, 28 or 32 pair TINNED but unshielded cat-3 cable will make you very happy. If you are is a LARGE building even way smaller than 111 8th ave, 26 ga is not for you. VZ uses 26 ga where they can. for T1s too, but it is shielded. You only have 300 T1s so you can stay reasonably sane no matter what you do, but this is NOT the place to cheap out. The ability to jack in and do testing or an end to end frontal patch in an emergency is great. Be absolutely POSITIVE you get the I and O connections done correctly and your head and arms around the DSX terminology. O is OUT - and that is NOT going away from you but is pissing bits at you from the "TRANSMIT" side of whatever is connected there. O is what the Monitor port is padded down from, also. I is IN and where you stick bits IN towards whatever is connected beyond that port and it would be "RECEIVE" in datacomm terminology at that device. With 32 pair champ connectors on the back it is trivial to FIX if some tech mixes things up on the back of the DSX (unlike when EVERYTHING was wirewrapped), but just don't start letting the techs mess up. If you do ONE wrong, cross connects to it will no
Re: Why choose 120 volts?
Seth Mattinen wrote: I have a pure curiosity question for the NANOG crowd here. If you run your facility/datacenter/cage/rack on 120 volts, why? I've been running my facility at 208 for years because I can get away with lower amperage circuits. I'm curious about the reasons for using high-amp 120 volt circuits to drive racks of equipment instead of low-amp 208 or 240 volt circuits And you have been doing something that is a right step in the right direction, but may well not be the best ultimate solution. Lets half ignore codes. Not to be illegal, but they CAN be changed and you can often get specific exceptions if you are not just in the back room of an office but are in a clearly professionally managed facility with well trained staff plugging in equipment. Every time I look at a nameplate and I see 100-250VAC I get very frustrated. If only that had been perhaps 100-300VAC, I could then run it on 277VAC and that is especailly nice for many reasons. Most large USA buildings already have 277 and probably all their flourescent lighting is run off it (277VAC ballasts are readily available and what look like rats-ass wall switches but are higher rated ones are readily available - both even at home despot if you look hard enough), so nothing terribly new has to be learned by electricians, etc. 277 is the phase leg to NEUTRAL voltage of a 277/480 WYE system that most everything except small human plugged appliances use in any but the very largest USA office building. It is what typically comes in from the power company. 120/208 is the output of typically a delta/wye transformer that steps that down for the dumb humans to safely use and you are paying the penalty of the WHOLE LOAD having to go through a second less than 100% efficient transformer. The beauty of 277 is that on a single breaker pole (unlike 208 where you are most likely to have 2 HOT legs and need a 2 pole simultaneous trip breaker) on reasonable size branch circuits that you are still allowed to plug MULTIPLE loads into without individual fuses or breakers (that is "allowed to" - you may chose to protect each outlet in the rack, but that is not compulsory) you get 277/120=2.31 times as much power available. Sadly routers, servers, switches, etc. typically are rated to 250VAC, so using raw 277 won't work. But let us see how close HP/IBM/ACP and many many others are getting still using ONE breaker pole per much more efficient branch circuit. NB that as you go to larger branch circuits in AMPS, you MUST be supplying just ONE load or MUST have additioanl breakers or fuses as you split it up. We all know 120/208 and 277/480. What about another NEW pair of voltages in WYE connection! Lets use 240/415. It is exactly twice 120/208 (well it is not stated as 240/416 I'd guess since 240 x 1.73259 = 415.82 they just truncate rather than round - though 2400/4160 is a standard designation...) and is inside the 250V max rating of the switching power supplies. It still uses a single breaker pole. Your get EXACTLY twice as much power out of a 240/415 WYE branch circuit as you would out of a 120/208 at the same AMPs. But you may save a transformer and its continuous power waste or at least part of it in between. How do you get to 240/415 is the next issue. If you have 2400/4160 or 7,960/13,800 primary into your building, and you do all your own transformers, getting 240/415(6) can be a single transformer step for you, and you will probably have many transformers so can also create seperate 277/480 for modest size AC inits and lighting, While LARGE chillers can be ordered at the higher voltages, and for the relatively small amount of 120/208 you probably should come off 277/480 into standard 120/208 delta wye transformers because normal electricians can do that rather then the 13K gods(). But if you are a smaller building the only voltage that makes sense that the utility is supporting is 277/480. Rather than take all your rack power through another transformer step with the losses and the extra heat to eject from the building, consider instead using buck (as in the classic BOOST/BUCK transformers) to knock that 277/480 down to 240/415. It can be packaged as a 3 phase unit for less than three singles, and will be smaller and less costly to have wired up, but the three singles may be available from stock. It is the same sort of device you must have in front of a load that needs 240 or 250 and can't handle 208, but in that case is wired BOOSTING rather than BUCKING. FWIW an electric range burner or a hot water heater element rated for 240 produces EXACTLY 75% of the heat if run on 208 (go do the math...), but you should NOT use boost bucks for such a simple situation because optional heating elements can be ordered originally OR bought as replacements for less than $10 each and easily replaced in the field to give the original 240 rated wattage on 208 supply. In any case the 3 phase b
Re: Out of warranty APC PDU repair
- Original Message - From: "Oliver Hookins" To: Sent: Wednesday, May 27, 2009 12:47 AM Subject: Out of warranty APC PDU repair Hi all, hopefully this isn't too off topic (since it's datacentre related). We have an APC AP7952 rack PDU which has stopped working. I believe the management module is faulty, and it is about 5 years old. APC don't service these outside of warranty at all so I'm trying to find a 3rd-party repairer. So far APC doesn't seem to understand this concept. Can anyone suggest a company that would work on these units in Sydney, Australia or nearby? One company here in the USA that might talk you through figuring out what is wrong and then can supply repair pars is Gruber Power Services in Phoenix. Google for GRUBER POWER as they have several domains , though www.gruber.com probably will work, too. They are one division of Gruber out there that also custom builds fiber jumpers and makes custom rack shelves and makes 1U 24 and 32 port RJ45 to variously wired 25 and 32 pair champ connectors. and a myriad of other things. They buy refurbish and sell LARGE UPSes and related gear of many brands. Also have good prices on UPS batteries for folks here in the USA in spite of UnitedParcelService shipping prices... If you are ordering pallet loads, truck becomes very attractive. Small quantities, use their EBAY store as it is very automated and they give you a slightly better price that way on small orders. Good folks! We did a site inspection of their facilities years ago right after a Nanog meeting that was in Scottsdale. By then they were already making several custom things for us and we were delighted to see where it all came from. They are flexible and versitile and like doing custom projects. They can easily do custom punched and bent, drilled, tapped, etc steel or aluminum "things" (typically RACK hardware) that can be left raw metal or powder coated. Best price is NOT at some 10/50/110 like step but is related to how many they can make of that item out of each standard sheet of incoming metal. For example, they custom made us dense 96 port OC-48 BNC panels in 19" and shorter in 23" sizes. D shaped holes and we let idle NOC fingers load bulk bought 75 ohm insulated BNC feed-throughs when and as needed. VASTLY less expensive than commercial offerings. Also got a lot of 1U 16 hole blanks as we just string 8 pack of 735 cable to a customer's racktop if they need T3s. That will take two such cables. Unless they have a lot of RAS boxes that need many T3s, most anyone else will prefer OC-x if they are getting much more capacity than that. Anyway, I hope they can help with your problem and it is INTERESTING to hear how long or how short APC chooses to support users...
Re: Subnet Size for BGP peers.
- Original Message - From: "Jim Wininger" To: Sent: Wednesday, July 29, 2009 3:59 PM Subject: Subnet Size for BGP peers. I have a question about the subnet size for BGP peers. Typically when we turn up a new BGP customer we turn them up on a /29 or a /30. That seems to be the "norm". We connect to many of our BGP peers with ethernet. It would be a simple So what is wrong with a /31? We use /30s but if you are short on IP space, look at using /31 rather than /30 links. Cuts your space usage in half. If I remember correctly, the BIG problem with using /31s when they first became "legal" was to decide if the customer still gets the higher numbered IP address (or you the lower one), or if you still get the ODD number. No kidding, it is a problem for some! Where you are on ethernet, use a seperate 802.1q vlan per customer and have your switch give the customer untagged packets. If you have downstreams in your COLO, and either free or as a paid service, offer to setup private vlans in your switch for any pair or group of customers that need to also connect to each other privately for whatever they are doing. In that latter case, they will be getting tagged packets but their routers or switches should have no problem dealing with them. We don't charge for physical crossconnects, so this has saved us having to do physical crossconnects between customers, and has saved customers router ports.
Re: Dutch ISPs to collaborate and take responsibility for bottedclients
Exactly correct. The number one priority, which trumps all others, is making the abuse stop. Yes, there are many other things that can and should be done, but that's the first one. Stopping the abuse is fine, but cutting service to the point that a family using VOIP only for their phone service can't call 911 and several children burn to death could bring all sorts of undesirable regulation let alone the bad press and legal expenses.
Re: So I've got this 2.5gig wave, what do I do with it?
Due to the vagaries of telecom pricing, I've ended up with a 2.5gig wavelength service between two locations when what I really wanted was a gig-e or two. I'm really not sure if this is a "transparent" wave service or not... the carrier is using gear from Ciena to hand it off to us and they seem to be big on transparent waves, so maybe it is, but nobody seems to be able to say for sure. OTU-1 ? Is it coming off a ciena CN 4200 ? OTN / G.709 ? Go see: http://www.ciena.com/products/flexselect_otn.htm or this one, (but the OTN/G.709 link seems to be failing for some reason) http://www.sonet.com/EDU/edu.htm OTU-1 can carry an intact OC-48 that is really part of someone elses network with no undesired interactions (think "tunnel"). OTU-2 carries an OC-192 the same way, and OTU-3 is for OC-768. With the right hardware, OTU-n transports random other optical (ethernet, storage, video, etc) signals without having them on an OC-x SONET signal. And Sonet traffic can also be muxed in and carried with all the random other stuff on the same OTU-n pipe. They can probably cheerily hand you either an OC-48 capable connection or an OTU-1 one to match what you are getting for hardware, but ask. Their OTU-1 based service they are providing you might be an individual wave/lambda/color but more likely is on an OTU-2 or even OTU-3 on a system that potentially carries many dozens of them on the same fiber using DWDM.. The OTN/G.709 stuff is relatively new and very desirable, Ciena has is as well and some others - see if that is what they are using. You may want to see what Ciena has to offer you to make use of it.. They can take in an OTU-n signal in what at first glance seems to be just a transponder or possible a mux-ponder card, but they are doing a complete OEO transformation and the E part, unlike some of their competition, is used effectively to let random spare ports even on other cards in the chassis be used for components being muxed up onto this particular OTU-x signal. It is quite flexible, and that could be why you are getting confusing information about what you actually have available to you. You could get old Cerent 454 aka Cisco 15454 gear pretty cheap, but the original 15454 gig-e cards will not make you happy. Newer cards can give you 2 full Gig-es and probably a modest amount of spare DS3 or DS1 tdm capacity on an OC-48, but then the price will be higher. They probably want more $$s if you ask them to hand you your "OC-48" as multiple smaller pieces (eg 2 x Gig-e), and I would think you should want to do the muxing yourself., but ask, if that is easiest for you. It will help if you KNOW what their hardware is actually capable of. So, how can I best make use of this beast in my ethernet-centered world? 1- Since it doesn't cost me anything, I'm going to try to send a gigabit LX signal and see what happens. The handoff from the carrier to me seems to be a plain 1310 type of signal... Light goes on, light goes off, maybe the carrier's gear doesn't care that it's not modulating "fast enough" and I'll get lucky? Unlikely. They probably have to know exactly what is coming at them, though it may be easily configured to be any of several possibilities at or around the same speed (eg OTU-1 vs OC-48) 2- MRV's EM2009-GM2 card takes 2xGE and spits out a 2.5G signal of some sort. However they won't guarantee me that it'll work when talking to the Ciana card the carrier is using. I should probably inquire about their return policy and go for it. Don't play with "returns", have them DEMO it for you for a few weeks. If they won't, shop elsewhere. MRV has all sorts of nice "bag or tricks" gadgets, but I'd first ask Ciena what small chassis they have that can play here. It will also be a "free" education into what the carrier's own capabilities probably are, and Ciena should know what equipment is in use there.
