Re: Max Prefix Out, was Re: Verizon 701 Route leak?
On Sat, 2 Sep 2017 at 05:41, Randy Bush wrote: > >>> i have 142 largish bgp customers, a large enough number that the number > >>> of prefixes i receive from them varies annoyingly. how do i reasonably > >>> automate setting of my outbound prefix limit? > >> > >> First, it seems you know the inbound so automating the outbound is > simple > >> arithmetic. > > > > I would have said the same... i ought to know high-water marks for your > > inbound peer count(s), and can work out a +20% outbound... > > you just assumed that the transitive closure of everybody's cones > implement and propagate count. ain't gonna happen. I am not sure what the issue here is. If I can tell my peering partner a recommended maximum prefix value for them to set on their side, surely I can configure that same value on my side as the upper outbound limit. Kind regards, Job
Re: Max Prefix Out, was Re: Verizon 701 Route leak?
> i have 142 largish bgp customers, a large enough number that the > number of prefixes i receive from them varies annoyingly. how do > i reasonably automate setting of my outbound prefix limit? First, it seems you know the inbound so automating the outbound is simple arithmetic. >>> >>> I would have said the same... i ought to know high-water marks for >>> your inbound peer count(s), and can work out a +20% outbound... >> >> you just assumed that the transitive closure of everybody's cones >> implement and propagate count. ain't gonna happen. > > I am not sure what the issue here is. If I can tell my peering partner > a recommended maximum prefix value for them to set on their side, > surely I can configure that same value on my side as the upper > outbound limit. which is why i do not tell peers a max count. this stuff works for small isps, in the lab, ... but not at scale; especially when you have isps as customers. i wish it did. bgp at scale is rather dynamic. i suspect your $dayjob's irr filters being exact help a bit. randy
Re: Max Prefix Out, was Re: Verizon 701 Route leak?
On Sat, Sep 02, 2017 at 04:27:03PM +0900, Randy Bush wrote: > > I am not sure what the issue here is. If I can tell my peering > > partner a recommended maximum prefix value for them to set on their > > side, surely I can configure that same value on my side as the upper > > outbound limit. > > which is why i do not tell peers a max count. I think you'll find that some of your peers will make an educated guess and set an inbound limit anyway. Actively requesting that no limit is applied may make one part of a fringe minority. Most networks publish a baseline number via a rendezvous point like PeeringDB, this makes it easy to signal to larger groups what the recommended values are. > this stuff works for small isps, in the lab, ... but not at scale; > especially when you have isps as customers. i wish it did. In this context "small ISPs" may account for the majority of the target audience. It appears there are about 50,000 "origin only" ASNs [1], for the majority of those it'll be straightforward to decide on a sensible max-out value. BGP speaking CDN caching nodes are also low hanging fruit. But even for a network like NTT I can see benefits of a max-out limit in a number of scenarios. > bgp at scale is rather dynamic. i suspect your $dayjob's irr filters > being exact help a bit. Yes, BGP is dynamic, but these days a lot of the topology at the wholesale level has been firmly pinned down through mechanisms like 'peerlock' [2]. Speaking as an ISP for ISPs: NTT/2914 applies an inbound maximum-prefix limit on each and every EBGP session. Kind regards, Job [1]: http://bgp.potaroo.net/cgi-bin/plota?file=%2fvar%2fdata%2fbgp%2fas2%2e0%2fbgp-as-term%2etxt&descr=Origin%20only%20ASes&ylabel=Origin%20only%20ASes&with=step [2]: http://instituut.net/~job/peerlock_manual.pdf
Re: Max Prefix Out, was Re: Verizon 701 Route leak?
(from earlier randy) > you just assumed that the transitive closure of everybody's cones > implement and propagate count. ain't gonna happen. well, I was thinking that you can survey your customers to know their approximate inbound number, you can implement a max-prefix in from them with that (ideally you're already doing that). You can figure out the output from you as well in a similar fashion. In either case you're not implementing a limit that's 1% larger than the actual number, you're hedging the number for at least operational overhead reasons to 20-40%. Even a large ISP is sending (today) less than 100k prefixes when the peer isn't asking for 'full routes'. So, I'd imagine you bucket your customers as: default only - limit 10 customer prefixes only - limit +30% of your customer routes set full transit - +20% of current full table (yes, you may have more buckets than me, meh) and those are good starting points, if you keep these bucketed you can just ratchet up the limits as time requires. The prefix-limits (in or out) isn't to stop jim-isp from sending 2 of jane-isp's routes, it's to keep jim-isp from making a bad situation very bad. You (ideally!) have prefix-lists to limit jim from sending jane's routes. On Sat, Sep 2, 2017 at 4:16 AM, Job Snijders wrote: > On Sat, Sep 02, 2017 at 04:27:03PM +0900, Randy Bush wrote: > > > I am not sure what the issue here is. If I can tell my peering > > > partner a recommended maximum prefix value for them to set on their > > > side, surely I can configure that same value on my side as the upper > > > outbound limit. > > > > which is why i do not tell peers a max count. > > I think you'll find that some of your peers will make an educated guess > and set an inbound limit anyway. Actively requesting that no limit is > applied may make one part of a fringe minority. > > This is a quick survey of your peers and setting the buckets from above at 'sane' limits, right? > Most networks publish a baseline number via a rendezvous point like > PeeringDB, this makes it easy to signal to larger groups what the > recommended values are. > > > this stuff works for small isps, in the lab, ... but not at scale; > > especially when you have isps as customers. i wish it did. > > In this context "small ISPs" may account for the majority of the target > audience. It appears there are about 50,000 "origin only" ASNs [1], for > the majority of those it'll be straightforward to decide on a sensible > max-out value. BGP speaking CDN caching nodes are also low hanging > fruit. But even for a network like NTT I can see benefits of a max-out > limit in a number of scenarios. > > > bgp at scale is rather dynamic. i suspect your $dayjob's irr filters > > being exact help a bit. > > Yes, BGP is dynamic, but these days a lot of the topology at the > wholesale level has been firmly pinned down through mechanisms like > 'peerlock' [2]. > > Speaking as an ISP for ISPs: NTT/2914 applies an inbound maximum-prefix > limit on each and every EBGP session. > > you can answer this if you want, or not.. but I'm curious, is this tuned-per-peer? or via some bucket form as I proposed above? I expect ntt COULD per-peer, since I think almost-all-config is auto-generated, but I'd be curious still if you decided at the bucket level instead because it's saner to think about it that way (for me anyway) or just went 'current +N%' for each peer? > Kind regards, > > Job > > [1]: http://bgp.potaroo.net/cgi-bin/plota?file=%2fvar%2fdata% > 2fbgp%2fas2%2e0%2fbgp-as-term%2etxt&descr=Origin%20only% > 20ASes&ylabel=Origin%20only%20ASes&with=step > [2]: http://instituut.net/~job/peerlock_manual.pdf >
Re: Max Prefix Out, was Re: Verizon 701 Route leak?
On Sat, Sep 02, 2017 at 12:08:41PM -0400, Christopher Morrow wrote: > > I think you'll find that some of your peers will make an educated > > guess and set an inbound limit anyway. Actively requesting that no > > limit is applied may make one part of a fringe minority. > > This is a quick survey of your peers and setting the buckets from > above at 'sane' limits, right? yes > > Speaking as an ISP for ISPs: NTT/2914 applies an inbound > > maximum-prefix limit on each and every EBGP session. > > you can answer this if you want, or not.. but I'm curious, is this > tuned-per-peer? or via some bucket form as I proposed above? I expect > ntt COULD per-peer, since I think almost-all-config is auto-generated, > but I'd be curious still if you decided at the bucket level instead > because it's saner to think about it that way (for me anyway) or just > went 'current +N%' for each peer? I can contribute two examples: NTT (AS 2914): We use both approaches. For downstream customers a simple bucket system is used (currently with just one bucket: 31000 for IPv4, 2000 for IPv6). On the peering side of things the announcement count for each peer is polled at regular intervals and a +N% limit is set. In both approaches an override option is available in case someone emails the NOC "hey, we are about to turn up something big, can you ensure there is sufficient headroom". Coloclue (AS 8283): For every peering partner, data is fetched from the PeeringDB API and the fields visible here https://www.peeringdb.com/asn/2914 as 'IPv4 Prefixes' and 'IPv6 Prefixes' are used as input into the router configuration process. Coloclue's formula is simple, if the field's value is less than 100, set the limit to 100, if the value is over 100: add 10% to whatever value was published. This process is executed every 12 hours. In case no PDB record for the ASN exists: set 10,000 for IPv4 / 1,000 for IPv6. A manual override mechanism exists. If I compare the two: NTT's method emphasizes business continuity and has no external dependencies, Coloclue (being a network for experimentation) explores how to avoid explicit noc-to-noc coordination and relies on self-published data being kept up to date. Whatever your cooking method, maximum prefix limits should never get in the way of normal operations (e.g. organic growth), but exist only to try to nip obvious route leaks in the bud. This means one can be quite generous when picking values. Kind regards, Job
Re: Moving fibre trunks: interruptions?
On 2017-09-01 18:38, Ricky Beam wrote: > Buried stuff requires a great deal of planning, permitting, and insurance. Are cables in railway right of way considered "burried stuff" from the point of view of all the regulatory approvals since it is on private land (railway's) ? I take it that it is the railway which burries a new cable in its ballast (since it knows where other cables are burried, has to handle cable crossing its bridges etc)? In the specific case of Turcot in Montréal, the government was in charge of cleaning the land, removing any obstructions (such as a major sewer collector which had to be moved) etc, and even drained and compressed the ground before handing it over to CN to build its tracks. So CN got a clean slate, ready to lay tarp, ballast and tracks (and later string fibre). (ironically, that land used to belong to CN and was the Turcot rail yards).
Re: Moving fibre trunks: interruptions?
Aerial's not that rare in Europe (rural areas, sometimes even close to metro). Cheers, mh Le 01/09/2017 à 21:52, Rod Beck a écrit : > I don't think there is virtually any aerial in Europe. So given the cost > difference why is virtually all fiber buried on this side of the Atlantic? > > > > From: NANOG on behalf of Jared Mauch > > Sent: Friday, September 1, 2017 9:37 PM > To: Michael Loftis > Cc: Nanog@nanog.org > Subject: Re: Moving fibre trunks: interruptions? > > > >> On Sep 1, 2017, at 3:32 PM, Michael Loftis wrote: >> >> If it is in the railroad RoW they may be restricted to daylight working >> only. Check with your provider or OSP crew. >> > > Yup. Railroad work is complex just because you have to coordinate with the > railroad owner and they have to be onsite for all work. The cost of going > underground vs aerial is also astronomical in many cases. > > - Jared
Re: Moving fibre trunks: interruptions?
That depends on the country. Here in Denmark it is not possible to get rights to put up any aerial at all. The cost difference is irrelevant when you have no option but to put it in the ground. Not only is there no new aerial installations here but the old ones are taken down. Very little is left by now and in a few years it will all be gone. The municipalities want it pretty and wires in the air is ugly. One advantage however is that buried stuff usually survives storms better. Den 1. sep. 2017 21.53 skrev "Rod Beck" : I don't think there is virtually any aerial in Europe. So given the cost difference why is virtually all fiber buried on this side of the Atlantic?
Re: Moving fibre trunks: interruptions?
Le 02/09/2017 à 21:25, Baldur Norddahl a écrit : > That depends on the country. Here in Denmark it is not possible to get > rights to put up any aerial at all. The cost difference is irrelevant when > you have no option but to put it in the ground. > > Not only is there no new aerial installations here but the old ones are > taken down. Very little is left by now and in a few years it will all be > gone. The municipalities want it pretty and wires in the air is ugly. > > One advantage however is that buried stuff usually survives storms better. Right. Here in France it (aerial running along with copper) happens even close to metropoles (like Paris). mh > > Den 1. sep. 2017 21.53 skrev "Rod Beck" : > > I don't think there is virtually any aerial in Europe. So given the cost > difference why is virtually all fiber buried on this side of the Atlantic?
Re: Max Prefix Out, was Re: Verizon 701 Route leak?
> On Sep 2, 2017, at 12:41 PM, Job Snijders wrote: > > Coloclue (AS 8283): > >For every peering partner, data is fetched from the PeeringDB API >and the fields visible here https://www.peeringdb.com/asn/2914 as >'IPv4 Prefixes' and 'IPv6 Prefixes' are used as input into the >router configuration process. Coloclue's formula is simple, if the >field's value is less than 100, set the limit to 100, if the value >is over 100: add 10% to whatever value was published. This process >is executed every 12 hours. In case no PDB record for the ASN >exists: set 10,000 for IPv4 / 1,000 for IPv6. A manual override >mechanism exists. > > If I compare the two: NTT's method emphasizes business continuity and > has no external dependencies, Coloclue (being a network for > experimentation) explores how to avoid explicit noc-to-noc coordination > and relies on self-published data being kept up to date. How has the Coloclue max-prefix method described worked out? This sounds pretty effective for this type of network. How often has manual intervention (beyond a pre-arranged manual override) been required? Theodore Baschak - AS395089 - Hextet Systems https://bgp.guru/ - https://hextet.net/ http://mbix.ca/ - http://mbnog.ca/
Re: Max Prefix Out, was Re: Verizon 701 Route leak?
> well, I was thinking that you can survey your customers to know their > approximate inbound number, you can implement a max-prefix in from them > with that (ideally you're already doing that). > > You can figure out the output from you as well in a similar fashion. > > In either case you're not implementing a limit that's 1% larger than the > actual number, you're hedging the number for at least operational overhead > reasons to 20-40%. Even a large ISP is sending (today) less than 100k > prefixes when the peer isn't asking for 'full routes'. > > So, I'd imagine you bucket your customers as: > default only - limit 10 > customer prefixes only - limit +30% of your customer routes set > full transit - +20% of current full table > (yes, you may have more buckets than me, meh) > > and those are good starting points, if you keep these bucketed you can just > ratchet up the limits as time requires. The prefix-limits (in or out) isn't > to stop jim-isp from sending 2 of jane-isp's routes, it's to keep jim-isp > from making a bad situation very bad. You (ideally!) have prefix-lists to > limit jim from sending jane's routes. first, i have no magic bullet. sure wish i did. and i do not mean ill using ntt as an example; after all, job assures us they are very very important and very smart :) even pulling from peering.db, which is about as well-maintained as the irr (a race to the bottom), as job suggests, this relies on manual maintenance. it assumes the same count at all peerings, etc. etc. and the registered counts are horrifyingly approximate; ntt could leak 10k prefixes and not hit the limit as published. that they are gross approximations shows that they are not at all rigorous, calculated, ... this is not to say that any reasonable prefix count would have allowed the full-table goog leak to vz. and vz could have used an as-path filter not allowing _goog_(lotso-tier-ones)_ (which ntt uses, for example). but without a rigorous source of ground truth, prefix count limits will be approximate upper bounds and hence allow large mis-announcements. it is one tool in a sadly sparse toolbox, and not a strong one. randy
Re: Moving fibre trunks: interruptions?
The the USA, we have tornadoes, hurricanes, nasty wind and lightning, ice accumulation on lines, and idiot squirrels that like to eat fiber. Buried fiber over time will end up being cheaper than aerial once you factor in maintenance and repair. Add to that the additional cost of pole studies, replacement and attachments that the electric utility demands to be on their poles. When you're paying them for a pole study so that they can tell you that the pole isn't strong enough or tall enough to provide clearance, then paying them to replace the pole to make it sufficient, and then paying them monthly after that to be on the pole, just putting it in the ground starts making a lot of sense. We were affected by a fiber cut a while back caused by a truck that wiped out several electric poles. It was over 24 hours before the fiber company was even allowed access to the poles, because the electric utility wouldn't release them until they were finished. There's a lot to be said about controlling your own destiny by putting the fiber in the ground where you can work on it whenever you need to. You need insurance regardless of aerial or buried. 1/2 mile might be an exaggeration. Telcom doesn't do much trenching. Vibratory plow in open areas where there's nothing to contend with, but in urban, it's almost always directional boring. The only time we hit anything is when the other utilities fail to locate at all or fail to locate correctly. The easiest thing is to contract with a cable construction company that already has all the skills, insurance and equipment, and let them deal with it. On Fri, Sep 1, 2017 at 5:38 PM, Ricky Beam wrote: > On Fri, 01 Sep 2017 15:52:40 -0400, Rod Beck < > rod.b...@unitedcablecompany.com> wrote: > >> I don't think there is virtually any aerial in Europe. So given the cost >> difference why is virtually all fiber buried on this side of the Atlantic? >> > > Aerial is simple and fast... pull the cable through a stringer, move to > the next pole and repeat; when a section (about a mile) is done, it's > hoisted into the air and tied to the pole. The stringers are then moved to > the next mile of poles and the process repeats. > > Buried stuff requires a great deal of planning, permitting, and insurance. > You have to know everything that's ever been stuffed in the ground within > half a mile of where you're working to avoid the inevitable cutting of > something important -- gas, water, sewer, power, other telcom, even vacuum > tube lines and subways. And then you need trenching gear to get stuff in > the ground, and crews to come along behind to remediate the "environmental > damage". > > (Once the conduit is in the ground, it's a trivial matter to blow whatever > you need through it.) >
