BGP responding with wrong IP address.
Hi there, I am running OpenBGPd on an OpenBSD 4.4 router. Some times when traffic goes over one peer and finally gets to our router, the last hop will respond as a different peer. For example: Level3 IP 10.0.0.1 Global Crossing IP 192.168.0.1 Traffic traverses Global crossing all the way, last hop, 10.0.0.1 responds and vice versa. I noticed If i set one default gateway, this is always the IP to respond at the last hop. If i set multi-path default gateways then random IPs respond at the last hop. Is there a common way that I am missing to get around this? Am i overlooking something on the man pages? Thanks for your help! Regards, Ken
Re: BGP responding with wrong IP address.
On Sat, May 23, 2009 at 4:08 AM, Stuart Henderson wrote: > On 2009-05-22, Justin Credible wrote: > > I am running OpenBGPd on an OpenBSD 4.4 router. > > > > Some times when traffic goes over one peer and finally gets to our > router, > > the last hop will respond as a different peer. For example: > > > > Level3 IP 10.0.0.1 > > Global Crossing IP 192.168.0.1 > > > > Traffic traverses Global crossing all the way, last hop, 10.0.0.1 > responds > > and vice versa. > > > > I noticed If i set one default gateway, this is always the IP to respond > at > > the last hop. If i set multi-path default gateways then random IPs > respond > > at the last hop. > > > > Is there a common way that I am missing to get around this? Am i > overlooking > > something on the man pages? > > > > Thanks for your help! > > > > Regards, > > > > Ken > > > > > > If you're doing a traceroute to an address configured on the router > itself, it should reply from that address. Are you seeing something > different there? You definitely shouldn't. (ip_icmp.c:656) > > If you traceroute _through the router to another host_ (ip_icmp.c:668) > it will do a route lookup for the source, and use that as the source > address of the ICMP message (which is what shows in traceroute). > > What routes do you carry besides the default? No matter where default > points, if you have a specific route for the source of the traceroute > packets then it shouldn't be using the default. i.e. if you carry full > tables, you shouldn't see this. > > I point default at "localhost -reject" and take full tables and never > saw any strange behaviour with source address selection. > > Thanks for the response Stuart! I should have phrased that differently, sorry about that! What I mean by "the last hop" is "the last hop before the destination" so the second-last hop - My bad. So here would be another example. A traceroute should be: Traceroute 172.16.1.1 1.192.168.253.211 AS3549 2.192.168.24.5 AS3549 3. 192.168.0.1 AS3549 4. 172.16.1.1 MYASN But instead it would look something like this: Traceroute 172.16.1.1 1.192.168.253.211 AS3549 2.192.168.24.5 AS3549 3. 10.0.0.1 AS3356 4. 172.16.1.1 MYASN So the IP address which i use to peer with Level3 responds at the second last hop, rather than the Global Crossing IP since it traversed the entire way through Global Crossing. Both of the IPs which respond at the second last hop are on my router so the problem is on my end. It doesn't appear to be a BGP problem as much as a default route problem. I tried adding "reply-to" rules in my pf.conf so that traffic that comes in on one interface will go out the same interface but that doesn't seem to work either, since the reply from the wrong address happens before or during the state that stateful connections are being established. That's all that I can understand of the problem. I hope the explanation is satisfactory! Regards, Ken
Re: BGP responding with wrong IP address.
> I tried adding "reply-to" rules in my pf.conf so that traffic that comes in > on one interface will go out the same interface but that doesn't seem to > work either, since the reply from the wrong address happens before or during > the state that stateful connections are being established. > I should also add that the only two interfaces causing this problem are both Virtual interfaces using 802.1Q encapsulation (VLANs) on the same physical interface since both of these peers traverse the same wire.
Re: BGP responding with wrong IP address.
On Sat, May 23, 2009 at 3:25 PM, Stuart Henderson wrote: > On 2009/05/23 13:57, Justin Credible wrote: > > On Sat, May 23, 2009 at 4:08 AM, Stuart Henderson >wrote: > > > > > On 2009-05-22, Justin Credible > wrote: > > > > I am running OpenBGPd on an OpenBSD 4.4 router. > > > > > > > > Some times when traffic goes over one peer and finally gets to our > > > router, > > > > the last hop will respond as a different peer. For example: > > > > > > > > Level3 IP 10.0.0.1 > > > > Global Crossing IP 192.168.0.1 > > > > > > > > Traffic traverses Global crossing all the way, last hop, 10.0.0.1 > > > responds > > > > and vice versa. > > > > > > > > I noticed If i set one default gateway, this is always the IP to > respond > > > at > > > > the last hop. If i set multi-path default gateways then random IPs > > > respond > > > > at the last hop. > > > > > > > > Is there a common way that I am missing to get around this? Am i > > > overlooking > > > > something on the man pages? > > > > > > > > Thanks for your help! > > > > > > > > Regards, > > > > > > > > Ken > > > > > > > > > > > > > > If you're doing a traceroute to an address configured on the router > > > itself, it should reply from that address. Are you seeing something > > > different there? You definitely shouldn't. (ip_icmp.c:656) > > > > > > If you traceroute _through the router to another host_ (ip_icmp.c:668) > > > it will do a route lookup for the source, and use that as the source > > > address of the ICMP message (which is what shows in traceroute). > > > > > > What routes do you carry besides the default? No matter where default > > > points, if you have a specific route for the source of the traceroute > > > packets then it shouldn't be using the default. i.e. if you carry full > > > tables, you shouldn't see this. > > > > > > I point default at "localhost -reject" and take full tables and never > > > saw any strange behaviour with source address selection. > > > > > > > > Thanks for the response Stuart! > > > > I should have phrased that differently, sorry about that! What I mean by > > "the last hop" is "the last hop before the destination" so the > second-last > > hop - My bad. > > > > So here would be another example. > > > > A traceroute should be: > > > > Traceroute 172.16.1.1 > > 1.192.168.253.211 AS3549 > > 2.192.168.24.5 AS3549 > > 3. 192.168.0.1 AS3549 > > 4. 172.16.1.1 MYASN > > > > But instead it would look something like this: > > > > Traceroute 172.16.1.1 > > 1.192.168.253.211 AS3549 > > 2.192.168.24.5 AS3549 > > 3. 10.0.0.1 AS3356 > > 4. 172.16.1.1 MYASN > > > > So the IP address which i use to peer with Level3 responds at the second > > last hop, rather than the Global Crossing IP since it traversed the > entire > > way through Global Crossing. Both of the IPs which respond at the second > > last hop are on my router so the problem is on my end. It doesn't appear > to > > be a BGP problem as much as a default route problem. > > So, this part of my mail applies: > > "If you traceroute _through the router to another host_ (ip_icmp.c:668) > it will do a route lookup for the source, and use that as the source > address of the ICMP message (which is what shows in traceroute). > > What routes do you carry besides the default? No matter where default > points, if you have a specific route for the source of the traceroute > packets then it shouldn't be using the default. i.e. if you carry full > tables, you shouldn't see this." > > Do you carry full tables? Yes sir > > > > I tried adding "reply-to" rules in my pf.conf so that traffic that comes > in > > on one interface will go out the same interface but that doesn't seem to > > work either, since the reply from the wrong address happens before or > during > > the state that stateful connections are being established. > > PF isn't involved in this address selection, it's a message from the > router's IP stack because the TTL was exceeded, the lookup is entirely > done in the stack, reply-to isn't used. > > Ok I think I understand that... So what should be my next move?
OpenBGP default route selection process
Hi there, I am running OpenBSD 4.4 with OpenBGPD and multiple peers. For some reason the device is selecting Level3 as the default route for absolutely everything which is not statically set. On Level3 config i have set: set localpref 100 softreconfig in yes max-prefix 100 restart 300 For the others I have not set max-prefix. Also set nexthop qualify via bgp rde route-age evaluate and then stopped the session for Level 3 and started it again so it would seem "less stable" to the decision engine since it is a newer session, it is still the default for every single route. I even did a route flush and flushed them all, and did a refresh from another peer, at which point all routes came back, defaulting to Level3! How do i stop this from being my default route? The reason why is because it is not the best route most of the time. E.g. to some parts of the US it takes 16 hops through Level3, whereas Global Crossing will do it in 1 hop, Abovenet in 3, etc. Thanks! Ken
Re: OpenBGP default route selection process
On Sat, May 23, 2009 at 6:35 PM, Justin Credible wrote: > > Hi there, > > I am running OpenBSD 4.4 with OpenBGPD and multiple peers. > > For some reason the device is selecting Level3 as the default route for absolutely everything which is not statically set. > > On Level3 config i have set: > > set localpref 100 > softreconfig in yes > max-prefix 100 restart 300 > > For the others I have not set max-prefix. > > Also set > > nexthop qualify via bgp > rde route-age evaluate > > and then stopped the session for Level 3 and started it again so it would seem "less stable" to the decision engine since it is a newer session, it is still the default for every single route. I even did a route flush and flushed them all, and did a refresh from another peer, at which point all routes came back, defaulting to Level3! > > How do i stop this from being my default route? > > The reason why is because it is not the best route most of the time. E.g. to some parts of the US it takes 16 hops through Level3, whereas Global Crossing will do it in 1 hop, Abovenet in 3, etc. > > Thanks! > > Ken If you need more examples here you go: # bgpctl show rib 199.185.137.3 flags: * = Valid, > = Selected, I = via IBGP, A = Announced origin: i = IGP, e = EGP, ? = Incomplete flags destination gateway lpref med aspath origin *>199.185.136.0/2364.x.x.x 200 1 3549 812 812 812 812 3602 22512 i * 199.185.136.0/23212.x.x.x 100 500 3356 6453 812 3602 22512 i # route -n show | grep 199.185.136.0/23 # route -n show | grep 199.185.136 199.185.136/23 212.x.x.x UG100 -48 vlan400 # route delete 199.185.136/23 delete net 199.185.136/23 # ping 199.185.137.3 PING 199.185.137.3 (199.185.137.3): 56 data bytes 64 bytes from 199.185.137.3: icmp_seq=0 ttl=245 time=150.000 ms 64 bytes from 199.185.137.3: icmp_seq=1 ttl=245 time=155.865 ms --- 199.185.137.3 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 150.000/152.932/155.865/2.958 ms # route -n show | grep 199.185.136 199.185.136/23 212.x.x.x UG100 -48 vlan400 # bgpctl show rib 199.185.137.3 flags: * = Valid, > = Selected, I = via IBGP, A = Announced origin: i = IGP, e = EGP, ? = Incomplete flags destination gateway lpref med aspath origin *>199.185.136.0/2364.x.x.x 200 1 3549 812 812 812 812 3602 22512 i * 199.185.136.0/23212.x.x.x 100 500 3356 6453 812 3602 22512 i I've even set my config to be EXTREMELY biased against Level3 but it (the 212 address) still populates my routing tables: BGP routing table entry for 199.185.136.0/23 3549 812 812 812 812 3602 22512 Nexthop 64.x.x.x (via 212.x.x.x) from gblx-p1 (208.48.250.230) Origin IGP, metric 1, localpref 200, external, valid, best Last update: 00:26:45 ago Communities: 3549:4356 3549:8013 3549:8023 3549:8043 3549:8073 3549:8090 3549:8163 3549:8173 3549:8223 3549:8233 3549:30840 BGP routing table entry for 199.185.136.0/23 3356 6453 812 3602 22512 Nexthop 212.x.x.x (via 212.x.x.x) from level3-p2 (4.69.187.4) Origin IGP, metric 500, localpref 100, external, valid Last update: 00:26:45 ago # traceroute -n 199.185.137.3 traceroute to 199.185.137.3 (199.185.137.3), 64 hops max, 40 byte packets 1 212.x.x.x 0.550 ms 0.555 ms 0.448 ms 2 4.69.136.93 0.529 ms 0.445 ms 0.575 ms 3 4.69.136.90 11.273 ms 17.935 ms 11.317 ms 4 4.69.139.73 11.396 ms 11.439 ms 11.317 ms 5 4.68.63.106 16.769 ms 17.935 ms 17.939 ms 6 195.219.195.37 11.772 ms 195.219.83.2 11.687 ms 195.219.195.89 11.562 ms 7 195.219.243.14 12.17 ms 195.219.195.22 164.349 ms 164.471 ms 8 195.219.144.10 83.354 ms 195.219.144.1 12.184 ms 12.62 ms 9 195.219.144.10 83.355 ms 83.270 ms 216.6.98.1 109.634 ms 10 216.6.98.1 109.835 ms 109.880 ms 216.6.98.30 163.602 ms 11 216.6.98.30 163.552 ms 163.741 ms 64.86.115.38 178.523 ms 12 64.86.115.38 178.788 ms 179.88 ms 24.153.7.137 203.204 ms 13 24.153.7.137 180.416 ms 210.443 ms 238.549 ms 14 24.153.4.77 177.923 ms 178.712 ms 24.153.3.38 173.844 ms 15 24.153.3.38 173.921 ms 174.215 ms 173.595 ms 16 204.50.251.202 196.411 ms 207.107.204.178 177.465 ms 176.209 ms 17 207.107.204.178 177.542 ms 177.960 ms 176.719 ms 18 199.185.230.2 177.924 ms 199.185.137.3 177.712 ms 199.185.230.2 176.215 ms # route add 199.185.137.3 64.x.x.x add host 199.185.137.3: gateway 64.x.x.x # traceroute -n 199.185.137.3 traceroute to 199.185.137.3 (199.185.137.3), 64 hops max, 40 byte packets 1 64.x.x.x 10.505 ms 10.427 ms 10.316 ms 2 64.208.169.150 98.472 ms 98.635 ms 98.513 ms 3 69.63.248.98 97.96 ms 102.9 ms 97.141 ms 4 66.185.80.186 138.946 ms 107.131 ms 107.136 ms 5 24.153.4.74 149.191 ms 152.977 ms 159.35
Re: OpenBGP default route selection process
On Sat, May 23, 2009 at 8:55 PM, Justin Credible wrote: > On Sat, May 23, 2009 at 6:35 PM, Justin Credible > wrote: >> >> Hi there, >> >> I am running OpenBSD 4.4 with OpenBGPD and multiple peers. >> >> For some reason the device is selecting Level3 as the default route for absolutely everything which is not statically set. >> >> On Level3 config i have set: >> >> set localpref 100 >> softreconfig in yes >> max-prefix 100 restart 300 >> >> For the others I have not set max-prefix. >> >> Also set >> >> nexthop qualify via bgp >> rde route-age evaluate >> >> and then stopped the session for Level 3 and started it again so it would seem "less stable" to the decision engine since it is a newer session, it is still the default for every single route. I even did a route flush and flushed them all, and did a refresh from another peer, at which point all routes came back, defaulting to Level3! >> >> How do i stop this from being my default route? >> >> The reason why is because it is not the best route most of the time. E.g. to some parts of the US it takes 16 hops through Level3, whereas Global Crossing will do it in 1 hop, Abovenet in 3, etc. >> >> Thanks! >> >> Ken > > If you need more examples here you go: > > # bgpctl show rib 199.185.137.3 > flags: * = Valid, > = Selected, I = via IBGP, A = Announced > origin: i = IGP, e = EGP, ? = Incomplete > flags destination gateway lpref med aspath origin > *>199.185.136.0/2364.x.x.x 200 1 3549 812 812 812 812 > 3602 22512 i > * 199.185.136.0/23212.x.x.x 100 500 3356 6453 812 3602 22512 i > # route -n show | grep 199.185.136.0/23 > # route -n show | grep 199.185.136 > 199.185.136/23 212.x.x.x UG100 -48 vlan400 > # route delete 199.185.136/23 > delete net 199.185.136/23 > # ping 199.185.137.3 > PING 199.185.137.3 (199.185.137.3): 56 data bytes > 64 bytes from 199.185.137.3: icmp_seq=0 ttl=245 time=150.000 ms > 64 bytes from 199.185.137.3: icmp_seq=1 ttl=245 time=155.865 ms > --- 199.185.137.3 ping statistics --- > 2 packets transmitted, 2 packets received, 0.0% packet loss > round-trip min/avg/max/std-dev = 150.000/152.932/155.865/2.958 ms > # route -n show | grep 199.185.136 > 199.185.136/23 212.x.x.x UG100 -48 vlan400 > # bgpctl show rib 199.185.137.3 > flags: * = Valid, > = Selected, I = via IBGP, A = Announced > origin: i = IGP, e = EGP, ? = Incomplete > flags destination gateway lpref med aspath origin > *>199.185.136.0/2364.x.x.x 200 1 3549 812 812 812 812 > 3602 22512 i > * 199.185.136.0/23212.x.x.x 100 500 3356 6453 812 3602 22512 i > > > I've even set my config to be EXTREMELY biased against Level3 but it > (the 212 address) still populates my routing tables: > > > BGP routing table entry for 199.185.136.0/23 >3549 812 812 812 812 3602 22512 >Nexthop 64.x.x.x (via 212.x.x.x) from gblx-p1 (208.48.250.230) >Origin IGP, metric 1, localpref 200, external, valid, best >Last update: 00:26:45 ago >Communities: 3549:4356 3549:8013 3549:8023 3549:8043 3549:8073 > 3549:8090 3549:8163 3549:8173 3549:8223 3549:8233 3549:30840 > BGP routing table entry for 199.185.136.0/23 >3356 6453 812 3602 22512 >Nexthop 212.x.x.x (via 212.x.x.x) from level3-p2 (4.69.187.4) >Origin IGP, metric 500, localpref 100, external, valid >Last update: 00:26:45 ago > > > # traceroute -n 199.185.137.3 > traceroute to 199.185.137.3 (199.185.137.3), 64 hops max, 40 byte packets > 1 212.x.x.x 0.550 ms 0.555 ms 0.448 ms > 2 4.69.136.93 0.529 ms 0.445 ms 0.575 ms > 3 4.69.136.90 11.273 ms 17.935 ms 11.317 ms > 4 4.69.139.73 11.396 ms 11.439 ms 11.317 ms > 5 4.68.63.106 16.769 ms 17.935 ms 17.939 ms > 6 195.219.195.37 11.772 ms 195.219.83.2 11.687 ms 195.219.195.89 11.562 ms > 7 195.219.243.14 12.17 ms 195.219.195.22 164.349 ms 164.471 ms > 8 195.219.144.10 83.354 ms 195.219.144.1 12.184 ms 12.62 ms > 9 195.219.144.10 83.355 ms 83.270 ms 216.6.98.1 109.634 ms > 10 216.6.98.1 109.835 ms 109.880 ms 216.6.98.30 163.602 ms > 11 216.6.98.30 163.552 ms 163.741 ms 64.86.115.38 178.523 ms > 12 64.86.115.38 178.788 ms 179.88 ms 24.153.7.137 203.204 ms > 13 24.153.7.137 180.416 ms 210.443 ms 238.549 ms > 14 24.153.4.77 177.923 ms 178.712 ms 24.153.3.38 173.844 ms > 15 24.153.3.38 173.921 ms 174.215 ms 173.595 ms > 16 204.50.251.202 196.411 ms 207.107.204.178 177.465 ms 176.209 ms > 17 207.107.204.178 177.542 ms 177.960 ms 176.719
Re: OpenBGP default route selection process
On Sun, May 24, 2009 at 12:22 PM, Claudio Jeker wrote: > On Sun, May 24, 2009 at 11:54:37AM -0600, Justin Credible wrote: > > > >> I figured this one out. This particular problem was caused because i had set: >> >> nexthop qualify via bgp >> > > Why did you set this? Just for fun or was there a reason behind it. It was a last resort ro another problem which I was having (See thread titled "BGP responding with wrong IP address."). I thought that If i set this, that the correct interface would respond at the router since it seems as though random interfaces were responding. > >> I don't know why that setting in particular set all of my routes to >> point at Level 3 regardless of the preferential settings against it, >> but how i got around it is simple. >> >> route add -mpath default gw1 >> route add -mpath default gw2 >> etc... >> >> Then change that setting to >> >> nexthop qualify via default >> > > Why did you set this? Just for fun or was there a reason behind it. > If you need to use "nexthop qualify via default" or "nexthop qualify via > bgp" then you have an error in your network setup. Either you should use > an IGP (like ospfd) or have a static route to the bgp router. I toggled this setting to see if it would make a difference on the routing tables. My main reason for setting preferred routes is because BGP was selecting Level3 as default for our route to Latin America. We implemented GBLX since there are only two or three hops to Latin America through them, but BGP wasn't selecting them by default, therefore i had to manually intervene. This setting seems to have fixed that particular problem but the problem in thread "BGP responding with wrong IP address." still exists somewhat. In a way they are related, in more ways they are not, that's why I have them as separate threads. I don't use OSPFD yet since I am very new to BGP. Once I master BGP I will move on to learning about the other things which plug in to it, rather than screwing everything up all at once (especially in a live production environment). Thanks!
Workaround to recent Juniper Announcement?
Hi There, We have OpenBSD routers running OpenBGPD at the edge of our network and behind that we use Juniper Firewalls running JunOS which need to be patched due to: http://ptresearch.blogspot.com/2010/01/juniper-junos-remote-kernel-crash-flaw.html Since we have so many Junipers it will take a while to patch them and will be faster for us to add some mitigating rules on the OpenBSD routers, I am wondering if anyone knows of a quick and simple rule that we can Implement on OpenBSD to combat this issue? Thanks and regards, Justin
Re: Workaround to recent Juniper Announcement?
On Thu, Jan 7, 2010 at 12:23 PM, James Records wrote: > > It looks like their (Junipers) policy is to only tell customers with > support contracts what the specific option is that causes this. Only big customers apparently - Some people had several hours advance notice, some had several days advanced notice. We haven't had any notice yet at all (and we have scores of devices from Juniper). It has been a known issue since last year, however wasn't considered a security issue until last night apparently. I don't even want to use Juniper - We use them due to corporate policy, even though we have already proven if our OpenBSD machines die catastrophically we can replace them in about 30 minutes, as opposed to the top level Juniper support of getting someone on site within 4 hours just with the device before configuration begins. This is why we put OpenBSD in front of them. Justin
