RE: Possible Comcast Load Balancing issues in Chicago?
Seems to have cleared up about 30 minutes ago. Thanks, Mark Mayfield City of Roseville – AS 54371 Network Systems Engineer 2660 Civic Center Drive Roseville, MN 55113 651-792-7098 Office -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Smith, Courtney Sent: Thursday, May 15, 2014 10:32 AM To: Metro-Inet Netops Cc: nanog@nanog.org Subject: RE: Possible Comcast Load Balancing issues in Chicago? Looking into. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Metro-Inet Netops Sent: Thursday, May 15, 2014 11:10 AM To: nanog@nanog.org Subject: Possible Comcast Load Balancing issues in Chicago? It’s possible I’m missing something, but it looks like traffic from Comcast business customers in the Minneapolis/St. Paul area is not all making it through Chicago to my VPN endpoints. Testing to some of my address space from one of the Comcast remote VPN sites, I can get to some numbers and not others (odds/evens switch roles depending on the block I trace) fall apart hopping though Chicago. Testing to the same addresses from the HE looking glass, everything seems fine. Any input or even knowledge that it’s being worked on would be appreciated. Traceroutes below: Fails: HUGO-CH-2921#trace 199.249.109.13 Type escape sequence to abort. Tracing the route to vpn.metro-inet.us (199.249.109.13) VRF info: (vrf in name/id, vrf out name/id) 1 75-149-158-38-minnesota.hfc.comcastbusiness.net (75.149.158.38) 0 msec 0 msec 0 msec 2 73.237.168.1 12 msec 12 msec 16 msec 3 te-4-2-ur02.whitebear.mn.minn.comcast.net (68.85.167.37) 16 msec 16 msec 16 msec 4 te-2-1-ur02.oakdale.mn.minn.comcast.net (68.87.174.138) 16 msec 16 msec 16 msec 5 te-8-3-ur01.oakdale.mn.minn.comcast.net (68.87.174.109) 20 msec 12 msec 16 msec 6 te-0-3-0-5-ar01.crosstown.mn.minn.comcast.net (69.139.176.61) 16 msec 16 msec 20 msec 7 pos-0-0-0-0-ar01.roseville.mn.minn.comcast.net (68.87.174.194) 16 msec 16 msec 12 msec 8 he-1-12-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.94.77) 24 msec 24 msec 28 msec 9 * * * 10 * Gets Through: HUGO-CH-2921#trace 199.249.109.14 Type escape sequence to abort. Tracing the route to 199.249.109.14 VRF info: (vrf in name/id, vrf out name/id) 1 75-149-158-38-minnesota.hfc.comcastbusiness.net (75.149.158.38) 4 msec 0 msec 0 msec 2 73.237.168.1 8 msec 12 msec 8 msec 3 te-4-2-ur01.whitebear.mn.minn.comcast.net (68.85.167.33) 12 msec 12 msec 12 msec 4 te-2-1-ur02.shoreview.mn.minn.comcast.net (68.87.174.129) 8 msec 12 msec 12 msec 5 te-8-3-ur01.shoreview.mn.minn.comcast.net (68.87.174.125) 12 msec 8 msec 8 msec 6 te-0-3-0-6-ar01.roseville.mn.minn.comcast.net (68.87.174.178) 12 msec 12 msec 12 msec 7 he-1-11-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.94.73) 28 msec 20 msec 24 msec 8 ix-7-0-3-0.tcore1.CT8-Chicago.as6453.net (64.86.137.33) 16 msec 16 msec 20 msec 9 206.82.141.90 20 msec 20 msec 20 msec 10 hurricane-ic-124397-chi-bb1.c.telia.net (213.248.104.214) 24 msec 24 msec 28 msec 11 100ge13-1.core1.msp1.he.net (184.105.223.178) 36 msec 24 msec 28 msec 12 city-of-roseville.gigabitethernet3-8.core1.msp1.he.net (184.105.249.218) 28 msec 28 msec 24 msec 13 199.249.109.227 28 msec 24 msec 28 msec Gets Through: HUGO-CH-2921#trace 67.88.181.25 Type escape sequence to abort. Tracing the route to ip67-88-181-25.z181-88-67.customer.algx.net(67.88.181.25) VRF info: (vrf in name/id, vrf out name/id) 1 pubwp.comcast.net (75.149.158.38) 0 msec 0 msec 0 msec 2 73.237.168.1 8 msec 8 msec 8 msec 3 te-4-2-ur01.whitebear.mn.minn.comcast.net (68.85.167.33) 12 msec 12 msec 8 msec 4 te-2-1-ur02.shoreview.mn.minn.comcast.net (68.87.174.129) 8 msec 8 msec 12 msec 5 te-8-3-ur01.shoreview.mn.minn.comcast.net (68.87.174.125) 12 msec 12 msec 8 msec 6 te-0-3-0-6-ar01.roseville.mn.minn.comcast.net (68.87.174.178) 8 msec 12 msec 12 msec 7 he-1-12-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.94.77) 24 msec 20 msec 16 msec 8 ix-2-3-0-0.tcore1.CT8-Chicago.as6453.net (64.86.137.41) 24 msec 16 msec 40 msec 9 p5-1.ir1.chicago2-il.us.xo.net (206.111.2.33) 68 msec 20 msec 80 msec 10 207.88.14.193.ptr.us.xo.net (207.88.14.193) 32 msec 32 msec 36 msec 11 ae0d0.mcr1.minneapolis-mn.us.xo.net (216.156.0.170) 28 msec 28 msec 28 msec Fails: HUGO-CH-2921#trace 67.88.181.26 Type escape sequence to abort. Tracing the route to webvpn.metro-inet.us (67.88.181.26) VRF info: (vrf in name/id, vrf out name/id) 1 pubwp.comcast.net (75.149.158.38) 0 msec 0 msec 0 msec 2 73.237.168.1 8 msec 8 msec 8 msec 3 te-4-2-ur02.whitebear.mn.minn.comcast.net (68.85.167.37) 8 msec 12 msec 8 msec 4 te-2-1-ur02.oakdale.mn.minn.comcast.net (68.87.174.138) 8 msec 12 msec 20 msec 5 te-8-3-ur01.oakdale.mn.minn.comcast.net (68.87.174.109) 8 msec 8 msec 12 msec 6 te-0-3-0-5-ar01.crosstown.mn.minn.comcast.net (69.139.176.61) 12 msec 12 msec 12 msec 7 pos
Possible Sudden Uptick in ASA DOS?
Come in this morning to find one failover pair of ASA's had the primary crash and failover, then a couple hours later, the secondary crash and failover, back to the primary. Another pair running the same code had the primary crash and fail in the same time window. So, three crashes in 4 hours in our environment. Open a TAC case on one of these for post-mortem analysis, and they interpreted the crash dump to point at a DOS bug first published in Oct. The very interesting thing; on the phone the TAC engineer said this was "the 10th one of these I've dealt with this morning". Here's the bug they reference: https://tools.cisco.com/bugsearch/bug/CSCul36176/?reffering_site=dumpcr Anyone else have observations to add on this? Mark Mayfield City of Roseville - AS 54371 Network Systems Engineer 2660 Civic Center Drive Roseville, MN 55113 651-792-7098 Office
RE: Possible Sudden Uptick in ASA DOS?
Thank you sir. I read your presentation quite some time ago, probably one of the first times you posted to the list. It has definitely informed many of my design processes; particularly with regard to server publishing, and been a major part of my supporting documentation in arguments with others at my organization over the last few years. Of course, these particular ASA implementations are for law enforcement applications, so we are mandated to implement in ways that auditors from the state and federal agencies approve of. However, this makes me consider the need to more aggressively ACL inbound traffic at the router level before these particular firewalls, which I can do, and may help mitigate such events, so thank you for the reminder! Mark Mayfield City of Roseville - AS 54371 Network Systems Engineer 2660 Civic Center Drive Roseville, MN 55113 651-792-7098 Office -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Roland Dobbins Sent: Wednesday, July 08, 2015 12:18 To: nanog@nanog.org Subject: Re: Possible Sudden Uptick in ASA DOS? On 8 Jul 2015, at 23:58, Mark Mayfield wrote: > Come in this morning to find one failover pair of ASA's had the > primary crash and failover, then a couple hours later, the secondary > crash and failover, back to the primary. See this preso: <https://app.box.com/s/a3oqqlgwe15j8svojvzl> --- Roland Dobbins
RE: VPN over Comcast
In June of last year, when Comcast did firmware updates on the business gateways in the MSP area, we lost all (3) of our sites with Netgear gateways, but not the sites SMC gateways (the management interface is almost identical, but the brand is marked on the modem). Business support was apparently aware of a Cisco VPN problem through the Netgear, and simply replaced the Netgear units with SMC, and we haven't had issues since. This is IOS to ASA site-to-site VPN. Mark Mayfield City of Roseville Network Systems Engineer 2660 Civic Center Drive Roseville, MN 55113 -Original Message- From: Michael Malitsky [mailto:malit...@netabn.com] Sent: Tuesday, April 27, 2010 12:43 PM To: nanog@nanog.org Subject: VPN over Comcast I will probably be laughed at, but I'll ask just in case. We are having particularly bad luck trying to run VPN tunnels over Comcast cable in the Chicago area. The symptoms are basically complete loss of connectivity (lasting minutes to sometimes hours), or sometimes flapping for a period of time. More often than not, a reboot of the cable modem is required. The most interesting ones involve the following: a PIX or ASA configured as an EZvpn client, connecting to a 3000 concentrator, authentication over RADIUS. When I go to look at the RADIUS logs, I see connections from the same box with small intervals. Timeout is 8 hours, so theoretically I should see 3 connections in a 24-hr period. In some cases, I see dozens, in the most egregious cases, thousands over a 24-hour period. I am taking that as an indicator of a really unstable Comcast circuit. We have not had this problem with any other ISP, anywhere in the country. I am pretty much down to telling customers to find another provider... Any thoughts or ideas on the matter will be appreciated. PS. To be fair (?) to Comcast, this is not a ubiquitous problem. It affects about 25% of the installations I get to see. Sincerely, Michael Malitsky Confidentiality Statement: The documents accompanying this transmission contain confidential information that is legally privileged. This information is intended only for the use of the individuals or entities listed above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or action taken in reliance on the contents of these documents is strictly prohibited. If you have received this information in error, please notify the sender immediately and arrange for the return or destruction of these documents.
RE: POE switches and lightning
About a month ago, we had a lightning strike near our main campus. We lost one POE Cisco 3560 completely (apparently blown power supply), and in a separate but nearby building, another 3560 lost the ability to deliver POE, but continued to operate as a switch. Both had to be replaced. Both were on wiring closet type UPS'es with surge suppression, and those were unaffected. Mark -Original Message- From: Caleb Tennis [mailto:caleb.ten...@gmail.com] Sent: Thursday, May 13, 2010 10:37 AM To: North American Network Operators Group Subject: POE switches and lightning We had a lightning strike nearby yesterday that looks to have come inside our facility via a feeder circuit that goes outdoors underground to our facility's gate. What's interesting is that various POE switches throughout the entire building seemed to be affected in that some of their ports they just shut down/off. Rebooting these switches brought everything back to life. It didn't impact anything non-POE, and even then, only impacted some devices. But it was spread across the whole building, across multiple switches. I was just curious if anyone had seen anything similar to this before? Our incoming electrical power has surge suppression, and the power to the switches is all through double conversion UPS, so I'm not quite sure why any of them would have been impacted at all. I'm guessing that the strike had some impact on the electrical ground, but I don't know what we can do to prevent future strikes from causing the same issues. Thoughts? Confidentiality Statement: The documents accompanying this transmission contain confidential information that is legally privileged. This information is intended only for the use of the individuals or entities listed above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or action taken in reliance on the contents of these documents is strictly prohibited. If you have received this information in error, please notify the sender immediately and arrange for the return or destruction of these documents.
