IP Geolocation
Anyone else have issues where their IP block gets randomly set to China? We have been trying to track down this issue for months and our customers are starting to get upset. We get a /29 from our upstream provider that we CGNAT (yeah I know, working on implementing IPV6) to all of our customers at 1 particular site. No other sites have any issues. We had our upstream provider allocate us a new IP block from a different subnet which fixed the issue for a while but now it's back. The state and town are correct but the country states China. This is having issues with Speedtests, NetFlix and others. The upstream is claiming that we are purposely using a proxy or VPN to china which causes this. We have checked all our configurations and even replaced all hardware in case something was hacked. Any ideas? Thanks Travis Garrison
Re: Comcast outages continue even in areas with PG&E power restored
On Fri, 11 Oct 2019, Michael Thomas wrote: On 10/11/19 4:31 PM, Sean Donelan wrote: The FCC asked a half-dozen carriers about their network resilience plans last month. Comcast was not one of the service providers askedd about their plans. The FCC should have looked closer at Comcast in California. While it was expected many people would loose home Internet, voice, video service when their Customer Premise Equipment lost power. The FCC no longer requires battery backup for CPE. That is now a customer responsibility. It turns out, Comcast's outside plant was woefully unprepared to handle long, i.e. 24 hour, power outages. And even when power is restored to people's homes, Comcast service is often still down. So I knew that telcos are required to battery backup pots, but are isp's too? I have a dinky little provider who also provides pots, but i have never been clear whether dsl stays up too in a blackout. Mike First of all DSL is not pots. Traditional voice services run on a subscriber loop which is a pair of copper lines running from the central office to the customer end point. This analog voice service is almost always backed up with a bank of batteries so that the service continues to run in the event of an emergency. DSL is a data service that runs on the subscriber loop at the same time as the voice service. This service is not required to be battery backed and will invariably stop working when power is cut at the customer end point. Ted
Re: IP Geolocation
> On Oct 14, 2019, at 9:14 AM, Travis Garrison wrote: > > Anyone else have issues where their IP block gets randomly set to China? We > have been trying to track down this issue for months and our customers are > starting to get upset. We get a /29 from our upstream provider that we CGNAT > (yeah I know, working on implementing IPV6) to all of our customers at 1 > particular site. No other sites have any issues. We had our upstream provider > allocate us a new IP block from a different subnet which fixed the issue for > a while but now it's back. The state and town are correct but the country > states China. This is having issues with Speedtests, NetFlix and others. The > upstream is claiming that we are purposely using a proxy or VPN to china > which causes this. We have checked all our configurations and even replaced > all hardware in case something was hacked. Any ideas? I’ve seen some people do their geolocation on a /24 boundary, so if someone else in that same /24 is located there, it might be an issue. I know in a prior life I had that issue with some CDNs and we eventually worked with them to resolve the issue. - Jared
Re: IP Geolocation
http://thebrotherswisp.com/index.php/geo-and-vpn/ Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Mon, Oct 14, 2019 at 11:38 AM Jared Mauch wrote: > > > > On Oct 14, 2019, at 9:14 AM, Travis Garrison > wrote: > > > > Anyone else have issues where their IP block gets randomly set to China? > We have been trying to track down this issue for months and our customers > are starting to get upset. We get a /29 from our upstream provider that we > CGNAT (yeah I know, working on implementing IPV6) to all of our customers > at 1 particular site. No other sites have any issues. We had our upstream > provider allocate us a new IP block from a different subnet which fixed the > issue for a while but now it's back. The state and town are correct but the > country states China. This is having issues with Speedtests, NetFlix and > others. The upstream is claiming that we are purposely using a proxy or VPN > to china which causes this. We have checked all our configurations and even > replaced all hardware in case something was hacked. Any ideas? > > I’ve seen some people do their geolocation on a /24 boundary, so if > someone else in that same /24 is located there, it might be an issue. I > know in a prior life I had that issue with some CDNs and we eventually > worked with them to resolve the issue. > > - Jared
Re: IP Geolocation
Agreed, I’ve seen this before across wider boundaries. Even /22s. -Ben > On Oct 14, 2019, at 8:38 AM, Jared Mauch wrote: > > > >> On Oct 14, 2019, at 9:14 AM, Travis Garrison wrote: >> >> Anyone else have issues where their IP block gets randomly set to China? We >> have been trying to track down this issue for months and our customers are >> starting to get upset. We get a /29 from our upstream provider that we CGNAT >> (yeah I know, working on implementing IPV6) to all of our customers at 1 >> particular site. No other sites have any issues. We had our upstream >> provider allocate us a new IP block from a different subnet which fixed the >> issue for a while but now it's back. The state and town are correct but the >> country states China. This is having issues with Speedtests, NetFlix and >> others. The upstream is claiming that we are purposely using a proxy or VPN >> to china which causes this. We have checked all our configurations and even >> replaced all hardware in case something was hacked. Any ideas? > > I’ve seen some people do their geolocation on a /24 boundary, so if someone > else in that same /24 is located there, it might be an issue. I know in a > prior life I had that issue with some CDNs and we eventually worked with them > to resolve the issue. > > - Jared
New telemetry system ideas
I am a researcher working on developing a new switch-based on-the-fly telemetry system that takes a flow chart as input to describe a particular detection task (rather than just features or information elements as in IPFIX). For an example of what I mean by "flow chart" see the figure here: https://ieeexplore.ieee.org/mediastore_new/IEEE/content/media/8048782/8048856/8048939/8048939-fig-4-source-hires.gif. Might anyone have pointers to a source of more such flow charts? The other issue I'm worried about is that it might take a couple rounds before an event is detected (since the system has to step through the flow chart and possibly look at different traffic features in the process). What is a typical duration of the types of events people might want to catch with a telemetry system like this? Do these kind of events generate the same type of traffic throughout their durations, or do traffic features change as the event progresses? Thanks! Chris
Re: IP Geolocation
Is this an indication of a prefix that was highjacked? Sent from my iPhone > On Oct 14, 2019, at 9:19 AM, Ben Cannon wrote: >
.COM Zone DNSSEC Operational Update -- ZSK length change
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 All, Verisign is in the process of increasing the size and strength of the DNSSEC Zone Signing Keys (ZSKs) for the top-level domains that it operates. As part of this process, the ZSK for the .COM zone will be increased in size from 1024 to 1280 bits. On October 10, 2019 the 1280 bit ZSK was pre-published in the .COM zone. On October 15, we plan to sign the .COM zone with the 1280 bit ZSK. On October 20, we plan to remove the old 1024 bit ZSK from the zone. We do not anticipate any problems from this upgrade. In accordance with our normal operating procedures we have a rollback process should it become necessary to revert to the 1024 bit ZSK. DW -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBCAAGBQJdpL5zAAoJEGyZpGmowJiNKZsH/3D1uNJ959pOoZjiRWHd/bZ5 V6mNXkKA34jKudC0ugPt1RQGEExspsNHBTjSK//SO+nayKl8GQUdrnL5AINJKQjU C61nhFORQXxhrE7JTbbbLeiVWVMwBhWmefi9zNIi/2MANWhX0UzuKREfsAnUgyVL 9mbfxxFCmAC6HPVkE3wZS5Z8h+3AP4zkctLOWQv/wi0aCd6o6/k9fejpmN2YhFQy 7OrFqKpTq0wa7K1zx8xXyBNPstWlH0709zn3Cm6aYkQ7mLoCNl2H6ODJCmvtTZeo BYVKGojABUWXAurOiPSaljGHRAEC1sN5cV5rkeWmXjoRo+dGPOA60bzbdNsTfss= =J3+s -END PGP SIGNATURE- smime.p7s Description: S/MIME cryptographic signature
APRICOT 2020 Call for PC Volunteers
Hi everyone, The APRICOT 2020 Programme Committee is responsible for the solicitation and selection of suitable presentation and tutorial content for the APRICOT 2020 conference (https://2020.apricot.net/). The APRICOT PC Chairs are now seeking nominations from the community to join the APRICOT 2020 PC to assist with the development of the programme for APRICOT 2020. Eligible PC candidates are those who have attended APRICOT conferences in the recent past, have broad technical knowledge of Internet operations, and have reasonable familiarity with the format of APRICOT conferences. Having constructive opinions and ideas about how the programme content might be improved is of high value too. PC members are expected to work actively to solicit content and review submissions for technical merit. The PC meets by conference call, weekly in frequency during the three months prior to APRICOT. If you are interested in joining the PC and meet the above eligibility criteria, please send a brief note to "pc-chairs at apricot.net". The note should include affiliation (if any) and contact details (including e-mail address), and a brief description of why you would make a good addition to the PC. PC members who are active, successfully solicit content, contribute to regular PC meetings and submit reviews receive complimentary registration for APRICOT 2020. The PC Chairs will accept nominations received by 17:00 UTC+8 on Monday 21st October, 2019, and will announce the new PC shortly thereafter. Many thanks! Mark Tinka, Marijana Novakovic & Philip Smith APRICOT 2020 PC Chairs
Re: Comcast outages continue even in areas with PG&E power restored
On Fri, 11 Oct 2019, Ted Hatfield wrote: First of all DSL is not pots. [] DSL is a data service that runs on the subscriber loop at the same time as the voice service. This service is not required to be battery backed and will invariably stop working when power is cut at the customer end point. That is not why people are surprised. When the house doesn't have power, and doesn't have home generator or UPS, (most) people are less surprised their DSL or Cable modem and VOIP doesn't work anymore. The reasons I saw people angry on twitter was no Comcast service even when they had power at the house (utility, generator, UPS). Their Comcast service died quickly, even when the home had power but the Comcast outside plant didn't seem to have any backup power. DSL modems also need power at the home, but the telco providers seem to have more backup power in the outside plant or central offices. That meant DSL worked as long as the house had power (or a home generator or UPS). I know, rich people problems. Rich people can afford backup generators and got upset when their Internet and TV didn't work.
Re: Comcast outages continue even in areas with PG&E power restored
On 10/14/19 3:06 PM, Sean Donelan wrote: That is not why people are surprised. When the house doesn't have power, and doesn't have home generator or UPS, (most) people are less surprised their DSL or Cable modem and VOIP doesn't work anymore. The reasons I saw people angry on twitter was no Comcast service even when they had power at the house (utility, generator, UPS). Their Comcast service died quickly, even when the home had power but the Comcast outside plant didn't seem to have any backup power. DSL modems also need power at the home, but the telco providers seem to have more backup power in the outside plant or central offices. That meant DSL worked as long as the house had power (or a home generator or UPS). So it turns out that our local telco/isp does keep dsl running via the same mechanism as they keep pots power backed up (i'm guessing it's a diesel generator at the co, but am not sure). It seems that a lot of the pedestals terminating the local loop these days do the conversion to IP right there with sip/h.248/mgcp/rtp. I'm not sure how they get power to the pedestal, but these were all a home run to the co at one time so it probably wasn't hard to power them from the co. For all i know, that's how they're all powered all the time, with a transfer switch at the co, rather than tapping the local grid next to the pedestal. Of course this is a lot of conjecture on my part... be glad to be clued in by folks in know. Mike
Re: Comcast outages continue even in areas with PG&E power restored
On Mon, 14 Oct 2019, Michael Thomas wrote: Of course this is a lot of conjecture on my part... be glad to be clued in by folks in know. An old news story, but telco's usually have backup batteries in their outside plant, cell towers, etc. During power outages, they shuttle small generators between outside cabinets to re-charge the batteries. Remote Terminal Units (RTUs) use local power, i.e. look for the utility meter nearby. There is often a generator plug and battery cabinet next to the RTU. They aren't powered from the central office. Some cable systems have battery and/or generator backup on their "I-Net" cable plant serving government and major businesses, but not on their residential cable plant. I don't know Comcast's business practices. Old news story: https://www.multichannel.com/news/att-will-replace-batteries-after-fires-130936 ORIGINAL: JAN 18, 2008 AT&T Will Replace Batteries After Fires City officials, long critical of the size and placement of powering cabinets needed to back up AT&T’s U-verse TV video service, now have concerns beyond aesthetics. Sometimes, the cabinets explode. AT&T acknowledged the problem and said it would replace 17,000 lithium batteries in outdoor cabinets around the country. [...] The steel cabinets house controls and backup power supplies for the video network. “They’ve been pretty cooperative,” Kesner said of AT&T. “We’re in a holding pattern” regarding the video deployment, he said. [...]
Re: Comcast outages continue even in areas with PG&E power restored
On 10/14/19 4:16 PM, Sean Donelan wrote: On Mon, 14 Oct 2019, Michael Thomas wrote: Of course this is a lot of conjecture on my part... be glad to be clued in by folks in know. An old news story, but telco's usually have backup batteries in their outside plant, cell towers, etc. During power outages, they shuttle small generators between outside cabinets to re-charge the batteries. Remote Terminal Units (RTUs) use local power, i.e. look for the utility meter nearby. There is often a generator plug and battery cabinet next to the RTU. They aren't powered from the central office. Interesting! And so primitive! So they go to all of the expense of laying fiber, but not power too? Mike
Re: Comcast outages continue even in areas with PG&E power restored
On 10/14/19 6:38 PM, Michael Thomas wrote: So it turns out that our local telco/isp does keep dsl running via the same mechanism as they keep pots power backed up (i'm guessing it's a diesel generator at the co, but am not sure). It seems that a lot of the pedestals terminating the local loop these days do the conversion to IP right there with sip/h.248/mgcp/rtp. I'm not sure how they get power to the pedestal, but these were all a home run to the co at one time so it probably wasn't hard to power them from the co. For all i know, that's how they're all powered all the time, with a transfer switch at the co, rather than tapping the local grid next to the pedestal. Of course this is a lot of conjecture on my part... be glad to be clued in by folks in know. Legacy carriers with existing copper plant being overbuilt by fiber often use "span power" at 48V or ~190VDC from the nearest central facility (which may be an old-school CO or RTU with utility power) to power their distributed plant that's deep into the network near the customer edge. There's usually then also some local battery as well but not much. The assumption is that the legacy copper, now being used simply for power, follows essentially the same routes as the fiber and will drop at the same time. In a cable cut situation this is often true, but in a "cable rot" situation it's obviously not, and fiber paths aren't always the same as power paths. The CO and older RTUs from the POTS and early ADSL era will have utility power as primary with somewhat extensive battery facilities. Old-school CO will have a lot more battery capacity than an RTU. An RTU will usually end up with a portable genset being delivered during an extended power outage as mentioned. These RTUs still host legacy POTS and TDM services that either have serious SLAs on them or regulatory uptime considerations whereas distributed peds are normally best-effort, non-regulated data services only in many cases precisely to keep the reliability requirements (and therefore cost) on them down. SBC/AT&T's pedestals they built for Lightspeed (U-Verse) do usually have local utility with a few hours of backup. I'm not sure why the went that route rather than span power. Might be that the early VDSL DSLAMs just used too much power for that. They seem to only have maybe a day or so of battery before they need a portable generator brought around which AT&T at least has procedures for (whether they are executed or not is another matter). They're something of a hybrid between a conventional RTU and modern distributed pedestal. All the conventional telcos are far more focused on keeping voice service alive since they get raked over the coals by the FCC if it drops due to lack of 911. That includes wireless if they are both a wireline and wireless operator. Interestingly, VoIP service delivered to the customer as such (even if there's an ATA built into the customer's "modem") often gets a pass on this since it's not considered POTS. Same goes for SLA'd business services, too especially T1s since those may host regulated voice. Coax operators have historically had less need for reliability as they were originally built purely for convenience services (cable television) and have been pressed into service for more modern data needs. Those "Alpha boxes" you may see around providing line power do usually have some batteries in them, though they're often ill-maintained and only provide maybe an hour of hold-up at most. Exclusively residential areas sometimes have zero hold-up ability at all. They'll drop at least outside node based digital services (e.g. DOCSIS) as soon as power falls along with anything being distributed into the field on AM fiber carriers. It's not unheard of for conventional linear TV to still be delivered into the field from a head-end at baseband on coax and sometimes that'll stay up longer depending on RF power/split budget as long as the local RF head end still has power. Pure-play fiber carriers, especially PON-based, get to turn what is often a curse in terms of design into a blessing, here. They usually have almost no active outside plant or, if they do, it's less distributed and can afford reasonable backup power infrastructure. While it's annoying to have no power available as you approach customer prem, that forces you to make choices such that there's no worry about backup until you can (usually) just make it the customer's responsibility right at the demarc. Your CO/head-end needs backup, of course, but that's usually a facility that can afford it. -- Brandon Martin
Re: Comcast outages continue even in areas with PG&E power restored
All true telco equipment is powered by batteries. Commerical power or generators just recharge the batteries. No switch over when commerical power is lost. Except when the generators(where equiped) switch over to recharge the batteries. Comcast and telcos do not put batteries in all remote powered terminals. I have an Enterprise grade Ethernet over coax connection. The headend it's distributed from doesn't have batteries. If it loses power, doesn't matter if I have power or a generator or ups to take over. This Internet connection goes down. For telcos(when I worked there), they usually had batteries that would last 4 to 8 hrs at remote terminals with powered equipment. And a connection for a splice crew to come out and connect their generator to it for power in case of an extended outage. Back then that was also how most cell phone towers were outfited. I also have some knowledge of the commerical power grid in my local area. It's not unheard of for the Comcast headend to lose power but my office doesn't. Lyle Giese LCR Computer Services, Inc. On 10/14/19 17:38, Michael Thomas wrote: On 10/14/19 3:06 PM, Sean Donelan wrote: That is not why people are surprised. When the house doesn't have power, and doesn't have home generator or UPS, (most) people are less surprised their DSL or Cable modem and VOIP doesn't work anymore. The reasons I saw people angry on twitter was no Comcast service even when they had power at the house (utility, generator, UPS). Their Comcast service died quickly, even when the home had power but the Comcast outside plant didn't seem to have any backup power. DSL modems also need power at the home, but the telco providers seem to have more backup power in the outside plant or central offices. That meant DSL worked as long as the house had power (or a home generator or UPS). So it turns out that our local telco/isp does keep dsl running via the same mechanism as they keep pots power backed up (i'm guessing it's a diesel generator at the co, but am not sure). It seems that a lot of the pedestals terminating the local loop these days do the conversion to IP right there with sip/h.248/mgcp/rtp. I'm not sure how they get power to the pedestal, but these were all a home run to the co at one time so it probably wasn't hard to power them from the co. For all i know, that's how they're all powered all the time, with a transfer switch at the co, rather than tapping the local grid next to the pedestal. Of course this is a lot of conjecture on my part... be glad to be clued in by folks in know. Mike
Re: Comcast outages continue even in areas with PG&E power restored
On 10/14/19 4:39 PM, Brandon Martin wrote: All the conventional telcos are far more focused on keeping voice service alive since they get raked over the coals by the FCC if it drops due to lack of 911. That includes wireless if they are both a wireline and wireless operator. Interestingly, VoIP service delivered to the customer as such (even if there's an ATA built into the customer's "modem") often gets a pass on this since it's not considered POTS. Same goes for SLA'd business services, too especially T1s since those may host regulated voice. So when we were working on this 20 years ago at Cisco, there was a tremendous amount of effort to deal with the issue of e911 and generally battery backup. I'm really surprised to hear that though we went through a lot of effort to deal with the CPE, that the cable plant was the actual problem. The cable companies should, imo, be held to the same standard as the telcos. Maybe even moreso these days since IP has taken over everything. The need for reliable e911 hasn't gone away just because the bits have turned into IP bit these days. Mike
Re: Comcast outages continue even in areas with PG&E power restored
On 10/14/19 8:26 PM, Michael Thomas wrote: So when we were working on this 20 years ago at Cisco, there was a tremendous amount of effort to deal with the issue of e911 and generally battery backup. I'm really surprised to hear that though we went through a lot of effort to deal with the CPE, that the cable plant was the actual problem. The cable companies should, imo, be held to the same standard as the telcos. Maybe even moreso these days since IP has taken over everything. The need for reliable e911 hasn't gone away just because the bits have turned into IP bit these days. They get around it, at least in part, by selling it as a "VoIP" service rather than "phone service". AT&T does the same with U-Verse voice. You can still buy POTS from AT&T, but it's a separate product with a completely different pricing structure from the U-Verse voice product. Voice over HFC networks is sometimes sold as a POTS-like service. I've only heard of this happening in places where the LEC and cable provider happen to end up being one-in-the-same. In those cases, yeah uptime is a big deal. I think what happens is that the standards get written and equipment designed with the assumption that everybody will be deploying all sorts of SLA'd, guaranteed services, then 99% of deployments end up being exclusively best-effort because it's so much easier and cheaper to deploy. GPON seems to be an interesting case of this since it's commonly deployed by telcos rather than cable MSOs, and, in greenfield applications, is often deployed exclusive to copper plant at all. It's pretty common to find GPON ONTs with inbuild UPS monitoring and communications as well as ATAs designed to deliver POTS-like service, but then a lot of SPs who are NOT the LEC of record just use that infrastructure to deliver VoIP-like services and push the UPS responsibility off onto the customer. -- Brandon Martin
Re: Comcast outages continue even in areas with PG&E power restored
On Mon, 14 Oct 2019, Michael Thomas wrote: deal with the CPE, that the cable plant was the actual problem. The cable companies should, imo, be held to the same standard as the telcos. Maybe even moreso these days since IP has taken over everything. The need for reliable e911 hasn't gone away just because the bits have turned into IP bit these days. Oh, but they are equal. The telco's went to the regulators and got the FCC and state PUCs to reduce or make backup power a customerresponsbility... Just like the cable companies. So now they are equal -- in the race to the bottom. Service providers must "OFFER" customers an OPTION for 8/24-hour standby backup power. The decision to puchase backup power is up to the customer. I assume you read the fine print on the back of your bill or the order terms (subject to change at anytime, without notice). The FCC is looking at standby power for cellular towers, but hasn't been paying attention to wireline and cable systems outside plant power. As I mentioned a few postings ago, cellular/wireless systems have been getting more resiliant. Wireline/cable systems have been getting less reliable.
Re: Comcast outages continue even in areas with PG&E power restored
Sean Donelan wrote: Given that providers can't supply power to mobile phones, that sending power over fiber is extremely eye unsafe and that most CPEs are routers which themselves are useless without end systems, it is reasonable that providers are not required to supply power to home. But, The FCC is looking at standby power for cellular towers, but hasn't been paying attention to wireline and cable systems outside plant power. why they are not equal? Masataka Ohta
