Hi Rick, Thanks for the suggestions and constructive feedback!
We are going to exempt same-origin fetches and see what difference this makes in the data: crbug.com/1382068. We expect this to help split-horizon DNS / VPN users, and will report back with updated metrics. On Fri, Nov 4, 2022 at 4:28 PM Rick Byers <[email protected]> wrote: > On Wed, Nov 2, 2022 at 12:05 PM 'Titouan Rigoudy' via blink-dev < > [email protected]> wrote: > >> Hi all, >> >> Thanks for your patience, I was travelling last week, then out for a >> couple days, and was unable to spend much time on this until now. >> >> On Wed, Oct 12, 2022 at 6:35 PM Titouan Rigoudy <[email protected]> >> wrote: >> >>> Thanks for getting back to us! >>> >>> On Wed, Oct 12, 2022 at 6:11 PM Daniel Bratell <[email protected]> >>> wrote: >>> >>>> We (Yoav, Philip, MikeT and I) discussed this at the API OWNERS meeting >>>> today and we are still not certain about the compatibility situation. The >>>> use counters are lower after compensating for likely malicious usage but >>>> they are still not quite as low as we'd like them to be. >>>> >>>> One thing we considered was whether the data you analyzed is >>>> representative for all platforms or whether Android is different in some >>>> way. We don't know. Do you know? If not, could you take a look at the >>>> Desktop data and check that it's the same pattern as for Android? >>>> >>> It's a good question! I'll take a look and see what I can tell from the >>> data. >>> >> >> The data on Desktop suggests that most usage is legitimate. ChromeOS >> usage is dominated by educational websites: school districts and the like. >> This could be driven by large providers like Clever or VPN usage. On >> Windows and Mac we mostly see intranet-looking websites (CRMs, portals). On >> Linux, we see some of the former and mostly the latter. >> >> On desktop platforms in general, the UseCounter indicates ~0.1% (Linux, >> ChromeOS) to ~0.2% (Windows, MacOS) of page visits are affected. >> >> If that's too much, we could consider shipping on Android first, where >> our previous analysis suggests the impact is an order of magnitude lower. >> We could also ship on desktop to beta only for a few milestones, in order >> to give websites more time to notice the change. >> > > Yikes, that sure sounds like a huge amount of breakage, and so not > something likely to survive a launch (partner escalations, negative press, > etc.). In fact that seems suspiciously high to me. You're saying that one > in 500 public Internet pages loaded on Windows has a legitimate need to > load a sub-resource from a private IP space? That's a HUGE > amount, hopefully it's over-counting somehow by a few orders of magnitude > :-) > > In terms of risk reduction, launching first on Android is one option, but > that sounds like it still has a non-trivial level of risk too. Here's a few > other ideas to possibly help form a pragmatic launch strategy (mostly drawn > from experience captured in bit.ly/blink-compat). > > - Survey some of the legitimate looking cases in the data and see if > you can reproduce the breakage. How severe is it in practice? > > We've tried this and come up short. None of the websites display the affected behavior, and most are hidden behind login pages. I tend to believe that the users of affected websites on desktop are seeing issues due to VPN use. For example, we've seen users run into issues when using Zscaler in crbug.com/1352591. We are hoping to alleviate this by exempting same-origin fetches as mentioned above. > > - Pick a random set of impacted sites and work with partnerships / > gtech teams to do targeted outreach. Is it easy to get sites to fix issues? > Can we learn anything about how to help make it easier for them, and/or how > to filter out hits from our data that aren't actually a problem in > practice? > > On Android, we tried reaching out but usage was dominated by websites we could not engage with. We could try on desktop! > > - Do a test deployment to beta channel and see what level of feedback > we get. Plan for a finch trial on stable of <1% of users (but IMHO current > data suggests we're not ready even for a small stable trial). > > I think rolling out to beta only is the best next course of action at this point, if exempting same-origin fetches does not result in a big reduction of the estimated compatibility impact. > > - Ensure failures are hooked up to NEL (Network Error Logging > <https://web.dev/network-error-logging/>) and work with some bigger > enterprises (eg. Google) to monitor, classify and debug breakage in their > environment > > We also considered this. Unfortunately this type of reporting would leak cross-origin information to the initiator website, so we scrapped it :( Cheers, Titouan > Cheers, >> Titouan >> >>> Another question is the likely impact of a failed request. That is a >>>> question that is much harder to answer, but maybe you have an idea? >>>> >>> In the case of malicious usage, a failed request is a win for user >>> security and privacy! :) Seriously though, typically it means one of two >>> things: a) the core functionality of the page does not work, because its >>> whole purpose was to talk to a private network server, or b) some >>> subresources on the page fail to load because of weirdness around servers >>> with multiple IP addresses (some private, some public) or VPN usage (site >>> used to be public, now is on a private IP). In the case of b), a reload >>> usually fixes the issue. >>> >>>> A third question is whether the situation is likely to improve or >>>> worsen over time. Do you have any insights about that? >>>> >>> Optimistically, the situation should improve as a result of developers >>> noticing the warnings we've been flashing in DevTools for a while. >>> >> > FWIW, past experiences for major breaking changes suggest that this is not > particularly effective. "hopeful deprecations" have pretty much never > worked out :-). > > >> Pessimistically (and unfortunately this has been more our experience so >>> far), the situation will stagnate and/or get worse until the deprecation >>> starts, and people sign up for the deprecation trial. We can always invest >>> some more in classifying uses and reaching out to many small websites, it's >>> simply a tradeoff of time and effort spent trying to get ahead of the issue >>> vs patching the security hole earlier and getting on to fixing other things. >>> >>> Cheers, >>> Titouan >>> >>>> /Daniel >>>> >>>> >>>> On 2022-10-06 12:12, Titouan Rigoudy wrote: >>>> >>>> Exactly! Somewhere in the ballpark of 0.01% is my rough estimate. >>>> >>>> Cheers, >>>> Titouan >>>> >>>> On Thu, Oct 6, 2022 at 11:06 AM Yoav Weiss <[email protected]> >>>> wrote: >>>> >>>>> So if the use counters are at 0.1-0.2%, we can expect only >>>>> 0.007-0.014% to be legitimate (roughly speaking)? >>>>> >>>>> On Thu, Oct 6, 2022 at 10:54 AM Titouan Rigoudy <[email protected]> >>>>> wrote: >>>>> >>>>>> Sure! I went through the UKM data from Android (where we can be >>>>>> reasonably sure that extensions are not messing with the data) and >>>>>> classified a bunch of the top websites. I ended up classifying ~2/3 of >>>>>> the >>>>>> traffic in volume. Out of that, it seemed that ~7% was legitimate. There >>>>>> was no indication that the rest of the traffic would strongly buck that >>>>>> trend, I was just hitting diminishing returns for low usecount websites. >>>>>> >>>>>> Cheers, >>>>>> Titouan >>>>>> >>>>>> On Wed, Oct 5, 2022 at 6:40 PM Chris Harrelson <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Thanks Titouan, two of them is definitely enough, no need for three. >>>>>>> :) >>>>>>> >>>>>>> Another question: have you done an analysis of how much of the >>>>>>> UseCounter traffic is "illegitimate use"? Two of them are at 0.1% or >>>>>>> 0.2%, >>>>>>> which is a risky level. But hopefully the percentage of "legitimate" >>>>>>> traffic is a lot lower? >>>>>>> >>>>>>> Chris >>>>>>> >>>>>>> On Wed, Oct 5, 2022 at 9:37 AM 'Titouan Rigoudy' via blink-dev < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> There are two enterprise policies for this [1]! One for disabling >>>>>>>> it entirely, and one for disabling it per origin. >>>>>>>> >>>>>>>> We have been engaging with the enterprise team since the start of >>>>>>>> PNA, a dozen or so milestones ago. This has been announced repeatedly >>>>>>>> in >>>>>>>> enterprise release notes over the past several milestones. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Titouan >>>>>>>> >>>>>>>> [1] >>>>>>>> https://developer.chrome.com/blog/private-network-access-preflight/#disable-with-enterprise-policy >>>>>>>> >>>>>>>> On Wed, Oct 5, 2022 at 6:10 PM Daniel Bratell <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I'm curious about the enterprise situation here. This seems to me >>>>>>>>> like something that could be in use in enterprise applications, and we >>>>>>>>> would not really know about it. ( >>>>>>>>> https://www.chromium.org/developers/enterprise-changes/ is a good >>>>>>>>> checklist for this) >>>>>>>>> >>>>>>>>> Is there an enterprise policy for this that can be used for those >>>>>>>>> that need the old behaviour, if not, could one be added? >>>>>>>>> >>>>>>>>> Also, have you reached out to the enterprise community? (See link >>>>>>>>> above) >>>>>>>>> >>>>>>>>> /Daniel >>>>>>>>> On 2022-10-05 15:28, Jonathan Hao wrote: >>>>>>>>> >>>>>>>>> Context Since M104, we've started sending preflight requests >>>>>>>>> before private network access, but ignoring the preflight result (or >>>>>>>>> the >>>>>>>>> lack of it). After analyzing the URL-keyed metrics, we found that >>>>>>>>> most of >>>>>>>>> them don't seem legit, most likely used for fingerprinting purposes, >>>>>>>>> so we >>>>>>>>> decided to start enforcing the preflight response, which means that >>>>>>>>> the >>>>>>>>> websites will not be able to fetch subresources from less-public ip >>>>>>>>> address >>>>>>>>> space without getting proper preflight responses. >>>>>>>>> >>>>>>>>> An intent focusing on the deprecation trial was sent earlier in >>>>>>>>> this thread: >>>>>>>>> https://groups.google.com/a/chromium.org/g/blink-dev/c/k8osI88QbKs/m/16ytAQ-BAwAJ?utm_medium=email&utm_source=footer. >>>>>>>>> There, we were suggested to send a separate intent to remove the >>>>>>>>> functionality, and hence this intent email. >>>>>>>>> >>>>>>>>> Contact emails [email protected], [email protected], >>>>>>>>> [email protected], [email protected], [email protected] >>>>>>>>> >>>>>>>>> Explainer >>>>>>>>> https://github.com/WICG/private-network-access/blob/main/explainer.md >>>>>>>>> >>>>>>>>> Specification https://wicg.github.io/private-network-access >>>>>>>>> >>>>>>>>> Design docs >>>>>>>>> >>>>>>>>> https://docs.google.com/document/d/1FYPIeP90MQ_pQ6UAo0mCB3g2Z_AynfPWHbDnHIST6VI/edit >>>>>>>>> >>>>>>>>> Summary >>>>>>>>> >>>>>>>>> Sends a CORS preflight request ahead of any private network >>>>>>>>> requests for subresources, asking for explicit permission from the >>>>>>>>> target >>>>>>>>> server. A private network request is any request from a public >>>>>>>>> website to a >>>>>>>>> private IP address or localhost, or from a private website (e.g. >>>>>>>>> intranet) >>>>>>>>> to localhost. Sending a preflight request mitigates the risk of >>>>>>>>> cross-site >>>>>>>>> request forgery attacks against private network devices such as >>>>>>>>> routers, >>>>>>>>> which are often not prepared to defend against this threat. >>>>>>>>> >>>>>>>>> >>>>>>>>> Blink component Blink>SecurityFeature>CORS>PrivateNetworkAccess >>>>>>>>> <https://bugs.chromium.org/p/chromium/issues/list?q=component:Blink%3ESecurityFeature%3ECORS%3EPrivateNetworkAccess> >>>>>>>>> >>>>>>>>> TAG review https://github.com/w3ctag/design-reviews/issues/572 >>>>>>>>> >>>>>>>>> TAG review status Issues addressed >>>>>>>>> >>>>>>>>> Risks >>>>>>>>> >>>>>>>>> >>>>>>>>> Interoperability and Compatibility >>>>>>>>> >>>>>>>>> The main interoperability risk, as always, is if other browser >>>>>>>>> engines do not implement this. Compat risk is straightforward: web >>>>>>>>> servers >>>>>>>>> that do not handle the new preflight requests will eventually break, >>>>>>>>> once >>>>>>>>> the feature ships. The plan to address this is as follows: 1. Send >>>>>>>>> preflight request, ignore result, always send actual request. Failed >>>>>>>>> preflight requests will result in a warning being shown in devtools. >>>>>>>>> 2. >>>>>>>>> Wait for 3 milestones. 3. Gate actual request on preflight request >>>>>>>>> success, >>>>>>>>> with deprecation trial for developers to buy some more time. 4. End >>>>>>>>> deprecation trial 4 milestones later. UseCounters: >>>>>>>>> https://chromestatus.com/metrics/feature/timeline/popularity/3753 >>>>>>>>> https://chromestatus.com/metrics/feature/timeline/popularity/3755 >>>>>>>>> https://chromestatus.com/metrics/feature/timeline/popularity/3757 >>>>>>>>> The above measure pages that make at least one private network >>>>>>>>> request for >>>>>>>>> which we would now send a preflight request. >>>>>>>>> >>>>>>>>> >>>>>>>>> *Gecko*: Worth prototyping ( >>>>>>>>> https://github.com/mozilla/standards-positions/issues/143) >>>>>>>>> >>>>>>>>> *WebKit*: No signal ( >>>>>>>>> https://lists.webkit.org/pipermail/webkit-dev/2021-November/032040.html) >>>>>>>>> Pending response. >>>>>>>>> >>>>>>>>> *Web developers*: No signals Anecdotal evidence so far suggests >>>>>>>>> that most web developers are OK with this new requirement, though >>>>>>>>> some do >>>>>>>>> not control the target endpoints and would be negatively impacted. >>>>>>>>> >>>>>>>>> *Other signals*: >>>>>>>>> >>>>>>>>> Ergonomics >>>>>>>>> >>>>>>>>> None. >>>>>>>>> >>>>>>>>> >>>>>>>>> Activation >>>>>>>>> >>>>>>>>> Gating access to the private network overnight on preflight >>>>>>>>> requests would likely result in widespread breakage. This is why the >>>>>>>>> plan >>>>>>>>> is to first send requests but not act on their result, giving server >>>>>>>>> developers time to implement code handling these requests. Deprecation >>>>>>>>> warnings will be surfaced in DevTools to alert web/client developers >>>>>>>>> when >>>>>>>>> the potential for breakage later on is detected. Enforcement will be >>>>>>>>> turned >>>>>>>>> on later (aiming for 3 milestones), along with a deprecation trial for >>>>>>>>> impacted web developers to buy themselves some more time. Experience >>>>>>>>> suggests a large fraction of developers will not notice the advance >>>>>>>>> deprecation warnings until things break. >>>>>>>>> >>>>>>>>> >>>>>>>>> Security >>>>>>>>> >>>>>>>>> This change aims to be security-positive, preventing CSRF attacks >>>>>>>>> against soft and juicy targets such as router admin interfaces. It >>>>>>>>> does not >>>>>>>>> cover navigation requests and workers, which are to be addressed in >>>>>>>>> followup launches. DNS rebinding threats were of particular concern >>>>>>>>> during >>>>>>>>> the design of this feature: >>>>>>>>> https://docs.google.com/document/d/1FYPIeP90MQ_pQ6UAo0mCB3g2Z_AynfPWHbDnHIST6VI/edit#heading=h.189j5gnadts9 >>>>>>>>> >>>>>>>>> >>>>>>>>> WebView application risks >>>>>>>>> >>>>>>>>> Does this intent deprecate or change behavior of existing APIs, >>>>>>>>> such that it has potentially high risk for Android WebView-based >>>>>>>>> applications? >>>>>>>>> >>>>>>>>> Not going to ship on Android WebView >>>>>>>>> >>>>>>>>> Goals for experimentation Give websites time to make sure they >>>>>>>>> respond to the preflights >>>>>>>>> >>>>>>>>> Reason this experiment is being extended N/A >>>>>>>>> >>>>>>>>> Ongoing technical constraints N/A >>>>>>>>> >>>>>>>>> Debuggability >>>>>>>>> >>>>>>>>> Relevant information (client and resource IP address space) is >>>>>>>>> already piped into the DevTools network panel. Deprecation warnings >>>>>>>>> and >>>>>>>>> errors will be surfaced in the DevTools issues panel explaining the >>>>>>>>> problem >>>>>>>>> when it arises. >>>>>>>>> >>>>>>>>> >>>>>>>>> Will this feature be supported on all six Blink platforms >>>>>>>>> (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? No >>>>>>>>> >>>>>>>>> Not on Android WebView given previous difficulty in supporting PNA >>>>>>>>> changes due to the lack of support for deprecation trials. Support for >>>>>>>>> WebView will be considered separately. >>>>>>>>> >>>>>>>>> >>>>>>>>> Is this feature fully tested by web-platform-tests >>>>>>>>> <https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md> >>>>>>>>> ? Yes >>>>>>>>> >>>>>>>>> DevTrial instructions >>>>>>>>> https://github.com/WICG/private-network-access/blob/main/HOWTO.md >>>>>>>>> >>>>>>>>> Flag name PrivateNetworkAccessRespectPreflightResults >>>>>>>>> >>>>>>>>> Requires code in //chrome? False >>>>>>>>> >>>>>>>>> Tracking bug https://crbug.com/591068 >>>>>>>>> >>>>>>>>> Launch bug https://crbug.com/1274149 >>>>>>>>> >>>>>>>>> Estimated milestones >>>>>>>>> OriginTrial desktop last 112 >>>>>>>>> OriginTrial desktop first 109 >>>>>>>>> DevTrial on desktop 98 >>>>>>>>> OriginTrial Android last 112 >>>>>>>>> OriginTrial Android first 109 >>>>>>>>> DevTrial on Android 98 >>>>>>>>> >>>>>>>>> Link to entry on the Chrome Platform Status >>>>>>>>> https://chromestatus.com/feature/5737414355058688 >>>>>>>>> >>>>>>>>> Links to previous Intent discussions Intent to prototype: >>>>>>>>> https://groups.google.com/a/chromium.org/g/blink-dev/c/PrB0xnNxaHs/m/jeoxvNjXCAAJ >>>>>>>>> Intent to Ship: >>>>>>>>> https://groups.google.com/a/chromium.org/g/blink-dev/c/72CK2mxD47c >>>>>>>>> >>>>>>>>> >>>>>>>>> This intent message was generated by Chrome Platform Status >>>>>>>>> <https://chromestatus.com/>. >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "blink-dev" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOC%3DiP%2BEcXFNTOfg829uzFh2YMov%3DTsmAzdP9VDn8MoLuHqjog%40mail.gmail.com >>>>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOC%3DiP%2BEcXFNTOfg829uzFh2YMov%3DTsmAzdP9VDn8MoLuHqjog%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "blink-dev" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPATO9dKw30wJL3Aj5OOmYeT5Gxy1Bb7Yf%2B7VFEmAjAEyySFWw%40mail.gmail.com >>>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPATO9dKw30wJL3Aj5OOmYeT5Gxy1Bb7Yf%2B7VFEmAjAEyySFWw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >> You received this message because you are subscribed to the Google Groups >> "blink-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPATO9eW%3DAvds4MTkgez7rVdF9gt1GrD9kmL-YqQV4YBpGSniw%40mail.gmail.com >> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPATO9eW%3DAvds4MTkgez7rVdF9gt1GrD9kmL-YqQV4YBpGSniw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPATO9cpKCrPZJQHguZVs2FYjStdhLHJTbJfCJNkzWfYrjJrwQ%40mail.gmail.com.
