Thanks for the follow-up. It sounds like there's a bunch of steps that can be taken to both reduce the amount of real breakage and perhaps better understand and quantify the level of breakage. I hope we can find a configuration that indicates <0.001% of page views would be broken in a user-observable way before rolling this out broadly (eg. to beta or 1% of stable users).
Good luck and let us know if there's anything API owners can do to help work through the risk analysis! Rick On Tue, Nov 8, 2022 at 12:11 PM Titouan Rigoudy <[email protected]> wrote: > Hi Rick, > > Thanks for the suggestions and constructive feedback! > > We are going to exempt same-origin fetches and see what difference this > makes in the data: crbug.com/1382068. We expect this to help > split-horizon DNS / VPN users, and will report back with updated metrics. > > On Fri, Nov 4, 2022 at 4:28 PM Rick Byers <[email protected]> wrote: > >> On Wed, Nov 2, 2022 at 12:05 PM 'Titouan Rigoudy' via blink-dev < >> [email protected]> wrote: >> >>> Hi all, >>> >>> Thanks for your patience, I was travelling last week, then out for a >>> couple days, and was unable to spend much time on this until now. >>> >>> On Wed, Oct 12, 2022 at 6:35 PM Titouan Rigoudy <[email protected]> >>> wrote: >>> >>>> Thanks for getting back to us! >>>> >>>> On Wed, Oct 12, 2022 at 6:11 PM Daniel Bratell <[email protected]> >>>> wrote: >>>> >>>>> We (Yoav, Philip, MikeT and I) discussed this at the API OWNERS >>>>> meeting today and we are still not certain about the compatibility >>>>> situation. The use counters are lower after compensating for likely >>>>> malicious usage but they are still not quite as low as we'd like them to >>>>> be. >>>>> >>>>> One thing we considered was whether the data you analyzed is >>>>> representative for all platforms or whether Android is different in some >>>>> way. We don't know. Do you know? If not, could you take a look at the >>>>> Desktop data and check that it's the same pattern as for Android? >>>>> >>>> It's a good question! I'll take a look and see what I can tell from the >>>> data. >>>> >>> >>> The data on Desktop suggests that most usage is legitimate. ChromeOS >>> usage is dominated by educational websites: school districts and the like. >>> This could be driven by large providers like Clever or VPN usage. On >>> Windows and Mac we mostly see intranet-looking websites (CRMs, portals). On >>> Linux, we see some of the former and mostly the latter. >>> >>> On desktop platforms in general, the UseCounter indicates ~0.1% (Linux, >>> ChromeOS) to ~0.2% (Windows, MacOS) of page visits are affected. >>> >>> If that's too much, we could consider shipping on Android first, where >>> our previous analysis suggests the impact is an order of magnitude lower. >>> We could also ship on desktop to beta only for a few milestones, in order >>> to give websites more time to notice the change. >>> >> >> Yikes, that sure sounds like a huge amount of breakage, and so not >> something likely to survive a launch (partner escalations, negative press, >> etc.). In fact that seems suspiciously high to me. You're saying that one >> in 500 public Internet pages loaded on Windows has a legitimate need to >> load a sub-resource from a private IP space? That's a HUGE >> amount, hopefully it's over-counting somehow by a few orders of magnitude >> :-) >> >> In terms of risk reduction, launching first on Android is one option, but >> that sounds like it still has a non-trivial level of risk too. Here's a few >> other ideas to possibly help form a pragmatic launch strategy (mostly drawn >> from experience captured in bit.ly/blink-compat). >> >> - Survey some of the legitimate looking cases in the data and see if >> you can reproduce the breakage. How severe is it in practice? >> >> We've tried this and come up short. None of the websites display the > affected behavior, and most are hidden behind login pages. I tend to > believe that the users of affected websites on desktop are seeing issues > due to VPN use. For example, we've seen users run into issues when using > Zscaler in crbug.com/1352591. We are hoping to alleviate this by > exempting same-origin fetches as mentioned above. > >> >> - Pick a random set of impacted sites and work with partnerships / >> gtech teams to do targeted outreach. Is it easy to get sites to fix >> issues? >> Can we learn anything about how to help make it easier for them, and/or >> how >> to filter out hits from our data that aren't actually a problem in >> practice? >> >> On Android, we tried reaching out but usage was dominated by websites we > could not engage with. We could try on desktop! > >> >> - Do a test deployment to beta channel and see what level of feedback >> we get. Plan for a finch trial on stable of <1% of users (but IMHO current >> data suggests we're not ready even for a small stable trial). >> >> I think rolling out to beta only is the best next course of action at > this point, if exempting same-origin fetches does not result in a big > reduction of the estimated compatibility impact. > >> >> - Ensure failures are hooked up to NEL (Network Error Logging >> <https://web.dev/network-error-logging/>) and work with some bigger >> enterprises (eg. Google) to monitor, classify and debug breakage in their >> environment >> >> We also considered this. Unfortunately this type of reporting would leak > cross-origin information to the initiator website, so we scrapped it :( > > Cheers, > Titouan > > >> Cheers, >>> Titouan >>> >>>> Another question is the likely impact of a failed request. That is a >>>>> question that is much harder to answer, but maybe you have an idea? >>>>> >>>> In the case of malicious usage, a failed request is a win for user >>>> security and privacy! :) Seriously though, typically it means one of two >>>> things: a) the core functionality of the page does not work, because its >>>> whole purpose was to talk to a private network server, or b) some >>>> subresources on the page fail to load because of weirdness around servers >>>> with multiple IP addresses (some private, some public) or VPN usage (site >>>> used to be public, now is on a private IP). In the case of b), a reload >>>> usually fixes the issue. >>>> >>>>> A third question is whether the situation is likely to improve or >>>>> worsen over time. Do you have any insights about that? >>>>> >>>> Optimistically, the situation should improve as a result of developers >>>> noticing the warnings we've been flashing in DevTools for a while. >>>> >>> >> FWIW, past experiences for major breaking changes suggest that this is >> not particularly effective. "hopeful deprecations" have pretty much never >> worked out :-). >> >> >>> Pessimistically (and unfortunately this has been more our experience so >>>> far), the situation will stagnate and/or get worse until the deprecation >>>> starts, and people sign up for the deprecation trial. We can always invest >>>> some more in classifying uses and reaching out to many small websites, it's >>>> simply a tradeoff of time and effort spent trying to get ahead of the issue >>>> vs patching the security hole earlier and getting on to fixing other >>>> things. >>>> >>>> Cheers, >>>> Titouan >>>> >>>>> /Daniel >>>>> >>>>> >>>>> On 2022-10-06 12:12, Titouan Rigoudy wrote: >>>>> >>>>> Exactly! Somewhere in the ballpark of 0.01% is my rough estimate. >>>>> >>>>> Cheers, >>>>> Titouan >>>>> >>>>> On Thu, Oct 6, 2022 at 11:06 AM Yoav Weiss <[email protected]> >>>>> wrote: >>>>> >>>>>> So if the use counters are at 0.1-0.2%, we can expect only >>>>>> 0.007-0.014% to be legitimate (roughly speaking)? >>>>>> >>>>>> On Thu, Oct 6, 2022 at 10:54 AM Titouan Rigoudy <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Sure! I went through the UKM data from Android (where we can be >>>>>>> reasonably sure that extensions are not messing with the data) and >>>>>>> classified a bunch of the top websites. I ended up classifying ~2/3 of >>>>>>> the >>>>>>> traffic in volume. Out of that, it seemed that ~7% was legitimate. There >>>>>>> was no indication that the rest of the traffic would strongly buck that >>>>>>> trend, I was just hitting diminishing returns for low usecount websites. >>>>>>> >>>>>>> Cheers, >>>>>>> Titouan >>>>>>> >>>>>>> On Wed, Oct 5, 2022 at 6:40 PM Chris Harrelson < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Thanks Titouan, two of them is definitely enough, no need for >>>>>>>> three. :) >>>>>>>> >>>>>>>> Another question: have you done an analysis of how much of the >>>>>>>> UseCounter traffic is "illegitimate use"? Two of them are at 0.1% or >>>>>>>> 0.2%, >>>>>>>> which is a risky level. But hopefully the percentage of "legitimate" >>>>>>>> traffic is a lot lower? >>>>>>>> >>>>>>>> Chris >>>>>>>> >>>>>>>> On Wed, Oct 5, 2022 at 9:37 AM 'Titouan Rigoudy' via blink-dev < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> There are two enterprise policies for this [1]! One for disabling >>>>>>>>> it entirely, and one for disabling it per origin. >>>>>>>>> >>>>>>>>> We have been engaging with the enterprise team since the start of >>>>>>>>> PNA, a dozen or so milestones ago. This has been announced repeatedly >>>>>>>>> in >>>>>>>>> enterprise release notes over the past several milestones. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Titouan >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> https://developer.chrome.com/blog/private-network-access-preflight/#disable-with-enterprise-policy >>>>>>>>> >>>>>>>>> On Wed, Oct 5, 2022 at 6:10 PM Daniel Bratell <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I'm curious about the enterprise situation here. This seems to me >>>>>>>>>> like something that could be in use in enterprise applications, and >>>>>>>>>> we >>>>>>>>>> would not really know about it. ( >>>>>>>>>> https://www.chromium.org/developers/enterprise-changes/ is a >>>>>>>>>> good checklist for this) >>>>>>>>>> >>>>>>>>>> Is there an enterprise policy for this that can be used for those >>>>>>>>>> that need the old behaviour, if not, could one be added? >>>>>>>>>> >>>>>>>>>> Also, have you reached out to the enterprise community? (See link >>>>>>>>>> above) >>>>>>>>>> >>>>>>>>>> /Daniel >>>>>>>>>> On 2022-10-05 15:28, Jonathan Hao wrote: >>>>>>>>>> >>>>>>>>>> Context Since M104, we've started sending preflight requests >>>>>>>>>> before private network access, but ignoring the preflight result (or >>>>>>>>>> the >>>>>>>>>> lack of it). After analyzing the URL-keyed metrics, we found that >>>>>>>>>> most of >>>>>>>>>> them don't seem legit, most likely used for fingerprinting purposes, >>>>>>>>>> so we >>>>>>>>>> decided to start enforcing the preflight response, which means that >>>>>>>>>> the >>>>>>>>>> websites will not be able to fetch subresources from less-public ip >>>>>>>>>> address >>>>>>>>>> space without getting proper preflight responses. >>>>>>>>>> >>>>>>>>>> An intent focusing on the deprecation trial was sent earlier in >>>>>>>>>> this thread: >>>>>>>>>> https://groups.google.com/a/chromium.org/g/blink-dev/c/k8osI88QbKs/m/16ytAQ-BAwAJ?utm_medium=email&utm_source=footer. >>>>>>>>>> There, we were suggested to send a separate intent to remove the >>>>>>>>>> functionality, and hence this intent email. >>>>>>>>>> >>>>>>>>>> Contact emails [email protected], [email protected], >>>>>>>>>> [email protected], [email protected], [email protected] >>>>>>>>>> >>>>>>>>>> Explainer >>>>>>>>>> https://github.com/WICG/private-network-access/blob/main/explainer.md >>>>>>>>>> >>>>>>>>>> Specification https://wicg.github.io/private-network-access >>>>>>>>>> >>>>>>>>>> Design docs >>>>>>>>>> >>>>>>>>>> https://docs.google.com/document/d/1FYPIeP90MQ_pQ6UAo0mCB3g2Z_AynfPWHbDnHIST6VI/edit >>>>>>>>>> >>>>>>>>>> Summary >>>>>>>>>> >>>>>>>>>> Sends a CORS preflight request ahead of any private network >>>>>>>>>> requests for subresources, asking for explicit permission from the >>>>>>>>>> target >>>>>>>>>> server. A private network request is any request from a public >>>>>>>>>> website to a >>>>>>>>>> private IP address or localhost, or from a private website (e.g. >>>>>>>>>> intranet) >>>>>>>>>> to localhost. Sending a preflight request mitigates the risk of >>>>>>>>>> cross-site >>>>>>>>>> request forgery attacks against private network devices such as >>>>>>>>>> routers, >>>>>>>>>> which are often not prepared to defend against this threat. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Blink component Blink>SecurityFeature>CORS>PrivateNetworkAccess >>>>>>>>>> <https://bugs.chromium.org/p/chromium/issues/list?q=component:Blink%3ESecurityFeature%3ECORS%3EPrivateNetworkAccess> >>>>>>>>>> >>>>>>>>>> TAG review https://github.com/w3ctag/design-reviews/issues/572 >>>>>>>>>> >>>>>>>>>> TAG review status Issues addressed >>>>>>>>>> >>>>>>>>>> Risks >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Interoperability and Compatibility >>>>>>>>>> >>>>>>>>>> The main interoperability risk, as always, is if other browser >>>>>>>>>> engines do not implement this. Compat risk is straightforward: web >>>>>>>>>> servers >>>>>>>>>> that do not handle the new preflight requests will eventually break, >>>>>>>>>> once >>>>>>>>>> the feature ships. The plan to address this is as follows: 1. Send >>>>>>>>>> preflight request, ignore result, always send actual request. Failed >>>>>>>>>> preflight requests will result in a warning being shown in devtools. >>>>>>>>>> 2. >>>>>>>>>> Wait for 3 milestones. 3. Gate actual request on preflight request >>>>>>>>>> success, >>>>>>>>>> with deprecation trial for developers to buy some more time. 4. End >>>>>>>>>> deprecation trial 4 milestones later. UseCounters: >>>>>>>>>> https://chromestatus.com/metrics/feature/timeline/popularity/3753 >>>>>>>>>> https://chromestatus.com/metrics/feature/timeline/popularity/3755 >>>>>>>>>> https://chromestatus.com/metrics/feature/timeline/popularity/3757 >>>>>>>>>> The above measure pages that make at least one private network >>>>>>>>>> request for >>>>>>>>>> which we would now send a preflight request. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Gecko*: Worth prototyping ( >>>>>>>>>> https://github.com/mozilla/standards-positions/issues/143) >>>>>>>>>> >>>>>>>>>> *WebKit*: No signal ( >>>>>>>>>> https://lists.webkit.org/pipermail/webkit-dev/2021-November/032040.html) >>>>>>>>>> Pending response. >>>>>>>>>> >>>>>>>>>> *Web developers*: No signals Anecdotal evidence so far suggests >>>>>>>>>> that most web developers are OK with this new requirement, though >>>>>>>>>> some do >>>>>>>>>> not control the target endpoints and would be negatively impacted. >>>>>>>>>> >>>>>>>>>> *Other signals*: >>>>>>>>>> >>>>>>>>>> Ergonomics >>>>>>>>>> >>>>>>>>>> None. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Activation >>>>>>>>>> >>>>>>>>>> Gating access to the private network overnight on preflight >>>>>>>>>> requests would likely result in widespread breakage. This is why the >>>>>>>>>> plan >>>>>>>>>> is to first send requests but not act on their result, giving server >>>>>>>>>> developers time to implement code handling these requests. >>>>>>>>>> Deprecation >>>>>>>>>> warnings will be surfaced in DevTools to alert web/client developers >>>>>>>>>> when >>>>>>>>>> the potential for breakage later on is detected. Enforcement will be >>>>>>>>>> turned >>>>>>>>>> on later (aiming for 3 milestones), along with a deprecation trial >>>>>>>>>> for >>>>>>>>>> impacted web developers to buy themselves some more time. Experience >>>>>>>>>> suggests a large fraction of developers will not notice the advance >>>>>>>>>> deprecation warnings until things break. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Security >>>>>>>>>> >>>>>>>>>> This change aims to be security-positive, preventing CSRF attacks >>>>>>>>>> against soft and juicy targets such as router admin interfaces. It >>>>>>>>>> does not >>>>>>>>>> cover navigation requests and workers, which are to be addressed in >>>>>>>>>> followup launches. DNS rebinding threats were of particular concern >>>>>>>>>> during >>>>>>>>>> the design of this feature: >>>>>>>>>> https://docs.google.com/document/d/1FYPIeP90MQ_pQ6UAo0mCB3g2Z_AynfPWHbDnHIST6VI/edit#heading=h.189j5gnadts9 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> WebView application risks >>>>>>>>>> >>>>>>>>>> Does this intent deprecate or change behavior of existing APIs, >>>>>>>>>> such that it has potentially high risk for Android WebView-based >>>>>>>>>> applications? >>>>>>>>>> >>>>>>>>>> Not going to ship on Android WebView >>>>>>>>>> >>>>>>>>>> Goals for experimentation Give websites time to make sure they >>>>>>>>>> respond to the preflights >>>>>>>>>> >>>>>>>>>> Reason this experiment is being extended N/A >>>>>>>>>> >>>>>>>>>> Ongoing technical constraints N/A >>>>>>>>>> >>>>>>>>>> Debuggability >>>>>>>>>> >>>>>>>>>> Relevant information (client and resource IP address space) is >>>>>>>>>> already piped into the DevTools network panel. Deprecation warnings >>>>>>>>>> and >>>>>>>>>> errors will be surfaced in the DevTools issues panel explaining the >>>>>>>>>> problem >>>>>>>>>> when it arises. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Will this feature be supported on all six Blink platforms >>>>>>>>>> (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? >>>>>>>>>> No >>>>>>>>>> >>>>>>>>>> Not on Android WebView given previous difficulty in supporting >>>>>>>>>> PNA changes due to the lack of support for deprecation trials. >>>>>>>>>> Support for >>>>>>>>>> WebView will be considered separately. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Is this feature fully tested by web-platform-tests >>>>>>>>>> <https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md> >>>>>>>>>> ? Yes >>>>>>>>>> >>>>>>>>>> DevTrial instructions >>>>>>>>>> https://github.com/WICG/private-network-access/blob/main/HOWTO.md >>>>>>>>>> >>>>>>>>>> Flag name PrivateNetworkAccessRespectPreflightResults >>>>>>>>>> >>>>>>>>>> Requires code in //chrome? False >>>>>>>>>> >>>>>>>>>> Tracking bug https://crbug.com/591068 >>>>>>>>>> >>>>>>>>>> Launch bug https://crbug.com/1274149 >>>>>>>>>> >>>>>>>>>> Estimated milestones >>>>>>>>>> OriginTrial desktop last 112 >>>>>>>>>> OriginTrial desktop first 109 >>>>>>>>>> DevTrial on desktop 98 >>>>>>>>>> OriginTrial Android last 112 >>>>>>>>>> OriginTrial Android first 109 >>>>>>>>>> DevTrial on Android 98 >>>>>>>>>> >>>>>>>>>> Link to entry on the Chrome Platform Status >>>>>>>>>> https://chromestatus.com/feature/5737414355058688 >>>>>>>>>> >>>>>>>>>> Links to previous Intent discussions Intent to prototype: >>>>>>>>>> https://groups.google.com/a/chromium.org/g/blink-dev/c/PrB0xnNxaHs/m/jeoxvNjXCAAJ >>>>>>>>>> Intent to Ship: >>>>>>>>>> https://groups.google.com/a/chromium.org/g/blink-dev/c/72CK2mxD47c >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> This intent message was generated by Chrome Platform Status >>>>>>>>>> <https://chromestatus.com/>. >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "blink-dev" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOC%3DiP%2BEcXFNTOfg829uzFh2YMov%3DTsmAzdP9VDn8MoLuHqjog%40mail.gmail.com >>>>>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOC%3DiP%2BEcXFNTOfg829uzFh2YMov%3DTsmAzdP9VDn8MoLuHqjog%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "blink-dev" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPATO9dKw30wJL3Aj5OOmYeT5Gxy1Bb7Yf%2B7VFEmAjAEyySFWw%40mail.gmail.com >>>>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPATO9dKw30wJL3Aj5OOmYeT5Gxy1Bb7Yf%2B7VFEmAjAEyySFWw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "blink-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPATO9eW%3DAvds4MTkgez7rVdF9gt1GrD9kmL-YqQV4YBpGSniw%40mail.gmail.com >>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPATO9eW%3DAvds4MTkgez7rVdF9gt1GrD9kmL-YqQV4YBpGSniw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFUtAY-PpH8XKFbDp62_8bku_Ci26uqo6XfzN%2BOymEUVOFKPbQ%40mail.gmail.com.
