+1 to Philip's sentiment, thanks for putting the effort into this! Given that it got to 50% stable and this was the only issue reported and Twilio is happy with the plan, then I'm also reluctant to ask for more opt-outs. So LGTM2
But please circle back here if you hear of any other non-trivial breakage. If necessary we can postpone again with finch and add an enterprise opt-out. +1 to what Johnny says about enterprise policies being the only effective knob for some environments - we've heard of many cases where updating a software package is a multi-month process for enterprise customers of a software system, but where asking those customers to enable a policy in their fleet is relatively easy. It's generally not about software developed by the enterprise itself, but bought from a 3p who has a support contract. More details on the policy and purpose of it are here <http://WebRTC's RTCStats of type "track" and "stream"'>. Rick On Wed, Feb 8, 2023 at 5:54 PM Philip Jägenstedt <foo...@chromium.org> wrote: > LGTM1 for the plan to remove in 112 with reverse OT until 115. > > We discussed in today's meeting whether we should also have an enterprise > policy, but landed on not requesting it. The reason is that it's quite > possible that this change only breaks Twilio and nothing else, at all. > That's because of the shape of the API and that the change got to 50% > stable in M109 without other breakage being reported. An enterprise policy > would be the "belts and braces" approach, but there's also a cost to making > changes like these more laborious that has to be weighed against the risk. > > Removing things from the web platform is usually thankless work, but it is > important to reduce complexity and converge with other browsers > (interoperability) so I want to applaud the effort put into this 👏 > > On Wed, Feb 8, 2023 at 6:45 PM Johnny Stenback <jstenb...@chromium.org> > wrote: > >> Deprecation trials require changes to servers which an enterprise may not >> have access to or ability to affect changes to, whereas enterprise policies >> are entirely within the enterprise's control. I'd recommend reaching out to >> the enterprise team for their perspective on usage policies and perspective >> on whether a policy should be included here or not (I'm still of the >> opinion that it's in our interest to include one). >> >> Thanks, >> Johnny >> >> On Tue, Feb 7, 2023 at 2:36 AM Henrik Boström <h...@chromium.org> wrote: >> >>> I have the same concern as Harald regarding corporate policies. Why not >>> a Deprecation Origin Trial in that case for list of users and more concrete >>> timeline? >>> >>> On Tuesday, February 7, 2023 at 9:10:40 AM UTC+1 Harald Alvestrand wrote: >>> >>>> Do we have trackable statistics on the usage of corporate policies? >>>> ie if nobody uses the policy in 2 milestones, can we detect that and >>>> decide that it is not needed and delete it, or will we be as unsure as we >>>> are now? >>>> >>>> >>>> On Mon, Feb 6, 2023 at 9:41 PM Mike Taylor <miketa...@chromium.org> >>>> wrote: >>>> >>>>> We don't know what we don't know, but it's not hard to imagine an >>>>> in-house enterprise WebRTC application that is using "stats" or "track". >>>>> Twilio is the breakage we know about (because a developer took the time to >>>>> report a bug). Having a policy so an app continues to work while a fix is >>>>> made is a good thing - and comes with the nice side effect of appearing on >>>>> the Enterprise release notes, increasing awareness. >>>>> >>>>> On 2/4/23 3:28 AM, Harald Alvestrand wrote: >>>>> >>>>> What's the imagined scenario in which an enterprise policy would be >>>>> useful? >>>>> >>>>> The only place I could imagine it being relevant is if there exists a >>>>> WebRTC application that is only used within a single enterprise (neither >>>>> hosting nor usage exists outside the enterprise), and that WebRTC >>>>> application depends on non-upgraded Twilio libraries. >>>>> >>>>> I don't know that we have evidence that such applications exist. >>>>> >>>>> >>>>> >>>>> On Fri, Feb 3, 2023 at 6:14 PM Mike Taylor <miketa...@chromium.org> >>>>> wrote: >>>>> >>>>>> I agree with Johnny that an enterprise policy would be useful, at >>>>>> least for a few milestones. >>>>>> >>>>>> On 1/30/23 5:16 AM, 'Harald Alvestrand' via blink-dev wrote: >>>>>> >>>>>> I'm not sure an enterprise policy is appropriate - I see the same >>>>>> problem with sunsetting the policy as with sunsetting the stat in >>>>>> general, >>>>>> and usage of enterprise policies is (as far as I know) far more opaque to >>>>>> us than origin trials or Finch feature usage. >>>>>> >>>>>> >>>>>> On Mon, Jan 30, 2023 at 11:13 AM Henrik Boström <h...@chromium.org> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Friday, January 27, 2023 at 7:24:58 PM UTC+1 Johnny Stenback >>>>>>> wrote: >>>>>>> Is there an enterprise policy in place for this deprecation already? >>>>>>> If not, adding one seems appropriate given the challenges of rolling out >>>>>>> even simple fixes in some enterprise environments. >>>>>>> >>>>>>> One does not exist at the moment but I can add one >>>>>>> <https://chromium.googlesource.com/chromium/src/+/HEAD/docs/enterprise/add_new_policy.md> >>>>>>> . >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Johnny >>>>>>> >>>>>>> On Fri, Jan 27, 2023 at 5:16 AM Henrik Boström <h...@chromium.org> >>>>>>> wrote: >>>>>>> Delaying the enabled-by-default to M112 is fine by me but I'll wait >>>>>>> for a resolution here before taking action. Currently it is >>>>>>> enabled-by-default in Canary. >>>>>>> >>>>>>> On Friday, January 27, 2023 at 12:41:23 PM UTC+1 >>>>>>> philipp...@googlemail.com wrote: >>>>>>> Am Fr., 27. Jan. 2023 um 11:49 Uhr schrieb Henrik Boström < >>>>>>> h...@chromium.org>: >>>>>>> *Contact emails* >>>>>>> h...@chromium.org, h...@chromium.org >>>>>>> >>>>>>> *Background* >>>>>>> I attempted to remove this feature before but had forgotten to file >>>>>>> an intent to deprecate, background here >>>>>>> <https://groups.google.com/a/chromium.org/g/blink-dev/c/RsIktnGhHqw/> >>>>>>> . >>>>>>> >>>>>>> *Specification* >>>>>>> The getStats() API spec is here >>>>>>> <https://w3c.github.io/webrtc-stats/> and it contains all the >>>>>>> metrics. The deprecated metrics are also listed, but in the obsolete >>>>>>> section >>>>>>> <https://w3c.github.io/webrtc-stats/#obsolete-rtcmediastreamtrackstats-members>. >>>>>>> There's an open issue to remove obsolete metrics from the spec as soon >>>>>>> as >>>>>>> they are unshipped from modern browsers. This is considered a blocker >>>>>>> for >>>>>>> the document to reach Proposed Recommendation status. >>>>>>> >>>>>>> *Summary* >>>>>>> WebRTC is a set of JavaScript APIs (spec >>>>>>> <https://w3c.github.io/webrtc-pc/>) that allow real-time >>>>>>> communication between browsers. For the relevant metrics being removed, >>>>>>> we're only talking about the WebRTC use case that is sending or >>>>>>> receiving >>>>>>> audio or video (typically Video Conferencing use cases), not the data >>>>>>> channel use cases that is a popular WebRTC use case, since data channel >>>>>>> only use cases would never have any tracks/streams. >>>>>>> >>>>>>> RTCPeerConnection.getStats() returns a map of string-to-objects, >>>>>>> where each object is one of the dictionaries defined in the stats spec. >>>>>>> The >>>>>>> reason an app calls getStats() is mostly to report quality metrics (send >>>>>>> and receive resolutions, bitrates, glitches, video QP, etc) which can be >>>>>>> important for A/B experimentation. It can also be used in a way that >>>>>>> impacts app logic or even UX inside the app. Most common use case I can >>>>>>> think of: poll getStats() at 10 Hz and render volume bars for each >>>>>>> participant based on volume levels from stats objects. >>>>>>> >>>>>>> The deprecation in question is to remove some stats objects that >>>>>>> were made obsolete several years ago: all metrics on the "track" >>>>>>> dictionary >>>>>>> have been moved to non-obsolete objects ("inbound-rtp", "outbound-rtp", >>>>>>> "media-source"). Reasons for wanting to deprecate include: >>>>>>> >>>>>>> - Spec-compliance: needed for browser implementations to align >>>>>>> and for the spec to become Proposed Recommendation. >>>>>>> - Web compat: Firefox never implement "track" or "steam" >>>>>>> >>>>>>> <https://wpt.fyi/results/webrtc-stats/supported-stats.https.html?label=experimental&label=master&aligned> >>>>>>> due >>>>>>> to them being obsolete. >>>>>>> - Performance: the duplicated metrics make up ~40% of the stats >>>>>>> report size, which can be a significant number of bytes in larger >>>>>>> meetings >>>>>>> and it is common for apps to poll getStats() 10 times per second. >>>>>>> - Tech debt: unblock removal of 1400 LOC. >>>>>>> >>>>>>> In the meantime, the obsolete metrics is duplicated in several >>>>>>> places of the stats report. >>>>>>> >>>>>>> *Risks* >>>>>>> *- Impossible to properly measure usage* >>>>>>> Because stats objects are exposed as JavaScript dictionaries, and >>>>>>> because apps have to iterate all objects of the stats report in order to >>>>>>> find the ones they are interested in or if they just dump all the data >>>>>>> without filtering, there is no way to measure how big the dependency is >>>>>>> on >>>>>>> track in the real world. >>>>>>> >>>>>>> While we lack use counters, we have some positive signs: >>>>>>> >>>>>>> - Because Firefox does not have "track" or "stream" stats, any >>>>>>> app that can run on Firefox already exercises the paths of these not >>>>>>> existing. >>>>>>> >>>>>>> >>>>>>> - An experiment to "unship deprecated metrics" has been running >>>>>>> at 50% Canary since October >>>>>>> >>>>>>> <https://groups.google.com/a/chromium.org/g/blink-dev/c/RsIktnGhHqw/m/3iqjODsMBwAJ>, >>>>>>> giving developers testing Canary a heads-up. Nobody complained until >>>>>>> the >>>>>>> experiment reached Stable. >>>>>>> - We got to 50% Stable in M109 and while we're in the process of >>>>>>> rolling it back now due to breaking twilio-video.js >>>>>>> <https://github.com/twilio/twilio-video.js/issues/1968>, it's >>>>>>> interesting to note that this is the only breakage we are aware of >>>>>>> (that >>>>>>> does not mean there aren't more breakages, but I believe this at >>>>>>> least says >>>>>>> something about the severity). >>>>>>> >>>>>>> *- Selenium et al typically starts browsers from fresh profiles and >>>>>>> hence does not know the finch trial seed* >>>>>>> The most likely explanation for breakage is not testing Canary or >>>>>>> test environments not having access to Finch experiments. This makes the >>>>>>> behavior on Stable a surprise. >>>>>>> >>>>>>> *- To have a Reverse Origin Trial or not to have a Reverse Origin >>>>>>> Trial?* >>>>>>> Migrating should require so few lines of code (look for stats.type >>>>>>> == 'inbound-rtp' instead of stats.type == 'track', for example) that it >>>>>>> seems to be a bigger hurdle for a developer to enroll in the trial than >>>>>>> to >>>>>>> simply fix their code. >>>>>>> >>>>>>> *- Compatiblity risk* >>>>>>> There is one particular metric out of all metrics that, if you run >>>>>>> Safari, does not exist on "inbound-rtp" yet. This can be a problem, but >>>>>>> again is probably not a big problem because this particular metric was >>>>>>> never implemented on Firefox so apps already need to survive without it, >>>>>>> and it is very easy to write a fallback path for the Safari case: >>>>>>> >>>>>>> let trackIdentifer = null; // In Firefox this will never be set >>>>>>> regardless. >>>>>>> if (inboundRtp.trackIdentifier) { >>>>>>> // Spec-compliant browser. >>>>>>> trackIdentifier = inboundRtp.trackIdentifier; >>>>>>> } else if (inboundRtp.trackId) { >>>>>>> // Fallback-path for Safari or 1+ year old Chromium browsers. >>>>>>> trackIdentifier = report.get(inboundRtp.trackId).trackIdentifier; >>>>>>> } >>>>>>> >>>>>>> *Proposal* >>>>>>> Rollback to 0% Stable but keep the "unship deprecated" experiment at >>>>>>> 50% Canary/Beta. Wait for Twilio to fix their issue and do another >>>>>>> rollout >>>>>>> attempt. Keep a slower rollout pace next time. >>>>>>> >>>>>>> I see limited amount of value in a Reverse Origin Trial since it >>>>>>> appears to be more effort to register to the trial than to fix the >>>>>>> issue, >>>>>>> if you are affected. >>>>>>> >>>>>>> I do prefer to have the feature enabled-by-default in M111+ and >>>>>>> overwrite that default via Finch rather than the other way around as to >>>>>>> not >>>>>>> "turn off the fire alarm" for non-Finch testing environments. I realize >>>>>>> that is not perfect (what if you run in a non-Finch environment) but it >>>>>>> would reduce overall risk. >>>>>>> >>>>>>> Thank you Henrik. I agree with one suggestion: only do default-off >>>>>>> in M112+ (which is branching so you would just need to revert this >>>>>>> commit >>>>>>> <https://chromiumdash.appspot.com/commit/3a4d52f365df03413a856ea20366b36e8fb8ea0b> >>>>>>> on >>>>>>> the M111 branch). >>>>>>> This gives developers another month to update (which itself should >>>>>>> be quick) and then rolling it out to their customers and users (which >>>>>>> takes >>>>>>> time). >>>>>>> >>>>>>> I hope that the 50% rollout caused enough incidents (even though you >>>>>>> may never hear about some of them) to get the fixes in ASAP. >>>>>>> >>>>>>> cheers >>>>>>> >>>>>>> Philipp >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "blink-dev" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to blink-dev+unsubscr...@chromium.org. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink- >>>>>>> dev/5ecf1ea6-c16c-464a-b529-439e05e4feedn%40chromium.org >>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/5ecf1ea6-c16c-464a-b529-439e05e4feedn%40chromium.org?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "blink-dev" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to blink-dev+unsubscr...@chromium.org. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink- >>>>>>> dev/6a0b5bb9-addd-4a56-b053-1429bbaabe2dn%40chromium.org >>>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/6a0b5bb9-addd-4a56-b053-1429bbaabe2dn%40chromium.org?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "blink-dev" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to blink-dev+unsubscr...@chromium.org. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOqqYVG_P0ji6Z-n0DoCua4nzJZN7S48o23%3DUpx_ELerqphfUg%40mail.gmail.com >>>>>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOqqYVG_P0ji6Z-n0DoCua4nzJZN7S48o23%3DUpx_ELerqphfUg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> >>>>>> >>>>> -- >> You received this message because you are subscribed to the Google Groups >> "blink-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to blink-dev+unsubscr...@chromium.org. >> To view this discussion on the web visit >> https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACZRgz4SF%2BGDjoD8S%3DfYHgLYX6PxeTUj4hd%3DA1%2BS%2BeBD2ygtpQ%40mail.gmail.com >> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACZRgz4SF%2BGDjoD8S%3DfYHgLYX6PxeTUj4hd%3DA1%2BS%2BeBD2ygtpQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "blink-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to blink-dev+unsubscr...@chromium.org. > To view this discussion on the web visit > https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYcQdCft3PrYCEs93V35pEb%3DZy6%2BjfYNwOAojNPYdtzhKw%40mail.gmail.com > <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYcQdCft3PrYCEs93V35pEb%3DZy6%2BjfYNwOAojNPYdtzhKw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFUtAY8ZN81y53tuMZpdeDsPg-F69nNqu6GCOV0ZmFuYs%3DTj0Q%40mail.gmail.com.
