LGTM2 to ship, with a commitment to move this part of the spec to WICG if it gets removed from the mediacapture-extensions spec.
On Thu, Dec 14, 2023 at 11:06 AM Alex Russell <slightly...@chromium.org> wrote: > LGTM1 > > On Thursday, December 14, 2023 at 8:00:41 AM UTC-8 Rick Byers wrote: > >> Sounds good Henrik! Yes, from our brief discussion in the API owners >> meeting I believe you have support from at least 3 API owners to proceed in >> this direction. It's important to us that we engage constructively in the >> standards process even in areas of differing opinion, but it's also crucial >> to our process that we not let such differences in opinion and priority be >> an effective veto power over what we choose to ship in Chromium. I'd >> encourage you and the WebRTC group to formalize processes for amicably >> agreeing to disagree on the importance of different use cases, while still >> being open to technical feedback and doing what we reasonably can to >> maximize the chance that the APIs we ship can eventually be interoperable >> if priorities of the other engines someday change [*]. API owners would >> also be interested to hear any other arguments for why Chromium shipping >> these APIs would be bad for the web (on this list, or anywhere else). I >> know there's a messy history with WebRTC in particular and services coming >> to depend on Chromium-only APIs when suitable standards-track alternatives >> are available in other engines. That's IMHO definitely not a pattern we >> want to risk repeating. >> >> Of course you also need Chrome privacy and security reviews, since it's >> important that features like this don't create a hole in our careful >> balance of side channel attack mitigations. But I see you already have >> privacy approval so hopefully security isn't too far off. You might want to >> wait for a signal there before starting implementation. >> >> Personally I'm also less concerned about interoperability risks when it >> comes to metrics API. It's already the case that our top >> performance metrics (Core Web Vitals) have APIs exposed only in Chromium. >> There's certainly some interop risk there, eg. of sites optimizing in >> engine-specific ways. But in practice we've seen developers use these APIs >> mostly to make their sites faster in ways that generally apply to all >> engines. So in that case, Safari and Firefox are getting most of the >> benefit of these APIs existing without having to incur most of the cost, >> which seems like a fine outcome to me. Also, I'm confident that if we >> eventually agree with other engines on some better way to expose the same >> information, then we can deprecate and remove any API shape we ship today >> and customers can migrate over without causing user-visible breakage >> (worst-case we just return dummy values on the deprecated APIs). >> >> Rick >> >> [*] My favorite example of this is Pointer Events where Apple was opposed >> to the use case, but also had good technical critiques of the API. We >> eventually (after a lot of research, open debate, and some flip-flopping by >> me) shipped a version of the API that addressed the legitimate technical >> concerns without addressing Apple's objections around the use case. Years >> later when a use case materialized for Apple (the Apple Pen), they just >> shipped the API in a fully interoperable fashion. That (as well as cases of >> the inverse where we realize we were wrong and unship) is what we mean by >> the blink process being designed for "eventual interoperability". >> >> >> On Thu, Dec 14, 2023 at 4:34 AM Henrik Boström <h...@chromium.org> wrote: >> >>> Thanks, Rick. Responses inline. >>> >>> On Wednesday, December 13, 2023 at 6:39:48 PM UTC+1 Rick Byers wrote: >>> >>> I looked into the details of the standards debate on this issue. It >>> sounds like it's still unclear whether the spec for this has WG support or >>> not, right? I certainly wouldn't want to mislead anyone as to API maturity >>> / likely interoperability by shipping based on a WebRTC WG specification if >>> there is an unresolved process concern. >>> >>> >>> I think it's safe to say we don't have consensus on the frame counters >>> (exposing dropped/glitches), while capture latency hopefully be less >>> contentious. >>> >>> >>> That said, I think Chromium's position on the technical debate here is >>> pretty clear - we do believe there's value in such stats APIs, even IF they >>> can only represent browser bugs (it's why we ship a crash reporting API >>> <https://wicg.github.io/crash-reporting/>, which has been similarly >>> controversial with Mozilla and Apple). We disagree that "there's nothing >>> web developers can do about it". Maybe that's true for Apple and Mozilla, >>> but for Google we rely critically on developer feedback through both open ( >>> crbug.com) and private (Google partnerships) channels and we want to >>> make it as easy as possible for developers to report an issue they're >>> seeing to us in a way that's actionable. Our privacy policy limits Chrome's >>> visibility into what's going on in the wild. So usually we find this >>> requires a mix of both site-acquired telemetry and browser-required >>> telemetry, and find the two can often complement each other nicely. >>> >>> Henrik, my advice if the WG doesn't have consensus for this API is to >>> move it to some incubation venue (like a WICG group). You clearly have a >>> community of web developers who want it, so it's probably more productive >>> to focus standards energy among allies who share an interest for the use >>> case, right? If you're willing to promptly move this to a WICG spec in the >>> event the WG asks to remove it from their spec, then I don't think this >>> debate changes anything from a blink API owner's perspective so I'm OK >>> treating it as non-blocking. A subset of API owners met today (Daniel, Mike >>> Taylor, Philip and I), and agreed with this stance. WDYT? >>> >>> >>> Me moving this to a WICG spec in the event that the WG asks to remove >>> them from mediacapture-extensions sounds good to me. >>> If me and/or the WebRTC Audio Team has access to a WICG spec for these >>> things, that may also give us a venue for, in the future, exploring >>> *playout* glitch metrics in a more enthusiastic setting, which is in >>> the early stages of discussions internally. >>> >>> >>> In terms of testing, normally we ask to see the tests land before >>> approving an I2S. Any reason we shouldn't wait for that here? >>> >>> >>> Given the "controversy" around the glitch metrics, the WebRTC Audio Team >>> wanted to get some Blink owners signals before they spend the engineering >>> efforts to implement this (including WPTs). >>> But if Blink owners also see the value in these metrics and we have a >>> plan (= move to WICG in the event that they are removed) I see no reason >>> not to ask them to start implementation today. >>> >>> For reference, see the WPTs we added for the video track stats here >>> <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/external/wpt/mediacapture-extensions/MediaStreamTrack-video-stats.https.html>. >>> The audio track stats WPTs should be similar and also complemented by C++ >>> unit tests in lower layers. >>> >>> >>> Thanks, >>> Rick >>> >>> >>> >>> On Fri, Dec 8, 2023 at 11:53 AM Henrik Boström <h...@chromium.org> >>> wrote: >>> >>> Contact emails >>> h...@chromium.org, o...@chromium.org, h...@chromium.org >>> >>> Specification >>> https://w3c.github.io/mediacapture-extensions/#the- >>> mediastreamtrackaudiostats-interface >>> >>> Summary >>> >>> The MediaStreamTrack Statistics API, or `track.stats`, has already >>> shipped for video tracks: see previous I2S here >>> <https://groups.google.com/a/chromium.org/g/blink-dev/c/ttzYv-30gY4/m/2FvJpxqMGQAJ> >>> . >>> >>> >>> This is the same API but for audio tracks, also motivated by the app's >>> desire to measure capture quality. This is very important for conferencing >>> websites such as Google Meet, Microsoft Teams, Zoom, Goto Meetings, etc. >>> All of which has expressed an interest in the audio portion of this API. >>> >>> >>> The API is only available for getUserMedia() sourced audio tracks, i.e. >>> microphone, so the API is behind a user prompt and only available during >>> capture. >>> >>> >>> The new interface we want to ship is MediaStreamTrackAudioStats >>> <https://w3c.github.io/mediacapture-extensions/#the-mediastreamtrackaudiostats-interface> >>> which allow measuring two things from the audio capture pipeline: >>> >>> 1. The number of audio frames, including if any audio frames are dropped >>> by the device, OS or User Agent. This allows measuring glitches in captured >>> audio. >>> >>> 2. The input latency, such as due to buffering or other delays in >>> delivering the audio frames to the track's sinks. >>> >>> Blink component >>> Blink>GetUserMedia >>> <https://bugs.chromium.org/p/chromium/issues/list?q=component:Blink%3EGetUserMedia> >>> >>> Risks >>> Interoperability and Compatibility >>> >>> Because the API provides *statistics* about capture quality, rather >>> than providing capture *functionality, *the interop/compatibility risk >>> is small. >>> >>> *Gecko*: Standards position issue >>> <https://github.com/mozilla/standards-positions/issues/935> >>> *WebKit*: Standards position issue >>> <https://github.com/WebKit/standards-positions/issues/260> >>> >>> Standardization >>> While the audio stats API is written by the W3C WebRTC Working Group and >>> track statistics overall is not controversial, there is an ongoing >>> disagreement with Mozilla about whether or not dropped frames (= >>> totalFrames - deliveredFrames) should be exposed to the web in the audio >>> case. The disagreement is tracked by this issue >>> <https://github.com/w3c/mediacapture-extensions/issues/129>. Our need >>> for this metric has been discussed at Virtual Interims, see recording >>> with 10:39 timestamp >>> <https://www.youtube.com/watch?v=xJMXnf3Qwh8&t=639s> but no consensus >>> could be reached (rough consensus was reached between everyone except >>> Mozilla). Youenn (Apple) has shown that frames can be dropped due to >>> Bluetooth connection >>> <https://github.com/w3c/mediacapture-extensions/issues/129#issuecomment-1822624904> >>> and >>> is not just "measuring browser bugs". >>> >>> *Web developers*: Positive >>> - The I2P >>> <https://groups.google.com/a/chromium.org/g/blink-dev/c/vUbD_psbPL8/m/wqq3kmZFBwAJ> >>> shows >>> support from Teams, Zoom and GoTo meetings. >>> - In the spec issue >>> <https://github.com/w3c/mediacapture-extensions/issues/129> regarding >>> the disagreement, more developer support is expressed (e.g. alfredh from >>> Nvidia and steely-glint). >>> >>> WebView application risks >>> >>> None >>> >>> Will this feature be supported on all six Blink platforms (Windows, Mac, >>> Linux, ChromeOS, Android, and Android WebView)? >>> Yes >>> >>> Is this feature fully tested by web-platform-tests >>> <https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md> >>> ? >>> Yes and WPTs will be written as part of implementing this, however unit >>> tests will also be needed to verify accuracy of metrics on a lower level. >>> WPTs will be more general like "frame counters increase over time during >>> capture" and that run-to-completion semantics are preserved. >>> >>> Link to entry on the Chrome Platform Status >>> https://chromestatus.com/feature/5141112910249984 >>> >>> Links to previous Intent discussions >>> Intent to prototype: >>> https://groups.google.com/a/chromium.org/d/msgid/blink- >>> dev/bb6c1af3-9eb3-4c6f-a136-dee709b7f906n%40chromium.org >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "blink-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to blink-dev+unsubscr...@chromium.org. >>> To view this discussion on the web visit https://groups.google.com/a/ >>> chromium.org/d/msgid/blink-dev/a594fc55-8030-4848-9fe6- >>> 549eccfdd8a8n%40chromium.org >>> <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/a594fc55-8030-4848-9fe6-549eccfdd8a8n%40chromium.org?utm_medium=email&utm_source=footer> >>> . >>> >>> -- > You received this message because you are subscribed to the Google Groups > "blink-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to blink-dev+unsubscr...@chromium.org. > To view this discussion on the web visit > https://groups.google.com/a/chromium.org/d/msgid/blink-dev/2e451af1-5e0a-4dd9-a12c-fc30c7dff7dbn%40chromium.org > <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/2e451af1-5e0a-4dd9-a12c-fc30c7dff7dbn%40chromium.org?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw-HbWWNeZ4jyRr38WR75FtxOA2DUhkwrHH0-OZnjkq9cg%40mail.gmail.com.
