Thanks, Rick. Responses inline. On Wednesday, December 13, 2023 at 6:39:48 PM UTC+1 Rick Byers wrote:
I looked into the details of the standards debate on this issue. It sounds like it's still unclear whether the spec for this has WG support or not, right? I certainly wouldn't want to mislead anyone as to API maturity / likely interoperability by shipping based on a WebRTC WG specification if there is an unresolved process concern. I think it's safe to say we don't have consensus on the frame counters (exposing dropped/glitches), while capture latency hopefully be less contentious. That said, I think Chromium's position on the technical debate here is pretty clear - we do believe there's value in such stats APIs, even IF they can only represent browser bugs (it's why we ship a crash reporting API <https://wicg.github.io/crash-reporting/>, which has been similarly controversial with Mozilla and Apple). We disagree that "there's nothing web developers can do about it". Maybe that's true for Apple and Mozilla, but for Google we rely critically on developer feedback through both open ( crbug.com) and private (Google partnerships) channels and we want to make it as easy as possible for developers to report an issue they're seeing to us in a way that's actionable. Our privacy policy limits Chrome's visibility into what's going on in the wild. So usually we find this requires a mix of both site-acquired telemetry and browser-required telemetry, and find the two can often complement each other nicely. Henrik, my advice if the WG doesn't have consensus for this API is to move it to some incubation venue (like a WICG group). You clearly have a community of web developers who want it, so it's probably more productive to focus standards energy among allies who share an interest for the use case, right? If you're willing to promptly move this to a WICG spec in the event the WG asks to remove it from their spec, then I don't think this debate changes anything from a blink API owner's perspective so I'm OK treating it as non-blocking. A subset of API owners met today (Daniel, Mike Taylor, Philip and I), and agreed with this stance. WDYT? Me moving this to a WICG spec in the event that the WG asks to remove them from mediacapture-extensions sounds good to me. If me and/or the WebRTC Audio Team has access to a WICG spec for these things, that may also give us a venue for, in the future, exploring *playout* glitch metrics in a more enthusiastic setting, which is in the early stages of discussions internally. In terms of testing, normally we ask to see the tests land before approving an I2S. Any reason we shouldn't wait for that here? Given the "controversy" around the glitch metrics, the WebRTC Audio Team wanted to get some Blink owners signals before they spend the engineering efforts to implement this (including WPTs). But if Blink owners also see the value in these metrics and we have a plan (= move to WICG in the event that they are removed) I see no reason not to ask them to start implementation today. For reference, see the WPTs we added for the video track stats here <https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/external/wpt/mediacapture-extensions/MediaStreamTrack-video-stats.https.html>. The audio track stats WPTs should be similar and also complemented by C++ unit tests in lower layers. Thanks, Rick On Fri, Dec 8, 2023 at 11:53 AM Henrik Boström <h...@chromium.org> wrote: Contact emails h...@chromium.org, o...@chromium.org, h...@chromium.org Specification https://w3c.github.io/mediacapture-extensions/#the- mediastreamtrackaudiostats-interface Summary The MediaStreamTrack Statistics API, or `track.stats`, has already shipped for video tracks: see previous I2S here <https://groups.google.com/a/chromium.org/g/blink-dev/c/ttzYv-30gY4/m/2FvJpxqMGQAJ> . This is the same API but for audio tracks, also motivated by the app's desire to measure capture quality. This is very important for conferencing websites such as Google Meet, Microsoft Teams, Zoom, Goto Meetings, etc. All of which has expressed an interest in the audio portion of this API. The API is only available for getUserMedia() sourced audio tracks, i.e. microphone, so the API is behind a user prompt and only available during capture. The new interface we want to ship is MediaStreamTrackAudioStats <https://w3c.github.io/mediacapture-extensions/#the-mediastreamtrackaudiostats-interface> which allow measuring two things from the audio capture pipeline: 1. The number of audio frames, including if any audio frames are dropped by the device, OS or User Agent. This allows measuring glitches in captured audio. 2. The input latency, such as due to buffering or other delays in delivering the audio frames to the track's sinks. Blink component Blink>GetUserMedia <https://bugs.chromium.org/p/chromium/issues/list?q=component:Blink%3EGetUserMedia> Risks Interoperability and Compatibility Because the API provides *statistics* about capture quality, rather than providing capture *functionality, *the interop/compatibility risk is small. *Gecko*: Standards position issue <https://github.com/mozilla/standards-positions/issues/935> *WebKit*: Standards position issue <https://github.com/WebKit/standards-positions/issues/260> Standardization While the audio stats API is written by the W3C WebRTC Working Group and track statistics overall is not controversial, there is an ongoing disagreement with Mozilla about whether or not dropped frames (= totalFrames - deliveredFrames) should be exposed to the web in the audio case. The disagreement is tracked by this issue <https://github.com/w3c/mediacapture-extensions/issues/129>. Our need for this metric has been discussed at Virtual Interims, see recording with 10:39 timestamp <https://www.youtube.com/watch?v=xJMXnf3Qwh8&t=639s> but no consensus could be reached (rough consensus was reached between everyone except Mozilla). Youenn (Apple) has shown that frames can be dropped due to Bluetooth connection <https://github.com/w3c/mediacapture-extensions/issues/129#issuecomment-1822624904> and is not just "measuring browser bugs". *Web developers*: Positive - The I2P <https://groups.google.com/a/chromium.org/g/blink-dev/c/vUbD_psbPL8/m/wqq3kmZFBwAJ> shows support from Teams, Zoom and GoTo meetings. - In the spec issue <https://github.com/w3c/mediacapture-extensions/issues/129> regarding the disagreement, more developer support is expressed (e.g. alfredh from Nvidia and steely-glint). WebView application risks None Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)? Yes Is this feature fully tested by web-platform-tests <https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md> ? Yes and WPTs will be written as part of implementing this, however unit tests will also be needed to verify accuracy of metrics on a lower level. WPTs will be more general like "frame counters increase over time during capture" and that run-to-completion semantics are preserved. Link to entry on the Chrome Platform Status https://chromestatus.com/feature/5141112910249984 Links to previous Intent discussions Intent to prototype: https://groups.google.com/a/chromium.org/d/msgid/blink- dev/bb6c1af3-9eb3-4c6f-a136-dee709b7f906n%40chromium.org -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion on the web visit https://groups.google.com/a/ chromium.org/d/msgid/blink-dev/a594fc55-8030-4848-9fe6- 549eccfdd8a8n%40chromium.org <https://groups.google.com/a/chromium.org/d/msgid/blink-dev/a594fc55-8030-4848-9fe6-549eccfdd8a8n%40chromium.org?utm_medium=email&utm_source=footer> . -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0ed3ec56-02a3-4284-86ec-0297cea0f68cn%40chromium.org.
