Contact emails h...@chromium.org, o...@chromium.org, h...@chromium.org Specification https://w3c.github.io/mediacapture-extensions/#the-mediastreamtrackaudiostats-interface
Summary The MediaStreamTrack Statistics API, or `track.stats`, has already shipped for video tracks: see previous I2S here <https://groups.google.com/a/chromium.org/g/blink-dev/c/ttzYv-30gY4/m/2FvJpxqMGQAJ> . This is the same API but for audio tracks, also motivated by the app's desire to measure capture quality. This is very important for conferencing websites such as Google Meet, Microsoft Teams, Zoom, Goto Meetings, etc. All of which has expressed an interest in the audio portion of this API. The API is only available for getUserMedia() sourced audio tracks, i.e. microphone, so the API is behind a user prompt and only available during capture. The new interface we want to ship is MediaStreamTrackAudioStats <https://w3c.github.io/mediacapture-extensions/#the-mediastreamtrackaudiostats-interface> which allow measuring two things from the audio capture pipeline: 1. The number of audio frames, including if any audio frames are dropped by the device, OS or User Agent. This allows measuring glitches in captured audio. 2. The input latency, such as due to buffering or other delays in delivering the audio frames to the track's sinks. Blink component Blink>GetUserMedia <https://bugs.chromium.org/p/chromium/issues/list?q=component:Blink%3EGetUserMedia> Risks Interoperability and Compatibility Because the API provides *statistics* about capture quality, rather than providing capture *functionality, *the interop/compatibility risk is small. *Gecko*: Standards position issue <https://github.com/mozilla/standards-positions/issues/935> *WebKit*: Standards position issue <https://github.com/WebKit/standards-positions/issues/260> Standardization While the audio stats API is written by the W3C WebRTC Working Group and track statistics overall is not controversial, there is an ongoing disagreement with Mozilla about whether or not dropped frames (= totalFrames - deliveredFrames) should be exposed to the web in the audio case. The disagreement is tracked by this issue <https://github.com/w3c/mediacapture-extensions/issues/129>. Our need for this metric has been discussed at Virtual Interims, see recording with 10:39 timestamp <https://www.youtube.com/watch?v=xJMXnf3Qwh8&t=639s> but no consensus could be reached (rough consensus was reached between everyone except Mozilla). Youenn (Apple) has shown that frames can be dropped due to Bluetooth connection <https://github.com/w3c/mediacapture-extensions/issues/129#issuecomment-1822624904> and is not just "measuring browser bugs". *Web developers*: Positive - The I2P <https://groups.google.com/a/chromium.org/g/blink-dev/c/vUbD_psbPL8/m/wqq3kmZFBwAJ> shows support from Teams, Zoom and GoTo meetings. - In the spec issue <https://github.com/w3c/mediacapture-extensions/issues/129> regarding the disagreement, more developer support is expressed (e.g. alfredh from Nvidia and steely-glint). WebView application risks None Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)? Yes Is this feature fully tested by web-platform-tests <https://chromium.googlesource.com/chromium/src/+/main/docs/testing/web_platform_tests.md> ? Yes and WPTs will be written as part of implementing this, however unit tests will also be needed to verify accuracy of metrics on a lower level. WPTs will be more general like "frame counters increase over time during capture" and that run-to-completion semantics are preserved. Link to entry on the Chrome Platform Status https://chromestatus.com/feature/5141112910249984 Links to previous Intent discussions Intent to prototype: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/bb6c1af3-9eb3-4c6f-a136-dee709b7f906n%40chromium.org -- You received this message because you are subscribed to the Google Groups "blink-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscr...@chromium.org. To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/a594fc55-8030-4848-9fe6-549eccfdd8a8n%40chromium.org.
