On Tue, Sep 24, 2024 at 7:16 AM Martin Storsjö <mar...@martin.st> wrote: > > On Mon, 23 Sep 2024, Cameron Gutman wrote: > > > On Mon, Sep 23, 2024 at 12:43 PM Zhao Zhili <quinkbl...@foxmail.com> wrote: > >> > >> > >> > >> > On Sep 24, 2024, at 01:24, Cameron Gutman <aicomman...@gmail.com> wrote: > >> > > >> > On Mon, Sep 23, 2024 at 6:07 AM Zhao Zhili <quinkbl...@foxmail.com> > >> > wrote: > >> >> > >> >> > >> >> > >> >>> On Sep 21, 2024, at 05:39, Martin Storsjö <mar...@martin.st> wrote: > >> >>> > >> >>> From: Jan Ekström <jee...@gmail.com> > >> >>> > >> >>> Co-authored-by: Ruslan Chernenko <ractyf...@gmail.com> > >> >>> Co-authored-by: Martin Storsjö <mar...@martin.st> > >> >>> --- > >> >>> This is a touched up version of Jan and Ruslan's patches for > >> >>> AV1 hwaccel via videotoolbox; I tried to polish the code a little > >> >>> by not overwriting avctx->extradata in > >> >>> ff_videotoolbox_av1c_extradata_create, and by factorizing out a > >> >>> new function ff_videotoolbox_buffer_append. > >> >> > >> >> LGTM, although I don’t have a device with AV1 support. > >> > > >> > I've asked for some testing from users with M3 MacBooks and it > >> > appears to have problems with certain resolutions (notably 4K). > >> > > >> > https://github.com/moonlight-stream/moonlight-qt/issues/1125 > >> > > >> > It's possible this is a Moonlight bug, but that seems unlikely > >> > because VideoToolbox HEVC decoding works fine at 4K and > >> > VideoToolbox AV1 works at 1080p and other resolutions. > >> > >> I can’t tell what’s going wrong from that bug report. Please test > >> with ffmpeg and/or ffplay cmdline and share the results. > >> > > > > I'm debugging this blind since I don't have hardware either, but I think > > we're mishandling Tile Group OBUs in this patch. > > > > Comparing working vs non-working logs, it looks like the encoder is using > > 2x1 tiling when encoding 4K and 1x1 for smaller unaffected resolutions. > > > > Working: > > [av1 @ 0x14f7b14c0] Frame 0: size 1280x720 upscaled 1280 render > > 1280x720 subsample 2x2 bitdepth 10 tiles 1x1. > > [av1 @ 0x14f7b14c0] Total OBUs on this packet: 4. > > [av1 @ 0x14f7b14c0] OBU idx:0, type:2, content available:1. > > [av1 @ 0x14f7b14c0] OBU idx:1, type:1, content available:1. > > [av1 @ 0x14f7b14c0] OBU idx:2, type:6, content available:1. > > [av1 @ 0x14f7b14c0] Format videotoolbox_vld chosen by get_format(). > > [av1 @ 0x14f7b14c0] Format videotoolbox_vld requires hwaccel > > av1_videotoolbox initialisation. > > [av1 @ 0x14f7b14c0] AV1 decode get format: videotoolbox_vld. > > > > Broken: > > [av1 @ 0x15128b530] Frame 0: size 3840x2160 upscaled 3840 render > > 3840x2160 subsample 2x2 bitdepth 10 tiles 2x1. > > [av1 @ 0x15128b530] Total OBUs on this packet: 4. > > [av1 @ 0x15128b530] OBU idx:0, type:2, content available:1. > > [av1 @ 0x15128b530] OBU idx:1, type:1, content available:1. > > [av1 @ 0x15128b530] OBU idx:2, type:3, content available:1. > > [av1 @ 0x15128b530] Format videotoolbox_vld chosen by get_format(). > > [av1 @ 0x15128b530] Format videotoolbox_vld requires hwaccel > > av1_videotoolbox initialisation. > > [av1 @ 0x15128b530] AV1 decode get format: videotoolbox_vld. > > [av1 @ 0x15128b530] OBU idx:3, type:4, content available:1. > > [av1 @ 0x15128b530] vt decoder cb: output image buffer is null: -17694 > > [av1 @ 0x15128b530] HW accel end frame fail. > > > > In the broken case, instead of a Frame OBU, we get a Frame Header OBU and > > a Tile Group OBU. To handle Tile Group OBUs, av1dec.c calls decode_slice() > > function, but videotoolbox_av1_decode_slice() in this patch simply returns > > without appending the OBU data to bitstream buffer. > > > > It looks like other AV1 hwaccels ignore the data buffer provided in the > > start_frame() callback and instead append to their bitstream buffers in > > decode_slice() instead. Maybe that's what we should do here too? > > That sounds plausible. > > I tried a modification like this on top: > > diff --git a/libavcodec/videotoolbox_av1.c b/libavcodec/videotoolbox_av1.c > index 9ccf65bb25..5fb1d06ddb 100644 > --- a/libavcodec/videotoolbox_av1.c > +++ b/libavcodec/videotoolbox_av1.c > @@ -74,15 +74,15 @@ static int videotoolbox_av1_start_frame(AVCodecContext > *avctx, > const uint8_t *buffer, > uint32_t size) > { > - VTContext *vtctx = avctx->internal->hwaccel_priv_data; > - return ff_videotoolbox_buffer_append(vtctx, buffer, size); > + return 0; > } > > static int videotoolbox_av1_decode_slice(AVCodecContext *avctx, > const uint8_t *buffer, > uint32_t size) > { > - return 0; > + VTContext *vtctx = avctx->internal->hwaccel_priv_data; > + return ff_videotoolbox_buffer_append(vtctx, buffer, size); > } > > static int videotoolbox_av1_end_frame(AVCodecContext *avctx) > > > Unfortunately this isn't quite enough to make it work - we probably need > to add some extra framing around the data we get in decode_slice. (The > data we get in decode_slice is around 20 bytes shorter than what we get in > start_frame.) > > I don't hit any issues with any AV1 samples that I have, I guess I don't > have any samples with tile groups? > > Can you or someone else grab and share a small sample of a stream that > fails to decode with this hwaccel, so we have a chance to debug it? >
Sure, here's a raw AV1 bitstream sample that should exhibit the issue: https://drive.google.com/file/d/1rp_O6pedhBYhDWFRuBCGTG1tfpNvtgrR/view _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
