On Mon, 23 Sep 2024, Cameron Gutman wrote:
On Mon, Sep 23, 2024 at 12:43 PM Zhao Zhili <quinkbl...@foxmail.com> wrote:
> On Sep 24, 2024, at 01:24, Cameron Gutman <aicomman...@gmail.com> wrote:
>
> On Mon, Sep 23, 2024 at 6:07 AM Zhao Zhili <quinkbl...@foxmail.com> wrote:
>>
>>
>>
>>> On Sep 21, 2024, at 05:39, Martin Storsjö <mar...@martin.st> wrote:
>>>
>>> From: Jan Ekström <jee...@gmail.com>
>>>
>>> Co-authored-by: Ruslan Chernenko <ractyf...@gmail.com>
>>> Co-authored-by: Martin Storsjö <mar...@martin.st>
>>> ---
>>> This is a touched up version of Jan and Ruslan's patches for
>>> AV1 hwaccel via videotoolbox; I tried to polish the code a little
>>> by not overwriting avctx->extradata in
>>> ff_videotoolbox_av1c_extradata_create, and by factorizing out a
>>> new function ff_videotoolbox_buffer_append.
>>
>> LGTM, although I don’t have a device with AV1 support.
>
> I've asked for some testing from users with M3 MacBooks and it
> appears to have problems with certain resolutions (notably 4K).
>
> https://github.com/moonlight-stream/moonlight-qt/issues/1125
>
> It's possible this is a Moonlight bug, but that seems unlikely
> because VideoToolbox HEVC decoding works fine at 4K and
> VideoToolbox AV1 works at 1080p and other resolutions.
I can’t tell what’s going wrong from that bug report. Please test
with ffmpeg and/or ffplay cmdline and share the results.
I'm debugging this blind since I don't have hardware either, but I think
we're mishandling Tile Group OBUs in this patch.
Comparing working vs non-working logs, it looks like the encoder is using
2x1 tiling when encoding 4K and 1x1 for smaller unaffected resolutions.
Working:
[av1 @ 0x14f7b14c0] Frame 0: size 1280x720 upscaled 1280 render
1280x720 subsample 2x2 bitdepth 10 tiles 1x1.
[av1 @ 0x14f7b14c0] Total OBUs on this packet: 4.
[av1 @ 0x14f7b14c0] OBU idx:0, type:2, content available:1.
[av1 @ 0x14f7b14c0] OBU idx:1, type:1, content available:1.
[av1 @ 0x14f7b14c0] OBU idx:2, type:6, content available:1.
[av1 @ 0x14f7b14c0] Format videotoolbox_vld chosen by get_format().
[av1 @ 0x14f7b14c0] Format videotoolbox_vld requires hwaccel
av1_videotoolbox initialisation.
[av1 @ 0x14f7b14c0] AV1 decode get format: videotoolbox_vld.
Broken:
[av1 @ 0x15128b530] Frame 0: size 3840x2160 upscaled 3840 render
3840x2160 subsample 2x2 bitdepth 10 tiles 2x1.
[av1 @ 0x15128b530] Total OBUs on this packet: 4.
[av1 @ 0x15128b530] OBU idx:0, type:2, content available:1.
[av1 @ 0x15128b530] OBU idx:1, type:1, content available:1.
[av1 @ 0x15128b530] OBU idx:2, type:3, content available:1.
[av1 @ 0x15128b530] Format videotoolbox_vld chosen by get_format().
[av1 @ 0x15128b530] Format videotoolbox_vld requires hwaccel
av1_videotoolbox initialisation.
[av1 @ 0x15128b530] AV1 decode get format: videotoolbox_vld.
[av1 @ 0x15128b530] OBU idx:3, type:4, content available:1.
[av1 @ 0x15128b530] vt decoder cb: output image buffer is null: -17694
[av1 @ 0x15128b530] HW accel end frame fail.
In the broken case, instead of a Frame OBU, we get a Frame Header OBU and
a Tile Group OBU. To handle Tile Group OBUs, av1dec.c calls decode_slice()
function, but videotoolbox_av1_decode_slice() in this patch simply returns
without appending the OBU data to bitstream buffer.
It looks like other AV1 hwaccels ignore the data buffer provided in the
start_frame() callback and instead append to their bitstream buffers in
decode_slice() instead. Maybe that's what we should do here too?
That sounds plausible.
I tried a modification like this on top:
diff --git a/libavcodec/videotoolbox_av1.c b/libavcodec/videotoolbox_av1.c
index 9ccf65bb25..5fb1d06ddb 100644
--- a/libavcodec/videotoolbox_av1.c
+++ b/libavcodec/videotoolbox_av1.c
@@ -74,15 +74,15 @@ static int videotoolbox_av1_start_frame(AVCodecContext
*avctx,
const uint8_t *buffer,
uint32_t size)
{
- VTContext *vtctx = avctx->internal->hwaccel_priv_data;
- return ff_videotoolbox_buffer_append(vtctx, buffer, size);
+ return 0;
}
static int videotoolbox_av1_decode_slice(AVCodecContext *avctx,
const uint8_t *buffer,
uint32_t size)
{
- return 0;
+ VTContext *vtctx = avctx->internal->hwaccel_priv_data;
+ return ff_videotoolbox_buffer_append(vtctx, buffer, size);
}
static int videotoolbox_av1_end_frame(AVCodecContext *avctx)
Unfortunately this isn't quite enough to make it work - we probably need
to add some extra framing around the data we get in decode_slice. (The
data we get in decode_slice is around 20 bytes shorter than what we get in
start_frame.)
I don't hit any issues with any AV1 samples that I have, I guess I don't
have any samples with tile groups?
Can you or someone else grab and share a small sample of a stream that
fails to decode with this hwaccel, so we have a chance to debug it?
// Martin
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
