(resent due to not adding mail-list into the CC) Hey there! Thanks for checking out the patch;
As for videotoolbox_av1_start_frame: For the av_fast_realloc call it's mostly the same thing as ff_videotoolbox_buffer_copy defined at libavcodec/videotoolbox.c:79. It takes a pointer to a bitstream, its allocated size and updates it. Also there is a videotoolbox_common_decode_slice defined at libavcodec/ videotoolbox:434, which does a pretty similar job to the one I mentioned, but a little different. As for videotoolbox_av1_decode_params: vtctx->bitstream at that moment shall be empty at that point, so does the vtctx->bitsream_size And as it turned out the sequence header obu needs to be copied into the bitstream for videotoolbox to decode a frame as mentioned in the thread on the apple forums(check the last but one comment)[1] So basically what's happening here: videotoolbox_av1_decode_params getting called the first in that decoding loop at libavcodec/av1dec.c:1243, it copies the sequence header obu into the bitstream as the first bytes(because at that point bitstream shall be zero), and increases the vtctx->bitstream_size by the size of the sequence header obu. Then videotoolbox_av1_start_frame is getting called at libavcodec/av1dec.c:1364 and copies the frame itself into the vtctx->bitstream, taking into account the shift needed by the vtctx->bitstream_size(which at that moment should be the size of the sequence header obu). [1] https://forums.developer.apple.com/forums/thread/739953 On Sun, 12 May 2024 at 11:52, Ruslan Chernenko <ractyf...@gmail.com> wrote: > Hey there! > Thanks for checking out the patch; > > As for videotoolbox_av1_start_frame: > For the av_fast_realloc call it's mostly the same thing as > ff_videotoolbox_buffer_copy defined at libavcodec/videotoolbox.c:79. It > takes a pointer to a bitstream, its allocated size and updates it. Also > there is a videotoolbox_common_decode_slice defined at > libavcodec/videotoolbox:434, which does a pretty similar job to the one I > mentioned, but a little different. > > As for videotoolbox_av1_decode_params: > vtctx->bitstream at that moment shall be empty at that point, so does the > vtctx->bitsream_size > And as it turned out the sequence header obu needs to be copied into the > bitstream for videotoolbox to decode a frame as mentioned in the thread on > the apple forums(check the last but one comment)[1] > > So basically what's happening here: > videotoolbox_av1_decode_params getting called the first in that decoding > loop at libavcodec/av1dec.c:1243, it copies the sequence header obu into > the bitstream as the first bytes(because at that point bitstream shall be > zero), and increases the vtctx->bitstream_size by the size of the sequence > header obu. > Then videotoolbox_av1_start_frame is getting called at > libavcodec/av1dec.c:1364 and copies the frame itself into the > vtctx->bitstream, taking into account the shift needed by the > vtctx->bitstream_size(which at that moment should be the size of the > sequence header obu). > > [1] https://forums.developer.apple.com/forums/thread/739953 > > > On Sun, 12 May 2024 at 04:52, Cameron Gutman <aicomman...@gmail.com> > wrote: > >> On Mon, May 6, 2024 at 3:45 PM Черненко Руслан <ractyf...@gmail.com> >> wrote: >> > >> > Signed-off-by: Chernenko Ruslan <ractyf...@gmail.com> >> > --- >> > libavcodec/videotoolbox_av1.c | 102 ++++++++++++++++++++++++---------- >> > 1 file changed, 72 insertions(+), 30 deletions(-) >> > >> > diff --git a/libavcodec/videotoolbox_av1.c >> b/libavcodec/videotoolbox_av1.c >> > index 7f7270c466..736f2548db 100644 >> > --- a/libavcodec/videotoolbox_av1.c >> > +++ b/libavcodec/videotoolbox_av1.c >> > @@ -19,50 +19,90 @@ >> > * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA >> > 02110-1301 USA >> > */ >> > -#include "libavformat/avio.h" >> > -#include "libavformat/avio_internal.h" >> > -#define CALLED_FROM_AVCODEC 1 >> > -#include "libavformat/av1.c" >> > -#undef CALLED_FROM_AVCODEC >> > -#include "libavutil/avutil.h" >> > -#include "libavutil/frame.h" >> > -#include "libavutil/pixfmt.h" >> > +#include "libavutil/mem.h" >> > #include "av1dec.h" >> > -#include "codec_id.h" >> > #include "hwaccel_internal.h" >> > #include "internal.h" >> > #include "vt_internal.h" >> > -CFDataRef ff_videotoolbox_av1c_extradata_create(AVCodecContext >> *avctx) >> > + >> > +CFDataRef ff_videotoolbox_av1c_extradata_create(AVCodecContext *avctx) >> { >> > + >> > + AV1DecContext *s = avctx->priv_data; >> > + avctx->extradata = av_fast_realloc(avctx->extradata, + >> > &avctx->extradata_size, + s->seq_data_ref->size + 4); >> > + avctx->extradata_size = s->seq_data_ref->size + 4; >> > + avctx->extradata[0] = 0x81; // version and marker (constant) >> > + avctx->extradata[1] = s->raw_seq->seq_profile << 5 | >> > s->raw_seq->seq_level_idx[0]; >> > + avctx->extradata[2] = s->raw_seq->seq_tier[0] << 7 | >> > + s->raw_seq->color_config.high_bitdepth << 6 | + >> > s->raw_seq->color_config.twelve_bit << 5 | >> > + s->raw_seq->color_config.mono_chrome << 4 | >> > + s->raw_seq->color_config.subsampling_x << 3 | >> > + s->raw_seq->color_config.subsampling_y << 2 | >> > + >> s->raw_seq->color_config.chroma_sample_position; >> > + + >> > + if (s->raw_seq->initial_display_delay_present_flag) + >> > avctx->extradata[3] = 0 << 5 | >> > s->raw_seq->initial_display_delay_present_flag << 4 | >> > s->raw_seq->initial_display_delay_minus_1[0]; >> > + else >> > + avctx->extradata[3] = 0x00; >> > + memcpy(avctx->extradata + 4, s->seq_data_ref->data, >> > s->seq_data_ref->size); >> > + CFDataRef data = CFDataCreate(kCFAllocatorDefault, >> > avctx->extradata, avctx->extradata_size); >> > + return data; >> > +}; >> > + >> > + >> > +static int videotoolbox_av1_decode_params(AVCodecContext *avctx, >> > + int header_type, >> > + const uint8_t *buffer, >> > + uint32_t size) >> > { >> > - AVIOContext *pb; >> > - uint8_t *buf; >> > - CFDataRef data = NULL; >> > - int buf_size = 0; >> > - int ret = avio_open_dyn_buf(&pb); >> > - if (ret < 0) >> > - return NULL; >> > + VTContext *vtctx = avctx->internal->hwaccel_priv_data; >> > + void *tmp; >> > - ret = ff_isom_write_av1c(pb, avctx->extradata, >> > avctx->extradata_size, 1); >> > - if (ret < 0) >> > - goto fail; >> > + tmp = av_fast_realloc(vtctx->bitstream, >> > + &vtctx->allocated_size, >> > + size); >> >> This should be 'size + vtctx->bitstream_size' right? >> >> You're going to write from vtctx->bitstream_size to >> (vtctx->bitstream_size + size). >> >> > - buf_size = avio_get_dyn_buf(pb, &buf); >> > - if (buf_size) >> > - data = CFDataCreate(kCFAllocatorDefault, buf, buf_size); >> > + if (!tmp) >> > + return AVERROR(ENOMEM); >> > -fail: >> > - ffio_free_dyn_buf(&pb); >> > + vtctx->bitstream = tmp; >> > - return data; >> > + // copy the sequence header obu into the bitstream >> > + memcpy(vtctx->bitstream + vtctx->bitstream_size, + >> > buffer, + size); >> >> It looks like your mail client mangled the patch here. >> >> > + >> > + vtctx->bitstream_size += size; >> > + return 0; >> > } >> > static int videotoolbox_av1_start_frame(AVCodecContext *avctx, >> > const uint8_t *buffer, >> > uint32_t size) >> > { >> > + >> > + + VTContext *vtctx = avctx->internal->hwaccel_priv_data; >> > + void *tmp; >> > + >> > + tmp = av_fast_realloc(vtctx->bitstream, >> > + &vtctx->allocated_size, >> > + size); >> > + >> >> This should also be size + vtctx->bitstream_size. >> >> > + if (!tmp) >> > + return AVERROR(ENOMEM); >> > + >> > + vtctx->bitstream = tmp; >> > + >> > + // copy the frame data into the bitstream >> > + memcpy(vtctx->bitstream + vtctx->bitstream_size, buffer, size); >> > + vtctx->bitstream_size += size; >> > return 0; >> > } >> > @@ -70,17 +110,18 @@ static int >> > videotoolbox_av1_decode_slice(AVCodecContext *avctx, >> > const uint8_t *buffer, >> > uint32_t size) >> > { >> > - VTContext *vtctx = avctx->internal->hwaccel_priv_data; >> > - >> > - return ff_videotoolbox_buffer_copy(vtctx, buffer, size); >> > + return 0; >> > } >> > static int videotoolbox_av1_end_frame(AVCodecContext *avctx) >> > { >> > const AV1DecContext *s = avctx->priv_data; >> > + VTContext *vtctx = avctx->internal->hwaccel_priv_data; >> > AVFrame *frame = s->cur_frame.f; >> > - return ff_videotoolbox_common_end_frame(avctx, frame); >> > + int ret = ff_videotoolbox_common_end_frame(avctx, frame); >> > + vtctx->bitstream_size = 0; >> > + return ret; >> > } >> > const FFHWAccel ff_av1_videotoolbox_hwaccel = { >> > @@ -89,6 +130,7 @@ const FFHWAccel ff_av1_videotoolbox_hwaccel = { >> > .p.id = AV_CODEC_ID_AV1, >> > .p.pix_fmt = AV_PIX_FMT_VIDEOTOOLBOX, >> > .alloc_frame = ff_videotoolbox_alloc_frame, >> > + .decode_params = videotoolbox_av1_decode_params, >> > .start_frame = videotoolbox_av1_start_frame, >> > .decode_slice = videotoolbox_av1_decode_slice, >> > .end_frame = videotoolbox_av1_end_frame, >> > -- >> > 2.39.3 (Apple Git-146) >> > >> > _______________________________________________ >> > ffmpeg-devel mailing list >> > ffmpeg-devel@ffmpeg.org >> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel >> > >> > To unsubscribe, visit link above, or email >> > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". >> > > > -- > C уважением, > Руслан. > > Mail: ractyf...@gmail.com > Telegram: @nenkoru > VK: https://vk.com/racty > -- C уважением, Руслан. Mail: ractyf...@gmail.com Telegram: @nenkoru VK: https://vk.com/racty _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
