ffmpeg | branch: release/6.0 | Josh Allmann <joshua.allm...@gmail.com> | Thu Jun 20 17:33:55 2024 -0700| [3610a803e7550563aae665ddc536926e316b93ec] | committer: Timo Rothenpieler
avcodec/nvenc: fix segfault in intra-only mode In intra-only mode, frameIntervalP is 0, which means the frame data array is smaller than the number of surfaces. Together with using the wrong size on deallocation of the frame_data_array, this lead to a crash. Signed-off-by: Timo Rothenpieler <t...@rothenpieler.org> (cherry picked from commit c9151ea50715c4ce47ad1c8df519781565db01f6) > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=3610a803e7550563aae665ddc536926e316b93ec --- libavcodec/nvenc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c index 3c68ed3930..4a4607c4db 100644 --- a/libavcodec/nvenc.c +++ b/libavcodec/nvenc.c @@ -973,7 +973,7 @@ static av_cold int nvenc_recalc_surfaces(AVCodecContext *avctx) // Output in the worst case will only start when the surface buffer is completely full. // Hence we need to keep at least the max amount of surfaces plus the max reorder delay around. - ctx->frame_data_array_nb = ctx->nb_surfaces + ctx->encode_config.frameIntervalP - 1; + ctx->frame_data_array_nb = FFMAX(ctx->nb_surfaces, ctx->nb_surfaces + ctx->encode_config.frameIntervalP - 1); return 0; } @@ -1860,7 +1860,7 @@ av_cold int ff_nvenc_encode_close(AVCodecContext *avctx) av_fifo_freep2(&ctx->unused_surface_queue); if (ctx->frame_data_array) { - for (i = 0; i < ctx->nb_surfaces; i++) + for (i = 0; i < ctx->frame_data_array_nb; i++) av_buffer_unref(&ctx->frame_data_array[i].frame_opaque_ref); av_freep(&ctx->frame_data_array); } _______________________________________________ ffmpeg-cvslog mailing list ffmpeg-cvslog@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-cvslog To unsubscribe, visit link above, or email ffmpeg-cvslog-requ...@ffmpeg.org with subject "unsubscribe".
