> > -----Original Message----- > > From: ffmpeg-devel <ffmpeg-devel-boun...@ffmpeg.org> On Behalf Of > > Chen, Wenbin > > Sent: Wednesday, November 10, 2021 4:03 AM > > To: ffmpeg-devel@ffmpeg.org > > Subject: Re: [FFmpeg-devel] [PATCH 2/4] libavutil/hwcontext_qsv: fix > > a bug when malloc handle_pairs_internal > > > > > This commandline cause core dumped: > > > ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \ > > > -hwaccel_output_format vaapi -i input.264 \ > > > -vf "hwmap=derive_device=qsv,format=qsv" \ > > > -c:v h264_qsv output.264 > > > > > > reason: We use nb_surfaces to assign surface to > > handle_pairs_internal > > > but handle_pairs_internal is alloced with the size of > > init_pool_size. > > > This lead to access to illegal address. > > > > > > Now change it to use nb_surfaces to allocate handle_pairs_internal > > and the > > I'm not sure about whether this is right. > > When we look at the top of the qsv_frames_derive_to function that you > are changing, there is this: > > > if (src_ctx->initial_pool_size == 0) { > av_log(dst_ctx, AV_LOG_ERROR, "Only fixed-size pools can be " > "mapped to QSV frames.\n"); > return AVERROR(EINVAL); > } > > It's because QSV doesn't support dynamic pool sizes. > > When we look at the vaapi_pool_alloc function in hwcontext_vaapi.c, we > can see that: > > when initial_pool_size is > 0, the pool cannot grow beyond this value, > so nb_surfaces cannot be > initial_pool_size > > So I'm wondering what could have caused the segfault? Which values did > you have there for nb_surfaces and initial_pool_size? > > > > > core dumped error is unseen. Also change D3D11VA to use nb_surfaces > > > to align to VAAPI and DXVA2. > > Those changes are unrelated to fixing the issue with VAAPI. > (besides that I don't think these are needed at all) > > Kind regards, > softworkz
You are right. The real cause is that vaapi_decode_make_config() is called twice. The init_pool_size is changed on the second call. I will resubmit patch to fix this Thanks Wenbin > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".