> We don't really leverage these extra functions of NVDEC because it > breaks many assumptions about hwaccels, which are meant to be exact > decoders.
Yeah I understand that, and expected this kind of feedback. Do you envision a way for hwaccels (perhaps in the future) to support some additional level of *hardware acceleration*, such as resizing and cropping? Let me try to motivate the rationale. Think about a common use case in the AI world we live in: you receive a bunch of simultaneous 4k (or 1080p) incoming rtsp streams and you want to decode the video and pass it through some ML model. The native hevc codec doesn't support resizing, so you decode video at full 4k on the gpu by leveraging the nvdec hwaccel, which allocates something like 5-10 surfaces at 3840x2160, totalling around 250MB of VRAM (GPU memory), and then you have to immediately take all of those frames, pass them through a filterchain, scale them down to e.g. 640x360. Leaving aside the waste of additional CUDA cores and added latency of having to synchronize the cuda stream twice (once to receive the frame out of the decoder, the second to pull from the filterchain), do this for 50 camera streams and you'll quickly run out of GPU memory with a GPU utilization under 10%... Instead, my patch allows decoding at 640x360 (or whatever you choose) directly, allocating only 6MB of GPU VRAM (instead of 250MB at 4k), so now you can fit 40x more video decoding streams in the same GPU card. I understand the challenges of allowing hwaccels to resize/crop as part of the decoding process and how it currently breaks the API in several ways I'm of course unfamiliar with. However, I'd love to help support this feature in the future, since it would add tremendous value and enable more efficient real-time Computer Vision pipelines to leverage the amazing framework FFmpeg is. > You cannot do that here. The frame size can change mid-stream, and > this would suppress any such change. > Additionally, if this code runs more then once, then the offset is > applied repeatedly without actually resetting width/height. Very good points! Indeed I'm not yet super familiar with the structure of the hevc codec implementation, nor with the available functions to reconfigure a hwaccel (I know the Video Codec SDK supports this, but have never implemented such functionality), but I see how this would break. Would you see value in me trying to figure out a way to support reconfiguring the codec if the width/height/cropping changes mid-stream or would you reject the patch/feature regardless? Thanks! On Fri, Sep 20, 2024 at 7:18 AM Hendrik Leppkes <h.lepp...@gmail.com> wrote: > On Fri, Sep 20, 2024 at 1:24 AM Carlos Ruiz <carlos.r.do...@gmail.com> > wrote: > > > > Hi! > > > > This is my first contribution to the project so please excuse any bad > > etiquette, I tried to read all the FAQs before posting. Would love to > start > > by thanking everyone for such an amazing framework you've built! > > > > Anyway, here's my proposed patch to support video resizing when using > NVDEC > > hwaccel to decode hevc video (I could look into a similar patch for h264, > > av1, etc if this looks useful). There's a bit more context/explanation in > > the commit description in the patch, but please let me know if the use > case > > isn't clear. > > > > We don't really leverage these extra functions of NVDEC because it > breaks many assumptions about hwaccels, which are meant to be exact > decoders. > If anything, just fudging the width/height is certainly an API > violation and will likely not be possible as it breaks many > assumptions in the code otherwise, see below. > > > --- > > libavcodec/hevc/hevcdec.c | 8 ++++++-- > > libavcodec/nvdec.c | 21 +++++++++++++++++---- > > 2 files changed, 23 insertions(+), 6 deletions(-) > > > > diff --git a/libavcodec/hevc/hevcdec.c b/libavcodec/hevc/hevcdec.c > > index d915d74d22..d63fc5875f 100644 > > --- a/libavcodec/hevc/hevcdec.c > > +++ b/libavcodec/hevc/hevcdec.c > > @@ -351,8 +351,12 @@ static void export_stream_params(HEVCContext *s, > const > > HEVCSPS *sps) > > avctx->pix_fmt = sps->pix_fmt; > > avctx->coded_width = sps->width; > > avctx->coded_height = sps->height; > > - avctx->width = sps->width - ow->left_offset - > > ow->right_offset; > > - avctx->height = sps->height - ow->top_offset - > > ow->bottom_offset; > > + if (avctx->width <= 0 || avctx->height <= 0) { > > + avctx->width = sps->width; > > + avctx->height = sps->height; > > + } > > + avctx->width = avctx->width - ow->left_offset - > ow->right_offset; > > + avctx->height = avctx->height - ow->top_offset - > ow->bottom_offset; > > You cannot do that here. The frame size can change mid-stream, and > this would suppress any such change. > Additionally, if this code runs more then once, then the offset is > applied repeatedly without actually resetting width/height. > > - Hendrik > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". > _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".