> -----Original Message----- > From: ffmpeg-devel <ffmpeg-devel-boun...@ffmpeg.org> On Behalf Of > m.kaindl0...@gmail.com > Sent: Tuesday, February 18, 2025 8:17 PM > To: 'FFmpeg development discussions and patches' <ffmpeg- > de...@ffmpeg.org> > Subject: Re: [FFmpeg-devel] [PATCH] avfilter/dnn: add zero-shot image > classification using CLIP models > > The new backend is an extension of the existing Torch backend rather than a > separate implementation. In general, we don't need to add a new backend for libtorch for another use case.
> > Inference in CLIP differs from other models as it encodes (embeds) both > images and tokenized text labels, then calculates the similarity between the > encoded vectors. As a result, its forward pass takes two inputs and produces > two outputs. Is the tokenizer expected to be used by other dnn filters? We may put it in files like libavfilter / dnn_filter_common.c > > To ensure clarity and modularity, I have created a separate > dnn_torch_backend_clip file instead of expanding dnn_torch_backend. This > keeps the main file manageable and allows for easy exclusion from the build > when the tokenizer-cpp library is not included. > > If preferred, I can implement a standalone tokenizer class, integrate it into > e.g. > libavutil, and move the remaining code to the backend. > > -----Ursprüngliche Nachricht----- > Von: ffmpeg-devel <ffmpeg-devel-boun...@ffmpeg.org> Im Auftrag von Guo, > Yejun > Gesendet: Tuesday, 18 February 2025 11:09 > An: FFmpeg development discussions and patches <ffmpeg- > de...@ffmpeg.org> > Betreff: Re: [FFmpeg-devel] [PATCH] avfilter/dnn: add zero-shot image > classification using CLIP models > > > > > -----Original Message----- > > From: ffmpeg-devel <ffmpeg-devel-boun...@ffmpeg.org> On Behalf Of > > Maximilian Kaindl > > Sent: Tuesday, February 18, 2025 12:29 AM > > To: FFmpeg development discussions and patches <ffmpeg- > > de...@ffmpeg.org> > > Subject: Re: [FFmpeg-devel] [PATCH] avfilter/dnn: add zero-shot image > > classification using CLIP models > > > > Hello Yejun Guo, > > > > yes i can do that and submit it in another patch. Do you also have > > some feedback for the clip backend? I have already made some small > > changes (cuda accel and new preprocessing) that i will submit along > > with the other patch, but i would like to hear your thoughts. > > > Could you share why we need a new backend? > > > Thanks > > > > ________________________________ > > From: ffmpeg-devel <ffmpeg-devel-boun...@ffmpeg.org> on behalf of Guo, > > Yejun <yejun.guo-at-intel....@ffmpeg.org> > > Sent: Sunday, February 16, 2025 7:09 AM > > To: FFmpeg development discussions and patches <ffmpeg- > > de...@ffmpeg.org> > > Subject: Re: [FFmpeg-devel] [PATCH] avfilter/dnn: add zero-shot image > > classification using CLIP models > > > > > > > > > -----Original Message----- > > > From: ffmpeg-devel <ffmpeg-devel-boun...@ffmpeg.org> On Behalf Of > > > m.kaindl0...@gmail.com > > > Sent: Thursday, January 30, 2025 4:33 AM > > > To: ffmpeg-devel@ffmpeg.org > > > Subject: [FFmpeg-devel] [PATCH] avfilter/dnn: add zero-shot image > > > classification using CLIP models > > > > > > Add a new filter 'dnn_clip' that performs zero-shot image > > > classification using CLIP (Contrastive Language-Image Pre-Training) > models. > > The filter supports: > > > > For image classification with new dnn models, we'd better add the new > > model support with dnn_classify at https://ffmpeg.org/ffmpeg- > > filters.html#dnn_005fclassify > > > > > > _______________________________________________ > > ffmpeg-devel mailing list > > ffmpeg-devel@ffmpeg.org > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > > > To unsubscribe, visit link above, or email > > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". > > _______________________________________________ > > ffmpeg-devel mailing list > > ffmpeg-devel@ffmpeg.org > > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > > > To unsubscribe, visit link above, or email > > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org > with subject "unsubscribe". > > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org > with subject "unsubscribe". _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
