Dear FFmpeg developers, I've been doing some experiments with out-of-tree filters [1] and I start feeling confident that the approach works well enough and, besides a "demo" filter written in C [2], I've implemented a filter in Rust [3].
That said, I propose: - extracting the DNN filters (dnn_detect, dnn_classify, dnn_processing) to another repository, within the ffmpeg umbrella, to be integrated to ffmpeg at build time by the users who require this feature. - rewriting those filters in C++. My rationale is: - the DNN code is quite niche, not used by most people. - There are some files on it written in C++, namely the libtorch backend, so moving it out would reduce the C++ code in the ffmpeg repository. - Besides the libtorch backend, the other backends (openvino and tensorflow) use C APIs which are essentially wrappers around C++ APIs (as both openvino and tensorflow base APIs are C++). I don't see much benefits on using C wrappers when the C++ API could be used directly, making the code cleaner and more reliable, as the C++ APIs offer more type safety than the C ones and, well, RAII. Outcomes: - Less C++ code on ffmpeg codebase. - Less ~6k LOC from the ffmpeg codebase, reducing a little maintenance burden. - The DNN filters can evolve separately, and even have a different release cycle, ownership and collaboration model. - After the C++ rewrite, the DNN code will be type safer, likely shorter, and with less dependencies, as it'll use the original C++ APIs instead of the C wrappers. Proposed steps: - polish and merge my patch that enables out-of-tree filters [4]. - extract the libavfilter/*dnn* code to an external repository, say git.ffmpeg.org/dnn-filters.git, keeping the git history. - adapt the dnn-filters repository to be built "out-of-tree" without breaking any public API. - rewrite the openvino backend and tensorflow backends to C++, removing the dependency on the C API wrappers. - rewrite the DNN filters in C++. Open issues: - Currently selecting the DNN backend be done via ./configure --enable-(libopenvino|libtensorflow|libtorch), which is hardcoded in the build system. I don't know how yet it can be done via the out-of-tree approach. An easy way is to simply pass the backend via environment, like in `DNN_BACKEND=openvino ./configure`, but I'd like to know how you folks would do it. - several macros used in the C code won't work on the C++ rewrite, requiring new C++ specific macros to be created. This affects only the last step of the aforementioned steps, though. - There is no way yet to integrate the documentation of out-of-tree filters into ffmpeg docs. - There will still be some "remains" of the DNN code in libavutil, namely the detection boxes related API. - During the extraction process, the DNN code should be "frozen", and any existing non merged patch should either be merged before the extraction, or be adapted to the the new repo. - Some extra coordination will be needed not to break any production code during the migration. I don't think this is the case, as no public API needs change, but I might be wrong. - I've performed splitting git repositories before, but never with signed commits, if there are any in the DNN code. It's possible that the signatures will break in the process, but I am not very sure right now. In any case, I would like to know whether the current proposal aligns with the needs of the community or what needs change. If it does, I'd like to volunteer myself to head the work on it, depending on the project roadmap. [1] https://ffmpeg.org/pipermail/ffmpeg-devel/2025-March/341105.html [2] https://gitlab.com/leandrosansilva/ffmpeg-extra-filter-example [3] https://gitlab.com/leandrosansilva/ffmpeg-track-sort-filter [4] https://patchwork.ffmpeg.org/project/ffmpeg/patch/2477728b-6217-446a-9a15-830f028ba...@gmail.com/ _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
