Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files
On 2025-05-23T13:55:57.000+02:00, Michael Niedermayer wrote: > On Fri, May 23, 2025 at 11:33:40AM +0200, Timothée wrote: > >> On 2025-05-23T02:57:36.000+02:00, Michael Niedermayer >> wrote: >> >>> On Fri, May 23, 2025 at 02:45:59AM +0200, Michael Niedermayer >>> wrote: >>> >>>>Hi Ronald On Thu, May 22, 2025 at 07:59:06AM -0400, Ronald S. >>>>Bultje wrote: >>>> >>>>> Hi, On Wed, May 21, 2025 at 9:34 AM Timothée < >>>>> timothee.informati...@regaud-chapuy.fr> wrote: >>>>> >>>>>> Hello, I am interested in expanding ffmpeg's capabilities >>>>>> to extract low-level data from video codecs. Specifically, >>>>>> I'd like to implement functionality that would allow >>>>>> exporting frame data, macroblock information, quantization >>>>>> tables, and similar codec-specific elements to binary >>>>>> files for further analysis. After searching through the >>>>>> documentation and existing features, I haven't found >>>>>> similar functionality, though I may have missed something. >>>>>> Has this been implemented before, or are there related >>>>>> features I should examine? >>>>> >>>>> Some older codecs implement minor variants for this, e.g. >>>>> grep for AV_FRAME_DATA_MOTION_VECTORS, which attaches a >>>>> frame's motion vectors to the picture data. I believe >>>>> there's an example app and possibly a filter to overlay MVs >>>>> on top of the video frame based on this concept. You could >>>>> extend this to cover other (macro)block info. There used to >>>>> be a variant of this for quant-tables also but I can't find >>>>> it, maybe it was removed. >>>> >>>> For motion vectors: ./ffplay -flags2 +export_mvs -i >>>>matrixbench_mpeg2.mpg -vf codecview=mv=pf+bf+bb For macroblock >>>>segmentation and type vissualization + also motion vectors: >>>>ffplay-3.4.13 -debug vis_mb_type matrixbench_mpeg2.mpg -vf >>>>codecview=mv=pf+bf+bb For QP vissualization + also motion >>>>vectors: ffplay-3.4.13 -debug vis_qp matrixbench_mpeg2.mpg -vf >>>>codecview=mv=pf+bf+bb For qp values dumped on the console >>>>./ffplay -debug qp -i matrixbench_mpeg2.mpg >>> >>>And this can easily be extended to other codecs, ATM it should >>> work with all 16x16 MB based codecs like >>> msmpeg4*/wmv*/mpeg1/2/4/h263/h264 mbtype and qp vissualization >>> need codecview to be extended or versions around 3.4 which >>> implemented it differently Implementing vissualization as done >>> currently with sidedata and codecview is simple and efficient. >>> It also would allow exporting the data to json by writing a >>> codec2json filter in place of codecview Also all decoders >>> already have all this data parsed and available so its simpler >>> than trying to do it in a decoder independant way I would thus >>> suggest implementations of this for modern codecs to follow the >>> same path as the existing code. thx >> >> Thanks for the helpful pointers! I will work on the codec2json >> filter. Looking at the code, I see where I can access sidedata but >> extracting qb table seems to fail. (in codecview.c l.233: >> ff_qp_table_extract() return 0 and qp_table is empty) (I use >> ./ffmpeg -flags2 +export_mvs -i input.mp4 -vf codecview=qp=1 >> output.mp4 -y) Is is qp extraction not implemented yet? Or is it >> because I have h264 video? If it's not implemented, I'm curious >> why there’s already code that appears to handle it. > > look at ff_print_debug_info2() theres probably something missing > The QP and MB type code was changed from being inside arrays of > pictures to sidedata. Something likely was lost/forgotten in the > process -debug qp works with h264 so likely teh export into sidedata > is not fully implemented Yes, this is exactly what is happening. After some research, I have found: `// FIXME qscale / qp ... stuff` in libavcodec/h264_slice.c, line 1871 and also in libavfilter/qp_table.h: 34 /** 35 * Normalize the qscale factor 36 * FIXME Add support for other values of enum AVVideoEncParamsType 37 * besides AV_VIDEO_ENC_PARAMS_MPEG2. 38 */ 39 static inline i
Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files
The 2025-05-22T14:33:22.000+02:00, Ramiro Polla wrote : > Hi, > > On Thu, May 22, 2025 at 1:59 PM Ronald S. Bultje wrote: > >> On Wed, May 21, 2025 at 9:34 AM Timothée < >> timothee.informati...@regaud-chapuy.fr> wrote: >> >>> I am interested in expanding ffmpeg's capabilities to extract >>> low-level data from video codecs. Specifically, I'd like to >>> implement functionality that would allow exporting frame data, >>> macroblock information, quantization tables, and similar >>> codec-specific elements to binary files for further analysis. >>> After searching through the documentation and existing features, >>> I haven't found similar functionality, though I may have missed >>> something. Has this been implemented before, or are there >>> related features I should examine? >> >> Some older codecs implement minor variants for this, e.g. grep >> for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's motion >> vectors to the picture data. I believe there's an example app and >> possibly a filter to overlay MVs on top of the video frame based >> on this concept. You could extend this to cover other (macro)block >> info. There used to be a variant of this for quant-tables also but >> I can't find it, maybe it was removed. > > If I recall correctly, we wanted to move away from exporting more > of this kind of codec-specific information as picture data. > Timothée, this kind of feature depends a lot on the codec that you > want to work with. It will also depend on which syntax elements you > need from the codec. In FFmpeg, there is the CBS code which > describes the bitstream for a few codecs, which might help you. I > have a separate project (called FFglitch), which lets you export > some elements (such as quantization tables, dct coefficients, and > motion vectors) from some codecs (jpeg, mpeg2, mpeg4) to JSON > format, and even modify them in the bitstream. In the end, it will > depend on what you want to do with the extracted low-level data, and > for which codecs. Ramiro Thank you for your response and for mentioning FFglitch - it looks like an excellent project that's very relevant to what I'm trying to accomplish. I have a question about FFglitch's capabilities: is it possible to extract quantization parameter (QP) tables using ffedit? I've reviewed the documentation but may have missed this functionality if it exists. Ideally I would want to extract data for all existing codecs, but I know this is impossible. My current focus is on implementing this feature for H.264, with plans to extend support to H.265 and AVI formats as well. I want to extract this data for scientific use. Thank you again for your time and suggestions. Timothée ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files
On 2025-05-23T02:57:36.000+02:00, Michael Niedermayer wrote: > On Fri, May 23, 2025 at 02:45:59AM +0200, Michael Niedermayer wrote: > >> Hi Ronald On Thu, May 22, 2025 at 07:59:06AM -0400, Ronald S. >> Bultje wrote: >> >>> Hi, On Wed, May 21, 2025 at 9:34 AM Timothée < >>> timothee.informati...@regaud-chapuy.fr> wrote: >>> >>>>Hello, I am interested in expanding ffmpeg's capabilities to >>>>extract low-level data from video codecs. Specifically, I'd >>>>like to implement functionality that would allow exporting >>>>frame data, macroblock information, quantization tables, and >>>>similar codec-specific elements to binary files for further >>>>analysis. After searching through the documentation and >>>>existing features, I haven't found similar functionality, >>>>though I may have missed something. Has this been implemented >>>>before, or are there related features I should examine? >>> >>>Some older codecs implement minor variants for this, e.g. grep >>> for AV_FRAME_DATA_MOTION_VECTORS, which attaches a frame's >>> motion vectors to the picture data. I believe there's an example >>> app and possibly a filter to overlay MVs on top of the video >>> frame based on this concept. You could extend this to cover >>> other (macro)block info. There used to be a variant of this for >>> quant-tables also but I can't find it, maybe it was removed. >> >> For motion vectors: ./ffplay -flags2 +export_mvs -i >> matrixbench_mpeg2.mpg -vf codecview=mv=pf+bf+bb For macroblock >> segmentation and type vissualization + also motion vectors: >> ffplay-3.4.13 -debug vis_mb_type matrixbench_mpeg2.mpg -vf >> codecview=mv=pf+bf+bb For QP vissualization + also motion vectors: >> ffplay-3.4.13 -debug vis_qp matrixbench_mpeg2.mpg -vf >> codecview=mv=pf+bf+bb For qp values dumped on the console ./ffplay >> -debug qp -i matrixbench_mpeg2.mpg > > And this can easily be extended to other codecs, ATM it should work > with all 16x16 MB based codecs like > msmpeg4*/wmv*/mpeg1/2/4/h263/h264 mbtype and qp vissualization need > codecview to be extended or versions around 3.4 which implemented it > differently Implementing vissualization as done currently with > sidedata and codecview is simple and efficient. It also would allow > exporting the data to json by writing a codec2json filter in place > of codecview Also all decoders already have all this data parsed and > available so its simpler than trying to do it in a decoder > independant way I would thus suggest implementations of this for > modern codecs to follow the same path as the existing code. thx Thanks for the helpful pointers! I will work on the codec2json filter. Looking at the code, I see where I can access sidedata but extracting qb table seems to fail. (in codecview.c l.233: ff_qp_table_extract() return 0 and qp_table is empty) (I use ./ffmpeg -flags2 +export_mvs -i input.mp4 -vf codecview=qp=1 output.mp4 -y) Is is qp extraction not implemented yet? Or is it because I have h264 video? If it's not implemented, I'm curious why there’s already code that appears to handle it. Timothée ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-devel] [FEATURE PROPOSAL] Extracting codec-level data to binary files
Hello, I am interested in expanding ffmpeg's capabilities to extract low-level data from video codecs. Specifically, I'd like to implement functionality that would allow exporting frame data, macroblock information, quantization tables, and similar codec-specific elements to binary files for further analysis. After searching through the documentation and existing features, I haven't found similar functionality, though I may have missed something. Has this been implemented before, or are there related features I should examine? I'd appreciate your feedback on whether this feature would be considered valuable for the project and if it aligns with ffmpeg's development goals. If there's interest, I can provide a more detailed technical proposal. I'm new to both ffmpeg development and open source contributions in general, so I welcome any guidance or information that might help me contribute effectively. Thank you for your time and consideration. Timothée ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
