Hi, some of you might remember that I had warmed-up the discusion about improving support for subtitle handling in ffmpeg sometime last year.
A lot of different voices were heard and the unfinished work from Matthieau Buron has been in focus. My takeaway was that it's a pretty difficult subject involving complex requirements, for which there is no simple solution in sight. Consequently, I had put this on hold at my side just like all others. Few days ago I had to revisit and re-evaluate the subtitle situation once again: the current sub2video "hack" implementation is slow and ineffective: it creates a full-size video frame from the source subtitles and that full-size frame needs to be overlaid/blended with/onto every single video frame. That's a pretty expensive operation and it needs to be performed for every single video frame, even when there's nothing to overlay at all (= no subs to show in current scene). And when there are subtitles to show, those are usually limited to smaller areas on the screen; again, there's no need to overlay a full-size image onto each frame. From a performance perspective, it would be much more efficient to overlay the individual rects onto each frame only instead of a full frame blend. >From a general perspective - even though it has always been considered as a 'hack' - the sub2video implementation has become a kind of standard behavior, despite of its shortcomings. Users have accommodated with the respective command lines and are expecting these to work. I'd expect any deviation from the current behavior to be filed as regression bugs. >From there, I had decided to go for a small-step improvement of that situation while staying inside these corner points, and the plan was simple: - keep the current sub2video functionality alive at its core - but stop creating full video frames (with alpha) for feeding into a filtergraph (and later overlay) - instead, use refcounting for AVSubtitle and attach it to frames of (new) type AVMediaType_Subtitle - Those subtitle frames can travel through a filtergraph without needing to materialize them to images first For more efficient overlay (processing the relevant rects only), I have created a new filter: overlay_subs: - Input0: video - Input1: subtitles (format: SUBTITLE_BITMAP) - Output0: video In order to keep compatibility with existing command lines, I have added another filter: sub2video (reminescent naming) sub2video creates video frames (with alpha) from an existing subtitle stream: - Input0: subrtitles (format: SUBTITLE_BITMAP) - Output0: video This filter gets auto-inserted to retain compatibility with current sub2video command lines. As AVSubtitle can carry both, textual and graphical subtitles, the ability for sending textual subtitles through a filtergraph came more or less for free. For testing purposes, I have added another filter: sleet (translates subtitles to 'L337 speak') - Input0: subtitles (format: SUBTITLE_ASS or SUBTITLE_TEXT) - Output0: subtitles (format: same as input) Why leet? Real world use is surely questionable, but it has two advantages that make it a great choice for testing: You can see from almost every single line whether it has been filtered, and the text is still readable which allows to verify that timings are still valid. Working Command Lines Using "overlay_subs" (better performance): -y -loglevel verbose -ss 00:02:30 -i INPUT -filter_complex "[0:0]format=bgra[main];[main][0:3]overlay_subs=x=0:y=-220:format=auto,forma t=yuv420p" output.mp4 Using regular Overlay and the new sub2video filter: -y -loglevel verbose -ss 00:02:30 -filter_threads 1 -i INPUT -filter_complex "[0:0]format=bgra[main];[0:3]sub2video[subs];[main][subs]overlay=x=0:y=-220: format=auto,format=yuv420p" output.mp4 Legacy command line compatibility: (using regular overlay; sub2video filter is inserted automatically) -y -loglevel verbose -ss 00:04:30 -filter_threads 1 -i INPUT -filter_complex "[0:0][0:3]overlay=x=0:y=-220:format=auto,format=yuv420p" output.mp4 To make this work, I roughly did the following: - Added a new property 'type' to AVFrame of type enum AVMediaType - Changed code that uses width/height for checking whether a frame is video or audio and check the 'type' property instead - Added ff_ssrc_sbuffer and ff_ssink_sbuffer filters - Changed filtergraph code to support frames with AVMediaType == AVMediaType_Subtitle - Transformed the existing sub2video code accordingly, while still keeping the heartbeat mechanism in place Next Steps - Create modified version of the ASS filter that works based on (text) subtitle filter input - Create modified EIA-608/708 filter (e.g. split_eia608) which has one video input and one video output, plus a second output of type 'subtitle' (format: text or ass) This will allow to burn-in eia subs or save them in any other subtitle format without needing the weird "movie:xxx" input filter construct. - First of all: Get this patchset right... Reaching out for Help... This is a huge task where I hadn't expected to get that far. I'd consider my ffmpeg expertise as "spot knowledge": I do know certain areas very well, but there are others that I'm less familiar with. Nonetheless, I have a strategy and a working proof of concept. I'm, sure that there are quite a few things that need adjustment, and I'd be glad about any help and assistance! Thanks, softworkz --- softworkz (9): lavu/frame: avframe add type property avfilter/subtitles: Add subtitles.c avfilter/avfilter: Handle subtitle frames avfilter/overlay_subs: Add overlay_subs filter avfilter/sub2video: Add sub2video filter avfilter/sbuffer: Add sbuffersrv and sbuffersink filters avfilter/sleet: Add sleet filter fftools/ffmpeg: Replace sub2video with subtitle frame filtering Adjust sub2video Fate tests configure | 2 +- fftools/ffmpeg.c | 324 +++--- fftools/ffmpeg.h | 7 +- fftools/ffmpeg_filter.c | 198 +++- fftools/ffmpeg_hw.c | 2 +- fftools/ffmpeg_opt.c | 3 +- libavfilter/Makefile | 6 + libavfilter/allfilters.c | 13 +- libavfilter/avfilter.c | 8 +- libavfilter/avfiltergraph.c | 5 + libavfilter/buffersink.c | 65 ++ libavfilter/buffersink.h | 15 + libavfilter/buffersrc.c | 65 ++ libavfilter/buffersrc.h | 1 + libavfilter/formats.c | 11 + libavfilter/internal.h | 8 + libavfilter/sf_sleet.c | 209 ++++ libavfilter/subtitles.c | 61 ++ libavfilter/subtitles.h | 42 + libavfilter/svf_sub2video.c | 260 +++++ libavfilter/vf_overlay_subs.c | 1173 +++++++++++++++++++++ libavfilter/vf_overlay_subs.h | 88 ++ libavutil/frame.c | 74 +- libavutil/frame.h | 39 +- libavutil/version.h | 2 +- tests/ref/fate/filter-overlay-dvdsub-2397 | 179 ++-- tests/ref/fate/sub-dvb | 162 +-- tests/ref/fate/sub2video | 49 +- tests/ref/fate/sub2video_basic | 178 ++-- tests/ref/fate/sub2video_time_limited | 4 +- 30 files changed, 2743 insertions(+), 510 deletions(-) create mode 100644 libavfilter/sf_sleet.c create mode 100644 libavfilter/subtitles.c create mode 100644 libavfilter/subtitles.h create mode 100644 libavfilter/svf_sub2video.c create mode 100644 libavfilter/vf_overlay_subs.c create mode 100644 libavfilter/vf_overlay_subs.h -- 2.28.0.windows.1 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
