Since it is not healthy to keep everything for myself, here is a summary of the projects I have in mind for the enhancement of libavfilter.
The thing is, there are a lot of dependencies between these projects. Working on them in a proper order would make achieving the goals easier. On the other hand, if task B depends on task A, then working B without addressing A involves a little more work for B, but also a lot more work for A, and a lot more work for everything else that depends on A. This is one reason I am writing this mail: to make clear what needs to be done, and how tasks depend on each-other and fit together. The earliest tasks are not very sexy. They are about making the code clearer or more robust and generic rather than adding features. It is always boring. But they are necessary, because parts of the process is quite shaky, and we cannot afford to build on shaky code. If you want to help, please consider this and you will be welcome. Now that is established the specifics: - Multi-stage negotiation. That means negotiating properties that depend on other properties. I think the colorspace stuff is in that case, it cannot be negotiated until the pixel format is known. - Media type negotiation. That means that filters that do not care about the contents of what they are filtering, filters that only act on the frame information fields, can also not care if they are filtering audio or video or other. There are a lot of these filters: pts manipulation, concat, split, selection, etc. We cannot afford to have to update every one of them if we want to add a new media type, so this is a prerequisite. And it is an aspect of multi-stage negotiation, since pixel format or sample format can only be negotiated once we know the media type. - Partial graph configuration. Right now, the negotiation happens in steps, first query_formats then merge, then pick. But pick can be done on some links as soon as the formats are merged to a singleton while other filters are still delaying their query_formats, and once pick is done, the graph can start running. That would allow filters with constraints more complex than what AVFilterFormats can express. (We have a mechanism for that, used by amerge and one or two other filters; I introduced it early and it was a mistake, it is much too fragile and only converges in the simplest cases.) - Partial graph re-configuration. If we can run a graph when not all filters are configured, then we can de-configure and re-configure part of a graph. That would bring us the support for changes in pixel format. - Global running API. Right now, we make libavfilter activate filters by pumping request_frame() on one of the buffersinks, more or less randomly, and we hope it will result in something useful. We need a better API, one where we say libavfilter "start running these graphs", we push frames on input as we have them, we are notified when frames arrive on output. - Inter-filter threading. This is a big one, and quite self-explanatory. It goes hand to hand with a global running API, because "start running these graphs" also means "start as many threads as you need and keep them ready". Note that the network protocols in libavformat requires the same thing. And I really do not want yet two other frigging threading subsystems. This is why I have started slowly working on an unique system suited for all needs, starting with both libavformat and libavfilter but keeping also libavcodec in mind. - Out-of-band side-data. This is a mechanism to notify filters that the work they are currently doing may become irrelevant, or other kinds of urgent information. Filters can ignore it, and just do the work for nothing. But smarter filters can take it into account and just skip until it becomes relevant again. - Seeking. This is when applications notify the outputs they want to jump at a specified time, and all the filters in the graph make sure the instruction reach the inputs. This is probably rather easy and does not depend on many other things. Out-of-band side-data is meant for that, among other things: notify a filter "you can skip processing the frames you have queued, because we are seeking anyway"; and if seeking fails, notify the user as fast as possible "we tried seeking, it will not happen". A few things need ironing out, though. Do we want to handle it in activate() or do we want a separate callback? Do we want a specific mechanism for seeking or something more generic for all the messages that go backwards in the graph? Do we want a queue for messages that go backwards or would a single message be enough? - Subtitles. This one has been under the spotlight recently. What it means is rather obvious. But it if far from trivial. For starters, it requires the negotiation of media type, because we cannot afford to update all the utility filters for subtitles, they have to work transparently. Then we have the issue that subtitles streams are sparse. A filter to render a subtitle on a video frame needs to know if the next subtitle is before or after the current frame, but if the subtitles are muxed with the video, then the next subtitle can arrive only in several minutes. The solution I have for that is heartbeat frames: subtitles frames that contain no information except for their timestamp and mean that no change has occurred. To generate them, link the buffersink for the subtitles to the buffersink for the video. There are other issues to consider when designing subtitles: pixel format and automatic conversions, overlapping, global styles, etc. - Data packets. This is about having packets of binary data inside libavfilter. That means allowing to use bitstream filters like real filters in a graph instead of the second, more limited, API just for them. It also means codecs can be data → audio/video or audio/video → data filters. It requires the negotiation of media type. It also requires thinking how to fit the information present in AVPacket into AVFrame. Some timestamps are not available, for example. I think that is all I have in mind for now. Regards, -- Nicolas George
signature.asc
Description: PGP signature
_______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
