Since it is not healthy to keep everything for myself, here is a summary
of the projects I have in mind for the enhancement of libavfilter.
The thing is, there are a lot of dependencies between these projects.
Working on them in a proper order would make achieving the goals easier.
On the other hand, if task B depends on task A, then working B without
addressing A involves a little more work for B, but also a lot more work
for A, and a lot more work for everything else that depends on A.
This is one reason I am writing this mail: to make clear what needs to
be done, and how tasks depend on each-other and fit together.
The earliest tasks are not very sexy. They are about making the code
clearer or more robust and generic rather than adding features. It is
always boring. But they are necessary, because parts of the process is
quite shaky, and we cannot afford to build on shaky code.
If you want to help, please consider this and you will be welcome.
Now that is established the specifics:
- Multi-stage negotiation.
That means negotiating properties that depend on other properties. I
think the colorspace stuff is in that case, it cannot be negotiated
until the pixel format is known.
- Media type negotiation.
That means that filters that do not care about the contents of what
they are filtering, filters that only act on the frame information
fields, can also not care if they are filtering audio or video or
other. There are a lot of these filters: pts manipulation, concat,
split, selection, etc.
We cannot afford to have to update every one of them if we want to add
a new media type, so this is a prerequisite. And it is an aspect of
multi-stage negotiation, since pixel format or sample format can only
be negotiated once we know the media type.
- Partial graph configuration.
Right now, the negotiation happens in steps, first query_formats then
merge, then pick. But pick can be done on some links as soon as the
formats are merged to a singleton while other filters are still
delaying their query_formats, and once pick is done, the graph can
start running. That would allow filters with constraints more complex
than what AVFilterFormats can express. (We have a mechanism for that,
used by amerge and one or two other filters; I introduced it early and
it was a mistake, it is much too fragile and only converges in the
simplest cases.)
- Partial graph re-configuration.
If we can run a graph when not all filters are configured, then we can
de-configure and re-configure part of a graph. That would bring us the
support for changes in pixel format.
- Global running API.
Right now, we make libavfilter activate filters by pumping
request_frame() on one of the buffersinks, more or less randomly, and
we hope it will result in something useful. We need a better API, one
where we say libavfilter "start running these graphs", we push frames
on input as we have them, we are notified when frames arrive on
output.
- Inter-filter threading.
This is a big one, and quite self-explanatory. It goes hand to hand
with a global running API, because "start running these graphs" also
means "start as many threads as you need and keep them ready".
Note that the network protocols in libavformat requires the same
thing. And I really do not want yet two other frigging threading
subsystems. This is why I have started slowly working on an unique
system suited for all needs, starting with both libavformat and
libavfilter but keeping also libavcodec in mind.
- Out-of-band side-data.
This is a mechanism to notify filters that the work they are currently
doing may become irrelevant, or other kinds of urgent information.
Filters can ignore it, and just do the work for nothing. But smarter
filters can take it into account and just skip until it becomes
relevant again.
- Seeking.
This is when applications notify the outputs they want to jump at a
specified time, and all the filters in the graph make sure the
instruction reach the inputs.
This is probably rather easy and does not depend on many other things.
Out-of-band side-data is meant for that, among other things: notify a
filter "you can skip processing the frames you have queued, because we
are seeking anyway"; and if seeking fails, notify the user as fast as
possible "we tried seeking, it will not happen".
A few things need ironing out, though. Do we want to handle it in
activate() or do we want a separate callback? Do we want a specific
mechanism for seeking or something more generic for all the messages
that go backwards in the graph? Do we want a queue for messages that
go backwards or would a single message be enough?
- Subtitles.
This one has been under the spotlight recently. What it means is
rather obvious. But it if far from trivial.
For starters, it requires the negotiation of media type, because we
cannot afford to update all the utility filters for subtitles, they
have to work transparently.
Then we have the issue that subtitles streams are sparse. A filter to
render a subtitle on a video frame needs to know if the next subtitle
is before or after the current frame, but if the subtitles are muxed
with the video, then the next subtitle can arrive only in several
minutes. The solution I have for that is heartbeat frames: subtitles
frames that contain no information except for their timestamp and mean
that no change has occurred. To generate them, link the buffersink for
the subtitles to the buffersink for the video.
There are other issues to consider when designing subtitles: pixel
format and automatic conversions, overlapping, global styles, etc.
- Data packets.
This is about having packets of binary data inside libavfilter. That
means allowing to use bitstream filters like real filters in a graph
instead of the second, more limited, API just for them. It also means
codecs can be data → audio/video or audio/video → data filters.
It requires the negotiation of media type.
It also requires thinking how to fit the information present in
AVPacket into AVFrame. Some timestamps are not available, for example.
I think that is all I have in mind for now.
Regards,
--
Nicolas George
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
