Hi, On Thu, Oct 30, 2014 at 1:59 PM, Michael Niedermayer <michae...@gmx.at> wrote:
> On Thu, Oct 30, 2014 at 11:50:46AM +0100, Stefano Sabatini wrote: > > Sorry for the slow reply. > > > > On date Wednesday 2014-10-22 23:45:42 +0200, Nicolas George encoded: > > > [ CCing Anton, as most that is written here also apply to libav too, > and > > > this would be a good occasion to try a cross-fork cooperation; if that > is > > > not wanted, please let us know so we can drop the cc. ] > > > > > > 1. Problems with the current design > > > > > > 1.1. Mixed input-/output-driven model > > > > > > Currently, lavfi is designed to work in a mixed input-driven and > > > output-driven model. That means the application needs sometimes to > add > > > input to buffersources and sometimes request output to > buffersinks. This > > > is a bit of a nuisance, because it requires the application to do > it > > > properly: adding input on the wrong input or requesting a frame on > the > > > wrong output will cause extra memory consumption or latency. > > > > > > With the libav API, it can not work at all since there is no > mechanism > > > to determine which input needs a frame in order to proceed. > > > > > > The libav API is clearly designed for a more output-driven > > > implementation, with FIFOs anywhere to prevent input-driven frames > to > > > reach unready filters. Unfortunately, since it is impossible from > the > > > outside to guess what output will get a frame next, that can cause > > > frames to accumulate anywhere in the filter graph, eating a lot of > > > memory unnecessarily. > > > > > > FFmpeg's API has eliminated FIFOs in favour of queues in filters > that > > > need it, but these queues can not be controlled for unusual filter > > > graphs with extreme needs. Also, there still is an implicit FIFO > inside > > > buffersink. > > > > > > 1.2. Recursive implementation > > > > > > All work in a filter graph is triggered by recursive invocations > of the > > > filters' methods. It makes debugging harder. It also can lead to > large > > > stack usage and makes frame- and filter-level multithreading > harder to > > > implement. It also prevents some diagnosis from working reliably. > > > > > > 1.3. EOF handling > > > > > > Currently, EOF is propagated only through the return value of the > > > request_frame() method. That means it only works in an > output-driven > > > scheme. It also means that it has no timestamp attached to it; > this is > > > an issue for filters where the duration of the last frame is > relevant, > > > like vf_fps. > > > > > > 1.4. Latency > > > > > > Some filters need to know the timestamp of the next frame in order > to > > > know when the current frame will stop and be able to process it: > > > overlay, fps are two examples. These filters will introduce a > latency of > > > one input frame that could otherwise be avoided. > > > > > > 1.5. Timestamps > > > > > > Some filters do not care about timestamps at all. Some check and > have a > > > proper handling of NOPTS values. Some filters just assume the > frames > > > will have timestamps, and possibly make extra assumptions on that: > > > monotony, consistency, etc. That is an inconsistent mess. > > > > > > 1.6. Sparse streams > > > > > > There is a more severe instance of the latency issue when the input > > > comes from an interleaved sparse stream: in that case, waiting for > the > > > next frame in order to find the end of the current one may require > > > demuxing a large chunk of input, in turn provoking a lot of > activity on > > > other inputs of the graph. > > > > Other issues. > > > > > S1. the filtergraph can't properly readapt to mid-stream > > changes involving assumed invariants (aspect ratio, size, timebase, > > pixel format, sample_rate). Indeed the framework was designed as > > though some of these properties (the ones defined by query_formats) > > were not allowed to change. > > no, no and no :) > the filtergraph should be able adapt to some changes like aspect, > resolution and pixel / sample format. These are not invariants, some > of this definitly worked when i tried it long ago > i posted a (incomplete) patchset that fixes bugs in this relation > long ago There are filters that can perfectly well handle changes in > resolution, aspect, formats, ... > and there are filters which are buggy but could when they are fixed > also equally well support such changes > and there are filters which fundamentally do not support some changes, > these need to either be reinited and loose state/history or a > scale/aresample be inserted before them when changes on their input > occur > for some filters reinit is not appropriate, examples are things that > are intended to collect global statistics > Also scale / aresample filters can serve as > "parameter change barriers", filters afterwards do not need to deal > with such changes > > > > > > S2. Another problem is that we initialize the filter before the > > filtergraph, so for example the single filter can't readapt to the > > filtergraph topology. For example it would be useful to have the split > > filter to change the number of outputs depending on the number of > > outputs specified, but this can't be easily achieved. (That's in my > > opinion a minor problem though). > > > > > S3. It is not possible to direct commands towards a specific > > filter. For this we can add an ID to each filter instance. We could > > have something has: > > color:left_color=c=red [left] > > color:right_color=c=blue [right] > > > > then you can send commands (e.g. with zmq) with: > > echo left_color c yellow | tools/zmqsend > > c sends to all filters > C sends to a specific filter > What problem is there with C ? > > > [...] > > > 2. Proposed API changes > > > > > > To fix/enhance all these issues, I believe a complete rethink of the > > > scheduling design of the library is necessary. I propose the > following > > > changes. > > > > > > Note: some of these changes are not 100% related to the issues I > raised, > > > but looked like a good idea while thinking on an API rework. > > > > > > 2.1. AVFrame.duration > > > > > > Add a duration field to AVFrame; if set, it indicates the duration > of > > > the frame. Thus, it becomes unnecessary to wait for the next frame > to > > > know when the current frame stops, reducing the latency. > > > > > > Another solution would be to add a dedicated function on buffersrc > to > > > inject a timestamp for end or activity on a link. That would avoid > the > > > need of adding a field to AVFrame. > > > > Currently we have pkt_duration in AVFrame. The main problem is that we > > would need to rescale the duration accordingly in all filters when we > > change the timebase. > > > > Alternatively we should add a timebase value to AVFrame. Currently > > that information is stored externally, and this was never easy to > > handle. For example the timestamp in a filter is currently interpreted > > according to the input link time base. Unfortunately this will > > introduce redundancy. > > yes, also theres a somewhat related patch: > Benoit Fouet (4.5K) [FFmpeg-devel] [PATCH] libavu: add pkt_timebase to > AVFrame. > > also speaking of patches, comments for > [FFmpeg-devel] [PATCH] avutil/avstring: support asymmetric escaping in > av_get_token() > are welcome > i cant just apply it as its not 100% compatible with past formating > though its 99.9% and mayb 100% for actual used strings > so this patch really needs comments from people ... > > [...] > -- > Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB > > While the State exists there can be no freedom; when there is freedom there > will be no State. -- Vladimir Lenin > > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > Nicolas, what is status of this? I'm currently interested in frame multithreading in lavfi. _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
