So, let us survey the state of blocking and non-blocking mode at this date in FFmpeg.
First, why and how. This is a question of how the program is designed. There are two big classes of design: output-driven and input-driven. An output-driven design decides it wants to produce some output and reads as much as input to do it, waiting for it if necessary. An input-driven design takes input as it comes, updates its internal state or buffers it, and produces output as it can. There are obviously intermediate designs. Output-driven design is easier because state can be kept in local variables, including the instruction pointer. In input-driven design, if the amount if input available does not allow to perform a full step of processing, it must be fragmented or delayed somehow. On the other hand, as soon as a program has several live inputs, input that can receive data at any time and require immediate attention, input-driven is the only solution. If you are old enough, you may remember the time when web browsers would completely freeze when the DNS was lagging, because the DNS was handled in an output-driven way, without letting the browser react even to its GUI input. Historically, Unix favors an output-driven design. FFmpeg closely imitates Unix's design. When it became obvious that was not sustainable, Unix first cut the problem like the Gordian knot: signals that bypass the normal code paths and cause immediate action: ^C is how you provide user "input" to a program that is blocked reading from its data input. Since that is not sufficient for more modern applications, they introduced non-blocking mode. In this mode, inputs do not wait for data, they return immediately a special code that means "no data available immediately". Programs need to be ready for that code and try other inputs immediately and this one later. This is not enough: trying the input later is bad, because it requires a guess at how much later means. If later is long, the program is unresponsive. If later is short, the program will wake often for nothing. This last point is especially crucial for embedded application, where unnecessary wake-ups can prevent the hardware from going into deep sleep, causing a huge drain on the battery. (FFmpeg should be usable for embedded applications.) To solve this issue, Unix introduced the select() and then poll() functions, that block until one of the specified inputs is ready. With non-blocking mode and poll(), the normal structure of a program with several inputs becomes a single loop, blocking waiting for any input in a single poll() call, and then processing the ones that are ready before starting again. This is called an event loop. A few extra notes before starting with FFmpeg itself. I have only spoken about inputs yet. Outputs are easier: just send the data. And take care of flow control, which is my next point. Flow control means that the program can stop an input from receiving data. It may or may not be possible, depending on the input: TCP has flow control, broadcast television does not. Plain files need flow control, or they will flood your input. There is another solution to handle several inputs in a program: threads. They promise to make everything about handling several inputs simpler: just have one thread per input. But the reality is not so bright. If you need to stop a read operation from the network because the user canceled it from the GUI, you need a way of controlling a thread that is blocked in a system call. For this very simple and common case, you can use pthread_cancel(), but we have seen how even a simple function like that can cause no end of portability problems. For more unusual cases, you will need to implement some kind of message passing with the thread, and since message passing is not integrated with blocking system calls, you will also need to implement an event loop in the thread. An event loop in each thread. Yet, threads can be a life-saving solution for a particular issue: they allow to take an output-driven piece of code and run it as input-driven: just leave the thread running when the code is blocked on an input, and wake it when the input is received. Note that the ability of threads to run concurrently is not necessary for this to work, only the ability to switch context and back. But nowadays, threads are probably more tested and portable than contexts. Now, the state of non-blocking in current FFmpeg code. Most input APIs of FFmpeg are output-driven: av_read_frame() will wait until input is received. (For comparison, filters, codecs and bitstream filters are now all input-driven.) Blocking mode is the default, and it works. On the other hand, non-blocking often does not work. There will be a lot of cases: - Simple protocols, i.e. protocols that interact directly with a network API or a library, mostly work in both non-blocking mode, and can be integrated in a poll()/select() call using ffurl_get_file_handle(), internally but not externally. - Complex protocols, i.e. protocols that rely on one or several other protocols (like HTTP relies on TCP, or even on TLS that itself relies on TCP) may or may not work in non-blocking mode. Mostly not. - Simple demuxers, i.e. demuxers that use a single AVIO stream for their input, mostly do not work in non-blocking mode. - Complex demuxers, i.e. demuxers that use other demuxers or several AVIO streams, probably do not work. - Some (most?) device demuxers, i.e. demuxers that directly read from a device, work. - Simple/complex/device muxers are mostly the same as demuxers, but since it is about output, it is less a problem. And finally, how do we make it better? What we need urgently is an event loop. Or even better, a scheduler, i.e. a multi-thread event loop. This is not a very complex task. The hard part is to carefully plan ahead to make sure it can handle all the cases we need, including flow control. Demuxers and protocols will then need to be adapted to integrate in the scheduler. But it will not happen at once, so we will need wrappers to run the current demuxers and protocols in threads connected to the scheduler. Then we have four cases, depending on whether the application uses the API with the scheduler or the current output-driven API, and whether is uses a current output-driven module or an updated scheduled module. To call a current output-driven module from the current output-driven API, nothing changes. To call an updated scheduled module with the scheduler API, that is straightforward. To call an updated scheduled module from the current output-driven API, we create a new instance of scheduler just for it, and we run it until we have out output. The scheduler needs to be light-weight to make this workable. To call a current output-driven module with the scheduler API, we call its blocking function from a separate thread. We do not need to create a thread per module, but only a thread per level of recursive blocking calls; for example concat→Matroska→HTTP→TCP requires four threads, even if there are thousand of Matroska files in the concat. What would it change in practice? I am quite sure it can be made reasonably simple. First, allocate a scheduler for your whole program: av_scheduler_create(&scheduler, ...);. Then attach the AVFormatContext and AVIO instances to the scheduler before using them: avformat_schedule(avf, scheduler);. Then, optionally, attach callbacks for messages from your demuxers: your function will be called each time the demuxer has produced output, the equivalent of av_read_frame() returning. Or, if you do not like callbacks, the default can be to stop the scheduler and return the frame, with all the metadata telling you from which demuxer it comes: process it and re-run the scheduler. I have been thinking about this for a long time, and I think it is mostly ready in my head. I can produce the outline of an actual API with its documentation to really start the conversation. Regards, -- Nicolas George
signature.asc
Description: PGP signature
_______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
