[FFmpeg-devel] Strategy to optimally playback audio & video
Hi all, My app needs best performance video playback in high resolutions. I have a graphic engine polling the video frames and an audio engine polling decoded audio packets. The problem is, sometimes, the audio thread needs data that are not ready, so it decodes all packets until it got the required audio data. In this case video frames are also decoded and pushed in a queue. But if it implies decoding 10 HD video frames, I won’t get the audio data in time and I’ll get audio issues. (I have a few videos in which I have 10 consecutive video frames, then audio content, then 10 frames ….) What should I do in this case ? I see various solutions: 1- I could put in cache more than one second of audio and video and I should be ok, but with 4k movies, keeping 30 frames could imply using a huge amount of memory and I would like to avoid that. 2- I could open the file twice, one for video and one for audio, but I would use even more RAM and it would decrease performances because parsing the file twice 3- I could avoid decoding video frames if I’m missing audio data but that would imply seeking back after to get a clean video frame (so decoding again from previous keyframe), so I think that’s not a option at all. 4- Any better option ? I hope I’m on the right mailing list. Thanks a lot, Kind regards, Matt Beghin ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] Strategy to optimally playback audio & video
Ok. Thanks for your replies. Keeping the audio in separate files is an idea, but I’d like to be able to work directly with a movie, without recompression. I found a solution to my problem: when the audio thread needs data, I read all packets but I only decompress audio packets, video packets are pushed in a queue and decompressed later from the graphic engine thread. Thanks and sorry for posting at the wrong place ! Matt > On 13 Sep 2016, at 00:10, Sven C. Dack wrote: > > On 11/09/16 20:37, Matthieu Beghin wrote: >> I see various solutions: >> >> 1- I could put in cache more than one second of audio and video and I >> should be ok, but with 4k movies, keeping 30 frames could imply using a huge >> amount of memory and I would like to avoid that. >> 2- I could open the file twice, one for video and one for audio, but I >> would use even more RAM and it would decrease performances because parsing >> the file twice >> 3- I could avoid decoding video frames if I’m missing audio data but that >> would imply seeking back after to get a clean video frame (so decoding again >> from previous keyframe), so I think that’s not a option at all. >> 4- Any better option ? >> > > Keep audio and video in separate files. YouTube seems to be doing this for > their 4K videos. > > If you cannot cache the video for long then cache the audio so it can cause > less of a problem. > > Divide and conquer... > > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
[FFmpeg-devel] libavcodec memory usage
Hi, My application is using libavcodec etc. to playback movies. Everything is ok with our 64 bits app. But I have a 32 bits app and there when playing a 8k we reach 3 GB memory usage, so when adding another 4k or a few HD movies, the app crashes. I haven’t found a way to reduce memory usage. The library stores lots of frames or data. I tried reducing thread_count to 1 to test, but it does not help and performances are not acceptable. Is there a way to limit the size of buffered data / number of buffered frames ? I’m using FFMPEG 2.8.7 (libavcodec.56.60.100 / libavformat.56.40.101). Thanks, Best Regards, Matt Beghin ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] libavcodec memory usage
> I'm assuming that you're talking about 8K H264 with pixfmt=yuv420p10? 8K > yuv420p10 frames are 100MB Correct > 32 plus delayed output and current_pic gives H264_MAX_PICTURE_COUNT = 36 > without threading: 36*100=3.6GB. What do you mean by "delayed output” ? Can you link me to a document ? > Or maybe consider ditching 32bit support? Unfortunately, that’s 10 years old app using OSX Carbon library (not 64 bits), and rewriting that part would be too much work. As an alternative, is there a way to know how much bytes are buffered ? Iterating buffered frames ? Thanks! > On 03 Jan 2017, at 17:09, Ronald S. Bultje wrote: > > Hi, > > On Tue, Jan 3, 2017 at 10:58 AM, Matthieu Beghin < > matthieu.beg...@garagecube.com> wrote: > >> Hi, >> >> My application is using libavcodec etc. to playback movies. Everything is >> ok with our 64 bits app. But I have a 32 bits app and there when playing a >> 8k we reach 3 GB memory usage, so when adding another 4k or a few HD >> movies, the app crashes. >> >> I haven’t found a way to reduce memory usage. The library stores lots of >> frames or data. I tried reducing thread_count to 1 to test, but it does not >> help and performances are not acceptable. >> >> Is there a way to limit the size of buffered data / number of buffered >> frames ? >> I’m using FFMPEG 2.8.7 (libavcodec.56.60.100 / libavformat.56.40.101). > > > I'm assuming that you're talking about 8K H264 with pixfmt=yuv420p10? 8K > yuv420p10 frames are 100MB, H264 can have up to 16 refs/field, for > interlaced content this means 32 plus delayed output and current_pic gives > H264_MAX_PICTURE_COUNT = 36 without threading: 36*100=3.6GB. > > There's not much you can do about this in the decoder side. You can tell > your encoder to use less references and discard references earlier. How to > do this depends on which encoder you're using. > > Or maybe consider ditching 32bit support? > > Ronald > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] libavcodec memory usage
> On 03 Jan 2017, at 17:55, Ronald S. Bultje wrote: > > Hi, > > On Tue, Jan 3, 2017 at 11:44 AM, Matthieu Beghin < > matthieu.beg...@garagecube.com> wrote: > >>> I'm assuming that you're talking about 8K H264 with pixfmt=yuv420p10? 8K >> yuv420p10 frames are 100MB >> >> Correct >> >>> 32 plus delayed output and current_pic gives H264_MAX_PICTURE_COUNT = 36 >> without threading: 36*100=3.6GB. >> >> What do you mean by "delayed output” ? Can you link me to a document ? > > > Out-of-order coded P-frames, they are buffered internally but not counted > (and thus extra) on top of the max_refs value. I believe the spec calls > this dpb (delayed pic buffer). > >> Or maybe consider ditching 32bit support? >> >> Unfortunately, that’s 10 years old app using OSX Carbon library (not 64 >> bits), and rewriting that part would be too much work. >> > > I'm frowning at you. Really badly. > > As an alternative, is there a way to know how much bytes are buffered ? >> Iterating buffered frames ? > > > What are you hoping to accomplish by knowing how many bytes are buffered? > It doesn't solve the problem? But anyway, check > H264Context->short/long_ref[]; delayed_pic[] gives you the delayed output. > I’m trying to let people use 8k movies if it will fit, but in case it would use more than 1 GB, refuse to load the file. My application is a live application, it cannot crash, and actually, it can... So to know how much bytes livavcodec is using, I have to know the number of frames / field, if it is interlaced and the number of delayed frames, that's correct ? How can I know the number of frames / field ? To know if it’s interlaced: AVFrame::interlaced_frame Number of delayed frames ? Another solution is to start playing the movie and to check the buffered data amount each time I decode a frame, then stop the movie if it uses more than 1 GB... In this case, just iterating H264Context->short/long_ref[]; delayed_pic[] would be enough ? How can I get the H264Context ? Thanks ! Matthieu > Ronald > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] libavcodec memory usage
Hi, > On 03 Jan 2017, at 18:26, Ronald S. Bultje wrote: > > Hi, > > On Tue, Jan 3, 2017 at 12:16 PM, Matthieu Beghin < > matthieu.beg...@garagecube.com> wrote: > >> >>> On 03 Jan 2017, at 17:55, Ronald S. Bultje wrote: >>> >>> Hi, >>> >>> On Tue, Jan 3, 2017 at 11:44 AM, Matthieu Beghin < >>> matthieu.beg...@garagecube.com> wrote: >>> >>>>> I'm assuming that you're talking about 8K H264 with pixfmt=yuv420p10? >> 8K >>>> yuv420p10 frames are 100MB >>>> >>>> Correct >>>> >>>>> 32 plus delayed output and current_pic gives H264_MAX_PICTURE_COUNT = >> 36 >>>> without threading: 36*100=3.6GB. >>>> >>>> What do you mean by "delayed output” ? Can you link me to a document ? >>> >>> >>> Out-of-order coded P-frames, they are buffered internally but not counted >>> (and thus extra) on top of the max_refs value. I believe the spec calls >>> this dpb (delayed pic buffer). >>> >>>> Or maybe consider ditching 32bit support? >>>> >>>> Unfortunately, that’s 10 years old app using OSX Carbon library (not 64 >>>> bits), and rewriting that part would be too much work. >>>> >>> >>> I'm frowning at you. Really badly. >>> >>> As an alternative, is there a way to know how much bytes are buffered ? >>>> Iterating buffered frames ? >>> >>> >>> What are you hoping to accomplish by knowing how many bytes are buffered? >>> It doesn't solve the problem? But anyway, check >>> H264Context->short/long_ref[]; delayed_pic[] gives you the delayed >> output. >>> >> >> I’m trying to let people use 8k movies if it will fit, but in case it >> would use more than 1 GB, refuse to load the file. My application is a live >> application, it cannot crash, and actually, it can... >> >> So to know how much bytes livavcodec is using, I have to know the number >> of frames / field, if it is interlaced and the number of delayed frames, >> that's correct ? >> >> How can I know the number of frames / field ? >> To know if it’s interlaced: AVFrame::interlaced_frame >> Number of delayed frames ? >> >> Another solution is to start playing the movie and to check the buffered >> data amount each time I decode a frame, then stop the movie if it uses more >> than 1 GB... >> In this case, just iterating H264Context->short/long_ref[]; delayed_pic[] >> would be enough ? >> How can I get the H264Context ? > > > You can't access H264Context outside libavcodec without hacks... > > Does something in libavcodec crash if you use too much mem? (If so: what? > We want to fix this...) Or something else? I’m generally blocked by crashes in Apple nVidia GL driver. When I’m not blocked by my own assertions (I should use a macro…) but anyway if the GL driver calls crash I can’t reach memory limit. > The idea is that on OOM, libavcodec mallocs fail and we return gracefully. > If that doesn't happen, we want to fix it. If I have any crash in FFMPEG when reaching memory limit, I’ll document it and send you the info. > Ronald > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
