> On Jul 25, 2018, at 11:12 PM, Liu Steven <l...@chinaffmpeg.org> wrote: > > > >> 在 2018年7月26日，上午7:04，Ronak Patel <ronak2121-at-yahoo....@ffmpeg.org> 写道： >> >> Hi, >> >> We’d like to start working on this real soon and want to make sure we >> properly fix this. We understand this code is used by more than just fMP4 >> assets and even live streaming use cases. >> >> Can anyone please help shed some light on all use cases we should make sure >> are working after we make our fixes? >> >> We’re afraid that the number of use cases here are unknown and we’ll be >> stuck fixing this for an unknown time. >> >> Can anyone please shed some light on this? > > you can try -f segment or -f hls or -f dash to do waht you want try, if all > of them cannot suuport your feature, you can submmit patch here. > or you can submmit a new hls muxer patch here if you cannot fix all the > parameters for the old hls muxer. > >> >> Thanks >> >> Ronak >> >>> On Jul 13, 2018, at 8:05 AM, Ronak Patel >>> <ronak2121-at-yahoo....@ffmpeg.org> wrote: >>> >>> I was wondering if anyone on the development list was familiar with this >>> code and could provide pointers on how to fix up the code due to the below >>> problem. >>> >>> >>>> On Jul 12, 2018, at 7:27 PM, Ronak Patel >>>> <ronak2121-at-yahoo....@ffmpeg.org> wrote: >>>> >>>> >>>> >>>>> On Jul 12, 2018, at 6:21 PM, Ronak Patel >>>>> <ronak2121-at-yahoo....@ffmpeg.org> wrote: >>>>> >>>>> Hey Carl, >>>>> >>>>> So I dug into this more today and I have root caused what's exactly >>>>> happening here. >>>>> >>>>> The problematic code is this: >>>>> https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/hlsenc.c#L1368 >>>>> <https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/hlsenc.c#L1368> >>>>> This is where the filename is set and the next line actually opens the >>>>> file. >>>>> >>>>> The logic for this hls_window method is the following: >>>>> >>>>> 1. Make a new temporary file. >>>>> 2. Write out a new HLS manifest header. >>>>> 3. Loop through all available segments and write out all of the entries >>>>> for them. >>>>> 4. Close the temporary file when finished. >>>>> 5. Rename the temporary file to the target file name. >>>>> 6. Rinse and repeat for every single fragment. > > do you mean append fragment info to the m3u8 list, don’t refresh it?
Yup that’s what I mean. The fact that ffmpeg is refreshing the entire manifest hundreds of thousands of times is the reason for this massive slow down. We tried changing the code a little to only write the manifest once once at end, and we can now fragment a 60 hour audio file in less than one minute. Less than one minute compared to 4 days is a BIG time savings. >>>>> >>>>> Therefore, if you can imagine a 153 hour audio file, we write out a >>>>> totally new HLS manifest 550800 times (153 * 60 * 60 assuming a 1s >>>>> fragment duration) that gets progressively larger as each fragment is >>>>> generated. >>>>> >>>>> This is a classic O(N^2) algorithm implementation, instead of: >>>>> >>>>> 1. Creating the destination file up front & write the manifest header. >>>>> 2. Append the new segment to the file. >>>>> 3. If this is the last segment, write out EXT-X-ENDLIST. >>>>> >>>>> There's no looping involved, nor the need to make temporary files. >>>>> >>>>> FYI that I've noticed the same sort of pattern being applied to MPEG >>>>> DASH: >>>>> https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/dashenc.c#L786 >>>>> <https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/dashenc.c#L786> >>>>> >>>>> To implement something like this, looks like we'd have to significantly >>>>> re-engineer the code. Do you have any pointers on how to go about doing >>>>> this? Or, would you be able to help do this? >>>>> >>>> >>>> I was thinking about this and I think it must have been implemented this >>>> way because of live hls streams? Not writing to the main manifest directly >>>> and keeping that stream open does release any file system locks that there >>>> would be otherwise. >>>> >>>> Is this the reason why this was done this way at all? >>>> >>>> >>>>> Thanks for all your help, >>>>> >>>>> Ronak >>>>> >>>>>> On Jun 27, 2018, at 2:04 PM, Carl Zwanzig <c...@tuunq.com> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I haven't traced it out completely, but take a look at the flag >>>>>> HLS_TEMP_FILE in libavformat/hlsenc.c. >>>>>> >>>>>> Later, >>>>>> >>>>>> z! >>>>> _______________________________________________ > > Thanks > > Steven Liu >> > > > > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
