On Thu, Dec 9, 2021 at 10:33 PM Daniel Cantarín <ca...@canta.com.ar> wrote:
> Hi there. > This is my first message to this list, so please excuse me if I > unintendedly break some rule. > > I've read the debate between Soft Works and others, and would like to > add something to it. > I don't have a deep knowledge of the libs as other people here show. My > knowledge comes from working with live streams for some years now. And I > do understand the issue about modifying a public API for some use case > under debate: I believe it's a legit line of questioning to Soft Works > patches. However, I also feel we live streaming people are often let > aside as "border case" when it comes to ffmpeg/libav usage, and this > bias is present in many subtitles/captions debates. > > I work with Digital TV signals as input, and several different target > outputs more related to live streaming (mobiles, PCs, and so on). The > target location is Latin America, and thus I need subtitles/captions for > when we use english spoken audio (we speak mostly Spanish in LATAM). TV > people send you TV subtitle formats: scte-27, dvb subs, and so on. And > live streaming people uses other subtitles formats, mostly vtt and ttml. > I've found that CEA-608 captions are the most compatible caption format, > as it's understood natively by smart tvs and other devices, as well as > non-natively by any other device using popular player-side libraries. > So, I've made my own filter for generating CEA-608 captions for live > streams, using ffmpeg with the previously available OCR filter. Tried > VTT first, but it was problematic for live-streaming packaging, and with > CEA-608 I could just ignore that part of the process. > > While doing those filters, besides the whole deal of implementing the > conversion from text to CEA-608, I struggled with stuff like this: > - the sparseness of input subtitles, leading to OOM in servers and > stalled players. > - the "libavfilter doesn't take subtitle frames" and "it's all ASS > internally" issues. > - the "captions timings vs video frame timings vs audio timings" > problems (people talk a lot about syncing subs with video frames, but > rarely against actual dialogue audio). > - other (meta)data problems, like screen positioning or text encoding. > > This are all problems Soft Works seems to have faced as well. > > But of all the problems regarding live streaming subtitles with ffmpeg > (and there are LOTS of it), the most annoying problem is always this: > almost every time someone talked about implementing subtitles in filters > (in mail lists, in tickets, in other places like stack overflow, > etcetera), they always asumed input files. When the people specifically > talked about live streams, their peers always reasoned with files > mindset, and stated live streaming subtitles/captions as "border case". > > Let me be clear: this are not "border case" issues, but actually appear > in the most common use cases of live streaming transcoding. They all > appear *inmediatelly* when you try to use subtitles/captions in live > streams. > > I got here (I mean this thread) while looking for ways to fixing some > issues in my setup. I was reconsidering VTT/TTML generation instead of > CEA-608 (as rendering behave significantly different from device to > device), and thus I was about to generate subtitle type output from some > filter, was about to create my own standalone "heartbeat" filter to > normalize the sparseness, and so on and so on: again, all stuff Soft > Works seems to be handling as well. So I was quite happy to find someone > working on this again; last time I've seen it in ffmpeg's > mailing/patchwork > ( > https://patchwork.ffmpeg.org/project/ffmpeg/patch/20161102220934.26010-...@pkh.me) > > the code there seemed to die, and I was already late to say anything > about it. However, reading the other devs reaction to Soft Works work > was worrying, as it felt as history wanted to repeat itself (take a look > at discussions back then). > > It has been years so far of this situation. This time I wanted to > annotate this, as this conversation is still warm, in order to help Soft > Works's code survive. So, dear devs: I love and respect your work, and > your opinion is very important to me. I do not claim to know better than > you do ffmpeg's code. I do not claim to know better what to do with > libavfilter's API. Please understand: I'm not here to be right, but to > note my point of view. I'm not better than you; quite on the contrary > most likely. But I also need to solve some very real problems, and can't > wait until everything else is in wonderful shape to do it. I can't also > add lots of conditions in order to just fix the most immediate issues; > like it's the case with sparseness and heartbeat frames, which was a > heated debate years ago and seems to still be one, while I find it to be > the most obvious common sense backwards-compatible solution > implementation. Stuff like "clean" or "well designed" can't be more > important than actually working use cases while not breaking previously > implemented ones: because it's far easier to fix little blocks of "bad" > code rather than design something everybody's happy with (and history of > the project seems to be quite eloquent about that, specially when it > comes to this particular use cases). Also, I have my own patches (which > I would like to upstream some day), and I can tell the API do change > quite normally: I understand that should be a curated process, but > adding a single property for live-streaming subtitles isn't also > anybody's death, and thus that shouldn't be the kind of issues that > blocks big and important code implementations like the ones Soft Works > is working on; I just don't have the time to do myself all that work > he/she's doing, and it could be another bunch of years until someone > else have it. I can't tell if Soft Works code is well enough for you, or > if the ideas behind it are the best there are, but I can tell you the > implementations are in the right track: as a live streaming worker, I > know the problems he/she mentions in their exchanges with you all, and I > can tell you they're all blocking issues when dealing with live > streaming. Soft Work is not "forcing it" into the API, and this are not > "border cases" but normal and frequent live streaming issues. So, > please, if you don't have the time Soft Works have, or the will to > tackle the issues he/she's tackling, I beg you at least don't kill the > code this time if it does not breaks working use cases. > > You can not do much than send mails, because this long ago stopped being technical issue but ego-booster issue. > > Thanks, > Daniel. > _______________________________________________ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel > > To unsubscribe, visit link above, or email > ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe". > _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
