Re: [FFmpeg-devel] FFMPEG support for x265 vbv-end and vbv-end-fr-adj

[Tom Vaughan]

Michael - Sorry I missed your response on 5/15. I led the x265 project from 
2013 to 2018, and I helped to define this feature, but I can't speak for the 
x265 development team today. I could put you directly in touch with the x265 
developers at Multicoreware to work this out. I will also encourage them to 
respond to this thread directly on the FFMPEG-devel mailing list.

Speaking as an end-user, I don't think we require frame-accurate precision. The 
vbv-end-fr-adj value specifies the point at which the vbv-end logic is 
activated. vbv-end-fr-adj is a value between 0 and 1, but should always be less 
than 1. Multiply vbv-end-fr-adj by 100 to get the % of the transcoding job in 
time where the vbv-end feature needs to be triggered. A flag that is passed 
from FFMPEG to x265 at this point would work. I'm aware of the challenges of 
frame-rate accuracy when transcoding. I would say that a "best effort" from 
FFMPEG to trigger the signal at the right point is good enough. Non-decodable 
frames should count as if they were decodable, to not throw off the time 
calculation.

As an example, if someone is chunking the encoding of a video title into 300 
second chunks, and they want the vbv-end enforcement to start 15 seconds before 
the end of the chunk encode, they would use vbv-end-fr-adj 0.95. When x265 gets 
to the last 5% of the encode (last 15 seconds), vbv-end logic would be 
activated. 1 or 2 frames inaccuracy won't matter.

Thanks,
Tom

On 5/15/24, 5:59 AM, "ffmpeg-devel on behalf of Michael Niedermayer" 
mailto:ffmpeg-devel-boun...@ffmpeg.org> on 
behalf of mich...@niedermayer.cc <mailto:mich...@niedermayer.cc>> wrote:


On Wed, May 01, 2024 at 09:10:51PM +, Tom Vaughan wrote:
> When I attempt to pass vbv-end and vbv-end-fr-adj parameters to x265, FFMPEG 
> fails.
> 
> x265 [error]: vbv-end-fr-adj cannot be enabled when total number of frames is 
> unknown
> 
> x265 needs to know how many frames it is encoding so that it knows when the 
> vbv-end logic needs to kick in. In this case it would be after 96% of the 
> source video is encoded. Despite specifying a duration of 30 seconds ( -t 30 
> ) on both input and output (using a 60 second source video), x265 does not 
> know the total number of frames (but FFMPEG surely does).
> 
> ffmpeg -i test_uhdsdr_1min.mov -t 30 -an -sn -dn -c:v libx265 -b:v 12000k 
> -maxrate 8M -bufsize 18M -x265-params 
> vbv-init=0.9:vbv-end=0.9:vbv-end-fr-adj=.96:rc-lookahead=48:qg-size=32:scenecut=0:no-open-gop=1:frame-threads=0:repeat-headers=1:nr-inter=400:nr-intra=100:psy-rd=0:cbqpoffs=0:crqpoffs=3
>  -t 30 test_uhdsdr_96.mp4


you can specify total_frames along with vbv-end-fr


there is in general not a 1:1 correspondance between duration and the number of 
frames
the frame rate may vary, there may be skiped frames or damaged frames that
where lost and so on.


in case of some 2 pass mode it should be possible to get the exact number of 
frames
in the 2nd pass.
Otherwise my gut feelng is that a time based instead of a frame number based
threshold would have been easier to work with


if you are thinking that the total frame number should be set automatically
and not by hand. I agree, its just not easy to do that in all cases.
It could maybe be done in specific cases but in the general case
it would need 2 passes to be reliable.


thx


[...]
-- 
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB


Dictatorship: All citizens are under surveillance, all their steps and
actions recorded, for the politicians to enforce control.
Democracy: All politicians are under surveillance, all their steps and
actions recorded, for the citizens to enforce control.



___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

[FFmpeg-devel] FFMPEG support for x265 vbv-end and vbv-end-fr-adj

[Tom Vaughan]

 / 1 intra
x265 [info]: ME / range / subpel / merge : hex / 57 / 2 / 3
x265 [info]: Lookahead / bframes / badapt: 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb   : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 3 / off / on
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress: ABR-12000 kbps / 0.60
x265 [info]: VBV/HRD buffer / max-rate / init / end / fr-adj: 18000 / 18000 / 
0.900 / 0.900 / 0.960
x265 [info]: tools: rd=3 psy-rd=2.00 early-skip rskip mode=1 signhide tmvp
x265 [info]: tools: b-intra strong-intra-smoothing lslices=8 deblock sao
x265 [info]: frame I: 30, Avg QP:33.26  kb/s: 53324.81
x265 [info]: frame P:318, Avg QP:35.26  kb/s: 21643.30
x265 [info]: frame B:   1091, Avg QP:37.73  kb/s: 8003.99
x265 [info]: Weighted P-Frames: Y:11.6% UV:9.4%

encoded 1439 frames in 937.04s (1.54 fps), 11962.94 kb/s, Avg QP:37.09

Tom Vaughan
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Patch for libx265 memory leak

[Tom Vaughan]

Any further details you or others could provide to help the x265 dev team 
resolve the issue would be appreciated.

Can you share your steps to reproduce this issue? What parameters were changed? 
Max CTU size?

Is this documentation inadequate? 
https://x265.readthedocs.io/en/master/api.html#build-considerations

On 11/11/24, 10:46 PM, "ffmpeg-devel on behalf of Damiano Galassi" 
mailto:ffmpeg-devel-boun...@ffmpeg.org> on 
behalf of dam...@gmail.com > wrote:


On Tue, Nov 12, 2024 at 3:38 AM Zhao Zhili mailto:quinkbl...@foxmail.com>> wrote:


>
> cleanup() release library static allocations, and I don’t see lock or
> reference count
> within x265_cleanup(). So call cleanup() will break other x265 encoder
> instance,
> right?




x265 already crashes when trying to run two encodes with different settings
in
the same process, because it's storing some instance specific data in
global variables
that are overwritten when starting a new encode.


So at least with this patch it won't leaks the global data,
but the root issue needs to be fixed in x265.
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org 
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel 



To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org  with 
subject "unsubscribe".



___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Captions SCC

[Tom Vaughan]

Soft Works said... "the CC data needs to get into the video frames - so that it 
exists as input to the video encoder.
For the same reason, it is also not possible to add CCs at the muxer level. It 
always requires video encoding because the data needs to get into the video 
stream itself, so there's no way to get data from an SCC file into a video 
stream currently."

You're confusing subtitles with closed captions. Subtitles are text that is 
composited into the video. CEA-708 closed captions are carried as metadata in 
SEI NAL units, muxed into the elementary video stream.

Tom

On 2/6/25, 11:48 PM, "ffmpeg-devel on behalf of Soft Works" 
mailto:ffmpeg-devel-boun...@ffmpeg.org> on 
behalf of softworkz-at-hotmail@ffmpeg.org 
> wrote:


> -Original Message-
> From: ffmpeg-devel  > On Behalf Of
> Devlist Archive
> Sent: Friday, February 7, 2025 6:13 AM
> To: FFmpeg development discussions and patches  de...@ffmpeg.org >
> Subject: Re: [FFmpeg-devel] Captions SCC
> 
Hi Zack,
that message from "Jack" had confused me for a moment, but on re-reading it 
appears to be an AI response.
The content is total nonsense. There is no "SCC" encoder in ffmpeg, and if 
there was one, it wouldn't help much because the CC data needs to get into the 
video frames - so that it exists as input to the video encoder.
For the same reason, it is also not possible to add CCs at the muxer level. It 
always requires video encoding because the data needs to get into the video 
stream itself, so there's no way to get data from an SCC file into a video 
stream currently.

The only way in the ffmpeg architecture to get CC data encoded would be at the 
filtering level. The Subtitle Filtering patchset that I have submitted a while 
ago has a "splitcc" filter which has video as input and a video plus a subtitle 
pin/pad at the output side.
What would be needed is a reverse pendant like a "mergecc" filter with video 
and subtitles input - plus a CEA-608/708 encoder.

softworkz

___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Captions SCC

[Tom Vaughan]

I know what an elementary video stream is. I founded and led development of 
x265, which produces elementary video streams. Elementary video streams can 
contain SEI messages.
It's common to refer to the integration of closed captions into an elementary 
video stream as multiplexing. For example, see 
https://doc.haivision.com/MakitoX4Enc/1.7/closed-captioning

It is common for people in the industry to refer to burned-in (or open) 
captions as subtitles. Those are the only type of captions that would require a 
video encoder. 

Closed captions are pure metadata, encapsulated in Network Access Layer Units 
(NALUs). They are embedded by being muxed into the elementary video stream. See 
https://www.closedcaptioncreator.com/blog/articles/how-to-embed-608-708-closed-captioning.html

Even though embedded captions may appear as a separate data track in a media 
container, they are embedded in the elementary video bitstream.

Tom

On 2/7/25, 10:55 AM, "ffmpeg-devel on behalf of Soft Works" 
mailto:ffmpeg-devel-boun...@ffmpeg.org> on 
behalf of softworkz-at-hotmail@ffmpeg.org 
<mailto:softworkz-at-hotmail@ffmpeg.org>> wrote:

> -Original Message-
> From: ffmpeg-devel  <mailto:ffmpeg-devel-boun...@ffmpeg.org>> On Behalf Of Tom
> Vaughan
> Sent: Friday, February 7, 2025 7:36 PM
> To: FFmpeg development discussions and patches  de...@ffmpeg.org <mailto:de...@ffmpeg.org>>
> Subject: Re: [FFmpeg-devel] Captions SCC
>
> Soft Works said... "the CC data needs to get into the video frames -
> so that it exists as input to the video encoder.
> For the same reason, it is also not possible to add CCs at the muxer
> level. It always requires video encoding because the data needs to
> get into the video stream itself, so there's no way to get data from
> an SCC file into a video stream currently."


Hi Tom,
> You're confusing subtitles with closed captions. 
No.

> Subtitles are text that is composited into the video.
No.
> CEA-708 closed captions are
> carried as metadata in SEI NAL units, muxed into the elementary video
> stream.
I wouldn't call this "muxing" but that's not worth a debate.
(you might want to spend a thought on why it's called "elementary stream")

I said "it is also not possible to add CCs at the muxer level".

By that I mean "muxing" in ffmpeg terminology which is unambiguous.
sw




> 
> On 2/6/25, 11:48 PM, "ffmpeg-devel on behalf of Soft Works"  devel-boun...@ffmpeg.org <mailto:devel-boun...@ffmpeg.org> 
> <mailto:ffmpeg-devel-boun...@ffmpeg.org 
> <mailto:ffmpeg-devel-boun...@ffmpeg.org>> on
> behalf of softworkz-at-hotmail@ffmpeg.org 
> <mailto:softworkz-at-hotmail@ffmpeg.org> <mailto:softworkz-at-
> hotmail@ffmpeg.org <mailto:hotmail@ffmpeg.org>>> wrote:
> 
> 
> > -Original Message-
> > From: ffmpeg-devel  > <mailto:ffmpeg-devel-boun...@ffmpeg.org> <mailto:ffmpeg-
> devel-boun...@ffmpeg.org <mailto:devel-boun...@ffmpeg.org>>> On Behalf Of
> > Devlist Archive
> > Sent: Friday, February 7, 2025 6:13 AM
> > To: FFmpeg development discussions and patches  > de...@ffmpeg.org <mailto:de...@ffmpeg.org> <mailto:de...@ffmpeg.org 
> > <mailto:de...@ffmpeg.org>>>
> > Subject: Re: [FFmpeg-devel] Captions SCC
> >
> Hi Zack,
> that message from "Jack" had confused me for a moment, but on re-
> reading it appears to be an AI response.
> The content is total nonsense. There is no "SCC" encoder in ffmpeg,
> and if there was one, it wouldn't help much because the CC data needs
> to get into the video frames - so that it exists as input to the
> video encoder.
> For the same reason, it is also not possible to add CCs at the muxer
> level. It always requires video encoding because the data needs to
> get into the video stream itself, so there's no way to get data from
> an SCC file into a video stream currently.
> 
> The only way in the ffmpeg architecture to get CC data encoded would
> be at the filtering level. The Subtitle Filtering patchset that I
> have submitted a while ago has a "splitcc" filter which has video as
> input and a video plus a subtitle pin/pad at the output side.
> What would be needed is a reverse pendant like a "mergecc" filter
> with video and subtitles input - plus a CEA-608/708 encoder.
> 
> softworkz
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org <mailto:ffmpeg-devel@ffmpeg.org>
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel 
> <https://ffmpeg.org/mailman/listinfo/ffmpeg-devel>
> 
> To unsubscribe, v

Re: [FFmpeg-devel] Captions SCC

[Tom Vaughan]

I'm excited to see this discussion. There are many leading video distributors 
(streaming services, etc.) that would love to see solid support in FFMPEG for 
closed captions. No promises, but my sense is that additional money can be 
raised to sponsor this development.

WebVTT as a sidecar captions file is the modern way to handle captions. The 
latest streaming client devices / applications support WebVTT captions. New 
movies and TV shows are produced with WebVTT captions. 

For legacy device support, it is still necessary to convert the WebVTT to 
embedded 708 captions.

Decoding embedded 708 (from older video content), converting to WebVTT or SCC 
would be a valuable capability.

Tom 

On 2/8/25, 6:44 AM, "ffmpeg-devel on behalf of Devin Heitmueller" 
mailto:ffmpeg-devel-boun...@ffmpeg.org> on 
behalf of devin.heitmuel...@ltnglobal.com 
> wrote:


On Fri, Feb 7, 2025 at 6:59 PM Marth64 mailto:mart...@proxyid.net>> wrote:
>
> Hi Devin & Softworks,
>
> I have been working on a slide deck in my spare time with suggestions
> and a roadmap to improve general digital Closed Captions support.
> I hope to share this with you and the community for feedback in the
> next 2 weeks. I meant to wrap it up sooner but had some IRL stuff come
> up.
> I have some ideas and proof of concepts that might help a lot.


I'm certainly happy to discuss. I've got a ton of 608/708 captioning
related code outside the ffmpeg tree, so depending on what you're
trying to accomplish I may be able to offer some code up to help. If
nothing else, I've got *alot* of experience in this area and might be
able to offer some feedback.


Devin


-- 
Devin Heitmueller, Senior Software Engineer
LTN Global Communications
o: +1 (301) 363-1001
w: https://ltnglobal.com e : 
devin.heitmuel...@ltnglobal.com 
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org 
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel 



To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org  with 
subject "unsubscribe".



___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-devel] Captions SCC

[Tom Vaughan]

I suspect WebVTT became the modern CC format because...
- It is natively supported by the HLS and DASH specifications
- 
https://developer.apple.com/documentation/http-live-streaming/hls-authoring-specification-for-apple-devices
- 
https://reference.dashif.org/dash.js/latest/samples/captioning/caption_vtt.html
- Streaming is HTTP - based, and WebVTT is a W3C specification.
- https://www.w3.org/TR/webvtt1/
- WebVTT is delivered as a component (separate "sidecar" file). As such a 
streaming title can be updated to fix or improve a bad captions track, or to 
add support for additional languages without re-encoding or re-muxing the 
encoded video files. Updated a title to add support for another language only 
requires encoding the additional audio track, then packaging the updated set of 
audio, video and WebVTT files.
- Wide industry support in HLS and DASH packaging libraries (open source and 
commercial)
- Wide industry support in streaming player client platforms - both software 
and device native players
- Better support of dynamic ad insertion (avoids timing errors that cause 
program captions to appear over an ad, or vice versa)

As Zach noted, WebVTT supports setting the font color and background color. It 
also supports cue styling, including bold, underlined and italics.
See https://developer.mozilla.org/en-US/docs/Web/API/WebVTT_API for a live 
example.

Tom

On 2/9/25, 9:42 AM, "ffmpeg-devel on behalf of Devlist Archive" 
mailto:ffmpeg-devel-boun...@ffmpeg.org> on 
behalf of devl...@rlb.org <mailto:devl...@rlb.org>> wrote:


>
> Not to start an argument, but WebVTT is kind of a terrible format.
> It's a lowest common denominator and loses most formatting information
> available even in 608 (which is now more than 40 years old). Stuff
> like rollup captions for live programming, color (to distinguish
> speakers) and caption positioning are pretty important to the hearing
> impaired.

From the reading I have done, the WebVTT does support some placement,
italics, and appearance information, but not all players or ripping
programs support those functions. It does appear that 708 has even more
options for the number of captions windows, font sizes, colors, etc that
can make a big difference in experience.

The new captions standard for ATSC 3.0 is actually ISMC1 profile TTML files
that are a sidecar configuration by default. I would highly recommend
focusing on 608 and 708 compatibility first, and only then make sure all
possible features of those standards can be exported to webVTT for those
who will sponsor it. ISMC1 profile TTML won't really be needed until the
majority of TV tuners ship with ATSC 3.0 which is still quite far off.
Implementation of TTML outputs from 708 tracks will require the ability to
extract the entire A/53 Part 4 encoded stream to a sidecar file so
implementing bitstream filters to both mux and demux the captions stream
from the video stream for the full A/53 Part 4 dataset will be required.

Zach




On Sun, Feb 9, 2025 at 6:03 AM Devin Heitmueller <
devin.heitmuel...@ltnglobal.com <mailto:devin.heitmuel...@ltnglobal.com>> wrote:


> On Sat, Feb 8, 2025 at 10:39 PM Tom Vaughan  <mailto:t...@tvaughan.com>> wrote:
> >
> > I'm excited to see this discussion. There are many leading video
> distributors (streaming services, etc.) that would love to see solid
> support in FFMPEG for closed captions. No promises, but my sense is that
> additional money can be raised to sponsor this development.
> >
> > WebVTT as a sidecar captions file is the modern way to handle captions.
> The latest streaming client devices / applications support WebVTT captions.
> New movies and TV shows are produced with WebVTT captions.
>
> Not to start an argument, but WebVTT is kind of a terrible format.
> It's a lowest common denominator and loses most formatting information
> available even in 608 (which is now more than 40 years old). Stuff
> like rollup captions for live programming, color (to distinguish
> speakers) and caption positioning are pretty important to the hearing
> impaired.
>
> Yesterday I watched a DVD I ripped (purely for personal use) of a TV
> show with VTT captions for something that I knew what the original 608
> looked like, and wow what a difference (the WebVTT looked dramatically
> worse than the original 608).
>
> To my point: no, I don't think normalizing everything down to WebVTT
> is a good idea.
>
> Much of the goal, at least in the work that I do, is to conform to the
> FCC requirements, which generally require that the original 608/708
> from the content provider be preserved.
>
> There can be money to support such efforts. Most of my work in
> captioning within ffmpeg was funded by my employer, and most of the
> work I did improving 608 captio