fre 2023-09-15 klockan 15:10 -0300 skrev James Almer:
> On 9/13/2023 6:34 AM, Tomas Härdin wrote:
> > ons 2023-09-06 klockan 16:16 -0300 skrev James Almer:
> > > On 9/6/2023 2:53 PM, Tomas Härdin wrote:
> > > > ons 2023-09-06 klockan 11:38 -0300 skrev James Almer:
> > > > > Signed-off-by: James Almer <jamr...@gmail.com>
> > > > > ---
> > > > > This is an initial proof of concept for AVStream groups,
> > > > > something
> > > > > that's
> > > > > needed for quite a few existing and upcoming formats that
> > > > > lavf
> > > > > has no
> > > > > way to
> > > > > currently export. Said formats define a single video or audio
> > > > > stream
> > > > > composed
> > > > > by merging several individualy multiplexed streams within a
> > > > > media
> > > > > file.
> > > > > This is the case of HEIF, a format defining a tiled image
> > > > > where
> > > > > each
> > > > > tile is a
> > > > > separate image (either hevc, av1, etc) all of which need to
> > > > > be
> > > > > decoded
> > > > > individualy and then stitched together for presentation using
> > > > > container level
> > > > > information;
> > > >
> > > > I remember this blocking HEIF as a GSoC project. Honestly the
> > > > way
> > > > that
> > > > format is designed is immensely horrible.
> > > >
> > > > > MPEG-TS programs, currently exported as
> > > > > AVProgram, which this new general purpose API would replace.
> > > >
> > > > I can foresee this being a nuisance for users accustomed to
> > > > AVProgram.
> > > > Also this feature borders on NLE territory. Not necessarily a
> > > > bad
> > > > thing, but FFmpeg is overall poorly architectured for NLE
> > > > stuff. I
> > > > believe I raised this issue back when lavfi was proposed, it
> > > > being
> > > > wholly unsuitable for NLE work.
> > > >
> > > >
> > > > > +typedef struct AVStreamGroup {
> > > > > + /**
> > > > > + * A class for @ref avoptions. Set on stream creation.
> > > > > + */
> > > > > + const AVClass *av_class;
> > > > > +
> > > > > + /**
> > > > > + * Group index in AVFormatContext.
> > > > > + */
> > > > > + int index;
> > > > > +
> > > > > + /**
> > > > > + * Format-specific group ID.
> > > > > + * decoding: set by libavformat
> > > > > + * encoding: set by the user, replaced by libavformat if
> > > > > left
> > > > > unset
> > > > > + */
> > > > > + int id;
> > > > > +
> > > > > + /**
> > > > > + * Codec parameters associated with this stream group.
> > > > > Allocated
> > > > > and freed
> > > > > + * by libavformat in avformat_new_stream_group() and
> > > > > avformat_free_context()
> > > > > + * respectively.
> > > > > + *
> > > > > + * - demuxing: filled by libavformat on stream group
> > > > > creation or
> > > > > in
> > > > > + * avformat_find_stream_info()
> > > > > + * - muxing: filled by the caller before
> > > > > avformat_write_header()
> > > > > + */
> > > > > + AVCodecParameters *codecpar;
> > > > > +
> > > > > + void *priv_data;
> > > > > +
> > > > > + /**
> > > > > + * Number of elements in AVStreamGroup.stream_index.
> > > > > + *
> > > > > + * Set by av_stream_group_add_stream() and
> > > > > av_stream_group_new_stream(), must not
> > > > > + * be modified by any other code.
> > > > > + */
> > > > > + int nb_stream_indexes;
> > > > > +
> > > > > + /**
> > > > > + * A list of indexes of streams in the group. New
> > > > > entries
> > > > > are
> > > > > created with
> > > > > + * av_stream_group_add_stream() and
> > > > > av_stream_group_new_stream().
> > > > > + *
> > > > > + * - demuxing: entries are created by libavformat in
> > > > > avformat_open_input().
> > > > > + * If AVFMTCTX_NOHEADER is set in ctx_flags,
> > > > > then
> > > > > new entries may also
> > > > > + * appear in av_read_frame().
> > > > > + * - muxing: entries are created by the user before
> > > > > avformat_write_header().
> > > > > + *
> > > > > + * Freed by libavformat in avformat_free_context().
> > > > > + */
> > > > > + int *stream_index;
> > > > > +} AVStreamGroup;
> > > >
> > > > I see no provisions for attaching metadata, for example HEIF
> > > > stitching.
> > > > Putting it in coderpar seems wrong, since it is container-level
> > > > metadata. We could just have an HEIF specific struct as
> > > > container
> > > > metadata.
> > >
> > > The doxy for AVCodecParameters says "This struct describes the
> > > properties of an encoded stream.", so It's not about container
> > > level
> > > props.
> >
> > It *is* container level props. The underlying codecs have no
> > concept of
> > this kind of stitching. The closest you're going to get is tiles in
> > JPEG2000, but I doubt HEIF support JPEG2000.
> >
> > We might say "well the resulting stream group has resolution so
> > it's
> > like a codec" but see below.
> >
> > > Although codecpar will be used to export the merged/stitched
> > > stream
> > > props like dimensions and channel layout, maybe you're right
> > > about
> > > the
> > > metadata because there would be a clash between actual
> > > HEVC/Opus/AAC/AV1
> > > extradata and the HEIF/IAMF/etc specific info if both use
> > > codecpar.extradata, even if one will be in AVStream and the other
> > > in
> > > AVStreamGroup.
> >
> > Yes, pretty much. But it's more that codecpar is pressed into
> > service
> > where it probably doesn't belong. It might be more appropriate to
> > call
> > these "essence parameters". I'm going to stick my neck out further
> > and
> > say that picture and sound essence should be handled with different
> > structs, not smushed together into one struct like
> > AVCodecParameters.
>
> Can you suggest how to approach this then, if not with
> AVCodecParameters? For the resulting merged/stitched stream, we need
> at
> the very least dimensions, and for audio we need channel layout, and
> for
> each different kind of group type (HEIF, IAMF, TS programs) we need
> specific parameters, like order of tiles for HEIF, mixing parameters
> for
> IAMF, and these parameters should be in a form easy for the user
> (like
> our CLI) to feed to lavfi, where the actual merging/stitching would
> take
> place with new or existing filters for this specific purpose.
>
> Maybe something like:
>
> ----
> enum AVStreamGroupParamsType {
> AV_STREAM_GROUP_PARAMS_NONE,
> AV_STREAM_GROUP_PARAMS_TS,
> AV_STREAM_GROUP_PARAMS_HEIF,
> AV_STREAM_GROUP_PARAMS_IAMF,
> };
>
> typedef struct AVStreamGroupTSParams {
> // Basically AVProgram
> } AVStreamGroupTSParams;
>
> typedef struct AVStreamGroupHEIFParams {
> // dimensions, tile order, etc
> } AVStreamGroupHEIFParams;
>
> typedef struct AVStreamGroupIAMFParams {
> // channel layout, mixing params
> } AVStreamGroupIAMFParams;
>
> typedef struct AVStreamGroup {
> [...]
> enum AVStreamGroupParamsType type;
> union {
> AVStreamGroupTSParams ts;
> AVStreamGroupHEIFParams heif;
> AVStreamGroupIAMFParams iamf;
> } essence;
> [...]
> } AVStreamGroup;
Sorry for not replying to this sooner.
Yes, a typed union like this should work nicely. This way we keep
things related to each type of stream group separate.
/Tomas
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
