On Thu, Apr 7, 2016 at 4:18 PM, wm4 <nfx...@googlemail.com> wrote: > On Fri, 18 Mar 2016 17:50:39 +0100 > Matthieu Bouron <matthieu.bou...@gmail.com> wrote: > > > From: Matthieu Bouron <matthieu.bou...@stupeflix.com> > > > > --- > > > > Hello, > > Can't say much about this, so just some minor confused comments. >
Thanks for your comments and sorry for the late reply. > > > > > The following patch add hwaccel support to the mediacodec (h264) decoder > by allowing > > the user to render the output frames directly on a surface. > > > > In order to do so the user needs to initialize the hwaccel through the > use of > > av_mediacodec_alloc_context and av_mediacodec_default_init functions. > The later > > takes a reference to an android/view/Surface as parameter. > > > > If the hwaccel successfully initialize, the decoder output frames pix > fmt will be > > AV_PIX_FMT_MEDIACODEC. The following snippet of code demonstrate how to > render > > the frames on the surface: > > > > AVMediaCodecBuffer *buffer = (AVMediaCodecBuffer *)frame->data[3]; > > av_mediacodec_release_buffer(buffer, 1); > > > > The last argument of av_mediacodec_release_buffer enable rendering of the > > buffer on the surface (or not if set to 0). > > > > I don't understand this (at all), but unreferencing the AVFrame should > unref the underlying surface. > In this case, the underlying surface will remain (it is owned by the codec itself) but the output buffer (that should be renderered to the surface) will be discarded. > > > Regarding the internal changes in the mediacodec decoder: > > > > MediaCodec.flush() discards both input and output buffers meaning that if > > MediaCodec.flush() is called all output buffers the user has a reference > on are > > now invalid (and cannot be used). > > This behaviour does not fit well in the avcodec API. > > > > When the decoder is configured to output software buffers, there is no > issue as > > the buffers are copied. > > > > Now when the decoder is configured to output to a surface, the user > might not > > want to render all the frames as fast as the decoder can go and might > want to > > control *when* the frame are rendered, so we need to make sure that the > > MediaCodec.flush() call is delayed until all the frames the user retains > has > > been released or rendered. > > > > Delaying the call to MediaCodec.flush() means buffering any inputs that > come > > the decoder until the user has released/renderer the frame he retains. > > > > This is a limitation of this hwaccel implementation, if the user retains > a > > frame (a), then issue a flush command to the decoder, the packets he > feeds to > > the decoder at that point will be queued in the internal decoder packet > queue > > (until he releases the frame (a)). This scenario leads to a memory usage > > increase to say the least. > > > > Currently there is no limitation on the size of the internal decoder > packet > > queue but this is something that can be added easily. Then, if the queue > is > > full, what would be the behaviour of the decoder ? Can it block ? Or > should it > > returns something like AVERROR(EAGAIN) ? > > The current API can't do anything like this. It has to output 0 or 1 > frame per input packet. (If it outputs nothing, the frame is either > discarded or queued internally. The queue can be emptied only when > draining the decoder at the end of the stream.) > > So it looks like all you can do is blocking. (Which could lead to a > deadlock in the API user, depending of how the user's code works?) > Yes if I block at some point, it can lead to a deadlock if the user never releases all the frames. I'm considering buffering a few input packets before blocking. > > > > > About the other internal decoder changes I introduced: > > > > The MediaCodecDecContext is now refcounted (using the lavu/atomic api) > since > > the (hwaccel) frames can be retained by the user, we need to delay the > > destruction of the codec until the user has released all the frames he > has a > > reference on. > > The reference counter of the MediaCodecDecContext is incremented each > time an > > (hwaccel) frame is outputted by the decoder and decremented each time a > > (hwaccel) frame is released. > > > > Also, when the decoder is configured to output to a surface the pts that > are > > given to the MediaCodec API are now rescaled based on the codec_timebase > as > > those timestamps values are propagated to the frames rendered on the > surface > > since Android M. Not sure if it's really useful though. > > That's all nice, but where is this stuff documented at all? > It is documented here: http://developer.android.com/reference/android/media/MediaCodec.html#queueInputBuffer%28int,%20int,%20int,%20long,%20int%29 > > > > > On the performance side: > > > > On a nexus 5, decoding an h264 stream (main profile) 1080p@60fps: > > - software output + rgba conversion goes at 59~60fps > > - surface output + render on a surface goes at 100~110fps > > > > Matthieu > > > > --- > > configure | 1 + > > libavcodec/Makefile | 6 +- > > libavcodec/allcodecs.c | 1 + > > libavcodec/mediacodec.c | 125 ++++++++++++++++++ > > libavcodec/mediacodec.h | 85 +++++++++++++ > > libavcodec/mediacodec_surface.c | 66 ++++++++++ > > libavcodec/mediacodec_surface.h | 31 +++++ > > libavcodec/mediacodec_wrapper.c | 5 +- > > libavcodec/mediacodecdec.c | 272 > +++++++++++++++++++++++++++++++++------- > > libavcodec/mediacodecdec.h | 17 +++ > > libavcodec/mediacodecdec_h264.c | 23 ++++ > > libavutil/pixdesc.c | 4 + > > libavutil/pixfmt.h | 2 + > > 13 files changed, 586 insertions(+), 52 deletions(-) > > create mode 100644 libavcodec/mediacodec.c > > create mode 100644 libavcodec/mediacodec.h > > create mode 100644 libavcodec/mediacodec_surface.c > > create mode 100644 libavcodec/mediacodec_surface.h > > > > diff --git a/configure b/configure > > index e5de306..4d66673 100755 > > --- a/configure > > +++ b/configure > > @@ -2530,6 +2530,7 @@ h264_d3d11va_hwaccel_select="h264_decoder" > > h264_dxva2_hwaccel_deps="dxva2" > > h264_dxva2_hwaccel_select="h264_decoder" > > h264_mediacodec_decoder_deps="mediacodec" > > +h264_mediacodec_hwaccel_deps="mediacodec" > > h264_mediacodec_decoder_select="h264_mp4toannexb_bsf h264_parser" > > h264_mmal_decoder_deps="mmal" > > h264_mmal_decoder_select="mmal" > > diff --git a/libavcodec/Makefile b/libavcodec/Makefile > > index 6bb1af1..a3dad7e 100644 > > --- a/libavcodec/Makefile > > +++ b/libavcodec/Makefile > > @@ -10,6 +10,7 @@ HEADERS = avcodec.h > \ > > dirac.h > \ > > dxva2.h > \ > > jni.h > \ > > + mediacodec.h > \ > > qsv.h > \ > > vaapi.h > \ > > vda.h > \ > > @@ -91,7 +92,7 @@ OBJS-$(CONFIG_LSP) += lsp.o > > OBJS-$(CONFIG_LZF) += lzf.o > > OBJS-$(CONFIG_MDCT) += mdct_fixed.o mdct_float.o > mdct_fixed_32.o > > OBJS-$(CONFIG_ME_CMP) += me_cmp.o > > -OBJS-$(CONFIG_MEDIACODEC) += mediacodecdec.o > mediacodec_wrapper.o mediacodec_sw_buffer.o > > +OBJS-$(CONFIG_MEDIACODEC) += mediacodecdec.o > mediacodec_surface.o mediacodec_wrapper.o mediacodec_sw_buffer.o > > OBJS-$(CONFIG_MPEG_ER) += mpeg_er.o > > OBJS-$(CONFIG_MPEGAUDIO) += mpegaudio.o mpegaudiodata.o > \ > > mpegaudiodecheader.o > > @@ -734,6 +735,7 @@ OBJS-$(CONFIG_H263_VAAPI_HWACCEL) += > vaapi_mpeg4.o > > OBJS-$(CONFIG_H263_VIDEOTOOLBOX_HWACCEL) += videotoolbox.o > > OBJS-$(CONFIG_H264_D3D11VA_HWACCEL) += dxva2_h264.o > > OBJS-$(CONFIG_H264_DXVA2_HWACCEL) += dxva2_h264.o > > +OBJS-$(CONFIG_H264_MEDIACODEC_HWACCEL) += mediacodec.o > > OBJS-$(CONFIG_H264_VAAPI_HWACCEL) += vaapi_h264.o > > OBJS-$(CONFIG_H264_VDA_HWACCEL) += vda_h264.o > > OBJS-$(CONFIG_H264_VDPAU_HWACCEL) += vdpau_h264.o > > @@ -947,7 +949,7 @@ SKIPHEADERS-$(CONFIG_LIBSCHROEDINGER) += > libschroedinger.h > > SKIPHEADERS-$(CONFIG_LIBUTVIDEO) += libutvideo.h > > SKIPHEADERS-$(CONFIG_LIBVPX) += libvpx.h > > SKIPHEADERS-$(CONFIG_LIBWEBP_ENCODER) += libwebpenc_common.h > > -SKIPHEADERS-$(CONFIG_MEDIACODEC) += mediacodecdec.h > mediacodec_wrapper.h mediacodec_sw_buffer.h > > +SKIPHEADERS-$(CONFIG_MEDIACODEC) += mediacodecdec.h > mediacodec_surface.h mediacodec_wrapper.h mediacodec_sw_buffer.h > > SKIPHEADERS-$(CONFIG_QSV) += qsv.h qsv_internal.h > > SKIPHEADERS-$(CONFIG_QSVDEC) += qsvdec.h > > SKIPHEADERS-$(CONFIG_QSVENC) += qsvenc.h > > diff --git a/libavcodec/allcodecs.c b/libavcodec/allcodecs.c > > index 2a25d66..79ce855 100644 > > --- a/libavcodec/allcodecs.c > > +++ b/libavcodec/allcodecs.c > > @@ -78,6 +78,7 @@ void avcodec_register_all(void) > > REGISTER_HWACCEL(H263_VIDEOTOOLBOX, h263_videotoolbox); > > REGISTER_HWACCEL(H264_D3D11VA, h264_d3d11va); > > REGISTER_HWACCEL(H264_DXVA2, h264_dxva2); > > + REGISTER_HWACCEL(H264_MEDIACODEC, h264_mediacodec); > > REGISTER_HWACCEL(H264_MMAL, h264_mmal); > > REGISTER_HWACCEL(H264_QSV, h264_qsv); > > REGISTER_HWACCEL(H264_VAAPI, h264_vaapi); > > diff --git a/libavcodec/mediacodec.c b/libavcodec/mediacodec.c > > new file mode 100644 > > index 0000000..51dd37c > > --- /dev/null > > +++ b/libavcodec/mediacodec.c > > @@ -0,0 +1,125 @@ > > +/* > > + * Android MediaCodec public API functions > > + * > > + * Copyright (c) 2016 Matthieu Bouron <matthieu.bouron stupeflix.com> > > + * > > + * This file is part of FFmpeg. > > + * > > + * FFmpeg is free software; you can redistribute it and/or > > + * modify it under the terms of the GNU Lesser General Public > > + * License as published by the Free Software Foundation; either > > + * version 2.1 of the License, or (at your option) any later version. > > + * > > + * FFmpeg is distributed in the hope that it will be useful, > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > + * Lesser General Public License for more details. > > + * > > + * You should have received a copy of the GNU Lesser General Public > > + * License along with FFmpeg; if not, write to the Free Software > > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA > > + */ > > + > > +#include "config.h" > > + > > +#if CONFIG_H264_MEDIACODEC_HWACCEL > > + > > +#include <jni.h> > > + > > +#include "libavcodec/avcodec.h" > > +#include "libavutil/atomic.h" > > +#include "libavutil/mem.h" > > + > > +#include "ffjni.h" > > +#include "mediacodec.h" > > +#include "mediacodecdec.h" > > + > > +AVMediaCodecContext *av_mediacodec_alloc_context(void) > > +{ > > + return av_mallocz(sizeof(AVMediaCodecContext)); > > +} > > + > > +int av_mediacodec_default_init(AVCodecContext *avctx, > AVMediaCodecContext *ctx, void *surface) > > +{ > > + int ret = 0; > > + JNIEnv *env = NULL; > > + int attached = 0; > > + > > + env = ff_jni_attach_env(&attached, avctx); > > + if (!env) { > > + return AVERROR_EXTERNAL; > > + } > > + > > + ctx->surface = (*env)->NewGlobalRef(env, surface); > > + if (ctx->surface) { > > + avctx->hwaccel_context = ctx; > > + } else { > > + av_log(avctx, AV_LOG_ERROR, "Could not create new global > reference\n"); > > + ret = AVERROR_EXTERNAL; > > + } > > + > > + if (attached) { > > + ff_jni_detach_env(avctx); > > + } > > + > > + return ret; > > +} > > + > > +void av_mediacodec_default_free(AVCodecContext *avctx) > > +{ > > + JNIEnv *env = NULL; > > + int attached = 0; > > + > > + AVMediaCodecContext *ctx = avctx->hwaccel_context; > > + > > + env = ff_jni_attach_env(&attached, avctx); > > + if (!env) { > > + return; > > + } > > + > > + if (ctx->surface) { > > + (*env)->DeleteGlobalRef(env, ctx->surface); > > + ctx->surface = NULL; > > + } > > + > > + if (attached) { > > + ff_jni_detach_env(avctx); > > + } > > + > > + av_freep(&avctx->hwaccel_context); > > +} > > + > > +int av_mediacodec_release_buffer(AVMediaCodecBuffer *buffer, int render) > > +{ > > + MediaCodecDecContext *ctx = buffer->ctx; > > + int released = avpriv_atomic_int_add_and_fetch(buffer->released, 1); > > + > > + if (released == 1) { > > + return ff_AMediaCodec_releaseOutputBuffer(ctx->codec, > buffer->index, render); > > + } > > + > > + return 0; > > +} > > + > > +#else > > + > > +AVMediaCodecContext *av_mediacodec_alloc_context(void) > > +{ > > + return NULL; > > +} > > + > > +int av_mediacodec_default_init(AVCodecContext *avctx, > AVMediaCodecContext *ctx, void *surface) > > +{ > > + return 0; > > +} > > + > > +void av_mediacodec_default_free(AVCodecContext *avctx) > > +{ > > +} > > + > > +int av_mediacodec_release_buffer(AVMediaCodecBuffer *buffer, int render) > > +{ > > + return 0; > > +} > > + > > +#endif > > diff --git a/libavcodec/mediacodec.h b/libavcodec/mediacodec.h > > new file mode 100644 > > index 0000000..f303c63 > > --- /dev/null > > +++ b/libavcodec/mediacodec.h > > @@ -0,0 +1,85 @@ > > +/* > > + * Android MediaCodec public API > > + * > > + * Copyright (c) 2016 Matthieu Bouron <matthieu.bouron stupeflix.com> > > + * > > + * This file is part of FFmpeg. > > + * > > + * FFmpeg is free software; you can redistribute it and/or > > + * modify it under the terms of the GNU Lesser General Public > > + * License as published by the Free Software Foundation; either > > + * version 2.1 of the License, or (at your option) any later version. > > + * > > + * FFmpeg is distributed in the hope that it will be useful, > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > + * Lesser General Public License for more details. > > + * > > + * You should have received a copy of the GNU Lesser General Public > > + * License along with FFmpeg; if not, write to the Free Software > > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA > > + */ > > + > > +#ifndef AVCODEC_MEDIACODEC_H > > +#define AVCODEC_MEDIACODEC_H > > + > > +#include "libavcodec/avcodec.h" > > + > > +/** > > + * This structure holds a reference to a android/view/Surface object > that will > > + * be used as output by the decoder. > > + * > > + */ > > +typedef struct AVMediaCodecContext { > > + > > + /** > > + * android/view/Surface object reference. > > + */ > > + void *surface; > > + > > +} AVMediaCodecContext; > > + > > +/** > > + * Allocate and initialize a MediaCodec context. > > + * > > + * When decoding with MediaCodec is finished, the caller must free the > > + * MediaCodec context with av_mediacodec_default_free. > > + * > > + * @return a pointer to a newly allocated AVMediaCodecContext on > success, NULL otherwise > > + */ > > +AVMediaCodecContext *av_mediacodec_alloc_context(void); > > + > > +/** > > + * Convenience function that sets up the MediaCodec context. > > + * > > + * @param avctx codec context > > + * @param ctx MediaCodec context to initialize > > + * @param surface reference to an android/view/Surface > > + * @return 0 on success, < 0 otherwise > > + */ > > +int av_mediacodec_default_init(AVCodecContext *avctx, > AVMediaCodecContext *ctx, void *surface); > > + > > +/** > > + * This function must be called to free the MediaCodec context > initialized with > > + * av_mediacodec_default_init(). > > + * > > + * @param avctx codec context > > + */ > > +void av_mediacodec_default_free(AVCodecContext *avctx); > > + > > +/** > > + * Opaque structure representing a MediaCodec buffer to render. > > + */ > > +typedef struct MediaCodecBuffer AVMediaCodecBuffer; > > Why is the typedef differently named than the struct type? > A leftover (fixed locally). > > > + > > +/** > > + * Release a MediaCodec buffer and render it onto the surface the > decoder is > > + * associated with. > > + * > > + * @param buffer the buffer to render > > + * @param render 1 to render the buffer onto the surface or 0 to > discard the buffer > > + * @return 0 on success, < 0 otherwise > > + */ > > +int av_mediacodec_release_buffer(AVMediaCodecBuffer *buffer, int > render); > > + > > +#endif /* AVCODEC_MEDIACODEC_H */ > > diff --git a/libavcodec/mediacodec_surface.c > b/libavcodec/mediacodec_surface.c > > new file mode 100644 > > index 0000000..903ebe4 > > --- /dev/null > > +++ b/libavcodec/mediacodec_surface.c > > @@ -0,0 +1,66 @@ > > +/* > > + * Android MediaCodec Surface functions > > + * > > + * Copyright (c) 2016 Matthieu Bouron <matthieu.bouron stupeflix.com> > > + * > > + * This file is part of FFmpeg. > > + * > > + * FFmpeg is free software; you can redistribute it and/or > > + * modify it under the terms of the GNU Lesser General Public > > + * License as published by the Free Software Foundation; either > > + * version 2.1 of the License, or (at your option) any later version. > > + * > > + * FFmpeg is distributed in the hope that it will be useful, > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > + * Lesser General Public License for more details. > > + * > > + * You should have received a copy of the GNU Lesser General Public > > + * License along with FFmpeg; if not, write to the Free Software > > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA > > + */ > > + > > +#include <jni.h> > > + > > +#include "ffjni.h" > > +#include "mediacodec_surface.h" > > + > > +void *ff_mediacodec_surface_ref(void *surface, void *log_ctx) > > +{ > > + int attached = 0; > > + JNIEnv *env = NULL; > > + > > + void *reference = NULL; > > + > > + env = ff_jni_attach_env(&attached, log_ctx); > > + if (!env) { > > + return NULL; > > + } > > + > > + reference = (*env)->NewGlobalRef(env, surface); > > + > > + if (attached) { > > + ff_jni_detach_env(log_ctx); > > + } > > + > > + return reference; > > +} > > + > > +int ff_mediacodec_surface_unref(void *surface, void *log_ctx) > > +{ > > + int attached = 0; > > + JNIEnv *env = NULL; > > + > > + env = ff_jni_attach_env(&attached, log_ctx); > > + if (!env) { > > + return AVERROR_EXTERNAL; > > + } > > + > > + (*env)->DeleteGlobalRef(env, surface); > > + > > + if (attached) { > > + ff_jni_detach_env(log_ctx); > > + } > > + > > + return 0; > > +} > > diff --git a/libavcodec/mediacodec_surface.h > b/libavcodec/mediacodec_surface.h > > new file mode 100644 > > index 0000000..0178b8a > > --- /dev/null > > +++ b/libavcodec/mediacodec_surface.h > > @@ -0,0 +1,31 @@ > > +/* > > + * Android MediaCodec Surface functions > > + * > > + * Copyright (c) 2016 Matthieu Bouron <matthieu.bouron stupeflix.com> > > + * > > + * This file is part of FFmpeg. > > + * > > + * FFmpeg is free software; you can redistribute it and/or > > + * modify it under the terms of the GNU Lesser General Public > > + * License as published by the Free Software Foundation; either > > + * version 2.1 of the License, or (at your option) any later version. > > + * > > + * FFmpeg is distributed in the hope that it will be useful, > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > > + * Lesser General Public License for more details. > > + * > > + * You should have received a copy of the GNU Lesser General Public > > + * License along with FFmpeg; if not, write to the Free Software > > + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA > > + */ > > + > > +#ifndef AVCODEC_MEDIACODEC_SURFACE_H > > +#define AVCODEC_MEDIACODEC_SURFACE_H > > + > > +#include "libavcodec/avcodec.h" > > + > > +void *ff_mediacodec_surface_ref(void *surface, void *log_ctx); > > +int ff_mediacodec_surface_unref(void *surface, void *log_ctx); > > + > > +#endif /* AVCODEC_MEDIACODEC_SURFACE_H */ > > diff --git a/libavcodec/mediacodec_wrapper.c > b/libavcodec/mediacodec_wrapper.c > > index 6b3f905..621e40b 100644 > > --- a/libavcodec/mediacodec_wrapper.c > > +++ b/libavcodec/mediacodec_wrapper.c > > @@ -1306,12 +1306,9 @@ int ff_AMediaCodec_configure(FFAMediaCodec* > codec, const FFAMediaFormat* format, > > int attached = 0; > > JNIEnv *env = NULL; > > > > - /* TODO: implement surface handling */ > > - av_assert0(surface == NULL); > > - > > JNI_ATTACH_ENV_OR_RETURN(env, &attached, codec, AVERROR_EXTERNAL); > > > > - (*env)->CallVoidMethod(env, codec->object, > codec->jfields.configure_id, format->object, NULL, NULL, flags); > > + (*env)->CallVoidMethod(env, codec->object, > codec->jfields.configure_id, format->object, surface, NULL, flags); > > if (ff_jni_exception_check(env, 1, codec) < 0) { > > ret = AVERROR_EXTERNAL; > > goto fail; > > diff --git a/libavcodec/mediacodecdec.c b/libavcodec/mediacodecdec.c > > index d385651..1e4d9dd 100644 > > --- a/libavcodec/mediacodecdec.c > > +++ b/libavcodec/mediacodecdec.c > > @@ -23,6 +23,7 @@ > > #include <string.h> > > #include <sys/types.h> > > > > +#include "libavutil/atomic.h" > > #include "libavutil/common.h" > > #include "libavutil/mem.h" > > #include "libavutil/log.h" > > @@ -33,6 +34,8 @@ > > #include "avcodec.h" > > #include "internal.h" > > > > +#include "mediacodec.h" > > +#include "mediacodec_surface.h" > > #include "mediacodec_sw_buffer.h" > > #include "mediacodec_wrapper.h" > > #include "mediacodecdec.h" > > @@ -118,6 +121,10 @@ static enum AVPixelFormat > mcdec_map_color_format(AVCodecContext *avctx, > > int i; > > enum AVPixelFormat ret = AV_PIX_FMT_NONE; > > > > + if (s->surface) { > > + return AV_PIX_FMT_MEDIACODEC; > > + } > > + > > if (!strcmp(s->codec_name, "OMX.k3.video.decoder.avc") && > color_format == COLOR_FormatYCbYCr) { > > s->color_format = color_format = > COLOR_TI_FormatYUV420PackedSemiPlanar; > > } > > @@ -134,7 +141,113 @@ static enum AVPixelFormat > mcdec_map_color_format(AVCodecContext *avctx, > > return ret; > > } > > > > -static int mediacodec_wrap_buffer(AVCodecContext *avctx, > > +static void ff_mediacodec_dec_ref(MediaCodecDecContext *s) > > +{ > > + avpriv_atomic_int_add_and_fetch(s->refcount, 1); > > +} > > + > > +static void ff_mediacodec_dec_unref(MediaCodecDecContext *s) > > +{ > > + if (!avpriv_atomic_int_add_and_fetch(s->refcount, -1)) { > > + if (s->codec) { > > + ff_AMediaCodec_delete(s->codec); > > + s->codec = NULL; > > + } > > + > > + if (s->format) { > > + ff_AMediaFormat_delete(s->format); > > + s->format = NULL; > > + } > > + > > + if (s->surface) { > > + ff_mediacodec_surface_unref(s->surface, NULL); > > + s->surface = NULL; > > + } > > + > > + av_freep(&s->codec_name); > > + av_freep(&s->refcount); > > + } > > +} > > + > > +static void mediacodec_buffer_release(void *opaque, uint8_t *data) > > +{ > > + AVMediaCodecBuffer *buffer = opaque; > > + MediaCodecDecContext *ctx = buffer->ctx; > > + int released = avpriv_atomic_int_get(buffer->released); > > + > > + if (!released) { > > + ff_AMediaCodec_releaseOutputBuffer(ctx->codec, buffer->index, > 0); > > + } > > + > > + ff_mediacodec_dec_unref(ctx); > > + av_freep(&buffer->released); > > + av_freep(&buffer); > > +} > > + > > +static int mediacodec_wrap_hw_buffer(AVCodecContext *avctx, > > + MediaCodecDecContext *s, > > + ssize_t index, > > + FFAMediaCodecBufferInfo *info, > > + AVFrame *frame) > > +{ > > + int ret = 0; > > + AVMediaCodecBuffer *buffer = NULL; > > + > > + frame->buf[0] = NULL; > > + frame->width = avctx->width; > > + frame->height = avctx->height; > > + frame->format = avctx->pix_fmt; > > + frame->pkt_pts = av_rescale_q(info->presentationTimeUs, > > + av_make_q(1, 1000000), > > + avctx->pkt_timebase); > > + > > + buffer = av_mallocz(sizeof(AVMediaCodecBuffer)); > > + if (!buffer) { > > + ret = AVERROR(ENOMEM); > > + goto fail; > > + } > > + > > + buffer->released = av_mallocz(sizeof(*buffer->released)); > > + if (!buffer->released) { > > + ret = AVERROR(ENOMEM); > > + goto fail; > > + } > > + > > + frame->buf[0] = av_buffer_create(NULL, > > + 0, > > + mediacodec_buffer_release, > > + buffer, > > + AV_BUFFER_FLAG_READONLY); > > + > > + if (!frame->buf[0]) { > > + ret = AVERROR(ENOMEM); > > + goto fail; > > + > > + } > > + > > + buffer->ctx = s; > > + ff_mediacodec_dec_ref(s); > > + > > + buffer->index = index; > > + buffer->pts = info->presentationTimeUs; > > + > > + frame->data[3] = (uint8_t *)buffer; > > + > > + return 0; > > +fail: > > + if (buffer) { > > + av_free(buffer->released); > > + av_free(buffer); > > + } > > + > > + av_buffer_unref(&frame->buf[0]); > > + > > + ff_AMediaCodec_releaseOutputBuffer(s->codec, index, 0); > > + > > + return ret; > > +} > > + > > +static int mediacodec_wrap_sw_buffer(AVCodecContext *avctx, > > MediaCodecDecContext *s, > > uint8_t *data, > > size_t size, > > @@ -307,14 +420,61 @@ static int > mediacodec_dec_parse_format(AVCodecContext *avctx, MediaCodecDecConte > > return ff_set_dimensions(avctx, width, height); > > } > > > > + > > +static int mediacodec_dec_flush_codec(AVCodecContext *avctx, > MediaCodecDecContext *s) > > +{ > > + FFAMediaCodec *codec = s->codec; > > + int status; > > + > > + s->queued_buffer_nb = 0; > > + s->dequeued_buffer_nb = 0; > > + > > + s->draining = 0; > > + s->flushing = 0; > > + > > + status = ff_AMediaCodec_flush(codec); > > + if (status < 0) { > > + av_log(NULL, AV_LOG_ERROR, "Failed to flush MediaCodec %p", > codec); > > + return AVERROR_EXTERNAL; > > + } > > + > > + s->first_buffer = 0; > > + s->first_buffer_at = av_gettime(); > > What is the system time doing here? > It is here for debugging/information purpose, to know how many time it took to the codec to buffer and output its first buffer when it starts or after a flush (discard). > > > + > > + return 0; > > +} > > + > > int ff_mediacodec_dec_init(AVCodecContext *avctx, MediaCodecDecContext > *s, > > const char *mime, FFAMediaFormat *format) > > { > > int ret = 0; > > int status; > > > > + enum AVPixelFormat pix_fmt; > > + enum AVPixelFormat pix_fmts[3] = { > > + AV_PIX_FMT_MEDIACODEC, > > + AV_PIX_FMT_NONE, > > + }; > > + > > s->first_buffer_at = av_gettime(); > > > > + s->refcount = av_mallocz(sizeof(*s->refcount)); > > + if (!s->refcount) { > > + av_log(avctx, AV_LOG_ERROR, "Failed to init decoder reference > counter\n"); > > + goto fail; > > + } > > + *s->refcount = 1; > > + > > + pix_fmt = ff_get_format(avctx, pix_fmts); > > + if (pix_fmt == AV_PIX_FMT_MEDIACODEC) { > > + AVMediaCodecContext *user_ctx = avctx->hwaccel_context; > > + > > + if (user_ctx && user_ctx->surface) { > > + s->surface = ff_mediacodec_surface_ref(user_ctx->surface, > avctx); > > + av_log(avctx, AV_LOG_INFO, "Using surface %p\n", > s->surface); > > + } > > + } > > + > > s->codec_name = ff_AMediaCodecList_getCodecNameByType(mime, > avctx->width, avctx->height, avctx); > > if (!s->codec_name) { > > ret = AVERROR_EXTERNAL; > > @@ -329,7 +489,7 @@ int ff_mediacodec_dec_init(AVCodecContext *avctx, > MediaCodecDecContext *s, > > goto fail; > > } > > > > - status = ff_AMediaCodec_configure(s->codec, format, NULL, NULL, 0); > > + status = ff_AMediaCodec_configure(s->codec, format, s->surface, > NULL, 0); > > if (status < 0) { > > char *desc = ff_AMediaFormat_toString(format); > > av_log(avctx, AV_LOG_ERROR, > > @@ -377,7 +537,7 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, > MediaCodecDecContext *s, > > { > > int ret; > > int offset = 0; > > - int need_flushing = 0; > > + int need_draining = 0; > > uint8_t *data; > > ssize_t index; > > size_t size; > > @@ -389,15 +549,21 @@ int ff_mediacodec_dec_decode(AVCodecContext > *avctx, MediaCodecDecContext *s, > > int64_t input_dequeue_timeout_us = INPUT_DEQUEUE_TIMEOUT_US; > > int64_t output_dequeue_timeout_us = OUTPUT_DEQUEUE_TIMEOUT_US; > > > > + if (s->flushing) { > > + av_log(avctx, AV_LOG_ERROR, "Decoder is flushing and cannot > accept new buffer " > > + "until all output buffers have been > released\n"); > > + return AVERROR_EXTERNAL; > > + } > > + > > if (pkt->size == 0) { > > - need_flushing = 1; > > + need_draining = 1; > > } > > > > - if (s->flushing && need_flushing && s->queued_buffer_nb <= 0) { > > + if (s->draining && need_draining && s->queued_buffer_nb <= 0) { > > return 0; > > } > > > > - while (offset < pkt->size || (need_flushing && !s->flushing)) { > > + while (offset < pkt->size || (need_draining && !s->draining)) { > > int size; > > > > index = ff_AMediaCodec_dequeueInputBuffer(codec, > input_dequeue_timeout_us); > > @@ -416,26 +582,37 @@ int ff_mediacodec_dec_decode(AVCodecContext > *avctx, MediaCodecDecContext *s, > > return AVERROR_EXTERNAL; > > } > > > > - if (need_flushing) { > > + if (need_draining) { > > + int64_t pts = pkt->pts; > > uint32_t flags = > ff_AMediaCodec_getBufferFlagEndOfStream(codec); > > > > + if (s->surface) { > > + pts = av_rescale_q(pts, avctx->pkt_timebase, > av_make_q(1, 1000000)); > > + } > > + > > av_log(avctx, AV_LOG_DEBUG, "Sending End Of Stream > signal\n"); > > > > - status = ff_AMediaCodec_queueInputBuffer(codec, index, 0, > 0, pkt->pts, flags); > > + status = ff_AMediaCodec_queueInputBuffer(codec, index, 0, > 0, pts, flags); > > if (status < 0) { > > av_log(avctx, AV_LOG_ERROR, "Failed to queue input > empty buffer (status = %d)\n", status); > > return AVERROR_EXTERNAL; > > } > > > > - s->flushing = 1; > > + s->draining = 1; > > break; > > } else { > > + int64_t pts = pkt->pts; > > + > > size = FFMIN(pkt->size - offset, size); > > > > memcpy(data, pkt->data + offset, size); > > offset += size; > > > > - status = ff_AMediaCodec_queueInputBuffer(codec, index, 0, > size, pkt->pts, 0); > > + if (s->surface) { > > + pts = av_rescale_q(pts, avctx->pkt_timebase, > av_make_q(1, 1000000)); > > + } > > + > > + status = ff_AMediaCodec_queueInputBuffer(codec, index, 0, > size, pts, 0); > > if (status < 0) { > > av_log(avctx, AV_LOG_ERROR, "Failed to queue input > buffer (status = %d)\n", status); > > return AVERROR_EXTERNAL; > > @@ -447,8 +624,8 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, > MediaCodecDecContext *s, > > } > > } > > > > - if (s->flushing) { > > - /* If the codec is flushing, block for a fair amount of time to > > + if (s->draining) { > > + /* If the codec is draining, block for a fair amount of time to > > * ensure we got a frame */ > > output_dequeue_timeout_us = OUTPUT_DEQUEUE_BLOCK_TIMEOUT_US; > > } else if (s->dequeued_buffer_nb == 0) { > > @@ -471,15 +648,22 @@ int ff_mediacodec_dec_decode(AVCodecContext > *avctx, MediaCodecDecContext *s, > > " flags=%" PRIu32 "\n", index, info.offset, info.size, > > info.presentationTimeUs, info.flags); > > > > - data = ff_AMediaCodec_getOutputBuffer(codec, index, &size); > > - if (!data) { > > - av_log(avctx, AV_LOG_ERROR, "Failed to get output > buffer\n"); > > - return AVERROR_EXTERNAL; > > - } > > + if (s->surface) { > > + if ((ret = mediacodec_wrap_hw_buffer(avctx, s, index, > &info, frame)) < 0) { > > + av_log(avctx, AV_LOG_ERROR, "Failed to wrap MediaCodec > buffer\n"); > > + return ret; > > + } > > + } else { > > + data = ff_AMediaCodec_getOutputBuffer(codec, index, &size); > > + if (!data) { > > + av_log(avctx, AV_LOG_ERROR, "Failed to get output > buffer\n"); > > + return AVERROR_EXTERNAL; > > + } > > > > - if ((ret = mediacodec_wrap_buffer(avctx, s, data, size, index, > &info, frame)) < 0) { > > - av_log(avctx, AV_LOG_ERROR, "Failed to wrap MediaCodec > buffer\n"); > > - return ret; > > + if ((ret = mediacodec_wrap_sw_buffer(avctx, s, data, size, > index, &info, frame)) < 0) { > > + av_log(avctx, AV_LOG_ERROR, "Failed to wrap MediaCodec > buffer\n"); > > + return ret; > > + } > > } > > > > *got_frame = 1; > > @@ -516,9 +700,9 @@ int ff_mediacodec_dec_decode(AVCodecContext *avctx, > MediaCodecDecContext *s, > > } else if (ff_AMediaCodec_infoOutputBuffersChanged(codec, index)) { > > ff_AMediaCodec_cleanOutputBuffers(codec); > > } else if (ff_AMediaCodec_infoTryAgainLater(codec, index)) { > > - if (s->flushing) { > > + if (s->draining) { > > av_log(avctx, AV_LOG_ERROR, "Failed to dequeue output > buffer within %" PRIi64 "ms " > > - "while flushing remaining > frames, output will probably lack last %d frames\n", > > + "while draining remaining > frames, output will probably lack last %d frames\n", > > output_dequeue_timeout_us / > 1000, s->queued_buffer_nb); > > } else { > > av_log(avctx, AV_LOG_DEBUG, "No output buffer available, > try again later\n"); > > @@ -533,39 +717,35 @@ int ff_mediacodec_dec_decode(AVCodecContext > *avctx, MediaCodecDecContext *s, > > > > int ff_mediacodec_dec_flush(AVCodecContext *avctx, MediaCodecDecContext > *s) > > { > > - FFAMediaCodec *codec = s->codec; > > - int status; > > - > > - s->queued_buffer_nb = 0; > > - s->dequeued_buffer_nb = 0; > > + if (!s->surface || avpriv_atomic_int_get(s->refcount) == 1) { > > + int ret; > > > > - s->flushing = 0; > > + if ((ret = mediacodec_dec_flush_codec(avctx, s)) < 0) { > > + return ret; > > + } > > > > - status = ff_AMediaCodec_flush(codec); > > - if (status < 0) { > > - av_log(NULL, AV_LOG_ERROR, "Failed to flush MediaCodec %p", > codec); > > - return AVERROR_EXTERNAL; > > + return 1; > > } > > > > - s->first_buffer = 0; > > - s->first_buffer_at = av_gettime(); > > - > > + s->flushing = 1; > > return 0; > > } > > > > int ff_mediacodec_dec_close(AVCodecContext *avctx, MediaCodecDecContext > *s) > > { > > - if (s->codec) { > > - ff_AMediaCodec_delete(s->codec); > > - s->codec = NULL; > > - } > > - > > - if (s->format) { > > - ff_AMediaFormat_delete(s->format); > > - s->format = NULL; > > - } > > - > > - av_freep(&s->codec_name); > > + ff_mediacodec_dec_unref(s); > > > > return 0; > > } > > + > > +int ff_mediacodec_dec_is_flushing(AVCodecContext *avctx, > MediaCodecDecContext *s) > > +{ > > + return s->flushing; > > +} > > + > > +AVHWAccel ff_h264_mediacodec_hwaccel = { > > + .name = "mediacodec", > > + .type = AVMEDIA_TYPE_VIDEO, > > + .id = AV_CODEC_ID_H264, > > + .pix_fmt = AV_PIX_FMT_MEDIACODEC, > > +}; > > diff --git a/libavcodec/mediacodecdec.h b/libavcodec/mediacodecdec.h > > index 36fdbf5..dae3d67 100644 > > --- a/libavcodec/mediacodecdec.h > > +++ b/libavcodec/mediacodecdec.h > > @@ -34,12 +34,17 @@ > > > > typedef struct MediaCodecDecContext { > > > > + int *refcount; > > + > > char *codec_name; > > > > FFAMediaCodec *codec; > > FFAMediaFormat *format; > > > > + void *surface; > > + > > int started; > > + int draining; > > int flushing; > > > > int width; > > @@ -79,4 +84,16 @@ int ff_mediacodec_dec_flush(AVCodecContext *avctx, > > int ff_mediacodec_dec_close(AVCodecContext *avctx, > > MediaCodecDecContext *s); > > > > +int ff_mediacodec_dec_is_flushing(AVCodecContext *avctx, > > + MediaCodecDecContext *s); > > + > > +typedef struct MediaCodecBuffer { > > + > > + MediaCodecDecContext *ctx; > > + ssize_t index; > > + int64_t pts; > > + int *released; > > + > > +} MediaCodecBuffer; > > + > > #endif /* AVCODEC_MEDIACODECDEC_H */ > > diff --git a/libavcodec/mediacodecdec_h264.c > b/libavcodec/mediacodecdec_h264.c > > index 2d1d525..4b74fb1 100644 > > --- a/libavcodec/mediacodecdec_h264.c > > +++ b/libavcodec/mediacodecdec_h264.c > > @@ -261,6 +261,29 @@ static int mediacodec_decode_frame(AVCodecContext > *avctx, void *data, > > av_fifo_generic_write(s->fifo, &input_ref, sizeof(input_ref), > NULL); > > } > > > > + /* > > + * MediaCodec.flush() discards both input and output buffers, thus > we > > + * need to delay the call to this function until the user has > released or > > + * renderered the frames he retains. > > + * > > + * After we have buffered an input packet, check if the codec is in > the > > + * flushing state. If it is, we need to call > ff_mediacodec_dec_flush. > > + * > > + * ff_mediacodec_dec_flush returns 0 if the flush cannot be > performed on > > + * the codec (because the user retains frames). The codec stays in > the > > + * flushing state. > > + * > > + * ff_mediacodec_dec_flush returns 1 if the flush can actually be > > + * performed on the codec. The codec leaves the flushing state and > can > > + * process again packets. > > + * > > + */ > > + if (ff_mediacodec_dec_is_flushing(avctx, &s->ctx)) { > > + if (!ff_mediacodec_dec_flush(avctx, &s->ctx)) { > > + return avpkt->size; > > + } > > + } > > + > > /* process buffered data */ > > while (!*got_frame) { > > /* prepare the input data -- convert to Annex B if needed */ > > diff --git a/libavutil/pixdesc.c b/libavutil/pixdesc.c > > index 981fa0e..8864a3d 100644 > > --- a/libavutil/pixdesc.c > > +++ b/libavutil/pixdesc.c > > @@ -1974,6 +1974,10 @@ static const AVPixFmtDescriptor > av_pix_fmt_descriptors[AV_PIX_FMT_NB] = { > > .name = "qsv", > > .flags = AV_PIX_FMT_FLAG_HWACCEL, > > }, > > + [AV_PIX_FMT_MEDIACODEC] = { > > + .name = "mediacodec", > > + .flags = AV_PIX_FMT_FLAG_HWACCEL, > > + }, > > [AV_PIX_FMT_MMAL] = { > > .name = "mmal", > > .flags = AV_PIX_FMT_FLAG_HWACCEL, > > diff --git a/libavutil/pixfmt.h b/libavutil/pixfmt.h > > index dbd2470..f8533b0 100644 > > --- a/libavutil/pixfmt.h > > +++ b/libavutil/pixfmt.h > > @@ -300,6 +300,8 @@ enum AVPixelFormat { > > AV_PIX_FMT_GBRAP12BE, ///< planar GBR 4:4:4:4 48bpp, big-endian > > AV_PIX_FMT_GBRAP12LE, ///< planar GBR 4:4:4:4 48bpp, little-endian > > > > + AV_PIX_FMT_MEDIACODEC, ///< hardware decoding through MediaCodec > > + > > Don't forget a minor API bump to libavutil (and libavcodec too). > Fixed locally. > > > AV_PIX_FMT_NB, ///< number of pixel formats, DO NOT USE THIS > if you want to link with shared libav* because the number of formats might > differ between versions > > }; > > > I will wait the hwaccel stuff from libav to be merged in order to resubmit (and push if ok) this patch. Thanks for the review, Matthieu [...] _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
