On Fri, Jun 24, 2016 at 06:18:02PM +0200, Michael Niedermayer wrote: > On Fri, Jun 24, 2016 at 11:17:41AM +0200, Matthieu Bouron wrote: > > On Thu, Apr 07, 2016 at 02:51:44PM +0200, Matthieu Bouron wrote: > > > On Wed, Mar 23, 2016 at 6:16 PM, Matthieu Bouron > > > <matthieu.bou...@gmail.com> > > > wrote: > > > > > > > > > > > > > > > On Tue, Mar 22, 2016 at 10:04 AM, Matthieu Bouron < > > > > matthieu.bou...@gmail.com> wrote: > > > > > > > >> > > > >> > > > >> On Fri, Mar 18, 2016 at 5:50 PM, Matthieu Bouron < > > > >> matthieu.bou...@gmail.com> wrote: > > > >> > > > >>> From: Matthieu Bouron <matthieu.bou...@stupeflix.com> > > > >>> > > > >>> --- > > > >>> > > > >>> Hello, > > > >>> > > > >>> The following patch add hwaccel support to the mediacodec (h264) > > > >>> decoder > > > >>> by allowing > > > >>> the user to render the output frames directly on a surface. > > > >>> > > > >>> In order to do so the user needs to initialize the hwaccel through the > > > >>> use of > > > >>> av_mediacodec_alloc_context and av_mediacodec_default_init functions. > > > >>> The later > > > >>> takes a reference to an android/view/Surface as parameter. > > > >>> > > > >>> If the hwaccel successfully initialize, the decoder output frames pix > > > >>> fmt will be > > > >>> AV_PIX_FMT_MEDIACODEC. The following snippet of code demonstrate how > > > >>> to > > > >>> render > > > >>> the frames on the surface: > > > >>> > > > >>> AVMediaCodecBuffer *buffer = (AVMediaCodecBuffer *)frame->data[3]; > > > >>> av_mediacodec_release_buffer(buffer, 1); > > > >>> > > > >>> The last argument of av_mediacodec_release_buffer enable rendering of > > > >>> the > > > >>> buffer on the surface (or not if set to 0). > > > >>> > > > >>> Regarding the internal changes in the mediacodec decoder: > > > >>> > > > >>> MediaCodec.flush() discards both input and output buffers meaning > > > >>> that if > > > >>> MediaCodec.flush() is called all output buffers the user has a > > > >>> reference > > > >>> on are > > > >>> now invalid (and cannot be used). > > > >>> This behaviour does not fit well in the avcodec API. > > > >>> > > > >>> When the decoder is configured to output software buffers, there is no > > > >>> issue as > > > >>> the buffers are copied. > > > >>> > > > >>> Now when the decoder is configured to output to a surface, the user > > > >>> might not > > > >>> want to render all the frames as fast as the decoder can go and might > > > >>> want to > > > >>> control *when* the frame are rendered, so we need to make sure that > > > >>> the > > > >>> MediaCodec.flush() call is delayed until all the frames the user > > > >>> retains > > > >>> has > > > >>> been released or rendered. > > > >>> > > > >>> Delaying the call to MediaCodec.flush() means buffering any inputs > > > >>> that > > > >>> come > > > >>> the decoder until the user has released/renderer the frame he retains. > > > >>> > > > >>> This is a limitation of this hwaccel implementation, if the user > > > >>> retains > > > >>> a > > > >>> frame (a), then issue a flush command to the decoder, the packets he > > > >>> feeds to > > > >>> the decoder at that point will be queued in the internal decoder > > > >>> packet > > > >>> queue > > > >>> (until he releases the frame (a)). This scenario leads to a memory > > > >>> usage > > > >>> increase to say the least. > > > >>> > > > >>> Currently there is no limitation on the size of the internal decoder > > > >>> packet > > > >>> queue but this is something that can be added easily. Then, if the > > > >>> queue > > > >>> is > > > >>> full, what would be the behaviour of the decoder ? Can it block ? Or > > > >>> should it > > > >>> returns something like AVERROR(EAGAIN) ? > > > >>> > > > >>> About the other internal decoder changes I introduced: > > > >>> > > > >>> The MediaCodecDecContext is now refcounted (using the lavu/atomic api) > > > >>> since > > > >>> the (hwaccel) frames can be retained by the user, we need to delay the > > > >>> destruction of the codec until the user has released all the frames he > > > >>> has a > > > >>> reference on. > > > >>> The reference counter of the MediaCodecDecContext is incremented each > > > >>> time an > > > >>> (hwaccel) frame is outputted by the decoder and decremented each time > > > >>> a > > > >>> (hwaccel) frame is released. > > > >>> > > > >>> Also, when the decoder is configured to output to a surface the pts > > > >>> that > > > >>> are > > > >>> given to the MediaCodec API are now rescaled based on the > > > >>> codec_timebase > > > >>> as > > > >>> those timestamps values are propagated to the frames rendered on the > > > >>> surface > > > >>> since Android M. Not sure if it's really useful though. > > > >>> > > > >>> On the performance side: > > > >>> > > > >>> On a nexus 5, decoding an h264 stream (main profile) 1080p@60fps: > > > >>> - software output + rgba conversion goes at 59~60fps > > > >>> - surface output + render on a surface goes at 100~110fps > > > >>> > > > >>> > > > >> [...] > > > >> > > > >> Patch updated with the following differences: > > > >> * the public mediacodec api is now always built (not only when > > > >> mediacodec is available) (and the build when mediacodec is not > > > >> available > > > >> has been fixed) > > > >> * the documentation of av_mediacodec_release_buffer has been > > > >> improved a > > > >> bit > > > >> > > > > > > > > Patch updated with the following differences: > > > > MediaCodecBuffer->released type is now a volatile int (instead of a > > > > int*) > > > > MediaCodecContext->refcount type is now a volatile int (instead of a > > > > int*) > > > > > > > > > > Ping. > > > > Rebased patch attached. > > > > Matthieu > > > configure | 1 > > libavcodec/Makefile | 6 > > libavcodec/allcodecs.c | 1 > > libavcodec/mediacodec.c | 133 ++++++++++++++++++++ > > libavcodec/mediacodec.h | 88 +++++++++++++ > > libavcodec/mediacodec_surface.c | 66 +++++++++ > > libavcodec/mediacodec_surface.h | 31 ++++ > > libavcodec/mediacodec_wrapper.c | 5 > > libavcodec/mediacodecdec.c | 265 > > +++++++++++++++++++++++++++++++++------- > > libavcodec/mediacodecdec.h | 17 ++ > > libavcodec/mediacodecdec_h264.c | 44 +++++- > > libavcodec/version.h | 2 > > > libavutil/pixdesc.c | 4 > > libavutil/pixfmt.h | 2 > > libavutil/version.h | 2 > > libavutil changes look good, i dont know mediacodec so cannot easily > comment about the rest
Thanks. I'll wait another 3 days and if there is no objection I will push the patch. Matthieu [...] _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
