For the case that ffmpeg is built with TENSORFLOW_BACKEND enabled, while there
is no TF at runtime, ff_get_dnn_module always returns valid pointer, and no
chance for the filter to fall back to native mode. Looks that we need a runtime
check in function ff_get_dnn_module.
static av_cold int init(AVFilterContext* context)
{
SRCNNContext* srcnn_context = context->priv;
#ifdef TENSORFLOW_BACKEND
srcnn_context->dnn_module = ff_get_dnn_module(DNN_TF);
if (!srcnn_context->dnn_module){
av_log(context, AV_LOG_INFO, "could not load tensorflow backend, using
native backend instead\n");
srcnn_context->dnn_module = ff_get_dnn_module(DNN_NATIVE);
}
#else
srcnn_context->dnn_module = ff_get_dnn_module(DNN_NATIVE);
#endif
DNNModule* ff_get_dnn_module(DNNBackendType backend_type)
{
...
case DNN_TF:
#ifdef TENSORFLOW_BACKEND
//add a runtime check here, possible?
dnn_module->load_model = &ff_dnn_load_model_tf;
dnn_module->load_default_model = &ff_dnn_load_default_model_tf;
dnn_module->execute_model = &ff_dnn_execute_model_tf;
dnn_module->free_model = &ff_dnn_free_model_tf;
#else
av_freep(dnn_module);
return NULL;
#endif
-----Original Message-----
From: ffmpeg-devel [mailto:ffmpeg-devel-boun...@ffmpeg.org] On Behalf Of Pedro
Arthur
Sent: Tuesday, May 29, 2018 8:45 PM
To: Sergey Lavrushkin <dual...@gmail.com>
Cc: FFmpeg development discussions and patches <ffmpeg-devel@ffmpeg.org>
Subject: Re: [FFmpeg-devel] [GSOC] [PATCH] DNN module introduction and SRCNN
filter update
2018-05-29 5:14 GMT-03:00 Sergey Lavrushkin <dual...@gmail.com>:
> 2018-05-29 4:08 GMT+03:00 Pedro Arthur <bygran...@gmail.com>:
>>
>> 2018-05-28 19:52 GMT-03:00 Sergey Lavrushkin <dual...@gmail.com>:
>> > 2018-05-28 9:32 GMT+03:00 Guo, Yejun <yejun....@intel.com>:
>> >
>> >> looks that no tensorflow dependency is introduced, a new model
>> >> format is
>> >> created together with some CPU implementation for inference. With
>> >> this
>> >> idea, Android Neural Network would be a very good reference, see
>> >> https://developer.android.google.cn/ndk/guides/neuralnetworks/. It
>> >> defines how the model is organized, and also provided a CPU
>> >> optimized inference implementation (within the NNAPI runtime, it is open
>> >> source).
>> >> It
>> >> is still under development but mature enough to run some popular
>> >> dnn models with proper performance. We can absorb some basic
>> >> design. Anyway, just a reference fyi. (btw, I'm not sure about
>> >> any IP issue)
>> >>
>> >
>> > The idea was to first introduce something to use when tensorflow is
>> > not available. Here is another patch, that introduces tensorflow backend.
>> I think it would be better for reviewing if you send the second patch
>> in a new email.
>
>
> Then we need to push the first patch, I think.
Not necessarily, 'git send-email' may give you a glimpse of how it is done.
>
>>
>> >
>> >
>> >> For this patch, I have two comments.
>> >>
>> >> 1. change from "DNNModel* (*load_default_model)(DNNDefaultModel
>> >> model_type);" to " DNNModel* (*load_builtin_model)(DNNBuiltinModel
>> >> model_type);"
>> >> The DNNModule can be invoked by many filters, default model is a
>> >> good name at the filter level, while built-in model is better
>> >> within the DNN scope.
>> >>
>> >> typedef struct DNNModule{
>> >> // Loads model and parameters from given file. Returns NULL if
>> >> it is not possible.
>> >> DNNModel* (*load_model)(const char* model_filename);
>> >> // Loads one of the default models
>> >> DNNModel* (*load_default_model)(DNNDefaultModel model_type);
>> >> // Executes model with specified input and output. Returns
>> >> DNN_ERROR otherwise.
>> >> DNNReturnType (*execute_model)(const DNNModel* model);
>> >> // Frees memory allocated for model.
>> >> void (*free_model)(DNNModel** model); } DNNModule;
>> >>
>> >>
>> >> 2. add a new variable 'number' for DNNData/InputParams As a
>> >> typical DNN concept, the data shape usually is: <number, height,
>> >> width, channel> or <number, channel, height, width>, the last
>> >> component denotes its index changes the fastest in the memory. We
>> >> can add this concept into the API, and decide to support <NHWC> or <NCHW>
>> >> or both.
>> >
>> >
>> > I did not add number of elements in batch because I thought, that
>> > we would not feed more than one element at once to a network in a
>> > ffmpeg filter.
>> > But it can be easily added if necessary.
>> >
>> > So here is the patch that adds tensorflow backend with the previous
>> > patch.
>> > I forgot to change include guards from AVUTIL_* to AVFILTER_* in it.
>> You moved the files from libavutil to libavfilter while it was
>> proposed to move them to libavformat.
>
>
> Not only, it was also proposed to move it to libavfilter if it is
> going to be used only in filters. I do not know if this module is
> useful anywhere else besides libavfilter.
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
