Why don't you try to encapsulate your keras model within a wrapper class (an estimator let's say), and you implement inside this wrapper class the two functions: __getstate__ and __setstate__
On Thu, May 3, 2018 at 5:27 PM erp12 <eddie.pantri...@gmail.com> wrote: > I would like to create a Spark UDF which returns the a prediction made > with a > trained Keras model. Keras models are not typically pickle-able, however I > have used the monkey patch approach to making Keras models pickle-able, as > described here: http://zachmoshe.com/2017/04/03/pickling-keras-models.html > > This allows for models to be sent from the PySpark driver to the workers, > however the worker python processes do not have the monkey patched Model > class, and thus cannot properly un-pickle the models. To fix this issue, I > know I must call the monkey patching function (make_keras_picklable()) once > on each worker, however I have been unable to figure out how to do this. > > I am curious to hear if anyone has a fix for this issue, or would like to > offer an alternative way to make predictions with a Keras model within a > Spark UDF. > > Here is a Stack Overflow question with more details: > > https://stackoverflow.com/questions/50007126/pickling-monkey-patched-keras-model-for-use-in-pyspark > > Thank you! > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
