Hi community,

I am exploring PyFlink and I wonder if it is possible to use PyFlink
together with all these ML libs that ML engineers normally use: PyTorch,
Tensorflow, Scikit Learn, Xgboost, LightGBM, etc.

According to this SO thread
<https://stackoverflow.com/questions/38187637/integrating-scikit-learn-with-pyspark>,
PySpark cannot use Scikit Learn directly inside UDF because Scikit Learn
algorithms are not implemented to be distributed, while Spark runs
distributedly.

Given PyFlink is similar to PySpark, I guess the answer may be "no". But I
would love to double check, and to see what I need to do to make PyFlink
able to define UDFs using these ML libs.


(This question is cross-posted on StackOverflow
https://stackoverflow.com/questions/66631859/can-i-use-pyflink-together-with-pytorch-tensorflow-scikitlearn-xgboost-lightgbm
)


Thanks.


Best,

Yik San

Reply via email to