Many thanks. Will try it. On Thu, Jun 8, 2017 at 8:41 AM Nick Pentreath <nick.pentre...@gmail.com> wrote:
> Spark 2.2 will support the recommend-all methods in ML. > > Also, both ML and MLLIB performance has been greatly improved for the > recommend-all methods. > > Perhaps you could check out the current RC of Spark 2.2 or master branch > to try it out? > > N > > On Thu, 8 Jun 2017 at 17:18, Sahib Aulakh [Search] < > sahibaul...@coupang.com> wrote: > >> Many thanks for your response. I already figured out the details with >> some help from another forum. >> >> >> 1. I was trying to predict ratings for all users and all products. >> This is inefficient and now I am trying to reduce the number of required >> predictions. >> 2. There is a nice example buried in Spark source code which points >> out the usage of ML side ALS. >> >> Regards. >> Sahib Aulakh. >> >> On Wed, Jun 7, 2017 at 8:17 PM, Ryan <ryan.hd....@gmail.com> wrote: >> >>> 1. could you give job, stage & task status from Spark UI? I found it >>> extremely useful for performance tuning. >>> >>> 2. use modele.transform for predictions. Usually we have a pipeline for >>> preparing training data, and use the same pipeline to transform data you >>> want to predict could give us the prediction column. >>> >>> On Thu, Jun 1, 2017 at 7:48 AM, Sahib Aulakh [Search] < >>> sahibaul...@coupang.com> wrote: >>> >>>> Hello: >>>> >>>> I am training the ALS model for recommendations. I have about 200m >>>> ratings from about 10m users and 3m products. I have a small cluster with >>>> 48 cores and 120gb cluster-wide memory. >>>> >>>> My code is very similar to the example code >>>> >>>> spark/examples/src/main/scala/org/apache/spark/examples/mllib/MovieLensALS.scala >>>> code. >>>> >>>> I have a couple of questions: >>>> >>>> >>>> 1. All steps up to model training runs reasonably fast. Model >>>> training is under 10 minutes for rank 20. However, the >>>> model.recommendProductsForUsers step is either slow or just does not >>>> work >>>> as the code just seems to hang at this point. I have tried user and >>>> product >>>> blocks sizes of -1 and 20, 40, etc, played with executor memory size, >>>> etc. >>>> Can someone shed some light here as to what could be wrong? >>>> 2. Also, is there any example code for the ml.recommendation.ALS >>>> algorithm? I can figure out how to train the model but I don't >>>> understand >>>> (from the documentation) how to perform predictions? >>>> >>>> Thanks for any information you can provide. >>>> Sahib Aulakh. >>>> >>>> >>>> -- >>>> Sahib Aulakh >>>> Sr. Principal Engineer >>>> >>> >>> >> >> >> -- >> Sahib Aulakh >> Sr. Principal Engineer >> > -- Sahib Aulakh Sr. Principal Engineer
