Yes, certainly debatable for word2vec. You have a good point that this could overrun the 2GB limit if the model is one big datum, for large but not crazy models. This model could probably easily be serialized as individual vectors in this case. It would introduce a backwards-compatibility issue but it's possible to read old and new formats, I believe.
On Fri, Jan 13, 2017 at 8:16 PM Asher Krim <ak...@hubspot.com> wrote: > I guess it depends on the definition of "small". A Word2vec model with > vectorSize=300 and vocabulary=3m takes nearly 4gb. While it does fit on a > single machine (so isn't really "big" data), I don't see the benefit in > having the model stored in one file. On the contrary, it seems that we > would want the model to be distributed: > * avoids shuffling of data to one executor > * allows the whole cluster to participate in saving the model > * avoids rpc issues ( > http://stackoverflow.com/questions/40842736/spark-word2vecmodel-exceeds-max-rpc-size-for-saving > ) > * "feature parity" with mllib (issues with one large model file already > solved for mllib in SPARK-11994 > <https://issues.apache.org/jira/browse/SPARK-11994>) > > > On Fri, Jan 13, 2017 at 1:02 PM, Nick Pentreath <nick.pentre...@gmail.com> > wrote: > > Yup - it's because almost all model data in spark ML (model coefficients) > is "small" - i.e. Non distributed. > > If you look at ALS you'll see there is no repartitioning since the factor > dataframes can be large > On Fri, 13 Jan 2017 at 19:42, Sean Owen <so...@cloudera.com> wrote: > > You're referring to code that serializes models, which are quite small. > For example a PCA model consists of a few principal component vector. It's > a Dataset of just one element being saved here. It's re-using the code path > normally used to save big data sets, to output 1 file with 1 thing as > Parquet. > > On Fri, Jan 13, 2017 at 5:29 PM Asher Krim <ak...@hubspot.com> wrote: > > But why is that beneficial? The data is supposedly quite large, > distributing it across many partitions/files would seem to make sense. > > On Fri, Jan 13, 2017 at 12:25 PM, Sean Owen <so...@cloudera.com> wrote: > > That is usually so the result comes out in one file, not partitioned over > n files. > > On Fri, Jan 13, 2017 at 5:23 PM Asher Krim <ak...@hubspot.com> wrote: > > Hi, > > I'm curious why it's common for data to be repartitioned to 1 partition > when saving ml models: > > sqlContext.createDataFrame(Seq(data)).repartition(1 > ).write.parquet(dataPath) > > This shows up in most ml models I've seen (Word2Vec > <https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/feature/Word2Vec.scala#L314>, > PCA > <https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/feature/PCA.scala#L189>, > LDA > <https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/clustering/LDA.scala#L605>). > Am I missing some benefit of repartitioning like this? > > Thanks, > -- > Asher Krim > Senior Software Engineer > > > > > -- > Asher Krim > Senior Software Engineer > >
