LogisticRegression models consumes all driver memory

2015-09-23 Thread Eugene Zhulenev
We are running Apache Spark 1.5.0 (latest code from 1.5 branch) We are running 2-3 LogisticRegression models in parallel (we'd love to run 10-20 actually), they are not really big at all, maybe 1-2 million rows in each model. Cluster itself, and all executors look good. Enough free memory and no

Re: LogisticRegression models consumes all driver memory

2015-09-23 Thread Eugene Zhulenev
08DF8D> > > On Wed, Sep 23, 2015 at 3:19 PM, Eugene Zhulenev < > eugene.zhule...@gmail.com> wrote: > >> We are running Apache Spark 1.5.0 (latest code from 1.5 branch) >> >> We are running 2-3 LogisticRegression models in parallel (we'd love to >>

Re: LogisticRegression models consumes all driver memory

2015-09-23 Thread Eugene Zhulenev
du/pks/lookup?search=0x59DF55B8AF08DF8D> > > On Wed, Sep 23, 2015 at 4:38 PM, Eugene Zhulenev < > eugene.zhule...@gmail.com> wrote: > >> It's really simple: >> https://gist.github.com/ezhulenev/886517723ca4a353 >> >> The same strange heap behavi

Re: LogisticRegression models consumes all driver memory

2015-09-25 Thread Eugene Zhulenev
> Sincerely, > > DB Tsai > -- > Blog: https://www.dbtsai.com > PGP Key ID: 0xAF08DF8D > <https://pgp.mit.edu/pks/lookup?search=0x59DF55B8AF08DF8D> > > On Wed, Sep 23, 2015 at 5:39 PM, Eugene Zhulenev < > eugene.zhule...@gmail.com> wrote: