Your code looks correct for me. How many # of features do you have in this training? How many tasks are running in the job?
Sincerely, DB Tsai ---------------------------------------------------------- Blog: https://www.dbtsai.com PGP Key ID: 0xAF08DF8D <https://pgp.mit.edu/pks/lookup?search=0x59DF55B8AF08DF8D> On Wed, Sep 23, 2015 at 4:38 PM, Eugene Zhulenev <eugene.zhule...@gmail.com> wrote: > It's really simple: https://gist.github.com/ezhulenev/7777886517723ca4a353 > > The same strange heap behavior we've seen even for single model, it takes > ~20 gigs heap on a driver to build single model with less than 1 million > rows in input data frame. > > On Wed, Sep 23, 2015 at 6:31 PM, DB Tsai <dbt...@dbtsai.com> wrote: > >> Could you paste some of your code for diagnosis? >> >> >> Sincerely, >> >> DB Tsai >> ---------------------------------------------------------- >> Blog: https://www.dbtsai.com >> PGP Key ID: 0xAF08DF8D >> <https://pgp.mit.edu/pks/lookup?search=0x59DF55B8AF08DF8D> >> >> On Wed, Sep 23, 2015 at 3:19 PM, Eugene Zhulenev < >> eugene.zhule...@gmail.com> wrote: >> >>> We are running Apache Spark 1.5.0 (latest code from 1.5 branch) >>> >>> We are running 2-3 LogisticRegression models in parallel (we'd love to >>> run 10-20 actually), they are not really big at all, maybe 1-2 million rows >>> in each model. >>> >>> Cluster itself, and all executors look good. Enough free memory and no >>> exceptions or errors. >>> >>> However I see very strange behavior inside Spark driver. Allocated heap >>> constantly growing. It grows up to 30 gigs in 1.5 hours and then everything >>> becomes super sloooooow. >>> >>> We don't do any collect, and I really don't understand who is consuming >>> all this memory. Looks like it's something inside LogisticRegression >>> itself, however I only see treeAggregate which should not require so much >>> memory to run. >>> >>> Any ideas? >>> >>> Plus I don't see any GC pause, looks like memory is still used by >>> someone inside driver. >>> >>> [image: Inline image 2] >>> [image: Inline image 1] >>> >> >> >
