+1 on SPARK-25004. We have found it quite useful to diagnose PySpark OOM. On Tue, Aug 7, 2018 at 1:21 PM Holden Karau <hol...@pigscanfly.ca> wrote:
> I'd like to suggest we consider SPARK-25004 (hopefully it goes in soon), > but solving some of the consistent Python memory issues we've had for years > would be really amazing to get in. > > On Tue, Aug 7, 2018 at 1:07 PM, Tom Graves <tgraves...@yahoo.com.invalid> > wrote: > >> I would like to get clarification on our avro compatibility story before >> the release. anyone interested please look at - >> https://issues.apache.org/jira/browse/SPARK-24924 . I probably should >> have filed a separate jira and can if we don't resolve via discussion there. >> >> Tom >> >> On Tuesday, August 7, 2018, 11:46:31 AM CDT, shane knapp < >> skn...@berkeley.edu> wrote: >> >> >> According to the status, I think we should wait a few more days. Any >> objections? >> >> >> none here. >> >> i'm also pretty certain that waiting until after the code freeze to start >> testing the GHPRB on ubuntu is the wisest course of action for us. >> >> shane >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> >> > > > -- > Twitter: https://twitter.com/holdenkarau > -- John
