Re: [External Sender] Memory issues in 3.0.2 but works well on 2.4.4

2021-05-21 Thread Femi Anthony
Post the stack trace and provide some more details about your configuration On Fri, May 21, 2021 at 7:52 AM Praneeth Shishtla wrote: > Hi, > I have a simple DecisionForest model and was able to train the model on > pyspark==2.4.4 without any issues. > However, when I upgraded to pyspark==3.0.2,

Memory issues in 3.0.2 but works well on 2.4.4

2021-05-21 Thread Praneeth Shishtla
Hi, I have a simple DecisionForest model and was able to train the model on pyspark==2.4.4 without any issues. However, when I upgraded to pyspark==3.0.2, the fit takes a lot of time and eventually errors out saying out of memory. Even tried reducing the number of samples for training but no luck.