Hi Jeroen, in case you are using HIVE partitions how many partitions do you have?
Also is there any chance that you might post the code? Regards, Gourav Sengupta On Tue, Jan 2, 2018 at 7:50 AM, Jeroen Miller <bluedasya...@gmail.com> wrote: > Hello Gourav, > > On 30 Dec 2017, at 20:20, Gourav Sengupta <gourav.sengu...@gmail.com> > wrote: > > Please try to use the SPARK UI from the way that AWS EMR recommends, it > should be available from the resource manager. I never ever had any problem > working with it. THAT HAS ALWAYS BEEN MY PRIMARY AND SOLE SOURCE OF > DEBUGGING. > > For some reason sometimes there is absolutely nothing showing up in the > Spark UI or the UI is not refreshed, e.g. for the current stage is #x while > the logs shows stage #y (with y > x) is currently under way. > > It may very well be that the source of this problem lies between the > keyboard and the chair, but if this is the case, I do not know how to solve > this. > > > Also, I ALWAYS prefer the maximize Resource Allocation setting in EMR to > be set to true. > > Thanks for the tip -- will try this setting in my next batch of > experiments! > > JM > >
