on quick glance, it appears that you're calling collect() in there which is bringing down a huge amount of data down to the single Driver. this is why, when you allocated more memory to the Driver, a different error emerges most -definitely related to stop-the-world GC to cause the node to become unresponsive.
in general, collect() is bad and should only be used on small datasets for debugging or sanity-check purposes. anything serious should be done within the executors running on worker nodes - not on the Driver node. think of the Driver not as simply a coordinator - coordinating and allocating tasks to workers with the help of the cluster resource manager (i.e. YARN, Spark Standalone, Mesos, etc) very little memory should be allocated to the Driver. just enough to coordinate and a little extra for Driver-side debugging, but that's it. leave the rest up to the cluster nodes. > On Dec 29, 2015, at 8:28 PM, vadimtk <[email protected]> wrote: > > table > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
