Re: OOM in spark driver

2015-09-04 Thread Akhil Das
t on hdfs and with the help > > of JavaSparkListener, copying file from hdfs to s3 with hadoop fileUtil > > in onApplicationEnd method. But my job is getting failed with OOM in > spark > > driver. > > > > *5/09/02 04:17:57 INFO cluster.YarnClusterSchedulerBackend: Asking

Re: OOM in spark driver

2015-09-02 Thread Mike Hynes
e Now i am saving result on hdfs and with the help > of JavaSparkListener, copying file from hdfs to s3 with hadoop fileUtil > in onApplicationEnd method. But my job is getting failed with OOM in spark > driver. > > *5/09/02 04:17:57 INFO cluster.YarnClusterSchedulerBackend: Asking ea

OOM in spark driver

2015-09-01 Thread ankit tyagi
the help of JavaSparkListener, copying file from hdfs to s3 with hadoop fileUtil in onApplicationEnd method. But my job is getting failed with OOM in spark driver. *5/09/02 04:17:57 INFO cluster.YarnClusterSchedulerBackend: Asking each executor to shut down* *15/09/02 04:17:59 INFO