Performance issue when running Spark-1.6.1 in yarn-client mode with Hadoop 2.6.0

2017-06-06 Thread satishjohn
Performance issue / time taken to complete spark job in yarn is 4 x slower, when considered spark standalone mode. However, in spark standalone mode jobs often fails with executor lost issue. Hardware configuration 32GB RAM 8 Cores (16) and 1 TB HDD 3 (1 Master and 2 Workers) Spark configurat

Exception when accessing Spark Web UI in yarn-client mode

2017-06-14 Thread satishjohn
When I click on the ApplicationMaster URL I see this exception. 500013 [qtp1921129349-1619] WARN o.s.j.server.AbstractHttpConnection - header full: java.lang.RuntimeException: Header>6144 500014 [qtp1921129349-1619] WARN o.s.j.server.AbstractHttpConnection - /null/null/null/null/null/null/null/