Thanks Marcelo - looks like it was my fault. Seems when we deployed the new version of spark it was picking up the wrong yarn site and setting the wrong proxy host. All good now!
On Wed, Jun 3, 2015 at 11:01 AM, Marcelo Vanzin <van...@cloudera.com> wrote: > That code hasn't changed at all between 1.3 and 1.4; it also has been > working fine for me. > > Are you sure you're using exactly the same Hadoop libraries (since you're > building with -Phadoop-provided) and Hadoop configuration in both cases? > > On Tue, Jun 2, 2015 at 5:29 PM, Night Wolf <nightwolf...@gmail.com> wrote: > >> Hi all, >> >> Trying out Spark 1.4 on MapR Hadoop 2.5.1 running in yarn-client mode. >> Seems the application master doesn't work anymore, I get a 500 connect >> refused, even when I hit the IP/port of the spark UI directly. The logs >> don't show much. >> >> I build spark with Java 6, hive & scala 2.10 and 2.11. I've tried with >> and without -Phadoop-provided >> >> *Build command;* >> >> ./make-distribution.sh --name mapr4.0.2_yarn_j6_2.10 --tgz -Pyarn -Pmapr4 >> -Phadoop-2.4 -Pmapr4 -Phive -Phadoop-provided >> -Dhadoop.version=2.5.1-mapr-1501 -Dyarn.version=2.5.1-mapr-1501 -DskipTests >> -e -X >> >> *Logs from spark shell;* >> >> 15/06/03 00:10:56 INFO server.AbstractConnector: Started >> SelectChannelConnector@0.0.0.0:4040 >> 15/06/03 00:10:56 INFO util.Utils: Successfully started service 'SparkUI' >> on port 4040. >> 15/06/03 00:10:56 INFO ui.SparkUI: Started SparkUI at >> http://172.31.10.14:4040 >> 15/06/03 00:10:57 INFO yarn.Client: Requesting a new application from >> cluster with 71 NodeManagers >> 15/06/03 00:10:57 INFO yarn.Client: Verifying our application has not >> requested more than the maximum memory capability of the cluster (112640 MB >> per container) >> 15/06/03 00:10:57 INFO yarn.Client: Will allocate AM container, with 896 >> MB memory including 384 MB overhead >> 15/06/03 00:10:57 INFO yarn.Client: Setting up container launch context >> for our AM >> 15/06/03 00:10:57 INFO yarn.Client: Preparing resources for our AM >> container >> 15/06/03 00:10:57 INFO yarn.Client: Uploading resource >> file:///apps/spark/spark-1.4.0-SNAPSHOT-bin-mapr4.0.2_yarn_j6_2.11/lib/spark-assembly-1.4.0-SNAPSHOT-hadoop2.5.1-mapr-1501.jar >> -> >> maprfs:/user/nw/.sparkStaging/application_1432690361766_0593/spark-assembly-1.4.0-SNAPSHOT-hadoop2.5.1-mapr-1501.jar >> 15/06/03 00:10:58 INFO yarn.Client: Uploading resource >> file:/tmp/spark-5e42f904-ff83-4c93-bd35-4c3e20226a8a/__hadoop_conf__983379693214711.zip >> -> >> maprfs:/user/nw/.sparkStaging/application_1432690361766_0593/__hadoop_conf__983379693214711.zip >> 15/06/03 00:10:58 INFO yarn.Client: Setting up the launch environment for >> our AM container >> 15/06/03 00:10:58 INFO spark.SecurityManager: Changing view acls to: nw >> 15/06/03 00:10:58 INFO spark.SecurityManager: Changing modify acls to: nw >> 15/06/03 00:10:58 INFO spark.SecurityManager: SecurityManager: >> authentication disabled; ui acls disabled; users with view permissions: >> Set(nw); users with modify permissions: Set(nw) >> 15/06/03 00:10:58 INFO yarn.Client: Submitting application 593 to >> ResourceManager >> 15/06/03 00:10:58 INFO security.ExternalTokenManagerFactory: Initialized >> external token manager class - >> com.mapr.hadoop.yarn.security.MapRTicketManager >> 15/06/03 00:10:58 INFO impl.YarnClientImpl: Submitted application >> application_1432690361766_0593 >> 15/06/03 00:10:59 INFO yarn.Client: Application report for >> application_1432690361766_0593 (state: ACCEPTED) >> 15/06/03 00:10:59 INFO yarn.Client: >> client token: N/A >> diagnostics: N/A >> ApplicationMaster host: N/A >> ApplicationMaster RPC port: -1 >> queue: default >> start time: 1433290258143 >> final status: UNDEFINED >> tracking URL: >> http://qtausc-pphd0101.hadoop.local:8088/proxy/application_1432690361766_0593/ >> user: nw >> 15/06/03 00:11:00 INFO yarn.Client: Application report for >> application_1432690361766_0593 (state: ACCEPTED) >> 15/06/03 00:11:01 INFO yarn.Client: Application report for >> application_1432690361766_0593 (state: ACCEPTED) >> 15/06/03 00:11:02 INFO yarn.Client: Application report for >> application_1432690361766_0593 (state: ACCEPTED) >> 15/06/03 00:11:03 INFO yarn.Client: Application report for >> application_1432690361766_0593 (state: ACCEPTED) >> 15/06/03 00:11:03 INFO >> cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster >> registered as AkkaRpcEndpointRef(Actor[akka.tcp:// >> sparkYarnAM@192.168.81.167:36542/user/YarnAM#1631897818]) >> 15/06/03 00:11:03 INFO cluster.YarnClientSchedulerBackend: Add WebUI >> Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, >> Map(PROXY_HOSTS -> qtausc-pphd0167.hadoop.local, PROXY_URI_BASES -> >> http://qtausc-pphd0167.hadoop.local:8088/proxy/application_1432690361766_0593), >> /proxy/application_1432690361766_0593 >> 15/06/03 00:11:03 INFO ui.JettyUtils: Adding filter: >> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter >> 15/06/03 00:11:04 INFO yarn.Client: Application report for >> application_1432690361766_0593 (state: RUNNING) >> 15/06/03 00:11:04 INFO yarn.Client: >> client token: N/A >> diagnostics: N/A >> ApplicationMaster host: 192.168.81.167 >> ApplicationMaster RPC port: 0 >> queue: default >> start time: 1433290258143 >> final status: UNDEFINED >> tracking URL: >> http://qtausc-pphd0101.hadoop.local:8088/proxy/application_1432690361766_0593/ >> user: nw >> 15/06/03 00:11:04 INFO cluster.YarnClientSchedulerBackend: Application >> application_1432690361766_0593 has started running. >> 15/06/03 00:11:04 INFO util.Utils: Successfully started service >> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45668. >> 15/06/03 00:11:04 INFO netty.NettyBlockTransferService: Server created on >> 45668 >> 15/06/03 00:11:04 INFO storage.BlockManagerMaster: Trying to register >> BlockManager >> 15/06/03 00:11:04 INFO storage.BlockManagerMasterEndpoint: Registering >> block manager 172.31.10.14:45668 with 265.4 MB RAM, >> BlockManagerId(driver, 172.31.10.14, 45668) >> 15/06/03 00:11:04 INFO storage.BlockManagerMaster: Registered BlockManager >> >> >> *Logs from AM logs page in YARN;* >> 15/06/03 10:11:01 INFO yarn.ApplicationMaster: Registered signal handlers >> for [TERM, HUP, INT] >> 15/06/03 10:11:02 INFO yarn.ApplicationMaster: ApplicationAttemptId: >> appattempt_1432690361766_0593_000001 >> 15/06/03 10:11:02 INFO spark.SecurityManager: Changing view acls to: nw >> 15/06/03 10:11:02 INFO spark.SecurityManager: Changing modify acls to: nw >> 15/06/03 10:11:02 INFO spark.SecurityManager: SecurityManager: >> authentication disabled; ui acls disabled; users with view permissions: >> Set(nw); users with modify permissions: Set(nw) >> 15/06/03 10:11:03 INFO slf4j.Slf4jLogger: Slf4jLogger started >> 15/06/03 10:11:03 INFO Remoting: Starting remoting >> 15/06/03 10:11:03 INFO Remoting: Remoting started; listening on addresses >> :[akka.tcp://sparkYarnAM@192.168.81.167:36542] >> 15/06/03 10:11:03 INFO util.Utils: Successfully started service >> 'sparkYarnAM' on port 36542. >> 15/06/03 10:11:03 INFO yarn.ApplicationMaster: Waiting for Spark driver >> to be reachable. >> 15/06/03 10:11:03 INFO yarn.ApplicationMaster: Driver now available: >> 172.31.10.14:59954 >> 15/06/03 10:11:03 INFO yarn.ApplicationMaster$AMEndpoint: Add WebUI >> Filter. >> AddWebUIFilter(org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,Map(PROXY_HOSTS >> -> qtausc-pphd0167.hadoop.local, PROXY_URI_BASES -> >> http://qtausc-pphd0167.hadoop.local:8088/proxy/application_1432690361766_0593 >> ),/proxy/application_1432690361766_0593) >> 15/06/03 10:11:03 INFO yarn.YarnRMClient: Registering the >> ApplicationMaster >> 15/06/03 10:11:04 INFO yarn.YarnAllocator: Will request 2 executor >> containers, each with 1 cores and 1408 MB memory including 384 MB overhead >> 15/06/03 10:11:04 INFO yarn.YarnAllocator: Container request (host: Any, >> capability: <memory:1408, vCores:1, disks:0.0>) >> 15/06/03 10:11:04 INFO yarn.YarnAllocator: Container request (host: Any, >> capability: <memory:1408, vCores:1, disks:0.0>) >> 15/06/03 10:11:04 INFO yarn.ApplicationMaster: Started progress reporter >> thread with (heartbeat : 3000, initial allocation : 200) intervals >> 15/06/03 10:11:04 INFO impl.AMRMClientImpl: Received new token for : >> qtausc-pphd0146.hadoop.local:55935 >> 15/06/03 10:11:04 INFO impl.AMRMClientImpl: Received new token for : >> qtausc-pphd0155.hadoop.local:45589 >> 15/06/03 10:11:04 INFO yarn.YarnAllocator: Launching container >> container_1432690361766_0593_01_000002 for on host >> qtausc-pphd0146.hadoop.local >> 15/06/03 10:11:04 INFO yarn.YarnAllocator: Launching ExecutorRunnable. >> driverUrl: akka.tcp:// >> sparkDriver@172.31.10.14:59954/user/CoarseGrainedScheduler, >> executorHostname: qtausc-pphd0146.hadoop.local >> 15/06/03 10:11:04 INFO yarn.YarnAllocator: Launching container >> container_1432690361766_0593_01_000003 for on host >> qtausc-pphd0155.hadoop.local >> 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Starting Executor Container >> 15/06/03 10:11:04 INFO yarn.YarnAllocator: Launching ExecutorRunnable. >> driverUrl: akka.tcp:// >> sparkDriver@172.31.10.14:59954/user/CoarseGrainedScheduler, >> executorHostname: qtausc-pphd0155.hadoop.local >> 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Starting Executor Container >> 15/06/03 10:11:04 INFO yarn.YarnAllocator: Received 2 containers from >> YARN, launching executors on 2 of them. >> 15/06/03 10:11:04 INFO impl.ContainerManagementProtocolProxy: >> yarn.client.max-nodemanagers-proxies : 500 >> 15/06/03 10:11:04 INFO impl.ContainerManagementProtocolProxy: >> yarn.client.max-nodemanagers-proxies : 500 >> 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Setting up >> ContainerLaunchContext >> 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Setting up >> ContainerLaunchContext >> 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Preparing Local resources >> 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Preparing Local resources >> 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Prepared Local resources >> Map(__spark__.jar -> resource { scheme: "maprfs" port: -1 file: >> "/user/nw/.sparkStaging/application_1432690361766_0593/spark-assembly-1.4.0-SNAPSHOT-hadoop2.5.1-mapr-1501.jar" >> } size: 124419029 timestamp: 1433290257972 type: FILE visibility: PRIVATE) >> 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Prepared Local resources >> Map(__spark__.jar -> resource { scheme: "maprfs" port: -1 file: >> "/user/nw/.sparkStaging/application_1432690361766_0593/spark-assembly-1.4.0-SNAPSHOT-hadoop2.5.1-mapr-1501.jar" >> } size: 124419029 timestamp: 1433290257972 type: FILE visibility: PRIVATE) >> 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Setting up executor with >> environment: Map(CLASSPATH -> >> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>/opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/mapr/hive/hive-current/lib/*, >> SPARK_LOG_URL_STDERR -> >> http://qtausc-pphd0155.hadoop.local:8042/node/containerlogs/container_1432690361766_0593_01_000003/nw/stderr?start=0, >> SPARK_DIST_CLASSPATH -> >> /opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/mapr/hive/hive-current/lib/*, >> SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1432690361766_0593, >> SPARK_YARN_CACHE_FILES_FILE_SIZES -> 124419029, SPARK_USER -> nw, >> SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE, SPARK_YARN_MODE -> true, >> SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1433290257972, SPARK_LOG_URL_STDOUT >> -> >> http://qtausc-pphd0155.hadoop.local:8042/node/containerlogs/container_1432690361766_0593_01_000003/nw/stdout?start=0, >> SPARK_YARN_CACHE_FILES -> >> maprfs:/user/nw/.sparkStaging/application_1432690361766_0593/spark-assembly-1.4.0-SNAPSHOT-hadoop2.5.1-mapr-1501.jar#__spark__.jar) >> 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Setting up executor with >> environment: Map(CLASSPATH -> >> {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>/opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/mapr/hive/hive-current/lib/*, >> SPARK_LOG_URL_STDERR -> >> http://qtausc-pphd0146.hadoop.local:8042/node/containerlogs/container_1432690361766_0593_01_000002/nw/stderr?start=0, >> SPARK_DIST_CLASSPATH -> >> /opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/mapr/hive/hive-current/lib/*, >> SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1432690361766_0593, >> SPARK_YARN_CACHE_FILES_FILE_SIZES -> 124419029, SPARK_USER -> nw, >> SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE, SPARK_YARN_MODE -> true, >> SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1433290257972, SPARK_LOG_URL_STDOUT >> -> >> http://qtausc-pphd0146.hadoop.local:8042/node/containerlogs/container_1432690361766_0593_01_000002/nw/stdout?start=0, >> SPARK_YARN_CACHE_FILES -> >> maprfs:/user/nw/.sparkStaging/application_1432690361766_0593/spark-assembly-1.4.0-SNAPSHOT-hadoop2.5.1-mapr-1501.jar#__spark__.jar) >> 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Setting up executor with >> commands: List({{JAVA_HOME}}/bin/java, -server, >> -XX:OnOutOfMemoryError='kill %p', -Xms1024m, -Xmx1024m, >> -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.driver.port=59954', >> -Dspark.yarn.app.container.log.dir=<LOG_DIR>, >> org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, >> akka.tcp://sparkDriver@172.31.10.14:59954/user/CoarseGrainedScheduler, >> --executor-id, 2, --hostname, qtausc-pphd0155.hadoop.local, --cores, 1, >> --app-id, application_1432690361766_0593, --user-class-path, >> file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr) >> 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Setting up executor with >> commands: List({{JAVA_HOME}}/bin/java, -server, >> -XX:OnOutOfMemoryError='kill %p', -Xms1024m, -Xmx1024m, >> -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.driver.port=59954', >> -Dspark.yarn.app.container.log.dir=<LOG_DIR>, >> org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, >> akka.tcp://sparkDriver@172.31.10.14:59954/user/CoarseGrainedScheduler, >> --executor-id, 1, --hostname, qtausc-pphd0146.hadoop.local, --cores, 1, >> --app-id, application_1432690361766_0593, --user-class-path, >> file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr) >> 15/06/03 10:11:04 INFO impl.ContainerManagementProtocolProxy: Opening >> proxy : qtausc-pphd0155.hadoop.local:45589 >> 15/06/03 10:11:04 INFO impl.ContainerManagementProtocolProxy: Opening >> proxy : qtausc-pphd0146.hadoop.local:55935 >> >> >> Any ideas what the problem is? >> >> Cheers, >> ~NW >> >> > > > -- > Marcelo >
