That code hasn't changed at all between 1.3 and 1.4; it also has been working fine for me.
Are you sure you're using exactly the same Hadoop libraries (since you're building with -Phadoop-provided) and Hadoop configuration in both cases? On Tue, Jun 2, 2015 at 5:29 PM, Night Wolf <nightwolf...@gmail.com> wrote: > Hi all, > > Trying out Spark 1.4 on MapR Hadoop 2.5.1 running in yarn-client mode. > Seems the application master doesn't work anymore, I get a 500 connect > refused, even when I hit the IP/port of the spark UI directly. The logs > don't show much. > > I build spark with Java 6, hive & scala 2.10 and 2.11. I've tried with and > without -Phadoop-provided > > *Build command;* > > ./make-distribution.sh --name mapr4.0.2_yarn_j6_2.10 --tgz -Pyarn -Pmapr4 > -Phadoop-2.4 -Pmapr4 -Phive -Phadoop-provided > -Dhadoop.version=2.5.1-mapr-1501 -Dyarn.version=2.5.1-mapr-1501 -DskipTests > -e -X > > *Logs from spark shell;* > > 15/06/03 00:10:56 INFO server.AbstractConnector: Started > SelectChannelConnector@0.0.0.0:4040 > 15/06/03 00:10:56 INFO util.Utils: Successfully started service 'SparkUI' > on port 4040. > 15/06/03 00:10:56 INFO ui.SparkUI: Started SparkUI at > http://172.31.10.14:4040 > 15/06/03 00:10:57 INFO yarn.Client: Requesting a new application from > cluster with 71 NodeManagers > 15/06/03 00:10:57 INFO yarn.Client: Verifying our application has not > requested more than the maximum memory capability of the cluster (112640 MB > per container) > 15/06/03 00:10:57 INFO yarn.Client: Will allocate AM container, with 896 > MB memory including 384 MB overhead > 15/06/03 00:10:57 INFO yarn.Client: Setting up container launch context > for our AM > 15/06/03 00:10:57 INFO yarn.Client: Preparing resources for our AM > container > 15/06/03 00:10:57 INFO yarn.Client: Uploading resource > file:///apps/spark/spark-1.4.0-SNAPSHOT-bin-mapr4.0.2_yarn_j6_2.11/lib/spark-assembly-1.4.0-SNAPSHOT-hadoop2.5.1-mapr-1501.jar > -> > maprfs:/user/nw/.sparkStaging/application_1432690361766_0593/spark-assembly-1.4.0-SNAPSHOT-hadoop2.5.1-mapr-1501.jar > 15/06/03 00:10:58 INFO yarn.Client: Uploading resource > file:/tmp/spark-5e42f904-ff83-4c93-bd35-4c3e20226a8a/__hadoop_conf__983379693214711.zip > -> > maprfs:/user/nw/.sparkStaging/application_1432690361766_0593/__hadoop_conf__983379693214711.zip > 15/06/03 00:10:58 INFO yarn.Client: Setting up the launch environment for > our AM container > 15/06/03 00:10:58 INFO spark.SecurityManager: Changing view acls to: nw > 15/06/03 00:10:58 INFO spark.SecurityManager: Changing modify acls to: nw > 15/06/03 00:10:58 INFO spark.SecurityManager: SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(nw); users with modify permissions: Set(nw) > 15/06/03 00:10:58 INFO yarn.Client: Submitting application 593 to > ResourceManager > 15/06/03 00:10:58 INFO security.ExternalTokenManagerFactory: Initialized > external token manager class - > com.mapr.hadoop.yarn.security.MapRTicketManager > 15/06/03 00:10:58 INFO impl.YarnClientImpl: Submitted application > application_1432690361766_0593 > 15/06/03 00:10:59 INFO yarn.Client: Application report for > application_1432690361766_0593 (state: ACCEPTED) > 15/06/03 00:10:59 INFO yarn.Client: > client token: N/A > diagnostics: N/A > ApplicationMaster host: N/A > ApplicationMaster RPC port: -1 > queue: default > start time: 1433290258143 > final status: UNDEFINED > tracking URL: > http://qtausc-pphd0101.hadoop.local:8088/proxy/application_1432690361766_0593/ > user: nw > 15/06/03 00:11:00 INFO yarn.Client: Application report for > application_1432690361766_0593 (state: ACCEPTED) > 15/06/03 00:11:01 INFO yarn.Client: Application report for > application_1432690361766_0593 (state: ACCEPTED) > 15/06/03 00:11:02 INFO yarn.Client: Application report for > application_1432690361766_0593 (state: ACCEPTED) > 15/06/03 00:11:03 INFO yarn.Client: Application report for > application_1432690361766_0593 (state: ACCEPTED) > 15/06/03 00:11:03 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: > ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka.tcp:// > sparkYarnAM@192.168.81.167:36542/user/YarnAM#1631897818]) > 15/06/03 00:11:03 INFO cluster.YarnClientSchedulerBackend: Add WebUI > Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, > Map(PROXY_HOSTS -> qtausc-pphd0167.hadoop.local, PROXY_URI_BASES -> > http://qtausc-pphd0167.hadoop.local:8088/proxy/application_1432690361766_0593), > /proxy/application_1432690361766_0593 > 15/06/03 00:11:03 INFO ui.JettyUtils: Adding filter: > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter > 15/06/03 00:11:04 INFO yarn.Client: Application report for > application_1432690361766_0593 (state: RUNNING) > 15/06/03 00:11:04 INFO yarn.Client: > client token: N/A > diagnostics: N/A > ApplicationMaster host: 192.168.81.167 > ApplicationMaster RPC port: 0 > queue: default > start time: 1433290258143 > final status: UNDEFINED > tracking URL: > http://qtausc-pphd0101.hadoop.local:8088/proxy/application_1432690361766_0593/ > user: nw > 15/06/03 00:11:04 INFO cluster.YarnClientSchedulerBackend: Application > application_1432690361766_0593 has started running. > 15/06/03 00:11:04 INFO util.Utils: Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45668. > 15/06/03 00:11:04 INFO netty.NettyBlockTransferService: Server created on > 45668 > 15/06/03 00:11:04 INFO storage.BlockManagerMaster: Trying to register > BlockManager > 15/06/03 00:11:04 INFO storage.BlockManagerMasterEndpoint: Registering > block manager 172.31.10.14:45668 with 265.4 MB RAM, > BlockManagerId(driver, 172.31.10.14, 45668) > 15/06/03 00:11:04 INFO storage.BlockManagerMaster: Registered BlockManager > > > *Logs from AM logs page in YARN;* > 15/06/03 10:11:01 INFO yarn.ApplicationMaster: Registered signal handlers > for [TERM, HUP, INT] > 15/06/03 10:11:02 INFO yarn.ApplicationMaster: ApplicationAttemptId: > appattempt_1432690361766_0593_000001 > 15/06/03 10:11:02 INFO spark.SecurityManager: Changing view acls to: nw > 15/06/03 10:11:02 INFO spark.SecurityManager: Changing modify acls to: nw > 15/06/03 10:11:02 INFO spark.SecurityManager: SecurityManager: > authentication disabled; ui acls disabled; users with view permissions: > Set(nw); users with modify permissions: Set(nw) > 15/06/03 10:11:03 INFO slf4j.Slf4jLogger: Slf4jLogger started > 15/06/03 10:11:03 INFO Remoting: Starting remoting > 15/06/03 10:11:03 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://sparkYarnAM@192.168.81.167:36542] > 15/06/03 10:11:03 INFO util.Utils: Successfully started service > 'sparkYarnAM' on port 36542. > 15/06/03 10:11:03 INFO yarn.ApplicationMaster: Waiting for Spark driver to > be reachable. > 15/06/03 10:11:03 INFO yarn.ApplicationMaster: Driver now available: > 172.31.10.14:59954 > 15/06/03 10:11:03 INFO yarn.ApplicationMaster$AMEndpoint: Add WebUI > Filter. > AddWebUIFilter(org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,Map(PROXY_HOSTS > -> qtausc-pphd0167.hadoop.local, PROXY_URI_BASES -> > http://qtausc-pphd0167.hadoop.local:8088/proxy/application_1432690361766_0593 > ),/proxy/application_1432690361766_0593) > 15/06/03 10:11:03 INFO yarn.YarnRMClient: Registering the ApplicationMaster > 15/06/03 10:11:04 INFO yarn.YarnAllocator: Will request 2 executor > containers, each with 1 cores and 1408 MB memory including 384 MB overhead > 15/06/03 10:11:04 INFO yarn.YarnAllocator: Container request (host: Any, > capability: <memory:1408, vCores:1, disks:0.0>) > 15/06/03 10:11:04 INFO yarn.YarnAllocator: Container request (host: Any, > capability: <memory:1408, vCores:1, disks:0.0>) > 15/06/03 10:11:04 INFO yarn.ApplicationMaster: Started progress reporter > thread with (heartbeat : 3000, initial allocation : 200) intervals > 15/06/03 10:11:04 INFO impl.AMRMClientImpl: Received new token for : > qtausc-pphd0146.hadoop.local:55935 > 15/06/03 10:11:04 INFO impl.AMRMClientImpl: Received new token for : > qtausc-pphd0155.hadoop.local:45589 > 15/06/03 10:11:04 INFO yarn.YarnAllocator: Launching container > container_1432690361766_0593_01_000002 for on host > qtausc-pphd0146.hadoop.local > 15/06/03 10:11:04 INFO yarn.YarnAllocator: Launching ExecutorRunnable. > driverUrl: akka.tcp:// > sparkDriver@172.31.10.14:59954/user/CoarseGrainedScheduler, > executorHostname: qtausc-pphd0146.hadoop.local > 15/06/03 10:11:04 INFO yarn.YarnAllocator: Launching container > container_1432690361766_0593_01_000003 for on host > qtausc-pphd0155.hadoop.local > 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Starting Executor Container > 15/06/03 10:11:04 INFO yarn.YarnAllocator: Launching ExecutorRunnable. > driverUrl: akka.tcp:// > sparkDriver@172.31.10.14:59954/user/CoarseGrainedScheduler, > executorHostname: qtausc-pphd0155.hadoop.local > 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Starting Executor Container > 15/06/03 10:11:04 INFO yarn.YarnAllocator: Received 2 containers from > YARN, launching executors on 2 of them. > 15/06/03 10:11:04 INFO impl.ContainerManagementProtocolProxy: > yarn.client.max-nodemanagers-proxies : 500 > 15/06/03 10:11:04 INFO impl.ContainerManagementProtocolProxy: > yarn.client.max-nodemanagers-proxies : 500 > 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Setting up > ContainerLaunchContext > 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Setting up > ContainerLaunchContext > 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Preparing Local resources > 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Preparing Local resources > 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Prepared Local resources > Map(__spark__.jar -> resource { scheme: "maprfs" port: -1 file: > "/user/nw/.sparkStaging/application_1432690361766_0593/spark-assembly-1.4.0-SNAPSHOT-hadoop2.5.1-mapr-1501.jar" > } size: 124419029 timestamp: 1433290257972 type: FILE visibility: PRIVATE) > 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Prepared Local resources > Map(__spark__.jar -> resource { scheme: "maprfs" port: -1 file: > "/user/nw/.sparkStaging/application_1432690361766_0593/spark-assembly-1.4.0-SNAPSHOT-hadoop2.5.1-mapr-1501.jar" > } size: 124419029 timestamp: 1433290257972 type: FILE visibility: PRIVATE) > 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Setting up executor with > environment: Map(CLASSPATH -> > {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>/opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/mapr/hive/hive-current/lib/*, > SPARK_LOG_URL_STDERR -> > http://qtausc-pphd0155.hadoop.local:8042/node/containerlogs/container_1432690361766_0593_01_000003/nw/stderr?start=0, > SPARK_DIST_CLASSPATH -> > /opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/mapr/hive/hive-current/lib/*, > SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1432690361766_0593, > SPARK_YARN_CACHE_FILES_FILE_SIZES -> 124419029, SPARK_USER -> nw, > SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE, SPARK_YARN_MODE -> true, > SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1433290257972, SPARK_LOG_URL_STDOUT > -> > http://qtausc-pphd0155.hadoop.local:8042/node/containerlogs/container_1432690361766_0593_01_000003/nw/stdout?start=0, > SPARK_YARN_CACHE_FILES -> > maprfs:/user/nw/.sparkStaging/application_1432690361766_0593/spark-assembly-1.4.0-SNAPSHOT-hadoop2.5.1-mapr-1501.jar#__spark__.jar) > 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Setting up executor with > environment: Map(CLASSPATH -> > {{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>/opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/mapr/hive/hive-current/lib/*, > SPARK_LOG_URL_STDERR -> > http://qtausc-pphd0146.hadoop.local:8042/node/containerlogs/container_1432690361766_0593_01_000002/nw/stderr?start=0, > SPARK_DIST_CLASSPATH -> > /opt/mapr/lib/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/yarn/*:/opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/common/lib/*:/opt/mapr/hive/hive-current/lib/*, > SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1432690361766_0593, > SPARK_YARN_CACHE_FILES_FILE_SIZES -> 124419029, SPARK_USER -> nw, > SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE, SPARK_YARN_MODE -> true, > SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1433290257972, SPARK_LOG_URL_STDOUT > -> > http://qtausc-pphd0146.hadoop.local:8042/node/containerlogs/container_1432690361766_0593_01_000002/nw/stdout?start=0, > SPARK_YARN_CACHE_FILES -> > maprfs:/user/nw/.sparkStaging/application_1432690361766_0593/spark-assembly-1.4.0-SNAPSHOT-hadoop2.5.1-mapr-1501.jar#__spark__.jar) > 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Setting up executor with > commands: List({{JAVA_HOME}}/bin/java, -server, > -XX:OnOutOfMemoryError='kill %p', -Xms1024m, -Xmx1024m, > -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.driver.port=59954', > -Dspark.yarn.app.container.log.dir=<LOG_DIR>, > org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, > akka.tcp://sparkDriver@172.31.10.14:59954/user/CoarseGrainedScheduler, > --executor-id, 2, --hostname, qtausc-pphd0155.hadoop.local, --cores, 1, > --app-id, application_1432690361766_0593, --user-class-path, > file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr) > 15/06/03 10:11:04 INFO yarn.ExecutorRunnable: Setting up executor with > commands: List({{JAVA_HOME}}/bin/java, -server, > -XX:OnOutOfMemoryError='kill %p', -Xms1024m, -Xmx1024m, > -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.driver.port=59954', > -Dspark.yarn.app.container.log.dir=<LOG_DIR>, > org.apache.spark.executor.CoarseGrainedExecutorBackend, --driver-url, > akka.tcp://sparkDriver@172.31.10.14:59954/user/CoarseGrainedScheduler, > --executor-id, 1, --hostname, qtausc-pphd0146.hadoop.local, --cores, 1, > --app-id, application_1432690361766_0593, --user-class-path, > file:$PWD/__app__.jar, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr) > 15/06/03 10:11:04 INFO impl.ContainerManagementProtocolProxy: Opening > proxy : qtausc-pphd0155.hadoop.local:45589 > 15/06/03 10:11:04 INFO impl.ContainerManagementProtocolProxy: Opening > proxy : qtausc-pphd0146.hadoop.local:55935 > > > Any ideas what the problem is? > > Cheers, > ~NW > > -- Marcelo
