Looks like you either have a misconfigured HDFS service, or you're using the wrong configuration on the client.
BTW, as I said in the previous response, the message you saw initially is *not* an error. If you're just trying things out, you don't need to do anything and Spark should still work. On Mon, Mar 12, 2018 at 6:13 PM, kant kodali <kanth...@gmail.com> wrote: > Hi, > > I read that doc several times now. I am stuck with the below error message > when I run ./spark-shell --master yarn --deploy-mode client. > > I have my HADOOP_CONF_DIR set to /usr/local/hadoop-2.7.3/etc/hadoop and > SPARK_HOME set to /usr/local/spark on all 3 machines (1 node for Resource > Manager and NameNode, 2 Nodes for Node Manager and DataNodes). > > Any idea? > > > > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /user/centos/.sparkStaging/application_1520898664848_0003/__spark_libs__2434167523839846774.zip > could only be replicated to 0 nodes instead of minReplication (=1). There > are 2 datanode(s) running and no node(s) are excluded in this operation. > > > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571) > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107) > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031) > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725) > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: at > org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: at > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: at > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: at > java.security.AccessController.doPrivileged(Native Method) > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: at > javax.security.auth.Subject.doAs(Subject.java:422) > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > 18/03/13 00:19:13 INFO LineBufferedStream: stdout: at > org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) > 18/03/13 > > > Thanks! > > > On Mon, Mar 12, 2018 at 4:46 PM, Marcelo Vanzin <van...@cloudera.com> wrote: >> >> That's not an error, just a warning. The docs [1] have more info about >> the config options mentioned in that message. >> >> [1] http://spark.apache.org/docs/latest/running-on-yarn.html >> >> On Mon, Mar 12, 2018 at 4:42 PM, kant kodali <kanth...@gmail.com> wrote: >> > Hi All, >> > >> > I am trying to use YARN for the very first time. I believe I configured >> > all >> > the resource manager and name node fine. And then I run the below >> > command >> > >> > ./spark-shell --master yarn --deploy-mode client >> > >> > I get the below output and it hangs there forever (I had been waiting >> > over >> > 10 minutes) >> > >> > 18/03/12 23:36:32 WARN Client: Neither spark.yarn.jars nor >> > spark.yarn.archive is set, falling back to uploading libraries under >> > SPARK_HOME. >> > >> > Any idea? >> > >> > Thanks! >> >> >> >> -- >> Marcelo > > -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
