I set the host and port of the driver and now the error slightly changed Using Spark's default log4j profile: > org/apache/spark/log4j-defaults.properties > 14/11/04 21:13:48 INFO CoarseGrainedExecutorBackend: Registered signal > handlers for [TERM, HUP, INT] > 14/11/04 21:13:48 INFO SecurityManager: Changing view acls to: > myuser,Myuser > 14/11/04 21:13:48 INFO SecurityManager: Changing modify acls to: > myuser,Myuser > 14/11/04 21:13:48 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(myuser, > Myuser); users with modify permissions: Set(myuser, Myuser) > 14/11/04 21:13:48 INFO Slf4jLogger: Slf4jLogger started > 14/11/04 21:13:48 INFO Remoting: Starting remoting > 14/11/04 21:13:49 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://driverPropsFetcher@myserver:37456] > 14/11/04 21:13:49 INFO Remoting: Remoting now listens on addresses: > [akka.tcp://driverPropsFetcher@myserver:37456] > 14/11/04 21:13:49 INFO Utils: Successfully started service > 'driverPropsFetcher' on port 37456. > 14/11/04 21:14:19 ERROR UserGroupInformation: PriviledgedActionException > as:Myuser cause:java.util.concurrent.TimeoutException: Futures timed out > after [30 seconds] > Exception in thread "main" java.lang.reflect.UndeclaredThrowableException: > Unknown exception in doAs > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1134) > at > org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:52) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:113) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:156) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala) > Caused by: java.security.PrivilegedActionException: > java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) > ... 4 more > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [30 seconds] > at > scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at > scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at > org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:125) > at > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:53) > at > org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:52) > ... 7 more >
Any ideas? Thanks. On Tue, Nov 4, 2014 at 11:29 AM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > If you want to run the spark application from a remote machine, then you > have to at least set the following configurations properly. > > *spark.driver.host* - points to the ip/host from where you are submitting > the job (make sure you are able to ping this from the cluster) > > *spark.driver.port* - set it to a port number which is accessible from > the spark cluster. > > You can look at more configuration options over here. > <http://spark.apache.org/docs/latest/configuration.html#networking> > > Thanks > Best Regards > > On Tue, Nov 4, 2014 at 6:07 AM, Saiph Kappa <saiph.ka...@gmail.com> wrote: > >> Hi, >> >> I am trying to submit a job to a spark cluster running on a single >> machine (1 master + 1 worker) with hadoop 1.0.4. I submit it in the code: >> «val sparkConf = new >> SparkConf().setMaster("spark://myserver:7077").setAppName("MyApp").setJars(Array("target/my-app-1.0-SNAPSHOT.jar"))». >> >> When I run this application on the same machine as the cluster everything >> works fine. >> >> But when I run it from a remote machine I get the following error: >> >> Using Spark's default log4j profile: >>> org/apache/spark/log4j-defaults.properties >>> 14/11/04 00:15:38 INFO CoarseGrainedExecutorBackend: Registered signal >>> handlers for [TERM, HUP, INT] >>> 14/11/04 00:15:38 INFO SecurityManager: Changing view acls to: >>> myuser,Myuser >>> 14/11/04 00:15:38 INFO SecurityManager: Changing modify acls to: >>> myuser,Myuser >>> 14/11/04 00:15:38 INFO SecurityManager: SecurityManager: authentication >>> disabled; ui acls disabled; users with view permissions: Set(myuser, >>> Myuser); users with modify permissions: Set(myuser, Myuser) >>> 14/11/04 00:15:38 INFO Slf4jLogger: Slf4jLogger started >>> 14/11/04 00:15:38 INFO Remoting: Starting remoting >>> 14/11/04 00:15:38 INFO Remoting: Remoting started; listening on >>> addresses :[akka.tcp://driverPropsFetcher@myserver:49190] >>> 14/11/04 00:15:38 INFO Remoting: Remoting now listens on addresses: >>> [akka.tcp://driverPropsFetcher@myserver:49190] >>> 14/11/04 00:15:38 INFO Utils: Successfully started service >>> 'driverPropsFetcher' on port 49190. >>> 14/11/04 00:15:38 WARN Remoting: Tried to associate with unreachable >>> remote address [akka.tcp://sparkDriver@mylaptop:57418]. Address is now >>> gated for 60000 ms, all messages to this address will be delivered to dead >>> letters. >>> 14/11/04 00:16:08 ERROR UserGroupInformation: PriviledgedActionException >>> as:Myuser cause:java.util.concurrent.TimeoutException: Futures timed out >>> after [30 seconds] >>> Exception in thread "main" >>> java.lang.reflect.UndeclaredThrowableException: Unknown exception in doAs >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1134) >>> at >>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:52) >>> at >>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:113) >>> at >>> org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:156) >>> at >>> org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala) >>> Caused by: java.security.PrivilegedActionException: >>> java.util.concurrent.TimeoutException: Futures timed out after [30 seconds] >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:415) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) >>> ... 4 more >>> Caused by: java.util.concurrent.TimeoutException: Futures timed out >>> after [30 seconds] >>> at >>> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) >>> at >>> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) >>> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) >>> at >>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) >>> at scala.concurrent.Await$.result(package.scala:107) >>> at >>> org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:125) >>> at >>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:53) >>> at >>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:52) >>> ... 7 more >>> >> >> I know this has something to do with hadoop permissions. I have checked >> and all necessary hadoop ports in the server are open and accessible from >> outside. >> >> How can I configure the right permissions? >> >> Thanks. >> > >
