just set your java class path properly export JAVA_HOME=/usr/lib/jvm/java-7-..... (somewhat like this...whatever version you having)
it will work.... Regards, Arpit On Wed, Apr 16, 2014 at 1:24 AM, ge ko <koenig....@gmail.com> wrote: > Hi, > > > > after starting the shark-shell > via /opt/shark/shark-0.9.1/bin/shark-withinfo -skipRddReload I receive lots > of output, including the exception that /bin/java cannot be executed. But > it is linked to /usr/bin/java ?!?! > > > > root#>ls -al /bin/java > > lrwxrwxrwx 1 root root 13 15. Apr 21:45 /bin/java -> /usr/bin/java > > root#>/bin/java -version > > java version "1.7.0_51" > OpenJDK Runtime Environment (rhel-2.4.4.1.el6_5-x86_64 u51-b02) > OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode) > > > > Starting the shark shell: > > > > [root@hadoop-pg-5 bin]# /opt/shark/shark-0.9.1/bin/shark-withinfo > -skipRddReload > -hiveconf hive.root.logger=INFO,console -skipRddReload > Starting the Shark Command Line Client > 14/04/15 21:45:57 WARN conf.HiveConf: DEPRECATED: Configuration property > hive.metastore.local no longer has any effect. Make sure to provide a valid > value for hive.metastore.uris if you are connecting to a remote metastore. > 14/04/15 21:45:58 WARN conf.HiveConf: DEPRECATED: Configuration property > hive.metastore.local no longer has any effect. Make sure to provide a valid > value for hive.metastore.uris if you are connecting to a remote metastore. > > Logging initialized using configuration in > jar:file:/opt/shark/shark-0.9.1/lib_managed/jars/edu.berkeley.cs.shark/hive-common/hive-common-0.11.0-shark-0.9.1.jar!/hive-log4j.properties > 14/04/15 21:45:58 INFO SessionState: > Logging initialized using configuration in > jar:file:/opt/shark/shark-0.9.1/lib_managed/jars/edu.berkeley.cs.shark/hive-common/hive-common-0.11.0-shark-0.9.1.jar!/hive-log4j.properties > Hive history > file=/tmp/root/hive_job_log_root_22574@hadoop-pg-5.cluster_201404152145_159664609.txt > 14/04/15 21:45:58 INFO exec.HiveHistory: Hive history > file=/tmp/root/hive_job_log_root_22574@hadoop-pg-5.cluster_201404152145_159664609.txt > 14/04/15 21:45:58 WARN conf.HiveConf: DEPRECATED: Configuration property > hive.metastore.local no longer has any effect. Make sure to provide a valid > value for hive.metastore.uris if you are connecting to a remote metastore. > 14/04/15 21:45:59 WARN conf.HiveConf: DEPRECATED: Configuration property > hive.metastore.local no longer has any effect. Make sure to provide a valid > value for hive.metastore.uris if you are connecting to a remote metastore. > 14/04/15 21:46:00 INFO slf4j.Slf4jLogger: Slf4jLogger started > 14/04/15 21:46:00 INFO Remoting: Starting remoting > 14/04/15 21:46:00 INFO Remoting: Remoting started; listening on addresses > :[akka.tcp://spark@hadoop-pg-5.cluster:38835] > 14/04/15 21:46:00 INFO Remoting: Remoting now listens on addresses: > [akka.tcp://spark@hadoop-pg-5.cluster:38835] > 14/04/15 21:46:00 INFO spark.SparkEnv: Registering BlockManagerMaster > 5,108: [GC 262656K->26899K(1005568K), 0,0409080 secs] > 14/04/15 21:46:00 INFO storage.DiskBlockManager: Created local directory > at /tmp/spark-local-20140415214600-9537 > 14/04/15 21:46:00 INFO storage.MemoryStore: MemoryStore started with > capacity 589.2 MB. > 14/04/15 21:46:00 INFO network.ConnectionManager: Bound socket to port > 51889 with id = ConnectionManagerId(hadoop-pg-5.cluster,51889) > 14/04/15 21:46:00 INFO storage.BlockManagerMaster: Trying to register > BlockManager > 14/04/15 21:46:00 INFO storage.BlockManagerMasterActor$BlockManagerInfo: > Registering block manager hadoop-pg-5.cluster:51889 with 589.2 MB RAM > 14/04/15 21:46:00 INFO storage.BlockManagerMaster: Registered BlockManager > 14/04/15 21:46:00 INFO spark.HttpServer: Starting HTTP Server > 14/04/15 21:46:00 INFO server.Server: jetty-7.6.8.v20121106 > 14/04/15 21:46:00 INFO server.AbstractConnector: Started > SocketConnector@0.0.0.0:59414 > 14/04/15 21:46:00 INFO broadcast.HttpBroadcast: Broadcast server started > at http://10.147.210.5:59414 > 14/04/15 21:46:01 INFO spark.SparkEnv: Registering MapOutputTracker > 14/04/15 21:46:01 INFO spark.HttpFileServer: HTTP File server directory is > /tmp/spark-cf56ada9-d950-4abc-a1c3-76fecdc4faa3 > 14/04/15 21:46:01 INFO spark.HttpServer: Starting HTTP Server > 14/04/15 21:46:01 INFO server.Server: jetty-7.6.8.v20121106 > 14/04/15 21:46:01 INFO server.AbstractConnector: Started > SocketConnector@0.0.0.0:45689 > 14/04/15 21:46:01 INFO server.Server: jetty-7.6.8.v20121106 > 14/04/15 21:46:01 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/storage/rdd,null} > 14/04/15 21:46:01 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/storage,null} > 14/04/15 21:46:01 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/stages/stage,null} > 14/04/15 21:46:01 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/stages/pool,null} > 14/04/15 21:46:01 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/stages,null} > 14/04/15 21:46:01 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/environment,null} > 14/04/15 21:46:01 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/executors,null} > 14/04/15 21:46:01 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/metrics/json,null} > 14/04/15 21:46:01 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/static,null} > 14/04/15 21:46:01 INFO handler.ContextHandler: started > o.e.j.s.h.ContextHandler{/,null} > 14/04/15 21:46:01 INFO server.AbstractConnector: Started > SelectChannelConnector@0.0.0.0:4040 > 14/04/15 21:46:01 INFO ui.SparkUI: Started Spark Web UI at > http://hadoop-pg-5.cluster:4040 > 14/04/15 21:46:01 INFO client.AppClient$ClientActor: Connecting to master > spark://hadoop-pg-5.cluster:7077... > 14/04/15 21:46:01 WARN shark.SharkEnv: Hive Hadoop shims detected local > mode, but Shark is not running locally. > 14/04/15 21:46:01 WARN shark.SharkEnv: Setting mapred.job.tracker to > 'Spark_1397591161823' (was 'local') > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Connected to > Spark cluster with app ID app-20140415214602-0010 > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: > app-20140415214602-0010/0 on worker-20140414105148-hadoop-pg-8.cluster-7078 > (hadoop-pg-8.cluster:7078) with 2 cores > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted > executor ID app-20140415214602-0010/0 on hostPort hadoop-pg-8.cluster:7078 > with 2 cores, 2.0 GB RAM > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: > app-20140415214602-0010/1 on worker-20140414105202-hadoop-pg-9.cluster-7078 > (hadoop-pg-9.cluster:7078) with 2 cores > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted > executor ID app-20140415214602-0010/1 on hostPort hadoop-pg-9.cluster:7078 > with 2 cores, 2.0 GB RAM > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/1 is now RUNNING > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/0 is now RUNNING > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/1 is now FAILED (class java.io.IOException: Cannot > run program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/1"): error=2, No such file or > directory) > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor > app-20140415214602-0010/1 removed: class java.io.IOException: Cannot run > program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/1"): error=2, No such file or > directory > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: > app-20140415214602-0010/2 on worker-20140414105202-hadoop-pg-9.cluster-7078 > (hadoop-pg-9.cluster:7078) with 2 cores > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted > executor ID app-20140415214602-0010/2 on hostPort hadoop-pg-9.cluster:7078 > with 2 cores, 2.0 GB RAM > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/0 is now FAILED (class java.io.IOException: Cannot > run program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/0"): error=2, No such file or > directory) > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor > app-20140415214602-0010/0 removed: class java.io.IOException: Cannot run > program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/0"): error=2, No such file or > directory > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: > app-20140415214602-0010/3 on worker-20140414105148-hadoop-pg-8.cluster-7078 > (hadoop-pg-8.cluster:7078) with 2 cores > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted > executor ID app-20140415214602-0010/3 on hostPort hadoop-pg-8.cluster:7078 > with 2 cores, 2.0 GB RAM > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/3 is now RUNNING > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/2 is now RUNNING > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/3 is now FAILED (class java.io.IOException: Cannot > run program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/3"): error=2, No such file or > directory) > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor > app-20140415214602-0010/3 removed: class java.io.IOException: Cannot run > program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/3"): error=2, No such file or > directory > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: > app-20140415214602-0010/4 on worker-20140414105148-hadoop-pg-8.cluster-7078 > (hadoop-pg-8.cluster:7078) with 2 cores > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted > executor ID app-20140415214602-0010/4 on hostPort hadoop-pg-8.cluster:7078 > with 2 cores, 2.0 GB RAM > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/2 is now FAILED (class java.io.IOException: Cannot > run program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/2"): error=2, No such file or > directory) > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor > app-20140415214602-0010/2 removed: class java.io.IOException: Cannot run > program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/2"): error=2, No such file or > directory > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: > app-20140415214602-0010/5 on worker-20140414105202-hadoop-pg-9.cluster-7078 > (hadoop-pg-9.cluster:7078) with 2 cores > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted > executor ID app-20140415214602-0010/5 on hostPort hadoop-pg-9.cluster:7078 > with 2 cores, 2.0 GB RAM > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/4 is now RUNNING > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/5 is now RUNNING > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/4 is now FAILED (class java.io.IOException: Cannot > run program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/4"): error=2, No such file or > directory) > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor > app-20140415214602-0010/4 removed: class java.io.IOException: Cannot run > program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/4"): error=2, No such file or > directory > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: > app-20140415214602-0010/6 on worker-20140414105148-hadoop-pg-8.cluster-7078 > (hadoop-pg-8.cluster:7078) with 2 cores > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted > executor ID app-20140415214602-0010/6 on hostPort hadoop-pg-8.cluster:7078 > with 2 cores, 2.0 GB RAM > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/5 is now FAILED (class java.io.IOException: Cannot > run program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/5"): error=2, No such file or > directory) > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor > app-20140415214602-0010/5 removed: class java.io.IOException: Cannot run > program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/5"): error=2, No such file or > directory > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: > app-20140415214602-0010/7 on worker-20140414105202-hadoop-pg-9.cluster-7078 > (hadoop-pg-9.cluster:7078) with 2 cores > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted > executor ID app-20140415214602-0010/7 on hostPort hadoop-pg-9.cluster:7078 > with 2 cores, 2.0 GB RAM > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/7 is now RUNNING > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/6 is now RUNNING > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/6 is now FAILED (class java.io.IOException: Cannot > run program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/6"): error=2, No such file or > directory) > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor > app-20140415214602-0010/6 removed: class java.io.IOException: Cannot run > program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/6"): error=2, No such file or > directory > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: > app-20140415214602-0010/8 on worker-20140414105148-hadoop-pg-8.cluster-7078 > (hadoop-pg-8.cluster:7078) with 2 cores > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted > executor ID app-20140415214602-0010/8 on hostPort hadoop-pg-8.cluster:7078 > with 2 cores, 2.0 GB RAM > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/7 is now FAILED (class java.io.IOException: Cannot > run program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/7"): error=2, No such file or > directory) > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor > app-20140415214602-0010/7 removed: class java.io.IOException: Cannot run > program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/7"): error=2, No such file or > directory > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: > app-20140415214602-0010/9 on worker-20140414105202-hadoop-pg-9.cluster-7078 > (hadoop-pg-9.cluster:7078) with 2 cores > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted > executor ID app-20140415214602-0010/9 on hostPort hadoop-pg-9.cluster:7078 > with 2 cores, 2.0 GB RAM > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/8 is now RUNNING > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/9 is now RUNNING > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/8 is now FAILED (class java.io.IOException: Cannot > run program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/8"): error=2, No such file or > directory) > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor > app-20140415214602-0010/8 removed: class java.io.IOException: Cannot run > program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/8"): error=2, No such file or > directory > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor added: > app-20140415214602-0010/10 on > worker-20140414105148-hadoop-pg-8.cluster-7078 (hadoop-pg-8.cluster:7078) > with 2 cores > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Granted > executor ID app-20140415214602-0010/10 on hostPort hadoop-pg-8.cluster:7078 > with 2 cores, 2.0 GB RAM > 14/04/15 21:46:02 INFO client.AppClient$ClientActor: Executor updated: > app-20140415214602-0010/9 is now FAILED (class java.io.IOException: Cannot > run program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/9"): error=2, No such file or > directory) > 14/04/15 21:46:02 INFO cluster.SparkDeploySchedulerBackend: Executor > app-20140415214602-0010/9 removed: class java.io.IOException: Cannot run > program "/bin/java" (in directory > "/var/run/spark/work/app-20140415214602-0010/9"): error=2, No such file or > directory > 14/04/15 21:46:02 ERROR client.AppClient$ClientActor: Master removed our > application: FAILED; stopping client > 14/04/15 21:46:02 WARN cluster.SparkDeploySchedulerBackend: Disconnected > from Spark cluster! Waiting for reconnection... > > > > I do not understand that java error, any hints ?!?! > > > > br, Gerd > > > > using CM,CDH5 incl. Spark parcel >
