Hi Judy, Are you somehow modifying Spark's classpath to include jars from Hadoop and Hive that you have running on the machine? The issue seems to be that you are somehow including a version of Hadoop that references the original guava package. The Hadoop that is bundled in the Spark jars should not do this.
- Patrick On Wed, Nov 26, 2014 at 1:45 AM, Judy Nash <judyn...@exchange.microsoft.com> wrote: > Looks like a config issue. I ran spark-pi job and still failing with the > same guava error > > Command ran: > > .\bin\spark-class.cmd org.apache.spark.deploy.SparkSubmit --class > org.apache.spark.examples.SparkPi --master spark://headnodehost:7077 > --executor-memory 1G --num-executors 1 > .\lib\spark-examples-1.2.1-SNAPSHOT-hadoop2.4.0.jar 100 > > > > Had used the same build steps on spark 1.1 and had no issue. > > > > From: Denny Lee [mailto:denny.g....@gmail.com] > Sent: Tuesday, November 25, 2014 5:47 PM > To: Judy Nash; Cheng Lian; u...@spark.incubator.apache.org > > > Subject: Re: latest Spark 1.2 thrift server fail with NoClassDefFoundError > on Guava > > > > To determine if this is a Windows vs. other configuration, can you just try > to call the Spark-class.cmd SparkSubmit without actually referencing the > Hadoop or Thrift server classes? > > > > > > On Tue Nov 25 2014 at 5:42:09 PM Judy Nash <judyn...@exchange.microsoft.com> > wrote: > > I traced the code and used the following to call: > > Spark-class.cmd org.apache.spark.deploy.SparkSubmit --class > org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 spark-internal > --hiveconf hive.server2.thrift.port=10000 > > > > The issue ended up to be much more fundamental however. Spark doesn't work > at all in configuration below. When open spark-shell, it fails with the same > ClassNotFound error. > > Now I wonder if this is a windows-only issue or the hive/Hadoop > configuration that is having this problem. > > > > From: Cheng Lian [mailto:lian.cs....@gmail.com] > Sent: Tuesday, November 25, 2014 1:50 AM > > > To: Judy Nash; u...@spark.incubator.apache.org > Subject: Re: latest Spark 1.2 thrift server fail with NoClassDefFoundError > on Guava > > > > Oh so you're using Windows. What command are you using to start the Thrift > server then? > > On 11/25/14 4:25 PM, Judy Nash wrote: > > Made progress but still blocked. > > After recompiling the code on cmd instead of PowerShell, now I can see all 5 > classes as you mentioned. > > However I am still seeing the same error as before. Anything else I can > check for? > > > > From: Judy Nash [mailto:judyn...@exchange.microsoft.com] > Sent: Monday, November 24, 2014 11:50 PM > To: Cheng Lian; u...@spark.incubator.apache.org > Subject: RE: latest Spark 1.2 thrift server fail with NoClassDefFoundError > on Guava > > > > This is what I got from jar tf: > > org/spark-project/guava/common/base/Preconditions.class > > org/spark-project/guava/common/math/MathPreconditions.class > > com/clearspring/analytics/util/Preconditions.class > > parquet/Preconditions.class > > > > I seem to have the line that reported missing, but I am missing this file: > > com/google/inject/internal/util/$Preconditions.class > > > > Any suggestion on how to fix this? > > Very much appreciate the help as I am very new to Spark and open source > technologies. > > > > From: Cheng Lian [mailto:lian.cs....@gmail.com] > Sent: Monday, November 24, 2014 8:24 PM > To: Judy Nash; u...@spark.incubator.apache.org > Subject: Re: latest Spark 1.2 thrift server fail with NoClassDefFoundError > on Guava > > > > Hm, I tried exactly the same commit and the build command locally, but > couldn't reproduce this. > > Usually this kind of errors are caused by classpath misconfiguration. Could > you please try this to ensure corresponding Guava classes are included in > the assembly jar you built? > > jar tf > assembly/target/scala-2.10/spark-assembly-1.2.1-SNAPSHOT-hadoop2.4.0.jar | > grep Preconditions > > On my machine I got these lines (the first line is the one reported as > missing in your case): > > org/spark-project/guava/common/base/Preconditions.class > > org/spark-project/guava/common/math/MathPreconditions.class > > com/clearspring/analytics/util/Preconditions.class > > parquet/Preconditions.class > > com/google/inject/internal/util/$Preconditions.class > > On 11/25/14 6:25 AM, Judy Nash wrote: > > Thank you Cheng for responding. > > > Here is the commit SHA1 on the 1.2 branch I saw this failure in: > > commit 6f70e0295572e3037660004797040e026e440dbd > > Author: zsxwing <zsxw...@gmail.com> > > Date: Fri Nov 21 00:42:43 2014 -0800 > > > > [SPARK-4472][Shell] Print "Spark context available as sc." only when > SparkContext is created... > > > > ... successfully > > > > It's weird that printing "Spark context available as sc" when creating > SparkContext unsuccessfully. > > > > Let me know if you need anything else. > > > > From: Cheng Lian [mailto:lian.cs....@gmail.com] > Sent: Friday, November 21, 2014 8:02 PM > To: Judy Nash; u...@spark.incubator.apache.org > Subject: Re: latest Spark 1.2 thrift server fail with NoClassDefFoundError > on Guava > > > > Hi Judy, could you please provide the commit SHA1 of the version you're > using? Thanks! > > On 11/22/14 11:05 AM, Judy Nash wrote: > > Hi, > > > > Thrift server is failing to start for me on latest spark 1.2 branch. > > > > I got the error below when I start thrift server. > > Exception in thread "main" java.lang.NoClassDefFoundError: > com/google/common/bas > > e/Preconditions > > at > org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configur > > ation.java:314).... > > > > Here is my setup: > > 1) Latest spark 1.2 branch build > > 2) Used build command: > > mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -Phive-thriftserver > -DskipTests clean package > > 3) Added hive-site.xml to \conf > > 4) Version on the box: Hive 0.13, Hadoop 2.4 > > > > Is this a real bug or am I doing something wrong? > > > > ----------------------------------- > > Full Stacktrace: > > Exception in thread "main" java.lang.NoClassDefFoundError: > com/google/common/bas > > e/Preconditions > > at > org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configur > > ation.java:314) > > at > org.apache.hadoop.conf.Configuration$DeprecationDelta.<init>(Configur > > ation.java:327) > > at > org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:409) > > > > at > org.apache.spark.deploy.SparkHadoopUtil.newConfiguration(SparkHadoopU > > til.scala:82) > > at > org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala: > > 42) > > at > org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala > > :202) > > at > org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.sca > > la) > > at > org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:1784) > > at > org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:105) > > at > org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:180) > > at org.apache.spark.SparkEnv$.create(SparkEnv.scala:292) > > at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:159) > > at org.apache.spark.SparkContext.<init>(SparkContext.scala:230) > > at > org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv. > > scala:38) > > at > org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveTh > > riftServer2.scala:56) > > at > org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThr > > iftServer2.scala) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. > > java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces > > sorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:353) > > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) > > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > Caused by: java.lang.ClassNotFoundException: > com.google.common.base.Precondition > > s > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
