Can you verify that the cluster is running the correct version of Spark. 1.5.2.
On Tue, Nov 17, 2015 at 7:23 PM, swetha kasireddy <swethakasire...@gmail.com > wrote: > Sorry <scope>compile</scope> makes it work locally. But, the cluster > still seems to have issues with <scope>provided</scope>. Basically it > does not seem to process any records, no data is shown in any of the tabs > of the Streaming UI except the Streaming tab. Executors, Storage, Stages > etc show empty RDDs. > > On Tue, Nov 17, 2015 at 7:19 PM, swetha kasireddy < > swethakasire...@gmail.com> wrote: > >> Hi TD, >> >> Basically, I see two issues. With <scope>provided</scope> the job does >> not start localy. It does start in Cluster but seems no data is getting >> processed. >> >> Thanks, >> Swetha >> >> On Tue, Nov 17, 2015 at 7:04 PM, Tim Barthram <tim.barth...@iag.com.au> >> wrote: >> >>> If you are running a local context, could it be that you should use: >>> >>> >>> >>> <scope>provided</scope> >>> >>> >>> >>> ? >>> >>> >>> >>> Thanks, >>> >>> Tim >>> >>> >>> >>> *From:* swetha kasireddy [mailto:swethakasire...@gmail.com] >>> *Sent:* Wednesday, 18 November 2015 2:01 PM >>> *To:* Tathagata Das >>> *Cc:* user >>> *Subject:* Re: Streaming Job gives error after changing to version 1.5.2 >>> >>> >>> >>> This error I see locally. >>> >>> >>> >>> On Tue, Nov 17, 2015 at 5:44 PM, Tathagata Das <t...@databricks.com> >>> wrote: >>> >>> Are you running 1.5.2-compiled jar on a Spark 1.5.2 cluster? >>> >>> >>> >>> On Tue, Nov 17, 2015 at 5:34 PM, swetha <swethakasire...@gmail.com> >>> wrote: >>> >>> >>> >>> Hi, >>> >>> I see java.lang.NoClassDefFoundError after changing the Streaming job >>> version to 1.5.2. Any idea as to why this is happening? Following are my >>> dependencies and the error that I get. >>> >>> <dependency> >>> <groupId>org.apache.spark</groupId> >>> <artifactId>spark-core_2.10</artifactId> >>> <version>${sparkVersion}</version> >>> <scope>provided</scope> >>> </dependency> >>> >>> >>> <dependency> >>> <groupId>org.apache.spark</groupId> >>> <artifactId>spark-streaming_2.10</artifactId> >>> <version>${sparkVersion}</version> >>> <scope>provided</scope> >>> </dependency> >>> >>> >>> <dependency> >>> <groupId>org.apache.spark</groupId> >>> <artifactId>spark-sql_2.10</artifactId> >>> <version>${sparkVersion}</version> >>> <scope>provided</scope> >>> </dependency> >>> >>> >>> <dependency> >>> <groupId>org.apache.spark</groupId> >>> <artifactId>spark-hive_2.10</artifactId> >>> <version>${sparkVersion}</version> >>> <scope>provided</scope> >>> </dependency> >>> >>> >>> >>> <dependency> >>> <groupId>org.apache.spark</groupId> >>> <artifactId>spark-streaming-kafka_2.10</artifactId> >>> <version>${sparkVersion}</version> >>> </dependency> >>> >>> >>> Exception in thread "main" java.lang.NoClassDefFoundError: >>> org/apache/spark/streaming/StreamingContext >>> at java.lang.Class.getDeclaredMethods0(Native Method) >>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2693) >>> at java.lang.Class.privateGetMethodRecursive(Class.java:3040) >>> at java.lang.Class.getMethod0(Class.java:3010) >>> at java.lang.Class.getMethod(Class.java:1776) >>> at >>> com.intellij.rt.execution.application.AppMain.main(AppMain.java:125) >>> Caused by: java.lang.ClassNotFoundException: >>> org.apache.spark.streaming.StreamingContext >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:372) >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:360) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Streaming-Job-gives-error-after-changing-to-version-1-5-2-tp25406.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >>> >>> >>> >>> _____________________________________________________________________ >>> >>> The information transmitted in this message and its attachments (if any) >>> is intended >>> only for the person or entity to which it is addressed. >>> The message may contain confidential and/or privileged material. Any >>> review, >>> retransmission, dissemination or other use of, or taking of any action >>> in reliance >>> upon this information, by persons or entities other than the intended >>> recipient is >>> prohibited. >>> >>> If you have received this in error, please contact the sender and delete >>> this e-mail >>> and associated material from any computer. >>> >>> The intended recipient of this e-mail may only use, reproduce, disclose >>> or distribute >>> the information contained in this e-mail and any attached files, with >>> the permission >>> of the sender. >>> >>> This message has been scanned for viruses. >>> _____________________________________________________________________ >>> >> >> >
