Getting to proto buff classes in Spark Context

2015-02-23 Thread necro351 .
Hello, I am trying to deserialize some data encoded using proto buff from within Spark and am getting class-not-found exceptions. I have narrowed the program down to something very simple that shows the problem exactly (see 'The Program' below) and hopefully someone can tell me the easy fix :) So

Re: Getting to proto buff classes in Spark Context

2015-02-23 Thread necro351 .
Ted Yu wrote: > bq. Caused by: java.lang.ClassNotFoundException: com.rick.reports.Reports$ > SensorReports > > Is Reports$SensorReports class in rick-processors-assembly-1.0.jar ? > > Thanks > > On Mon, Feb 23, 2015 at 8:43 PM, necro351 . wrote: > >> Hello, >

Re: Getting to proto buff classes in Spark Context

2015-02-23 Thread necro351 .
> The classname given in stack trace was com.rick.reports.Reports > > In the output from jar command the class is com.defend7.reports.Reports. > > FYI > > On Mon, Feb 23, 2015 at 9:33 PM, necro351 . wrote: > >> Hi Ted, >> >> Yes it appears to be: >> r

Re: Getting to proto buff classes in Spark Context

2015-02-25 Thread necro351 .
userClassPathFirst? > > On Tue, Feb 24, 2015 at 4:43 AM, necro351 . wrote: > > Hello, > > > > I am trying to deserialize some data encoded using proto buff from within > > Spark and am getting class-not-found exceptions. I have narrowed the > program > > down to so

Getting to proto buff classes in Spark Context

2015-02-26 Thread necro351 .
Hello everyone, We are trying to decode a message inside a Spark job that we receive from Kafka. The message is encoded using Proto Buff. The problem is when decoding we get class-not-found exceptions. We have tried remedies we found online in Stack Exchange and mail list archives but nothing seem

graceful shutdown not so graceful?

2015-03-17 Thread necro351 .
Hi all, I am trying to do a graceful shutdown of my spark streaming job and it appears that everything shuts down gracefully but the checkpointing thread, which continues to run until it crashes. I looked at the checkpoint thread in 1.3.0 ( https://github.com/apache/spark/blob/master/streaming/sr