Hi,
Can not reproduce your error on Spark 1.2.1 . It is not enough information.
What is your command line arguments wцру you starting spark-shell? what data
are you reading? etc.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-shell-and-StackOverFlowEr
I used the notation on JIRA where bq means quote.
FYI
On Mon, Aug 31, 2015 at 12:34 PM, Ashish Shrowty
wrote:
> Yes .. I am closing the stream.
>
> Not sure what you meant by "bq. and then create rdd"?
>
> -Ashish
>
> On Mon, Aug 31, 2015 at 1:02 PM Ted Yu wrote:
>
>> I am not familiar with yo
Yes .. I am closing the stream.
Not sure what you meant by "bq. and then create rdd"?
-Ashish
On Mon, Aug 31, 2015 at 1:02 PM Ted Yu wrote:
> I am not familiar with your code.
>
> bq. and then create the rdd
>
> I assume you call ObjectOutputStream.close() prior to the above step.
>
> Cheers
>
I am not familiar with your code.
bq. and then create the rdd
I assume you call ObjectOutputStream.close() prior to the above step.
Cheers
On Mon, Aug 31, 2015 at 9:42 AM, Ashish Shrowty
wrote:
> Sure .. here it is (scroll below to see the NotSerializableException).
> Note that upstream, I do
Sure .. here it is (scroll below to see the NotSerializableException). Note
that upstream, I do load up the (user,item,ratings) data from a file using
ObjectInputStream, do some calculations that I put in a map and then create
the rdd used in the code above from that map. I even tried checkpointing
Ashish:
Can you post the complete stack trace for NotSerializableException ?
Cheers
On Mon, Aug 31, 2015 at 8:49 AM, Ashish Shrowty
wrote:
> bcItemsIdx is just a broadcast variable constructed out of Array[(String)]
> .. it holds the item ids and I use it for indexing the MatrixEntry objects
>
bcItemsIdx is just a broadcast variable constructed out of Array[(String)]
.. it holds the item ids and I use it for indexing the MatrixEntry objects
On Mon, Aug 31, 2015 at 10:41 AM Sean Owen wrote:
> It's not clear; that error is different still and somehow suggests
> you're serializing a str
It's not clear; that error is different still and somehow suggests
you're serializing a stream somewhere. I'd look at what's inside
bcItemsIdx as that is not shown here.
On Mon, Aug 31, 2015 at 3:34 PM, Ashish Shrowty
wrote:
> Sean,
>
> Thanks for your comments. What I was really trying to do was
Yeah I see that now. I think it fails immediately because the map
operation does try to clean and/or verify the serialization of the
closure upfront.
I'm not quite sure what is going on, but I think it's some strange
interaction between how you're building up the list and what the
resulting repres
I'm not sure how to reproduce it? this code does not produce an error in master.
On Sun, Aug 30, 2015 at 7:26 PM, Ashish Shrowty
wrote:
> Do you think I should create a JIRA?
>
>
> On Sun, Aug 30, 2015 at 12:56 PM Ted Yu wrote:
>>
>> I got StackOverFlowError as well :-(
>>
>> On Sun, Aug 30, 201
Do you think I should create a JIRA?
On Sun, Aug 30, 2015 at 12:56 PM Ted Yu wrote:
> I got StackOverFlowError as well :-(
>
> On Sun, Aug 30, 2015 at 9:47 AM, Ashish Shrowty
> wrote:
>
>> Yep .. I tried that too earlier. Doesn't make a difference. Are you able
>> to replicate on your side?
>>
Yep .. I tried that too earlier. Doesn't make a difference. Are you able to
replicate on your side?
On Sun, Aug 30, 2015 at 12:08 PM Ted Yu wrote:
> I see.
>
> What about using the following in place of variable a ?
>
> http://spark.apache.org/docs/latest/programming-guide.html#broadcast-variab
I see.
What about using the following in place of variable a ?
http://spark.apache.org/docs/latest/programming-guide.html#broadcast-variables
Cheers
On Sun, Aug 30, 2015 at 8:54 AM, Ashish Shrowty
wrote:
> @Sean - Agree that there is no action, but I still get the
> stackoverflowerror, its ver
Using Spark shell :
scala> import scala.collection.mutable.MutableList
import scala.collection.mutable.MutableList
scala> val lst = MutableList[(String,String,Double)]()
lst: scala.collection.mutable.MutableList[(String, String, Double)] =
MutableList()
scala> Range(0,1).foreach(i=>lst+=(("1
That can't cause any error, since there is no action in your first
snippet. Even calling count on the result doesn't cause an error. You
must be executing something different.
On Sun, Aug 30, 2015 at 4:21 AM, ashrowty wrote:
> I am running the Spark shell (1.2.1) in local mode and I have a simple
15 matches
Mail list logo
