Re: registering Array of CompactBuffer to Kryo

2014-10-02 Thread Andras Barjak
; Class.forName("[Lorg.apache.spark.util.collection.CompactBuffer;") > > On Tue, Sep 30, 2014 at 5:33 PM, Andras Barjak > wrote: > > Hi, > > > > what is the correct scala code to register an Array of this private spark > > class to Kryo? > &

registering Array of CompactBuffer to Kryo

2014-09-30 Thread Andras Barjak
Hi, what is the correct scala code to register an Array of this private spark class to Kryo? "java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.util.collection.CompactBuffer[] Note: To register this class use: kryo.register(org.apache.spark.util.collection.CompactBuffe

EC2 instances missing SSD drives randomly?

2014-08-19 Thread Andras Barjak
Hi, Using the spark 1.0.1 ec2 script I launched 35 m3.2xlarge instances. (I was using Singapore region.) Some of the instances we got without the ephemeral internal (non-EBS) SSD devices that are supposed to be connected to them. Some of them have these drives but not all, and there is no sign fro

mounting SSD devices of EC2 r3.8xlarge instances

2014-06-03 Thread Andras Barjak
Hi, I have noticed that upon launching a cluster consisting of r3.8xlarge high-memory instances the standard /mnt /mnt2 /mnt3 /mnt4 temporary directories get created and set up for temp usage, however they will point to the root 8Gb filesystem. The 2x320GB SSD-s are not mounted and also they are no

Re: Spark and Hadoop

2014-05-20 Thread Andras Barjak
You can download any of them, I would go with the latest versions, or just download the source and build it yourself. For experimenting with basic things you can just launch the REPL and start right away in spark local mode not using any hadoop stuff. 2014-05-20 19:43 GMT+02:00 pcutil : > I'm a

Re: Spark runs applications in an inconsistent way

2014-04-23 Thread Andras Barjak
> >- Spark UI shows number of succeeded tasks is more than total number >of tasks, eg: 3500/3000. There are no failed tasks. At this stage the >computation keeps carrying on for a long time without returning an answer. > > No sign of resubmitted tasks in the command line logs either? Yo