Re: Launching with m3.2xlarge instances: /mnt and /mnt2 mounted on 7gb drive

I pulled the latest last night. I'm on commit 4da01e3. On Sun, Jul 20, 2014 at 2:08 PM, Matei Zaharia wrote: > Is this with the 1.0.0 scripts? I believe it's fixed in 1.0.1. > > Matei > > On Jul 20, 2014, at 1:22 AM, Chris DuBois wrote: > > > Using th

Launching with m3.2xlarge instances: /mnt and /mnt2 mounted on 7gb drive

Using the spark-ec2 script with m3.2xlarge instances seems to not have /mnt and /mnt2 pointing to the 80gb SSDs that come with that instance. Does anybody know whether extra steps are required when using this instance type? Thanks, Chris

Re: Errors accessing hdfs while in local mode

Hi Andrew, Thanks for the response. I believe I have HDFS set up correctly: all my slaves can access it fine, and I can access the files I'm storing there with ~/persistent-hdfs/bin/hadoop -ls, etc. However, when I do spark-submit --master local, I still get the following error: 14/07/18 16:20:58

Re: Error: No space left on device

ith m3.2xlarge instances and >> using the default settings (don't set anything in SparkConf). Submit >> the application with --driver-memory 20g. >> >> The running times are slower than what I remember, but it depends on >> the instance type. >> >> Best, &

Re: Error: No space left on device

re-launching a cluster with m3.2xlarge instances and > using the default settings (don't set anything in SparkConf). Submit > the application with --driver-memory 20g. > > The running times are slower than what I remember, but it depends on > the instance type. > > Best, >

Re: Error: No space left on device

we use /mnt/spark and /mnt2/spark as the > default local directory because they are local hard drives. Did your > last run of ALS on MovieLens 10M-100K with the default settings > succeed? -Xiangrui > > On Wed, Jul 16, 2014 at 8:00 AM, Chris DuBois > wrote: > > Hi Xiangrui, >

Re: Trouble with spark-ec2 script: --ebs-vol-size

Hi Ben, It worked for me, but only when using the default region. Using --region=us-west-2 resulted in errors about security groups. Chris On Wed, Jul 16, 2014 at 8:53 AM, Ben Horner wrote: > please add > > From: "Ben Horner [via Apache Spark User List]" <[hidden email] >

Errors accessing hdfs while in local mode

Hi all, When I try setMaster("local"), I get FileNotFound exceptions; without using setMaster my application is able to properly find my datasets at hdfs://[masterip]/data/blah.csv. Is there some other setting that I need to change in order to try running in local mode? I am running from the ec2

Re: Error: No space left on device

you recommend? Chris On Wed, Jul 16, 2014 at 1:09 AM, Chris DuBois wrote: > Hi Xiangrui, > > Here is the result on the master node: > $ df -i > FilesystemInodes IUsed IFree IUse% Mounted on > /dev/xvda1524288 273997 250291 53% / > tmpfs

Re: Error: No space left on device

ult, I > would recommend leaving this setting as the default value. > > Best, > Xiangrui > > On Wed, Jul 16, 2014 at 12:02 AM, Chris DuBois > wrote: > > Thanks for the quick responses! > > > > I used your final -Dspark.local.dir suggestion, but I see this duri

Re: Error: No space left on device

bly build may create many > > small files. -Xiangrui > > > > On Tue, Jul 15, 2014 at 11:35 PM, Chris DuBois > wrote: > >> Hi all, > >> > >> I am encountering the following error: > >> > >> INFO scheduler.TaskSetManager: Loss was d

Re: Error: No space left on device

The assembly build may create many > small files. -Xiangrui > > On Tue, Jul 15, 2014 at 11:35 PM, Chris DuBois > wrote: > > Hi all, > > > > I am encountering the following error: > > > > INFO scheduler.TaskSetManager: Loss was due to java.io.IOExceptio

Error: No space left on device

Hi all, I am encountering the following error: INFO scheduler.TaskSetManager: Loss was due to java.io.IOException: No space left on device [duplicate 4] For each slave, df -h looks roughtly like this, which makes the above error surprising. FilesystemSize Used Avail Use% Mounted on