Set N be the total number of cores on the cluster or less. sc.textFile
doesn't always give you that number, depends on the block size. For
MovieLens, I think the default behavior should be 2~3 partitions. You
need to call repartition to ensure the right number of partitions.
Which EC2 instance type did you use? I usually use m3.2xlarge or c?
instances that come with SSD and 1G or 10G network. For those
instances, you should see local drives mounted at /mnt, /mnt2, /mnt3,
... Make sure there was no error when you used the ec2 script to
launch the cluster.
It is a little strange to see 94% of / was used on a slave. Maybe
shuffle data went to /. I'm not sure which settings went wrong. I
recommend trying re-launching a cluster with m3.2xlarge instances and
using the default settings (don't set anything in SparkConf). Submit
the application with --driver-memory 20g.
The running times are slower than what I remember, but it depends on
the instance type.
Best,
Xiangrui
On Wed, Jul 16, 2014 at 10:18 PM, Chris DuBois <chris.dub...@gmail.com> wrote:
> Hi Xiangrui,
>
> I will try this shortly. When using N partitions, do you recommend N be the
> number of cores on each slave or the number of cores on the master? Forgive
> my ignorance, but is this best achieved as an argument to sc.textFile?
>
> The slaves on the EC2 clusters start with only 8gb of storage, and it
> doesn't seem that /mnt/spark and /mnt2/spark are mounted anywhere else by
> default. Looking at spark-ec2/setup-slaves.sh, it appears that these are
> only mounted if the instance type begins with r3. (Or am I not reading that
> right?) My slaves are a different instance type, and currently look like
> this:
> Filesystem Size Used Avail Use% Mounted on
> /dev/xvda1 7.9G 7.3G 515M 94% /
> tmpfs 7.4G 4.0K 7.4G 1% /dev/shm
> /dev/xvdv 500G 2.5G 498G 1% /vol
>
> I have been able to finish ALS on MovieLens 10M only twice, taking 221s and
> 315s for 20 iterations at K=20 and lambda=0.01. Does that timing sound about
> right, or does it point to a poor configuration? The same script with
> MovieLens 1M runs fine in about 30-40s with the same settings. (In both
> cases I'm training on 70% of the data.)
>
> Thanks for your help!
> Chris
>
>
> On Wed, Jul 16, 2014 at 4:29 PM, Xiangrui Meng <men...@gmail.com> wrote:
>>
>> For ALS, I would recommend repartitioning the ratings to match the
>> number of CPU cores or even less. ALS is not computation heavy for
>> small k but communication heavy. Having small number of partitions may
>> help. For EC2 clusters, we use /mnt/spark and /mnt2/spark as the
>> default local directory because they are local hard drives. Did your
>> last run of ALS on MovieLens 10M-100K with the default settings
>> succeed? -Xiangrui
>>
>> On Wed, Jul 16, 2014 at 8:00 AM, Chris DuBois <chris.dub...@gmail.com>
>> wrote:
>> > Hi Xiangrui,
>> >
>> > I accidentally did not send df -i for the master node. Here it is at the
>> > moment of failure:
>> >
>> > Filesystem Inodes IUsed IFree IUse% Mounted on
>> > /dev/xvda1 524288 280938 243350 54% /
>> > tmpfs 3845409 1 3845408 1% /dev/shm
>> > /dev/xvdb 10002432 1027 10001405 1% /mnt
>> > /dev/xvdf 10002432 16 10002416 1% /mnt2
>> > /dev/xvdv 524288000 13 524287987 1% /vol
>> >
>> > I am using default settings now, but is there a way to make sure that
>> > the
>> > proper directories are being used? How many blocks/partitions do you
>> > recommend?
>> >
>> > Chris
>> >
>> >
>> > On Wed, Jul 16, 2014 at 1:09 AM, Chris DuBois <chris.dub...@gmail.com>
>> > wrote:
>> >>
>> >> Hi Xiangrui,
>> >>
>> >> Here is the result on the master node:
>> >> $ df -i
>> >> Filesystem Inodes IUsed IFree IUse% Mounted on
>> >> /dev/xvda1 524288 273997 250291 53% /
>> >> tmpfs 1917974 1 1917973 1% /dev/shm
>> >> /dev/xvdv 524288000 30 524287970 1% /vol
>> >>
>> >> I have reproduced the error while using the MovieLens 10M data set on a
>> >> newly created cluster.
>> >>
>> >> Thanks for the help.
>> >> Chris
>> >>
>> >>
>> >> On Wed, Jul 16, 2014 at 12:22 AM, Xiangrui Meng <men...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi Chris,
>> >>>
>> >>> Could you also try `df -i` on the master node? How many
>> >>> blocks/partitions did you set?
>> >>>
>> >>> In the current implementation, ALS doesn't clean the shuffle data
>> >>> because the operations are chained together. But it shouldn't run out
>> >>> of disk space on the MovieLens dataset, which is small. spark-ec2
>> >>> script sets /mnt/spark and /mnt/spark2 as the local.dir by default, I
>> >>> would recommend leaving this setting as the default value.
>> >>>
>> >>> Best,
>> >>> Xiangrui
>> >>>
>> >>> On Wed, Jul 16, 2014 at 12:02 AM, Chris DuBois
>> >>> <chris.dub...@gmail.com>
>> >>> wrote:
>> >>> > Thanks for the quick responses!
>> >>> >
>> >>> > I used your final -Dspark.local.dir suggestion, but I see this
>> >>> > during
>> >>> > the
>> >>> > initialization of the application:
>> >>> >
>> >>> > 14/07/16 06:56:08 INFO storage.DiskBlockManager: Created local
>> >>> > directory at
>> >>> > /vol/spark-local-20140716065608-7b2a
>> >>> >
>> >>> > I would have expected something in /mnt/spark/.
>> >>> >
>> >>> > Thanks,
>> >>> > Chris
>> >>> >
>> >>> >
>> >>> >
>> >>> > On Tue, Jul 15, 2014 at 11:44 PM, Chris Gore <cdg...@cdgore.com>
>> >>> > wrote:
>> >>> >>
>> >>> >> Hi Chris,
>> >>> >>
>> >>> >> I've encountered this error when running Spark’s ALS methods too.
>> >>> >> In
>> >>> >> my
>> >>> >> case, it was because I set spark.local.dir improperly, and every
>> >>> >> time
>> >>> >> there
>> >>> >> was a shuffle, it would spill many GB of data onto the local drive.
>> >>> >> What
>> >>> >> fixed it was setting it to use the /mnt directory, where a network
>> >>> >> drive is
>> >>> >> mounted. For example, setting an environmental variable:
>> >>> >>
>> >>> >> export SPACE=$(mount | grep mnt | awk '{print $3"/spark/"}' | xargs
>> >>> >> |
>> >>> >> sed
>> >>> >> 's/ /,/g’)
>> >>> >>
>> >>> >> Then adding -Dspark.local.dir=$SPACE or simply
>> >>> >> -Dspark.local.dir=/mnt/spark/,/mnt2/spark/ when you run your driver
>> >>> >> application
>> >>> >>
>> >>> >> Chris
>> >>> >>
>> >>> >> On Jul 15, 2014, at 11:39 PM, Xiangrui Meng <men...@gmail.com>
>> >>> >> wrote:
>> >>> >>
>> >>> >> > Check the number of inodes (df -i). The assembly build may create
>> >>> >> > many
>> >>> >> > small files. -Xiangrui
>> >>> >> >
>> >>> >> > On Tue, Jul 15, 2014 at 11:35 PM, Chris DuBois
>> >>> >> > <chris.dub...@gmail.com>
>> >>> >> > wrote:
>> >>> >> >> Hi all,
>> >>> >> >>
>> >>> >> >> I am encountering the following error:
>> >>> >> >>
>> >>> >> >> INFO scheduler.TaskSetManager: Loss was due to
>> >>> >> >> java.io.IOException:
>> >>> >> >> No
>> >>> >> >> space
>> >>> >> >> left on device [duplicate 4]
>> >>> >> >>
>> >>> >> >> For each slave, df -h looks roughtly like this, which makes the
>> >>> >> >> above
>> >>> >> >> error
>> >>> >> >> surprising.
>> >>> >> >>
>> >>> >> >> Filesystem Size Used Avail Use% Mounted on
>> >>> >> >> /dev/xvda1 7.9G 4.4G 3.5G 57% /
>> >>> >> >> tmpfs 7.4G 4.0K 7.4G 1% /dev/shm
>> >>> >> >> /dev/xvdb 37G 3.3G 32G 10% /mnt
>> >>> >> >> /dev/xvdf 37G 2.0G 34G 6% /mnt2
>> >>> >> >> /dev/xvdv 500G 33M 500G 1% /vol
>> >>> >> >>
>> >>> >> >> I'm on an EC2 cluster (c3.xlarge + 5 x m3) that I launched using
>> >>> >> >> the
>> >>> >> >> spark-ec2 scripts and a clone of spark from today. The job I am
>> >>> >> >> running
>> >>> >> >> closely resembles the collaborative filtering example. This
>> >>> >> >> issue
>> >>> >> >> happens
>> >>> >> >> with the 1M version as well as the 10 million rating version of
>> >>> >> >> the
>> >>> >> >> MovieLens dataset.
>> >>> >> >>
>> >>> >> >> I have seen previous questions, but they haven't helped yet. For
>> >>> >> >> example, I
>> >>> >> >> tried setting the Spark tmp directory to the EBS volume at
>> >>> >> >> /vol/,
>> >>> >> >> both
>> >>> >> >> by
>> >>> >> >> editing the spark conf file (and copy-dir'ing it to the slaves)
>> >>> >> >> as
>> >>> >> >> well
>> >>> >> >> as
>> >>> >> >> through the SparkConf. Yet I still get the above error. Here is
>> >>> >> >> my
>> >>> >> >> current
>> >>> >> >> Spark config below. Note that I'm launching via
>> >>> >> >> ~/spark/bin/spark-submit.
>> >>> >> >>
>> >>> >> >> conf = SparkConf()
>> >>> >> >> conf.setAppName("RecommendALS").set("spark.local.dir",
>> >>> >> >> "/vol/").set("spark.executor.memory",
>> >>> >> >> "7g").set("spark.akka.frameSize",
>> >>> >> >> "100").setExecutorEnv("SPARK_JAVA_OPTS", "
>> >>> >> >> -Dspark.akka.frameSize=100")
>> >>> >> >> sc = SparkContext(conf=conf)
>> >>> >> >>
>> >>> >> >> Thanks for any advice,
>> >>> >> >> Chris
>> >>> >> >>
>> >>> >>
>> >>> >
>> >>
>> >>
>> >
>
>
