Fwd: Spark RDD cache memory usage

2014-04-30 Thread Han JU
Hi, As I understand, by default in Spark a fraction of the executor memory (60%) is reserved for RDD caching. So if there's no explicit caching in the code (eg. rdd.cache() etc.), or if we persist RDD with StorageLevel.DISK_ONLY, is this part of memory wasted? Does Spark allocates the RDD cache me

[EC2] r3 instance type

2014-05-12 Thread Han JU
Hi, I'm modifying the ec2 script for the new r3 instance support, but there's a problem with the instance storage. For example, `r3.large` has a single 32GB SSD disk, the problem is that it's a SSD with TRIM technology and is not automatically formatted and mounted, `lsblk` gives me this after ec