Sean Owen-2 wrote > Can you not just filter the range you want, then groupBy > timestamp/86400 ? That sounds like your solution 1 and is about as > fast as it gets, I think. Are you thinking you would have to filter > out each day individually from there, and that's why it would be slow? > I don't think that's needed. You also don't need to map to pairs.
I didn't make it clear in my first message that I want to obtain an RDD instead of an Iterable, and will be doing map-reduce like operations on the data by day. My problem is that groupBy returns an RDD[(K, Iterable[T])], but I really want an RDD[(K, RDD[T])]. Is there a better approach to this? I'm leaning towards partitioning my data by day on disk since all of my queries will always process data per day. However, the only problem I see with partitioning the data on disk is that it limits my system to cleanly work for a single timezone. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-separate-a-subset-of-an-RDD-by-day-tp9454p9464.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
