Re: How to lookup by a key in an RDD

2015-11-06 Thread swetha kasireddy
Hi, I have the following questions on IndexedRDD. 1. Does the IndexedRDD support the key types of String? As per the current documentation, it looks like it supports only Long? 2. Is IndexedRDD efficient when joined with another RDD. So, basically my usecase is that I need to create an Indexed

Re: How to lookup by a key in an RDD

2015-11-05 Thread swetha kasireddy
I read about the IndexedRDD. Is the IndexedRDD join with another RDD that is not an IndexedRDD efficient? On Mon, Nov 2, 2015 at 9:56 PM, Deenar Toraskar wrote: > Swetha > > Currently IndexedRDD is an external library and not part of Spark Core. > You can use it by adding a dependency and pull i

Re: How to lookup by a key in an RDD

2015-11-02 Thread Deenar Toraskar
Swetha Currently IndexedRDD is an external library and not part of Spark Core. You can use it by adding a dependency and pull it in. There are plans to move it to Spark core tracked in https://issues.apache.org/jira/browse/SPARK-2365. See https://spark-summit.org/2015/events/indexedrdd-efficient-f

Re: How to lookup by a key in an RDD

2015-11-02 Thread Ted Yu
Please take a look at SPARK-2365 On Mon, Nov 2, 2015 at 3:25 PM, swetha kasireddy wrote: > Hi, > > Is Indexed RDDs released yet? > > Thanks, > Swetha > > On Sun, Nov 1, 2015 at 1:21 AM, Gylfi wrote: > >> Hi. >> >> You may want to look into Indexed RDDs >> https://github.com/amplab/spark-indexed

Re: How to lookup by a key in an RDD

2015-11-02 Thread swetha kasireddy
Hi, Is Indexed RDDs released yet? Thanks, Swetha On Sun, Nov 1, 2015 at 1:21 AM, Gylfi wrote: > Hi. > > You may want to look into Indexed RDDs > https://github.com/amplab/spark-indexedrdd > > Regards, > Gylfi. > > > > > > > -- > View this message in context: > http://apache-spark-user-list

Re: How to lookup by a key in an RDD

2015-11-01 Thread Gylfi
Hi. You may want to look into Indexed RDDs https://github.com/amplab/spark-indexedrdd Regards, Gylfi. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243p25247.html Sent from the Apache Spark User List mailin

Re: How to lookup by a key in an RDD

2015-10-31 Thread Natu Lauchande
Hi, Looking here for the lookup function might help you: https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions Natu On Sat, Oct 31, 2015 at 6:04 PM, swetha wrote: > Hi, > > I have a requirement wherein I have to load data from hdfs, build an RDD > and