If you do this, you could simplify to: RDD.collect().last
However, this has the problem of collecting all data to the driver. Is your data sorted? If so, you could reverse the sort and take the first. Alternatively, a hackey implementation might involve a mapPartitionsWithIndex that returns an empty iterator for all partitions except for the last. For the last partition, you would filter all elements except for the last element in your iterator. This should leave one element, which is your last element. Frank Austin Nothaft fnoth...@berkeley.edu fnoth...@eecs.berkeley.edu 202-340-0466 On Apr 23, 2014, at 10:44 PM, Adnan Yaqoob <nsyaq...@gmail.com> wrote: > This function will return scala List, you can use List's last function to get > the last element. > > For example: > > RDD.take(RDD.count()).last > > > On Thu, Apr 24, 2014 at 10:28 AM, Sai Prasanna <ansaiprasa...@gmail.com> > wrote: > Adnan, but RDD.take(RDD.count()) returns all the elements of the RDD. > > I want only to access the last element. > > > On Thu, Apr 24, 2014 at 10:33 AM, Sai Prasanna <ansaiprasa...@gmail.com> > wrote: > Oh ya, Thanks Adnan. > > > On Thu, Apr 24, 2014 at 10:30 AM, Adnan Yaqoob <nsyaq...@gmail.com> wrote: > You can use following code: > > RDD.take(RDD.count()) > > > On Thu, Apr 24, 2014 at 9:51 AM, Sai Prasanna <ansaiprasa...@gmail.com> wrote: > Hi All, Some help ! > RDD.first or RDD.take(1) gives the first item, is there a straight forward > way to access the last element in a similar way ? > > I coudnt fine a tail/last method for RDD. !! > > > >
