Thanks Cheng !!
On Thu, Apr 24, 2014 at 5:43 PM, Cheng Lian wrote:
> You may try this:
>
> val lastOption = sc.textFile("input").mapPartitions { iterator =>
> if (iterator.isEmpty) {
> iterator
> } else {
> Iterator
> .continually((iterator.next(), iterator.hasNext()))
>
You may try this:
val lastOption = sc.textFile("input").mapPartitions { iterator =>
if (iterator.isEmpty) {
iterator
} else {
Iterator
.continually((iterator.next(), iterator.hasNext()))
.collect { case (value, false) => value }
.take(1)
}
}.collect().lastOption
It
Hi All, Finally i wrote the following code, which is felt does optimally if
not the most optimum one.
Using file pointers, seeking the byte after the last \n but backwards !!
This is memory efficient and i hope even unix tail implementation should be
something similar !!
import java.io.RandomAcces
Thanks Guys !
On Thu, Apr 24, 2014 at 11:29 AM, Sourav Chandra <
sourav.chan...@livestream.com> wrote:
> Also same thing can be done using rdd.top(1)(reverseOrdering)
>
>
>
> On Thu, Apr 24, 2014 at 11:28 AM, Sourav Chandra <
> sourav.chan...@livestream.com> wrote:
>
>> You can use rdd.takeOrder
Also same thing can be done using rdd.top(1)(reverseOrdering)
On Thu, Apr 24, 2014 at 11:28 AM, Sourav Chandra <
sourav.chan...@livestream.com> wrote:
> You can use rdd.takeOrdered(1)(reverseOrdrering)
>
> reverseOrdering is you Ordering[T] instance where you define the ordering
> logic. This y
You can use rdd.takeOrdered(1)(reverseOrdrering)
reverseOrdering is you Ordering[T] instance where you define the ordering
logic. This you have to pass in the method
On Thu, Apr 24, 2014 at 11:21 AM, Frank Austin Nothaft <
fnoth...@berkeley.edu> wrote:
> If you do this, you could simplify to:
What i observe is, this way of computing is very inefficient. It returns
all the elements of the RDD to a List which takes considerable amount of
time.
Then it calculates the last element.
I have a file of size 3 GB in which i ran a lot of aggregate operations
which dint took the time that this ta
If you do this, you could simplify to:
RDD.collect().last
However, this has the problem of collecting all data to the driver.
Is your data sorted? If so, you could reverse the sort and take the first.
Alternatively, a hackey implementation might involve a mapPartitionsWithIndex
that returns an
This function will return scala List, you can use List's last function to
get the last element.
For example:
RDD.take(RDD.count()).last
On Thu, Apr 24, 2014 at 10:28 AM, Sai Prasanna wrote:
> Adnan, but RDD.take(RDD.count()) returns all the elements of the RDD.
>
> I want only to access the la
Adnan, but RDD.take(RDD.count()) returns all the elements of the RDD.
I want only to access the last element.
On Thu, Apr 24, 2014 at 10:33 AM, Sai Prasanna wrote:
> Oh ya, Thanks Adnan.
>
>
> On Thu, Apr 24, 2014 at 10:30 AM, Adnan Yaqoob wrote:
>
>> You can use following code:
>>
>> RDD.take
Oh ya, Thanks Adnan.
On Thu, Apr 24, 2014 at 10:30 AM, Adnan Yaqoob wrote:
> You can use following code:
>
> RDD.take(RDD.count())
>
>
> On Thu, Apr 24, 2014 at 9:51 AM, Sai Prasanna wrote:
>
>> Hi All, Some help !
>> RDD.first or RDD.take(1) gives the first item, is there a straight
>> forward
You can use following code:
RDD.take(RDD.count())
On Thu, Apr 24, 2014 at 9:51 AM, Sai Prasanna wrote:
> Hi All, Some help !
> RDD.first or RDD.take(1) gives the first item, is there a straight forward
> way to access the last element in a similar way ?
>
> I coudnt fine a tail/last method for
12 matches
Mail list logo
