This works just fine without any compilation error:
def print() {
def foreachFunc = (rdd: RDD[T], time: Time) => {
val total = rdd.collect().toList
println ("-------------------------------------------")
println ("Time: " + time)
println ("-------------------------------------------")
total.foreach(println)
// val first11 = rdd.take(11)
// println ("-------------------------------------------")
// println ("Time: " + time)
// println ("-------------------------------------------")
// first11.take(10).foreach(println)
// if (first11.size > 10) println("...")
println()
}
new ForEachDStream(this,
context.sparkContext.clean(foreachFunc)).register()
}
On Wed, Mar 26, 2014 at 8:50 PM, Eduardo Costa Alfaia <
[email protected]> wrote:
> Thanks Guys for reply, but I have found this piece of code in streaming:
> def print() {
> 585 def foreachFunc = (rdd: RDD[T], time: Time) => {
> 586 //val first11 = rdd.take(11)
> 587 //val first100 = rdd.take(100)
> 588 val total = rdd.collect()
> 589 println ("-------------------------------------------")
> 590 println ("Time: " + time)
> 591 println ("-------------------------------------------")
> 592 //first100.take(100).foreach(println)
> 593 total.collect().foreach(println)
> 594 //if (first100.size > 99) println("(...)")
> 595 println()
> 596 }
> 597 new ForEachDStream(this,
> context.sparkContext.clean(foreachFunc)).register()
> 598 }
>
> I would like to print whole the RDD, I have seen I should use the
> collect() function instead of take(), I have done but I have gotten a
> compilation error.
> does anyone help me?
>
> BR
>
> Em 3/25/14, 13:48, Sourav Chandra escreveu:
>
> You can extend DStream and override print() method. Then you can create
> your own DSTream extending from this.
>
>
> On Tue, Mar 25, 2014 at 6:07 PM, Eduardo Costa Alfaia <
> [email protected]> wrote:
>
>> Hi Guys,
>> I think that I already did this question, but I don't remember if anyone
>> has answered me. I would like changing in the function print() the quantity
>> of words and the frequency number that are sent to driver's screen. The
>> default value is 10.
>>
>> Anyone could help me with this?
>>
>> Best Regards
>>
>> --
>> Informativa sulla Privacy: http://www.unibs.it/node/8155
>>
>
>
>
> --
>
> Sourav Chandra
>
> Senior Software Engineer
>
> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>
> [email protected]
>
> o: +91 80 4121 8723
>
> m: +91 988 699 3746
>
> skype: sourav.chandra
>
> Livestream
>
> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
> Block, Koramangala Industrial Area,
>
> Bangalore 560034
>
> www.livestream.com
>
>
>
> Informativa sulla Privacy: http://www.unibs.it/node/8155
>
--
Sourav Chandra
Senior Software Engineer
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
[email protected]
o: +91 80 4121 8723
m: +91 988 699 3746
skype: sourav.chandra
Livestream
"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
Block, Koramangala Industrial Area,
Bangalore 560034
www.livestream.com
