Re: jenkins failing on Kinesis shard limits

2015-07-25 Thread Tathagata Das
I have remove the flag in the PullRequestBuilder that enabled Kinesis tests. All the Kinesis tests should be ignored now. In the mean time we can fix the tests. On Sat, Jul 25, 2015 at 8:44 AM, Shixiong Zhu wrote: > The issue is in KinesisBackedBlockRDDSuite > > I have sent https://github.com/ap

Confidence in implicit factorization

2015-07-25 Thread Debasish Das
Hi, Implicit factorization is important for us since it drives recommendation when modeling user click/no-click and also topic modeling to handle 0 counts in document x word matrices through NMF and Sparse Coding. I am a bit confused on this code: val c1 = alpha * math.abs(rating) if (rating > 0

Parallelism of Custom receiver in spark

2015-07-25 Thread anshu shukla
1 - How to increase the level of *parallelism in spark streaming custom RECEIVER* . 2 - Will ssc.receiverstream(/**anything //) will *delete the data stored in spark memory using store(s) * logic . -- Thanks & Regards, Anshu Shukla

Re: non-deprecation compiler warnings are upgraded to build errors now

2015-07-25 Thread Iulian DragoČ™
On Fri, Jul 24, 2015 at 8:19 PM, Reynold Xin wrote: Jenkins only run Scala 2.10. I'm actually not sure what the behavior is > with 2.11 for that patch. > > iulian - can you take a look into it and see if it is working as expected? > It is, in the sense that warnings fail the build. Unfortunately

Re: jenkins failing on Kinesis shard limits

2015-07-25 Thread Shixiong Zhu
The issue is in KinesisBackedBlockRDDSuite I have sent https://github.com/apache/spark/pull/7661 to remove the streams created by KinesisBackedBlockRDDSuite and https://github.com/apache/spark/pull/7663 to fix the issue. Best Regards, Shixiong Zhu 2015-07-25 14:46 GMT+08:00 Prabeesh K. : > For

ReceiverStream SPARK not able to cope up with 20,000 events /sec .

2015-07-25 Thread anshu shukla
My eventGen is emitting 20,000 events/sec ,and I am using store(s1) in receive() method to push data to receiverStream . But this logic is working fine for upto 4000 events/sec and no batch are seen emitting for larger rate . *CODE:TOPOLOGY -* *JavaDStream sourcestream = ssc.receiverStream(