Re: Twitter streaming with apache spark stream only a small amount of tweets

2015-07-29 Thread Enno Shioji
If you start parallel Twitter streams, you will be in breach of their TOS. They allow a small number of parallel stream in practice, but if you do it on massive scale they'll ban you (I'm speaking from experience ;) ). If you really need that level of data, you need to talk to a company called Gni

Re: Twitter streaming with apache spark stream only a small amount of tweets

2015-07-29 Thread Zoran Jeremic
Actually, I posted that question :) I already implemented solution that Akhil suggested there , and that solution is using Sample tweets API, which returns only 1% of the tweets. It would not work in my scenario of use. For the hashtags I'm interested in, I need to catch each single tweet, not onl

Re: Twitter streaming with apache spark stream only a small amount of tweets

2015-07-29 Thread Peyman Mohajerian
'How to restart Twitter spark stream' i It may not be exactly what you are looking for, but i thought it did touch on some aspect of your question. On Wed, Jul 29, 2015 at 10:26 AM, Zoran Jeremic wrote: > Can you send me the subject of that email? I can't find any email > suggesting solution to

Re: Twitter streaming with apache spark stream only a small amount of tweets

2015-07-29 Thread Zoran Jeremic
Can you send me the subject of that email? I can't find any email suggesting solution to that problem. There is email "*Twitter4j streaming question*", but it doesn't have any sample code. It just confirms what I explained earlier that without filtering Twitter will limit to 1% of tweets, and if yo

Re: Twitter streaming with apache spark stream only a small amount of tweets

2015-07-29 Thread Peyman Mohajerian
This question was answered with sample code a couple of days ago, please look back. On Sat, Jul 25, 2015 at 11:43 PM, Zoran Jeremic wrote: > Hi, > > I discovered what is the problem here. Twitter public stream is limited to > 1% of overall tweets (https://goo.gl/kDwnyS), so that's why I can't >

Re: Twitter streaming with apache spark stream only a small amount of tweets

2015-07-25 Thread Zoran Jeremic
Hi, I discovered what is the problem here. Twitter public stream is limited to 1% of overall tweets (https://goo.gl/kDwnyS), so that's why I can't access all the tweets posted with specific hashtag using approach that I posted in previous email, so I guess this approach would not work for me. The