Hi,
I've implemented Twitter streaming as in the code given at the bottom of
email. It finds some tweets based on the hashtags I'm following. However,
it seems that a large amount of tweets is missing. I've tried to post some
tweets that I'm following in the application, and none of them was received
in application. I also checked some hashtags (e.g. #android) on Twitter
using Live and I could see that almost each second something was posted
with that hashtag, and my application received only 3-4 posts in one minute.
I didn't have this problem in earlier non-spark version of application
which used twitter4j to access user stream API. I guess this is some
trending stream, but I couldn't find anything that explains which Twitter
API is used in Spark Twitter Streaming and how to create stream that will
access everything posted on the Twitter.
I hope somebody could explain what is the problem and how to solve this.
Thanks,
Zoran
def initializeStreaming(){
> val config = getTwitterConfigurationBuilder.build()
> val auth: Option[twitter4j.auth.Authorization] = Some(new
> twitter4j.auth.OAuthAuthorization(config))
> val stream:DStream[Status] = TwitterUtils.createStream(ssc, auth)
> val filtered_statuses = stream.transform(rdd =>{
> val filtered = rdd.filter(status =>{
> var found = false
> for(tag <- hashTagsList){
> if(status.getText.toLowerCase.contains(tag)) {
> found = true
> }
> }
> found
> })
> filtered
> })
> filtered_statuses.foreachRDD(rdd => {
> rdd.collect.foreach(t => {
> println(t)
> })
> })
> ssc.start()
> }
>
