I run two apps dependent on the Twitter streaming API, so I set up stream processing on a separate EC2 instance, posting the data via HTTP to the appropriate Heroku app.
I don't like this set up for exactly the reason you specified, but I figure Heroku will have a solution eventually, so I'm just sitting tight. And, FWIW, if you want anything even close to reliability in the tweet stream, you have to use the streaming API. The search API is almost worthless. Jim On Thu, Mar 4, 2010 at 1:39 PM, Josh Cheek <[email protected]> wrote: > Hi, Twitter has a streaming API now, where you have a process that just > remains connected, and they push updates to your code as they occur. > http://apiwiki.twitter.com/Streaming-API-Documentation > > I found a gem for it called TweetStream > http://github.com/intridea/tweetstream > But if you run it, it will never complete, because it has to stay connected > to Twitter. > > So they have a class, TweetStream::Daemons, that will let you run it in the > background, using the daemons gem, which heroku has installed > http://installed-gems.heroku.com/ > > Anyway, this is what I am looking at right now, but it seems like I'll have > to have another dyno constantly running to handle the stream from Twitter, > which gets a little bit expensive, but is manageable. The problem is that I > want to also have it stream in posts for a given user, as well as watching > for a hash tag, which are two different types of connections (follow vs > track), so then I would need another dyno again to have it follow the user. > And, of course, if I wanted to monitor the daemons to restart them if they > go down, that would be yet another dyno. > > I'm curious what other developers are doing to integrate with Twitter, are > you guys using the streaming API? Do you have it set up through cron? > Something else that I haven't thought of? > > I'm just not sure what the best approach is to integrate in this manner > (I'd like it to be near real time, which is why I am leaning towards the > streaming api). > > I was also thinking maybe set up another computer to just run the daemons, > then whenever it gets a request, have it post the request to my app. Which > seems viable, but it means that the site isn't self-contained, and now I > have to maintain hardware also, so I'd prefer to do it all through heroku. > > Anyway, just looking for community input on what other people have found > that works, or thoughts of better ways to solve the issue. > > Thanks > -Josh > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<heroku%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
