On Sat, Feb 19, 2011 at 7:52 AM, Tom van der Woerdt <[email protected]> wrote: > On 2/19/11 1:49 PM, Paresh Nakhe wrote: >> >> Hi, >> From what i understand, there is no concept of rate limiting for >> streaming api. Actually it does make sense because if anyone is to use >> 'statuses/sample' method (say) the limit will soon be crossed. We are >> working on something that will heavily use the streaming api, so if rate >> limiting is imposed in future it could create some problems. Are there >> any chances of such a restriction being imposed? > > There's no limit on the amount of tweets you can receive, but there's a > limit on the amount of searches you can do, and the amount of connections > you can have open. > Responses here from Twitter staff and notes in the docs say that the sample (default) level of delivery is limited to 1% of all tweets. As long as your total flow is less than 1% of the firehose (all tweets) you are supposed to receive every tweet that matches your search terms. From my experience, you only receive about 95% of the tweets that match a set of search terms. I've never used a set of search terms that delivered anywhere near 1% of the total flow.
Here are some numbers: - The total flow of all tweets is in the order of magnitude of 100,000,000 per day. - The sample stream should thus receive up to 1,000,000 tweets per day. - When I collect tweets based on search words that deliver about 10,000 tweets a day, I don't receive about 5% of the tweets that do match these terms. If getting *every* tweet that matches these terms is essential, I recommend combining streaming with backfilling. By backfilling I mean using the search API or timeline calls in the REST API to collect tweets that may have been missed by the streaming API. The result is real-time delivery of 95% of the tweets you need and eventual delivery of the remainder. Don't bother asking about future restrictions. That question will not be answered by Twitter HQ. -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
