Hi Matt, Thanks for the explanation. I will file the bug report. I'd like to hear more about the sample size. I've read through the Streaming API docs a lot, and I haven't come across anything specific about the rate limits. Where can I read more?
Brian Maso On Jan 10, 5:24 pm, Matt Harris <[email protected]> wrote: > Hey Brian, > > When you use the Streaming API filter method we will stream to you all the > Tweets which match your track terms - up to your allowed sample size. > > What this means is over the course of a sampling window we apply your track > terms to the full firehose, and then return as many results as your sample > rate allows. If you exceed your allowed sample size we will return a > 'rate_limited' response containing the total number of matched Tweets > missed. > > When matching track terms we apply the 'track' keywords to the raw Tweet > text. This is different to the Search API which applies the track terms to > the raw Tweet text plus the expanded URL. (The Streaming API doesn't expand > URLs because it would delay the delivery of the Tweet). > > The issue you are describing is not caused by sampling limits or reduced > subsets, but is instead due to a retweet parsing issue our engineers are > looking into. What appears to be happening is the Streaming API is trying to > match against the truncated RT version of the Tweet instead of the original > Tweet text. > > If you file this in our issue tracker we can let you know when the issue is > resolved. The issue tracker can be found here: > http://code.google.com/p/twitter-api/issues/list > > Best, > > @themattharris > Developer Advocate, Twitterhttp://twitter.com/themattharris > > On Mon, Jan 10, 2011 at 4:48 PM, Brian Maso <[email protected]>wrote: > > > Sounds consistent with what I've been seeing. Where did you get your > > impression of how the streaming API is "optimized"? I am having a hard > > time finding any authoritative documentation describing what the > > powers that be at Twitter *intend* to be included in the stream (as > > opposed to what they actually *implemented*, which may differ from > > intentions for a variety of reasons). > > > If what you say is true, it kind of limits to use-cases of the > > streaming API to a far narrower set than what one would think by > > reading the Streaming API documentation. There's one section of the > > documentation that attempts to describe how to implement a system that > > utilizes the streaming API and avoids missing any tweets. Obviously if > > the stream of tweets is already a reduced subset, then it doesn't > > matter very much if you miss a few. > > > Brian Maso > > > On Jan 9, 4:06 pm, Bess <[email protected]> wrote: > > > Streaming API is build by Twitter while Search API is build by Startup > > > Summize acquired by Twitter. Search API is rate-limited. > > > > If you just use Twitter search feature, you may see everything. Using > > > Search API to display API returned results is limited by your > > > developer API. > > > > Streaming API may not show everything b/c it is optimized on the > > > content based on its logarithm. > > > > On Jan 9, 2:29 pm, Brian Maso <[email protected]> wrote: > > > > > What I did is opened up three separate normal browser tabs in Firefox, > > > > each using the Twitter search web interface to search for three > > > > different hashtags ("#ces", "ces11", and "nfl" -- examples of three > > > > tags that should have decent ongoing traffic). > > > > > At the same time I have an application capturing tweets from the same > > > > three hashtags using the streaming API ("filter.json? > > > > q=#ces,#ces11,#nfl", with appropriate URL encoding). > > > > > Irregardless of the amount of time, the streaming application captured > > > > about 25% fewer tweets. Detailed analysis of the tweet IDs captured by > > > > the browsers vs. those captured by the standalone application > > > > retrieving tweets via the streaming API verified that there were > > > > tweets delivered through the browsers that did not appear through the > > > > streaming API. There were no tweets delivered through the streaming > > > > API that did not also appear in the set of tweets delivewred through > > > > the browsers. > > > > > I would love it if anyone else would try a similar experiment and > > > > report back results. Maybe I'm doing something wrong, or maybe this is > > > > an anomaly, or maybe the streaming API just doesn't capture as much -- > > > > impossible for me to say. > > > > > I note that the streaming API documentation doesn't claim an intent to > > > > match accuracy with the search API (nor vice versa). At this point I'm > > > > thinking to get the greatest accuracy I should be collecting tweets > > > > from *both* APIs. > > > > > Brian Maso > > > > > On Jan 7, 5:08 pm, Bess <[email protected]> wrote: > > > > > > This is hard to believe. Streaming API is an approved API that should > > > > > not have any limit. It should give you everything without any limit. > > > > > On the other hand Search API has rate-limitation. > > > > > > Did you use any filter? > > > > > > On Jan 6, 9:42 pm, Brian Maso <[email protected]> wrote: > > > > > > > Hi All, > > > > > > > Using the Streaming API, I'm noticing about a 25% loss rate when > > > > > > tracking multiple hashtags vs. using the good old Search API. I'm > > > > > > fouind it hard to believe this is true, so I tested over and over, > > but > > > > > > I keep getting the same results. The Streaming API just seems to > > not > > > > > > provide a fair number of tweets. > > > > > > > Note that I have the lowest rate limit with the Streaming API -- > > > > > > perhaps highest rate limits have lower loss rates. > > > > > > > Has anyone else noticed the rate loss Streaming vs. Search API? Or > > am > > > > > > I on crack? > > > > > > > Does the loss rate get lower with the higher Streaming API account > > > > > > limits? > > > > > > > Brian Maso > > > -- > > Twitter developer documentation and resources:http://dev.twitter.com/doc > > API updates via Twitter:http://twitter.com/twitterapi > > Issues/Enhancements Tracker: > >http://code.google.com/p/twitter-api/issues/list > > Change your membership to this group: > >http://groups.google.com/group/twitter-development-talk > > -- Twitter developer documentation and resources: http://dev.twitter.com/doc API updates via Twitter: http://twitter.com/twitterapi Issues/Enhancements Tracker: http://code.google.com/p/twitter-api/issues/list Change your membership to this group: http://groups.google.com/group/twitter-development-talk
