Is it better to use our old HttpUrlConnection instead of URLFetch ? On Fri, Oct 26, 2012 at 2:10 AM, Jeff Schnitzer <[email protected]> wrote:
> Translated: "URLFetch is rock-solid, except because it uses a shared > IP pool it will erratically fail if you use it to fetch from almost > any third-party service that pays attention to load." Which really > isn't very rock-solid at all. > > The shared IP pool is a significant problem with URLFetch, and you > really need to be careful when using it. The standard workaround is > to set up your own proxy servers elsewhere on the net - a PITA but not > optional for many services. Here's the issue to star to hopefully get > Google to do something about the issue: > > http://code.google.com/p/googleappengine/issues/detail?id=6644 > > FWIW, I've also found that URLFetch is occasionally less than snappy. > But there are a lot of moving parts involved so it's hard to figure > out exactly where to lay blame. > > One thing to watch out for is that the default URLFetch timeout is > fairly short. I usually find it necessary to increase the timeout, > especially with services with erratic performance (eg Facebook). > > Jeff > > On Thu, Oct 25, 2012 at 1:29 PM, Vinny P <[email protected]> wrote: > > In my experience, the reliability of URLFetch is rock-solid. The problem > is > > the external server that you're connecting to. The external server can be > > relatively fast to respond (many web APIs, such as Google's goo.glshortener > > and so forth) or relatively slow and error-prone (the Reddit API in > > particular is just absolutely terrible to access from AppEngine; since > all > > GAE urlfetches come from the same pool of IPs, the Reddit servers > > deliberately throttle requests because they think all of the requests are > > coming from a single poorly-behaved app, not multiple apps. And no, the > > reddit api does not offer oauth or similar authentication). > > > > There's a couple of ways to mitigate this; you can use task queues to > keep > > retrying a urlfetch, backends to continuously urlfetch and cache the > > results, find a different 3rd party service, etc. > > > > -Vinny P > > > > > > On Thursday, October 25, 2012 11:58:26 AM UTC-5, Joshua Smith wrote: > >> > >> I use the python version, and get a couple failures a day. The easy > answer > >> is to treat it just like mail: always use a task, so that if it fails, > it > >> will retry. > >> > >> On Oct 25, 2012, at 12:12 PM, Deepak Singh <[email protected]> > wrote: > >> > >> Hi Alll, > >> > >> I want to discuss here your experience about GAE Java URLFetchService. > >> > >> We are using async feature of this service to retrieve data from 3rd > party > >> servers and our business mainly depends on the data received from their > >> servers. > >> I observe that UrlFetch fails many times with java.io exception and > thus > >> we lose our business. > >> > >> So i would like to know your experience about its reliability, > >> DeadlineExceededException cases, ways to handle it and all. > >> > >> Let us know how reliable is URLFetchService(GAE Java) ? > >> > >> Regards > >> Deepak Singh > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Google App Engine" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]. > >> > >> For more options, visit this group at > >> http://groups.google.com/group/google-appengine?hl=en. > >> > >> > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To view this discussion on the web visit > > https://groups.google.com/d/msg/google-appengine/-/RgAEOStwEtMJ. > > > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > > http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- Deepak Singh -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
