Any chance you have control over the client? If you are using caching, chances are good many of the requests are "no ops" and you are just checking for changes - in this case you should be polling for a version number, not the data itself. The client checks the version number against the locally last known version and re-requests the full data if needed.
Another alternate implementation is to generate a obfuscated URL, maybe using UUID + SHA1 hashing. Example: /some/url/891893984ajkskakska23 Send a redirect back after a user authenticates and proxy cache the obfuscated URL. For some weak security, time the data out hourly so users always need to receive the redirect to get the data. You have a lot of different requirements that are not particularly easy to engineer for; you either need to accept the costs or engineer workarounds. Some of these workarounds will require requirement compromises. Ikai Lan Developer Programs Engineer, Google App Engine Blog: http://googleappengine.blogspot.com Twitter: http://twitter.com/app_engine Reddit: http://www.reddit.com/r/appengine On Sat, May 14, 2011 at 4:47 AM, Ricky Button <[email protected]> wrote: > > Thanks I'll check this out. Is there anything I can to do optimize > bandwidth used. Each of those 50,000 requests are completely different > entities and have different uris, (/api/articles/46452) > > On May 13, 10:10 pm, Gopal Patel <[email protected]> wrote: > > you can use in-instance caching. It may not be always up to date but will > > give you extreme speeds. > > > > http://appengine-cookbook.appspot.com/recipe/decorator-to-getset-from... > > > > checkout fifth one. > > > > > > > > On Sat, May 14, 2011 at 4:37 AM, Ricky Button <[email protected]> > wrote: > > > At a peak, a user of my application can make around 30,000 - 50,000 > > > requests a day that returns some xml data. The xml data itself is > > > cached so the request is only doing two RPC calls (one for > > > authentication, one for the xml string). However, this seems to use a > > > lot of CPU time and outgoing bandwidth. I've tried using memcache for > > > the xml data but it turns out that it is usually a low hit ratio so it > > > is quite a bit slower. Is there any form of aggressive caching or > > > method I can use to limit the resources used over millions of requests > > > per day? > > > > > P.S. I have looked at this article > > >http://www.kyle-jensen.com/proxy-caching-on-google-appengine > > > and figured that it would not work well in my scenario because I want > > > to be able to authenticate on each request. > > > > > Thanks, > > > Ricky Button > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine" group. > > > To post to this group, send email to [email protected] > . > > > To unsubscribe from this group, send email to > > > [email protected]. > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
