You are right, the queue.py code should be put in gaehandler.py. I have cleaned up the code and posted it here:
http://appengine-cookbook.appspot.com/recipe/hot-handler It might be a good idea to disable it (comment it out) by default since it could confuse developers when they try to look at their logs. Robin On Oct 30, 12:38 am, mdipierro <mdipie...@cs.depaul.edu> wrote: > Let's if I understand. > > 1) we copy queue.py in gluon.contrib > 2) we edit the web2py app.yaml to include the above modification so > that that url is served by queue.py not web2py > 3) google keeps calling itself and the entire web2py app gets cached? > > Massimo > > On Oct 30, 12:30 am, Robin B <robi...@gmail.com> wrote: > > > Looks like google exponentially backs off a task when it fails, so > > here is a version that uses eta to schedule the next task 10s in the > > future. > > > ### in queue.yaml > > queue: > > - name: default > > rate: 20/m > > bucket_size: 1 > > > ### in app.yaml > > handlers: > > - url: /_ah/queue/default > > script: queue.py > > login: admin > > > ### in queue.py > > import os, sys, datetime > > from wsgiref.handlers import CGIHandler > > > from google.appengine.api.labs import taskqueue > > delta = datetime.timedelta(seconds=10) > > > def wsgi_app(env, res): > > taskqueue.add(eta=datetime.datetime.now() + delta) > > res('200 OK',[('Content-Type','text/plain')]) > > return [''] > > > def main(): > > CGIHandler().run(wsgi_app) > > > if __name__ == '__main__': > > main() > > > #### how to launch > > 1) visit "/_ah/queue/default" in your browser > > 2) login as an admin which inserts the first tasks > > 3) view the logs > > > By the way this is on the production server, the dev server does not > > run the tasks automatically. > > > ### view the logs > > > 10-29 10:28PM 11.010 /_ah/queue/default 200 17ms 7cpu_ms > > 10-29 10:28PM 01.018 /_ah/queue/default 200 28ms 9cpu_ms > > 10-29 10:27PM 51.007 /_ah/queue/default 200 16ms 5cpu_ms > > 10-29 10:27PM 41.011 /_ah/queue/default 200 18ms 5cpu_ms > > 10-29 10:27PM 31.007 /_ah/queue/default 200 38ms 14cpu_ms > > 10-29 10:27PM 21.009 /_ah/queue/default 200 23ms 7cpu_ms > > 10-29 10:27PM 11.009 /_ah/queue/default 200 20ms 7cpu_ms > > 10-29 10:27PM 01.009 /_ah/queue/default 200 20ms 10cpu_ms > > 10-29 10:26PM 51.007 /_ah/queue/default 200 44ms 35cpu_ms > > 10-29 10:26PM 41.017 /_ah/queue/default 200 24ms 12cpu_ms > > > Literally like clock work, a task hits your app once every 10 seconds. > > > Robin > > > On Oct 29, 11:47 pm, Robin B <robi...@gmail.com> wrote: > > > > I looked at the docs and came up with this code: > > > > ### somehow run this code once to insert atleast one task in the queue > > > from google.appengine.api.labs import taskqueue > > > taskqueue.add(url='/_ah/queue/default') # will return 404 > > > > ### add this to queue.yaml > > > queue: > > > - name: default > > > rate: 10/m > > > bucket_size: 1 > > > > ### look at the logs and see the task hitting your app about every 10 > > > sec or so, enough to keep a handler hot > > > > 10-29 09:42PM 46.443 /_ah/queue/default 404 7ms 4cpu_ms 0kb AppEngine- > > > Google; (+http://code.google.com/appengine) > > > 10-29 09:41PM 55.219 /_ah/queue/default 404 10ms 6cpu_ms 0kb AppEngine- > > > Google; (+http://code.google.com/appengine) > > > 10-29 09:41PM 29.602 /_ah/queue/default 404 6ms 3cpu_ms 0kb AppEngine- > > > Google; (+http://code.google.com/appengine) > > > 10-29 09:41PM 16.774 /_ah/queue/default 404 5ms 2cpu_ms 0kb AppEngine- > > > Google; (+http://code.google.com/appengine) > > > 10-29 09:40PM 54.808 /_ah/queue/default 404 5ms 2cpu_ms 0kb AppEngine- > > > Google; (+http://code.google.com/appengine) > > > 10-29 09:40PM 48.794 /_ah/queue/default 404 7ms 3cpu_ms 0kb AppEngine- > > > Google; (+http://code.google.com/appengine) > > > 10-29 09:40PM 42.776 /_ah/queue/default 404 10ms 4cpu_ms 0kb AppEngine- > > > Google; (+http://code.google.com/appengine) > > > 10-29 09:40PM 36.772 /_ah/queue/default 404 63ms 74cpu_ms 0kb > > > AppEngine-Google; (+http://code.google.com/appengine) > > > 10-29 09:40PM 13.176 /_ah/queue/default 404 6ms 2cpu_ms 0kb AppEngine- > > > Google; (+http://code.google.com/appengine) > > > 10-29 09:40PM 07.161 /_ah/queue/default 404 7ms 4cpu_ms 0kb AppEngine- > > > Google; (+http://code.google.com/appengine) > > > 10-29 09:40PM 07.161 /_ah/queue/default 404 7ms 4cpu_ms 0kb AppEngine- > > > Google; (+http://code.google.com/appengine) > > > > Robin > > > > On Oct 29, 10:32 pm, mdipierro <mdipie...@cs.depaul.edu> wrote: > > > > > do you have an example of code? > > > > > On Oct 29, 10:16 pm, Robin B <robi...@gmail.com> wrote: > > > > > > > neat - now I get it. That's a nice hack! > > > > > > Did you find that out from the Google App Engine group, or thought > > > > > > of > > > > > > it yourself? > > > > > > Thought of it myself. > > > > > > Robin > > > > > > On Oct 29, 12:09 am, Richard <richar...@gmail.com> wrote: > > > > > > > neat - now I get it. That's a nice hack! > > > > > > Did you find that out from the Google App Engine group, or thought > > > > > > of > > > > > > it yourself? > > > > > > > Richard > > > > > > > On Oct 29, 10:29 am, Robin B <robi...@gmail.com> wrote: > > > > > > > > > The docs say a task will run when resources are free. In your > > > > > > > > experience do the tasks generally run on time? > > > > > > > > Yes, they are trying to say that task queue webhooks are subject > > > > > > > to > > > > > > > the same quota restrictions as your app. > > > > > > > > > Also how would you make a task run periodically? > > > > > > > > This is answered in a previous message in this thread: > > > > > > > > "To keep an app hot, use the task > > > > > > > queue to hit a url that simply returns a non-200 HTTP status > > > > > > > code, so > > > > > > > the task will retry indefinitely at up to 10 Hz (10 > > > > > > > calls/second), you > > > > > > > can specify how often a task is called." > > > > > > > > Tasks are considered 'completed' and removed if they result in a > > > > > > > 200 > > > > > > > HTTP status code. > > > > > > > > > My understanding is > > > > > > > > they are for one off events and cron is for periodic. > > > > > > > > You are right, always returning a non-200 status code is > > > > > > > obviously not > > > > > > > the intended purpose of the task queue, but in this situation we > > > > > > > are > > > > > > > hacking the task queue to accomplish a specific goal of calling > > > > > > > the > > > > > > > task periodically since tasks can run much more frequently than > > > > > > > cron > > > > > > > which is necessary to keep a handler 'hot'. > > > > > > > > Robin > > > > > > > > > Richard > > > > > > > > > On Oct 28, 11:23 pm, Robin B <robi...@gmail.com> wrote: > > > > > > > > > > Oops typo: I said cron is 1/sec it actually its 1/minute, > > > > > > > > > task queue > > > > > > > > > is 10/second, so cron is not fast enough but task queue is > > > > > > > > > plenty > > > > > > > > > fast. > > > > > > > > > > Robin > > > > > > > > > > On Oct 28, 4:18 am, Richard <richar...@gmail.com> wrote: > > > > > > > > > > > that's really useful - thanks Robin! > > > > > > > > > > > if Python handlers last ~15 seconds then wouldn't cron be > > > > > > > > > > fast enough > > > > > > > > > > (at 1/sec)? > > > > > > > > > > Task queue is labelled experimental so I am wary to use it > > > > > > > > > > at the > > > > > > > > > > moment. > > > > > > > > > > > Richard > > > > > > > > > > > On Oct 28, 4:13 pm, Robin B <robi...@gmail.com> wrote: > > > > > > > > > > > > > How long does it stay cached? > > > > > > > > > > > > Inactive Python handlers used to last ~1 minute, recently > > > > > > > > > > > it is closer > > > > > > > > > > > to 15 seconds. > > > > > > > > > > > > Java servlets last longer, over a minute, maybe becuase > > > > > > > > > > > they are so > > > > > > > > > > > slow to boot (6 seconds +). > > > > > > > > > > > > One of the coming releases is supposed to speed up the > > > > > > > > > > > cold boot > > > > > > > > > > > times. > > > > > > > > > > > > Cron has a maximum frequency of 1 Hz (1 call/second), so > > > > > > > > > > > it is no > > > > > > > > > > > longer fast enough to do the job. To keep an app hot, > > > > > > > > > > > use the task > > > > > > > > > > > queue to hit a url that simply returns a non-200 HTTP > > > > > > > > > > > status code, so > > > > > > > > > > > the task will retry indefinitely at up to 10 Hz (10 > > > > > > > > > > > calls/second), you > > > > > > > > > > > can specify how often a task is called. > > > > > > > > > > > > Robin > > > > > > > > > > > > On Oct 27, 9:53 pm, mdipierro <mdipie...@cs.depaul.edu> > > > > > > > > > > > wrote: > > > > > > > > > > > > > How long does it stay cached? > > > > > > > > > > > > > On Oct 27, 9:26 pm, Thadeus Burgess > > > > > > > > > > > > <thade...@thadeusb.com> wrote: > > > > > > > > > > > > > > run a cron on your local computer to urllib the page > > > > > > > > > > > > > ? :) All I can come up > > > > > > > > > > > > > with. It is just one of the ways GAE works, when a > > > > > > > > > > > > > page is no longer needed, > > > > > > > > > > > > > it kills the processes until their needed again. Had > > > > > > > > > > > > > this problem with > > > > > > > > > > > > > dreamhost before they supported Passenger > > > > > > > > > > > > > > -Thadeus > > > > > > > > > > > > > > On Tue, Oct 27, 2009 at 6:57 PM, mdipierro > > > > > > > > > > > > > <mdipie...@cs.depaul.edu> wrote: > > > > > > > > > > > > > > > I do not know. > > > > > > > > > > > > > > > On Oct 27, 6:51 pm, Richard <richar...@gmail.com> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > is it possible to get web2py on GAE as responsive > > > > > > > > > > > > > > > as Slicehost? (See > > > > > > > > > > > > > > > Thadeus' app: surrenderthebooty.thadeusb.com) > > > > > > > > > > > > > > > > On Oct 26, 4:42 pm, Richard <richar...@gmail.com> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I noticed my GAE app is slow for the first > > > > > > > > > > > > > > > > webpage I load, but snappy > > > > > > > > > > > > > > > > for subsequent webpages. > > > > > > > > > > > > > > > > So it seems there is a caching issue, which has > > > > > > > > > > > > > > > > been discussed a few > > > > > > > > > > > > > > > > times (courtesy of Google site search): > > > > > > > > > > > > > >http://groups.google.com/group/web2py/browse_thread/thread/329388bec9.... > > > > > > > > > > > > > > .. > > > > > > > > > > > > > > > > > Is there a way to keep the web2py structures > > > > > > > > > > > > > > > > cached? Perhaps a CRON > > > > > > > > > > > > > > > > job to load a page every so often? > > > > > > > > > > > > > > > > > And are there other ways to tune apps on GAE, > > > > > > > > > > > > > > > > such as setting > > > > > > > > > > > > > > > > "migrate=False"? > > > > > > > > > > > > > > > > > thanks, > > > > > > > > > > > > > > > > Richard --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~----------~----~----~----~------~----~------~--~---
