Debian/nginx/systemd deployment: I made scheduler working with help: https://groups.google.com/d/msg/web2py/eHXwines4o0/i3WqDlKjCQAJ and https://groups.google.com/d/msg/web2py/jFWNnz5cl9U/UpBSkxf4_2kJ
Thank you very much Niphlod, Michael M, Brian M Dne čtvrtek 5. května 2016 13:06:04 UTC+2 Mirek Zvolský napsal(a): > > Yes. > I run with scheduler already. It is really nice and great ! > Going away from the ajax solution it was easy and there was almost no > problem. (I have very easy parameters for the task and I return nothing, > just I save into db.) > The result code is cleaner (one task starting call instead of rendering > hidden html element + js reading from it + ajax call + parsing args). > > Maybe my previous mistake (I mean my message here in this thread) will be > helpfull for others to go with scheduler. > > What I need to do now is deployment for the scheduler (on Debian and > nginx). > > PS: > It was fast but important > - find where I can see code errors (in schedulers db tables), > - how to set timeout (in the function call) > > Here is the code example - controller and models/scheduler.py: > def find(): > def onvalidation(form): > form.vars.asked = datetime.datetime.utcnow() > form = SQLFORM(db.question) > if form.process(onvalidation=onvalidation).accepted: > scheduler.queue_task(task_catalogize, > pvars={'question_id': form.vars.id, 'question': > form.vars.question, 'asked': str(form.vars.asked)}, # str to json > serialize datetime > timeout=300) > return dict(form=form) > > import datetime > from gluon.scheduler import Scheduler > def task_catalogize(question_id, question, asked): > asked = datetime.datetime.strptime(asked, '%Y-%m-%d %H:%M:%S.%f') # > deserialize datetime > inserted = some_db_actions(question) > db.question[question_id] = { > 'duration': round((datetime.datetime.utcnow() - > asked).total_seconds(), 0), # same/similar we have in scheduler db > tables > 'inserted': inserted} > db.commit() > scheduler = Scheduler(db) > > > > Dne úterý 3. května 2016 14:21:23 UTC+2 Niphlod napsal(a): >> >> NP: as everything it's not the silver bullet but with the redis >> incarnation I'm sure you can achieve less than 3 second (if you tune >> heartbeat even less than 1 second) from when the task gets queued to when >> it gets processed. >> >> On Tuesday, May 3, 2016 at 12:32:13 PM UTC+2, Mirek Zvolský wrote: >>> >>> Hi, Niphlod. >>> >>> After I have read something about scheduler, >>> I am definitively sorry for my previous notes >>> and I choose web2py scheduler of course. >>> >>> It will be my first use of it (with much older ~3 years web2py app I >>> have used cron only), >>> so it will take some time to learn with scheduler. But it is sure worth >>> to redesign it so. >>> >>> Thanks you are patient with me. >>> Mirek >>> >>> >>> >>> >>> Dne pondělí 2. května 2016 20:35:05 UTC+2 Mirek Zvolský napsal(a): >>>> >>>> You are right. >>>> At this time it works for me via ajax well and I will look carefully >>>> for problems. >>>> If so, I will move to scheduler. >>>> >>>> I see this is exactly what Massimo(?) writes at the bottom of Ajax >>>> chapter of the book. >>>> >>>> PS: about times: >>>> At notebook with mobile connection it takes 20-40s. So it could be >>>> danger. >>>> At cloud server with SSD it takes 2-10s. But this will be my case. And >>>> I feel better when the user can have typical response in 3s instead in 8s. >>>> >>>> >>>> >>>> >>>> >>>> Dne neděle 1. května 2016 22:10:31 UTC+2 Niphlod napsal(a): >>>>> >>>>> the statement "I don't need to use the scheduler, because I want to >>>>> start it as soon as possible" is flaky at best. If your "fetching" varies >>>>> from 2 to 20 seconds and COULD extend further to 60 seconds, waiting a >>>>> few >>>>> seconds for the scheduler to start the process is .... uhm... debatable. >>>>> Of course relying to ajax if your "feching" can be killed in the >>>>> process is the only other way. >>>>> >>>>> On Sunday, May 1, 2016 at 8:09:23 PM UTC+2, Mirek Zvolský wrote: >>>>>> >>>>>> Thanks for info and tips, 6 years later. >>>>>> >>>>>> What I try to do >>>>>> is a form with single input, where user gives a query string >>>>>> and then data about (usually ~300) books will be retrieved via z39 >>>>>> and marc protocol/format, parsed and saved into local database. >>>>>> >>>>>> Of course this will take a time (2? 5? 20? seconds) and I decided >>>>>> not to show the result immediately, >>>>>> but show the same form with possibility to enter the next query + >>>>>> there is a list of pending queries (and their status - via ajax testing >>>>>> every 5 seconds) >>>>>> >>>>>> So my idea was to provide a return from the controller fast and >>>>>> before the return to start a new thread to retrieve/parse/save/commit >>>>>> data. >>>>>> >>>>>> From this discussion I understand that open new thread isn't best >>>>>> idea. >>>>>> I think it could be still possible, because if my new thread could be >>>>>> killed 60s later from the web server together with the original thread - >>>>>> such possibility is not fatal problem for me here. >>>>>> >>>>>> However when (as I read here) this would be a little wild technology, >>>>>> and because other technologies mentioned here: >>>>>> https://en.wikipedia.org/wiki/Comet_(programming) -paragraph >>>>>> Aternatives, are too difficult for me, >>>>>> and because I don't want use a scheduler, because I need to start as >>>>>> soon as possible, >>>>>> >>>>>> I will solve it so, >>>>>> that I will make 2 http accesses from my page: one with submit (will >>>>>> validate/save the query to database) and one with ajax/javascript >>>>>> (onSubmit >>>>>> from the old page or better: onPageLoaded from the next page where I >>>>>> give >>>>>> the query in .html DOM as some hidden value), which will start the z39 >>>>>> protocol/retrieve/parse/save data. >>>>>> This will be much better, because web2py in the ajax call will >>>>>> prepare the db variable with proper db model for me (which otherwise I >>>>>> must >>>>>> handle myselves in the separate thread). >>>>>> Callback from this ajax call should/could be some dummy javascript >>>>>> function, because it is not sure, and not important, if the page still >>>>>> exists when the server job will finish. >>>>>> >>>>>> So, if somebody is interesting and will read this very old thread, >>>>>> maybe this can give him some idea for time consumming actions. >>>>>> And maybe somebody will add other important hints or comments (thanks >>>>>> in advance). >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Dne středa 26. května 2010 0:33:02 UTC+2 Giuseppe Luca Scrofani >>>>>> napsal(a): >>>>>>> >>>>>>> Hi all, as promised I'm here to prove you are patient and nice :) >>>>>>> I' have to make this little app where there is a function that read >>>>>>> the html content of several pages of another website (like a spider) >>>>>>> and if a specified keyword is found the app refresh a page where >>>>>>> there >>>>>>> is the growing list of "match". >>>>>>> Now, the spider part is already coded, is called search(), it uses >>>>>>> twill to log in the target site, read the html of a list of pages, >>>>>>> perform some searching procedures and keep adding the result to a >>>>>>> list. I integrated this in a default.py controller and make a call in >>>>>>> def index(): >>>>>>> This make the index.html page loading for a long time, because now it >>>>>>> have to finish to scan all pages before return all results. >>>>>>> What I want to achieve is to automatically refresh index every 2 >>>>>>> second to keep in touch with what is going on, seeing the list of >>>>>>> match growing in "realtime". Even better, if I can use some sort of >>>>>>> ajax magic to not refresh the entire page... but this is not vital, a >>>>>>> simple page refresh would be sufficient. >>>>>>> Question is: I have to use threading to solve this problem? >>>>>>> Alternative solutions? >>>>>>> I have to made the list of match a global to read it from another >>>>>>> function? It would be simpler if I made it write a text file, adding >>>>>>> a >>>>>>> line for every match and reading it from the index controller? If I >>>>>>> have to use thread it will run on GAE? >>>>>>> >>>>>>> Sorry for the long text and for my bad english :) >>>>>>> >>>>>>> gls >>>>>>> >>>>>>> -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.