I finally made it, at least for now.
There is a database table where all the processes are stored, both to
be processed and completed ones.
db.define_table('work',
SQLField('name','string'),
SQLField('status','string'),
SQLField('priority','integer'),
SQLField('input','text'),
SQLField('results','text'),
SQLField
('date_submitted','datetime',default=datetime.datetime.now()),
SQLField('date_completed','datetime'),
)
than there is a cron that each minute look a the table and if there
are jobs to process runs the first one.
def process_job():
queue=db(db.work.status=='to be done').select
(orderby=db.work.priority)
if queue:
job=queue[0]
#do stuff like:
input_string=job.input
output_string='Procesed by cron worker : '+str(input_string)
#submit results
db.work[job.id]=dict
(status='completed',results=output_string,date_completed=datetime.datetime.now
())
db.commit()
return
the problem with this is that I have to wait one minute for each table
lookup, and that I'm not sure it will work correctly under WSGI, since
I do not want to rely on system cron.
The fastest improvement would be to run all the process in queue each
time the controller is called by the cron,
one by one, but I'm still convinced that using cron is not the best
way to do this, even if it works.
Am I the only one with this kind of issues? If not maybe I can post
the "solution" to AlterEgo...
On Jul 10, 3:19 am, mdipierro <mdipie...@cs.depaul.edu> wrote:
> You can use the databae as long the objects are serializable
>
> On Jul 9, 6:00 pm, kralin <andrea.pierle...@gmail.com> wrote:
>
> > thanks massimo, very elegant...
> > I have to check for the lock thing, otherwise a list is more tha
> > sufficient for thequeue:
>
> >queue=cache.ram('queue',lambda:[],10**10)
> >queue.append(o)
> >queue.pop(0)
>
> > it should be also very easy to use a priority schedule by appending
> > [int(priority), task] to thequeuelist.
>
> > however I still need to cope with the guy that's going to do the heavy
> > job.
> > Is it possibile to set a Worker in web2py that is able to read the
> > input from thequeueand put the results in a database, or do I have
> > to launch a cron each minute to check for thequeue?
>
> > On 10 Lug, 00:00, mdipierro <mdipie...@cs.depaul.edu> wrote:
>
> > > You can do something like
>
> > > classQueue:
> > > forever = 10**10
> > > def __init__(self):
> > > import thread
> > > self.q=[]
> > > self.lock=thread.allocate_lock()
> > > def enque(self,o):
> > > self.lock.acquire()
> > > self.q.append(o)
> > > self.lock.release()
> > > def dequeue(self):
> > > self.lock.acquire()
> > > o = self.q[0]
> > > del q[0]
> > > self.lock.release()
> > > return o
> > > def __len__(self):
> > > self.lock.acquire()
> > > ell=len(self.q)
> > > self.lock.release()
> > > return ell
>
> > >queue=cache.ram('queue',lambda:Queue,Queue.forever)
> > >queue.enqueue('object')
> > > print len(queue)
> > > o =queue.dequeue()
>
> > > If you define it in a model it will be visible everywhere, including
> > > cron scripts. I am not sure if the lock is really necessary (because
> > > of the global interpreter lock) but it is safe.
>
> > >queue.append(
>
> > > On Jul 9, 4:07 pm, kralin <andrea.pierle...@gmail.com> wrote:
>
> > > > Hi All,
> > > > I'm diving into web2py that at the moment gives me the power and the
> > > > time do a lot of cool things. so thanks a lot to you guys.
> > > > I was wondering if any of you ever had the need to set aqueuefor a
> > > > heavy process.
> > > > Let's suppose I've got one heavy process that takes 2 mins to run on a
> > > > singol-processor machine.
> > > > now if 10 user submit the process at (more or less) the same time, is
> > > > there a way to schedule the process based on the first coming, first
> > > > to have response rule? each of this user should be able to see their
> > > > result once completed or see something like"wait, 8 jobs still in
> > > >queue"
>
> > > > I know it can be done easily with external job sheduler, but it will
> > > > require to use a separate process, not beeing portable, and a little
> > > > bit more headhache...
>
> > > > a quick and dirty solution wuld be to run a cron each, let's say, 2
> > > > mins and if a process isqueue, execute it, but I feel that can be
> > > > done in a much more elegant way.
>
> > > > what do youthink?
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~----------~----~----~----~------~----~------~--~---
