On 3 september 2013 15:25:42 UTC+2 Russell Keith-Magee wrote: > > > On Tue, Sep 3, 2013 at 9:50 PM, <mailb...@gmail.com <javascript:>> wrote: > >> In my Django application I need to schedule some tasks to run in future. >> Celery is not an option here because some crucial options - mainly, the >> ability to revoke() a task - are only available with fully-featured MQ >> backend and I only have a limited shared hosting where I can't install this >> kind of software. After some reading, I've found that supposedly "right" >> way to do that in Python is to spawn a subprocess and call sleep(). >> However, spawning a process does have an overhead, especially considering a >> rather simple nature of tasks I want to defer (retrieval and conditional >> deletion of one object). >> >> Then I decided to use threads instead. They have little to no overhead, >> share memory and Python has a nice little threading.Timer class which is >> exactly what I need. My code looks like this: >> >> http://pastebin.com/K6RukAX9 >> >> Here's when my questions come in. I admit my knowledge of threading is >> somewhat low and Python operates on high-level too, so nuances are not >> easily noticeable. From what I've learned, threads (can?) share memory with >> main thread/process. My primary concern here is that, as far as I >> understand, method=prefork (the default) creates a pool of processes >> waiting for work. It seems possible that it may impact my scheduler >> somehow, i.e. if process that spawned it sits idle (sleeps?) when countdown >> ends. >> >> On the other hand, threaded mode is supposedly more efficient because, >> being free from overhead of process forking, it can create child threads at >> will. In my mind this raises a similar concern to above: what if framework >> decides to destroy a thread that happens to be the one where Timer has been >> previously spawned? >> >> It is entirely possible that my concerns are completely baseless and both >> methods have nothing to do with my scheduler. I'm asking for input from >> smarter people than me :) >> > > Lets back up a bit here. > > Web servers aren't designed to do the sort of thing you're describing. Web > requests are not designed to be long lived, and web servers are tuned with > this in mind. If you try to make them long lived, you're going to hit all > sorts of problems with thread/process lifecycle management. > > The solution here isn't to try and make a web server do something it > wasn't designed to do. The solution is to work out how to get a task queue > working on your hardware setup. > > You say you can't install a fully-fledged MQ service. Ok - that's fine -- > but don't throw out the baby with the bathwater. There are other ways to > back Celery -- if need be, you can use a database as your data store. > > Or, you could consider using an entirely different task queue. If your > service provider will give you access to Redis, look into RQ. It's > lightweight, but > > Or, if neither of those are viable, you can do a "ghetto queue" -- a > combination of cron and a database can be used to implement task queue-like > behaviour, without any need for celery. > > It's difficult to give more specific advice without knowing the specifics > of your problem domain and your deployment options -- but I implore you -- > don't keep down the path you're on. That way leads to madness :-) > > Yours, > Russ Magee %-) >
I get the overall message. I would use threads because that method looks simple enough, but if it isn't - I won't insist on it. Regarding Celery, I was specifically referring to revoke() method, it's absolutely crucial for me to be able to revoke a scheduled task. Internet says it's only supported with AMQP and Redis ( http://stackoverflow.com/questions/7415657/celery-tasks-dont-get-revoked ) though now that I look at it, docs also claim compatibility with MongoDB. However, I didn't managed to get it to work on such a setup and complete lack of error messages didn't help (the call just fails silently). I don't have Redis on my host, but I guess a free 20 MB instance on a cloud will be enough for my needs, so I'll settle with one of these small-ish queuers. On a side note, I am a bit surprised that every sane method to do scheduled task execution depends on MQs or Redis. Those are fairly new tools while the problem definitely didn't appear yesterday. What were people using back in PHP days, cron all the way? Thank you very much for your remarks :) -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. For more options, visit https://groups.google.com/groups/opt_out.
