On Tue, Sep 3, 2013 at 9:50 PM, <mailbox....@gmail.com> wrote: > In my Django application I need to schedule some tasks to run in future. > Celery is not an option here because some crucial options - mainly, the > ability to revoke() a task - are only available with fully-featured MQ > backend and I only have a limited shared hosting where I can't install this > kind of software. After some reading, I've found that supposedly "right" > way to do that in Python is to spawn a subprocess and call sleep(). > However, spawning a process does have an overhead, especially considering a > rather simple nature of tasks I want to defer (retrieval and conditional > deletion of one object). > > Then I decided to use threads instead. They have little to no overhead, > share memory and Python has a nice little threading.Timer class which is > exactly what I need. My code looks like this: > > http://pastebin.com/K6RukAX9 > > Here's when my questions come in. I admit my knowledge of threading is > somewhat low and Python operates on high-level too, so nuances are not > easily noticeable. From what I've learned, threads (can?) share memory with > main thread/process. My primary concern here is that, as far as I > understand, method=prefork (the default) creates a pool of processes > waiting for work. It seems possible that it may impact my scheduler > somehow, i.e. if process that spawned it sits idle (sleeps?) when countdown > ends. > > On the other hand, threaded mode is supposedly more efficient because, > being free from overhead of process forking, it can create child threads at > will. In my mind this raises a similar concern to above: what if framework > decides to destroy a thread that happens to be the one where Timer has been > previously spawned? > > It is entirely possible that my concerns are completely baseless and both > methods have nothing to do with my scheduler. I'm asking for input from > smarter people than me :) >
Lets back up a bit here. Web servers aren't designed to do the sort of thing you're describing. Web requests are not designed to be long lived, and web servers are tuned with this in mind. If you try to make them long lived, you're going to hit all sorts of problems with thread/process lifecycle management. The solution here isn't to try and make a web server do something it wasn't designed to do. The solution is to work out how to get a task queue working on your hardware setup. You say you can't install a fully-fledged MQ service. Ok - that's fine -- but don't throw out the baby with the bathwater. There are other ways to back Celery -- if need be, you can use a database as your data store. Or, you could consider using an entirely different task queue. If your service provider will give you access to Redis, look into RQ. It's lightweight, but Or, if neither of those are viable, you can do a "ghetto queue" -- a combination of cron and a database can be used to implement task queue-like behaviour, without any need for celery. It's difficult to give more specific advice without knowing the specifics of your problem domain and your deployment options -- but I implore you -- don't keep down the path you're on. That way leads to madness :-) Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users. For more options, visit https://groups.google.com/groups/opt_out.
