The queue solution that Malcolm is talking about requires 3 types of
processes as far as I can tell:
- the fcgi/django processes that will be pushing jobs onto the
queue
- a manager process that is started independently from the fcgi
processes and is responsible for maintaining the shared job queue
- one or more a worker processes that are started by the manager
in response to jobs it pulls off the queue. The manager could do the
offline processing itself, but if you have a lot of cpus you'd be
missing out on some performance.
A simpler solution is to have a cron job run often and either check a
directory or a database message table for new jobs to run. I think
using the DB to communicate is actually a pretty nice option as
keeping a long running manager process alive and the queue uncorrupted
sounds trickier than it's worth (just look at some of the caveats to
using queues in the multiprocessing module docs)
I responded to a similar thread the other day, seems to be a popular
problem:
http://groups.google.com/group/django-users/browse_thread/thread/570f146785c4596d/b0e7338230d7b35e?lnk=gst&q=+Processing+a+file...#b0e7338230d7b35e
I've only used the above in a low volume intranet site, so if someone
else has more experience I'd like to hear from them too. Especially
if they have experience pushing the mp module around.
-Jacob
On Feb 15, 4:14 pm, Gregor Müllegger <gre...@muellegger.de> wrote:
> Thank you for your answer Malcolm.
>
> Though what would you suggest as an solution? Shall i use the
> multiprocessing package as a replacement for the threaded module (i've
> already tried that, see below) or is it better to execute a management
> command with Popen('python manage.py process_video_queue') from the
> subprocess module?
>
> I already tried it with multiprocessing, but it hasn't worked - just
> like threading. It doesn't get executed.
>
> Thanks again,
> Gregor
>
> 2009/2/15 Malcolm Tredinnick <malc...@pointy-stick.com>:
>
>
>
> > On Sat, 2009-02-14 at 11:31 -0800, Gregor Müllegger wrote:
> >> Hi djangonauts,
>
> >> at the moment i try to setup a youtube-like site. Users can upload
> >> videos which will be converted with ffmpeg to the flv format. The
> >> convertion process is fired up in a view. Though i use the "threaded"
> >> module to not interrupt the view for sending back the response.
>
> > This comes up quite often and the standard answer is that this isn't the
> > right approach to solving the problem. You are tying the background
> > processing to the lifecycle of the fastcgi process. Web processes can
> > stop and start for various reasons (including periodically being killed
> > after a number of requests to rest the memory base).
>
> > Better to use a queue-like setup where you make a record of any
> > conversions that need to be done and have another process (or processes)
> > that go through the queue and convert things. The lifecycle of the
> > latter processes aren't then tied to the web-stack lifecycles.
>
> > Regards,
> > Malcolm
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---
