On Jan 10, 11:05 am, lenin <lenin....@gmail.com> wrote:
> The server MPM is prefork.
>
> I had mod_wsgi in daemon mode I think, with this lines:
>
> WSGIDaemonProcess user_name user=user_name group=user_name
> WSGIProcessGroup user_name
> WSGIScriptAlias / /path/to/django.wsgi
Which is one process with 15 threads. That means at most 15 requests
can be handled at the same time.
The problem is that with what you are doing, is it is likely that the
requests take a long time if they doing video conversion. Thus if you
get 15 requests active at one time, other requests will be queued and
will appear to take a lot longer.
How long does a request which involves conversion normally take?
Knowing this is critical to determining how many processes/threads you
need to handle enough concurrent requests based on your load.
The next problem though is if you are doing the video conversion
within the Django process itself, then allowing more concurrent
requests and thus conversions, means that the process size will
balloon out if this process takes a lot of memory for each request.
So, are you doing conversion within the Django process and how much
memory does the process of conversion use.
You thus perhaps should not be doing video conversion within the same
process, but executing a separate application to perform the
conversion. Even then, you may need a queueing system or otherwise
some way of restricting how many conversions can occur at a time,
because even though performed by a separate application, it may still
take up a lot of transient memory while running.
Also, from what I understand Django's abilities to serve up static
files or streamed data, isn't that excellent. In particular, if
hosting with WSGI capable server, it doesn't use the wsgi.file_wrapper
extension. This extension ensures that files are streamed in blocks
and the file is not read all into memory first in order to be able to
send it. Also, if operating system provides it, Apache will use
sendfile() or memory mapping techniques to return data in a much more
efficient way. So, how exactly are you returning the converted data
for Django to respond with?
> That was at the moment of the first message, but now I have commented
> out the WSGIDaemonProcess and WSGIProcessGroup directives, and what
> happens is that the memory fills up until the server crashes, It runs
> out of RAM and swap and it just dies. But until it crashes it is very
> responsive.
Which is not surprising as prefork means that instead of a restricted
number of copies of Apache, based on your configuration you could have
up to 70. That is 70 copies of Django, plus memory from conversions.
This would quickly cripple a box without adequate memory. Just the
startup cost of the extra processes when Apache deems they need to be
created would cause the load average of the box to shoot up.
> I have tried putting processes=5 threads=1 maximum-
> requests=200 in the daemon configuration but it doesn't help.
Which is only 5 concurrent requests, which is worse than the 15 you
had before. You also have 5 potentially fat Django processes rather
than 1.
Graham
> On apache I have this configuration
> KeepAlive Off
> <IfModule mpm_prefork_module>
> StartServers 5
> MinSpareServers 5
> MaxSpareServers 10
> MaxClients 70
> MaxRequestsPerChild 150
> </IfModule>
>
> On nginx I have
> worker_processes 10;
> events {
> worker_connections 4096;
> use epoll;}
>
> location / {
> proxy_pass http://localhost:8080;
> proxy_redirect default;
> proxy_set_header Host $host;
> proxy_set_header X-Real-IP $remote_addr;
> proxy_set_header X-Forwarded-For
> $proxy_add_x_forwarded_for;
>
> client_max_body_size 10m;
> client_body_buffer_size 128k;
>
> proxy_connect_timeout 1000;
> proxy_send_timeout 1000;
> proxy_read_timeout 1000;
>
> proxy_buffer_size 4k;
> proxy_buffers 4 32k;
> proxy_busy_buffers_size 64k;
> proxy_temp_file_write_size 128k;
> }
> I only include the bits I think are relevant.
>
> The site is kickyoutube.com, you can download any youtube video in
> several formats by appending kick to the youtube video url. Although
> at this moment I have disabled the formats that need conversion.
>
> I'm using mysql for the database and it is on the same server, but it
> really doesn't do a lot of queries.
>
> I have memcached but I only cache the list of latest videos at the
> homepage.
>
> So it really is the video conversion what takes up all the memory,
> because when I disable it everything seems normal.
>
> On 9 ene, 15:06, Graham Dumpleton <graham.dumple...@gmail.com> wrote:
>
> > On Jan 10, 4:21 am, lenin <lenin....@gmail.com> wrote:
>
> > > Hello,
>
> > > I would like to know if you could help me solve an issue with my
> > > server setup. I have a Django application running with mod_wsgi on
> > > apache with nginx as frontend, and it got very popular so the server
> > > gets more than 10000 requests a day. The app does some video
> > > conversion with ffmpeg and mencoder, using the commands module to call
> > > the commands.
>
> > > The server has 1 GB of RAM and an Intel Xeon 3060 processor.
>
> > > At first the server would crash and I needed to reboot it, and the
> > > problem seemed to be that the system ran out of swap, so I thought it
> > > was a memory issue, did some modifications and now memory usage is
> > > stable.
>
> > > But now the problem is that the site takes too much to load some
> > > times, I think because of all the simultaneous requests. I was
> > > wondering what would be the optimal apache+mod_wsgi+nginx
> > > configuration for a setup like this.
>
> > > Any help will be very much appreciated.
>
> > What MPM is Apache running with, prefork or worker MPM? Run 'httpd -V'
> > if you don't know.
>
> > What is the mod_wsgi configuration you are using for setting up your
> > Django instance? Are you using embedded mode or daemon mode? Post
> > actual WSGI configuration bits, do not describe it.
>
> > BTW, 10000 requests a day is not much. ;-)
>
> > Graham
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django users" group.
To post to this group, send email to django-users@googlegroups.com
To unsubscribe from this group, send email to
django-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/django-users?hl=en
-~----------~----~----~----~------~----~------~--~---
