On Tue, Dec 22, 2009 at 8:50 AM, Javier Guerra <jav...@guerrag.com> wrote:
> On Tue, Dec 22, 2009 at 8:44 AM, Michael <newmani...@gmail.com> wrote: > > Nginx can also create static files from dynamically served pages, so you > > could serve pages directly from disk. > > this is one of the best ways to do it. you could setup a > mostly-complete static copy of your site, and make nginx call the > dynamic one in it's 404 handler to 'fill the voids'. some benchmarks > show nginx static file handling roughly as fast as going to memcached. > > another point, maybe your current solution isn't helping with the > appropriate Etag, if-modified-since, and similar headers. these are > very important to let other machines in the way cache your content, > and the spiders _should_ use them to know when not to re-read and > re-index it. > > of course, if you set the static copy as a 'front layer', you should > get those for free. > > This actually depends on IO. This is actually the last resort on a heavily loaded server. Remember the whole point of a database is to optimize a large number of calls to disk. Memcached is most certainly the best way to do this and no matter what, you should implement it. Only add additional complications if this still isn't working. Your point about Etags and other headers is a good one, however they don't come for free in static files, you will still need to configure those to best suit your needs in you server config. That said, if you are getting thousands of bot reads, my bet is you aren't getting the typical Google and Yahoo bots. Perhaps you should look into blocking the bots that are causing you so much pain. Bots shouldn't be causing server load. -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-us...@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
