How would you go about building such a site? Is LAMP really the right approach?
Two major problems I've noticed, don't know if they are universal, but they sure hurt the performance:
1) Some sites have not put any thought into caching - i.e. the application server is serving up images or every single page is dynamically generated even though all (or most) of it is static such that most of the requests just aren't cacheable.
2) Because a database is there, it gets used, even when it shouldn't, and it often gets used poorly - bad or no connection pooling, many trips to the database for each page generated, no table indices, bizarro database schemas.
Overall I'd say my first guess is that too much is being generated on the fly, often because it's just easier not to worry about cacheability, but a good web cache can provide orders of magnitude improvement in performance, so it's worth some extra thought.
One project we had involved the users navigating through a big set of data, narrowing down the set by making choices about different variables. At any point it would display the choices that had been made, the remaining choices, and the top few hits in the data set. We initially thought all the pages would have to be dynamically generated, but then we realized that each page really represented a distinct and finite state, so we went ahead and built the site with Zope + Postgres, but made it so that the URLs were input to Zope and told what state to generate.
The upshot of all this is that we then just ran a web spider against Zope any time the data changed (once a week or so), and so the site ended up "feeling" pretty dynamic to a user but pretty much everything came straight out of a cache.
-Dave -- http://mail.python.org/mailman/listinfo/python-list
