Avid Fan <m...@privacy.net> wrote: > Jonathan Gardner wrote: > >> >> I see it as a sign of maturity with sufficiently scaled software that >> they no longer use an SQL database to manage their data. At some point >> in the project's lifetime, the data is understood well enough that the >> general nature of the SQL database is unnecessary. >> > > I am really struggling to understand this concept. > > Is it the normalised table structure that is in question or the query > language? > > Could you give some sort of example of where SQL would not be the way to > go. The only things I can think of a simple flat file databases.
Probably one of the best known large non-sql databases is Google's bigtable. Xah Lee of course dismissed this as he decided to write how bad non-sql databases are without actually looking at the prime example. If you look at some of the uses of bigtable you may begin to understand the tradeoffs that are made with sql. When you use bigtable you have records with fields, and you have indices, but there are limitations on the kinds of queries you can perform: in particular you cannot do joins, but more subtly there is no guarantee that the index is up to date (so you might miss recent updates or even get data back from a query when the data no longer matches the query). By sacrificing some of SQL's power, Google get big benefits: namely updating data is a much more localised option. Instead of an update having to lock the indices while they are updated, updates to different records can happen simultaneously possibly on servers on the opposite sides of the world. You can have many, many servers all using the same data although they may not have identical or completely consistent views of that data. Bigtable impacts on how you store the data: for example you need to avoid reducing data to normal form (no joins!), its much better and cheaper just to store all the data you need directly in each record. Also aggregate values need to be at least partly pre-computed and stored in the database. Boiling this down to a concrete example, imagine you wanted to implement a system like twitter. Think carefully about how you'd handle a sufficiently high rate of new tweets reliably with a sql database. Now think how you'd do the same thing with bigtable: most tweets don't interact, so it becomes much easier to see how the load is spread across the servers: each user has the data relevant to them stored near the server they are using and index changes propagate gradually to the rest of the system. -- Duncan Booth http://kupuguy.blogspot.com -- http://mail.python.org/mailman/listinfo/python-list
