On Jan 30, 2008 6:01 PM, Mark Green <[EMAIL PROTECTED]> wrote: > Ahem, there's a huge difference between being confronted with > a spinner/progress bar or an error page. The former speaks > "Please wait", the latter speaks "Try again".
OK, so let's break this down. There are two potential cases where you run up against your database's concurrent connection limit: 1. Your average traffic level involves more concurrent connections than your database permits. 2. Your average traffic level is within the number of concurrent connections your database permits, but you are experiencing a temporary spike above that level. In case (1), the odds of a timeout/retry scheme yielding success within the time the average user is prepared to wait are low; your database is simply swamped, and the only options are to increase available resources (in the form of database connections) or refuse to serve some number of requests. So case (2) is the only one worth discussing as a target for a timeout/retry scheme. In this case, you are (apparently) asking for Django to provide three things: 1. A mechanism for specifying a set number of database connections which Django will persistently maintain across request/response boundaries. 2. A mechanism for specifying how long Django should wait, when attempting to obtain a connection, before timing out and concluding that no connection will be obtained. 3. A mechanism for specifying how many times, after failing to obtain a connection, Django should re-try the attempt. How, then, would we apply these hypothetical configuration directives to the situation of a traffic spike? There are two possibilities: 1. Set these directives in advance and have them be a permanent part of the running application's configuration. 2. Avoid setting these directives until a spike is imminent (difficult to do) or in progress, and leave them only so long as the spike persists. In case (1) you are flat-out wasting resources and complicating Django's operation, by holding resources which are not used and mandating more complex logic for obtaining those resources. In nearly all cases this is a bad trade-off to make. In case (2) the first option really isn't possible, because you can't reliably predict traffic spikes in advance. This leaves the second option, which requires you to be constantly watching the number of database connections in use and involves shutting down your application temporarily in order to insert the necessary configuration directives. It is also unlikely that you will be able to do so before at least some users have received error pages. So you must either waste resources, or accept increased monitoring overhead and the inevitability that some requests will not receive successful responses. Add to this the following disadvantages: * More complex configuration of Django (and hence more potential for configuration error). * More complex code base for Django (and hence more operating overhead and more potential bugs). * Loss of flexibility in that the application layer must now possess more information about the database layer. * Loss of flexibility in that Django must now maintain persistent resources between requests, and have some mechanism for equitably distributing them across arbitrarily-many server processes on one or more instances of the HTTP daemon on one or more virtual or physical machines. Given all of this, I don't see this level of configuration in the application layer as a worthwhile trade-off to make. You've mentioned JDBC, which points to one potential conflict between your expectations and the reality of Django's design: Java-based web applications typically run threaded within a single Java-based server instance which is able to contain persist resources across requests and manage them equitably across all threads. Django, on the other hand, typically runs multi-*process* behind or embedded in an HTTP daemon and shares no information across the boundaries of those processes, and deliberately seeks to minimize the number of things which must be persisted across the boundaries of request/response cycles. This is a fundamentally different architecture, and so decreases the portability of assumptions from a Java-style background. In particular, there is no master Django process which can create and maintain a resource pool and allocate resources from it to other Django processes (especially since processes may be running behind/embedded in other instances of the HTTP daemon on the same machine, or on remote machines). Which brings up yet another disadvantage: * The configuration of the number of connections Django should maintain, and the timeout/retry directives, must be calculated with the assumption that they will be applied by each and every process independently and without knowledge of any of the others. At that point even the calculations involved in determining the correct settings are probably impossible, since common server arrangements can and will increase or decrease the number of running processes on the fly; calculations for Apache/mod_python, for example, must be able to equitably and reliably manage your resources for any number of independent processes "n" such that StartServers <= n <= MaxClients, and must take into account the turnover of processes which reach MaxRequestsPerChild. Throw in load-balancing among multiple HTTP daemons, and the complexity becomes mind-boggling. Based on this, it's even more clear that attempting to do this at the application layer is a bad idea. Since you apparently place an extremely high value on availability of a timeout/retry feature, I recommend seeking out an external connection manager which offers that feature; you will gain all the benefits of that feature while retaining the simplicity and flexibility of Django's configuration and shared-nothing architecture. -- "Bureaucrat Conrad, you are technically correct -- the best kind of correct." --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
