I agree that the kind of queries that are repeated tend to be light weight ones. The only reason I started looking at this was due to some higher than average latency between servers in our hosting environment.
When you're spending an extra 10m per query, even the light ones count. http://bitkickers.blogspot.com/2011/10/django-performance-latency-kills.html -Chase On Nov 8, 11:26 am, Tom Evans <tevans...@googlemail.com> wrote: > On Tue, Nov 8, 2011 at 4:07 PM, Chase <chase.seib...@gmail.com> wrote: > > Has anyone else ever been surprised by the number of duplicate > > database queries when looking at the Django debug toolbar output? > > > In my application, we have models for Users, Profiles and Companies. > > Many of our methods take a user as a parameter, and then look up the > > related profile and/or company. It's very common for a single page > > request to end up performing the same profile or company query a dozen > > times. > > > Of course, the code could be refactored to pass these objects around > > as needed instead of looking them up again. In fact, I have already > > started using a middleware that populates these as attributes on the > > request object. But it got me thinking, is this a common pattern among > > Django apps? If so, I wanted to propose a solution. > > > What if by default query sets were cached for the duration of that > > request? For example, you could perform a User.objects.get(id=123) > > twice inside a single request/response cycle, and the second time it > > would be retrieved from a per-request cache in memory. > > > Obviously this could break existing code, so it would need to be > > optional. But I would think that in almost all cases, web applications > > behave consistently with this enabled. In the cases where you really > > do want a fresh query, there could be a new chained query set method > > like .refresh() or .live(). > > > I'm interested on other people's estimates of how many database > > queries this would save them on their real world projects. I'm > > estimating 20% for mine. > > A similar query came up earlier this week on django-users@ > > http://groups.google.com/group/django-users/browse_thread/thread/bd8e... > > Slight difference is he wanted to cache at the app level, rather than > per-request. > > At $JOB we did look at this sort of optimization in our in-house C++ > web application library. We had ~50 identical queries looking up the > user on some requests - layer after layer of indirection, I'm mostly > glad we are mainly using Django now (still miss static validation of > code via compilation!). > > We spent a couple of weeks trying to eradicate the duplicate queries, > and did quite well, but after re-benchmarking the app, we barely > noticed any difference. > The most important factors in our DB performance turned out to be > expensive queries and writes, where as these repeated queries were all > very cheap to perform, and were almost always served out of the > database's query cache. > > Optimising these queries away in our case saved us only milliseconds > in performance, and no noticeable improvement to the end user. Every > situation is different though! > > Cheers > > Tom -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.