Thanks Anssi for both your replies. These db interactions definitely are really beyond the scope of my knowledge, unluckily.
I did try and implement an iterator such as the one that you describe in your other message, based on index column filtering - what I dislike is that it's an additional function getting a qs parameter rather then naturally sitting in the queryset and (thus?) I did not manage to make it very generic... Eg. some querysets don't even contain the key column anymore.. (the outlines of these are in my SO question but ). I'm sure it can be done better, especially if it's part of the queryset rather then an external function that gets an evaluated qs. I still manage to use it both for pagination and for very complex functions that I can't run in the DB. As you say, it would still be nice to see this kind of generic iterator in core, even known the limitations (heck, the standard iterator just exploses the process by memory usage, so it's not that much better :) ). WITH HOLD does not look like a safe solution if you have to explicitly close it (would need a separate thread with timeout maybe?), but I also read some discussions about creating a connection pool and persistent connections, so maybe at some point all these issues will find a common base for a solution.. As usual, I'll eagerly keep reading these posts and try to play with code on my own hoping to be able to contribute at some point.. Stefano On Wednesday, March 27, 2013 10:36:02 AM UTC+1, Anssi Kääriäinen wrote: > > On 03/27/2013 10:53 AM, Stefano Crosta wrote: > > Thanks Aymeric, > > that's true, and I should have put that link too. > > sadly "akaariai" who's the only one who seemed to understand a bit > > about it (and is a core dev) did not seem interested in bringing this > > any forward, so I thought I'd try to raise some interest again! > The problem is that server side cursors will need dedicated API, and > there will be database specific problems when using server side cursors. > On SQLite changes done to rows are visible in the results, on other > databases not. On PostgreSQL you will need to use WITH HOLD cursors if > you want to use server side cursors outside transactions (that is, in > autocommit mode normal server side cursors do not work). When using WITH > HOLD cursors you must close the cursor explicitly or you will have > cursor leak... And on MySQL WITH HOLD cursors aren't available at all, > so you must be in transaction to use server side cursors. > > Oracle seems to be the only core DB that will work without problems. In > fact, using .iterator() on Oracle already works without memory problems. > > Maybe the dedicated API could be adding two new keywords to .iterator(): > server_side_cursor, and with_hold. with_hold=True implies > server_side_cursor=True. If you use with_hold=True you are responsible > for closing the iterator, too. The behaviour of server_side_cursor and > with_hold is database specific - it will be impossible to abstract the > differences away. > > - Anssi > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
