On Fri, Dec 21, 2012 at 2:20 AM, Johannes Bauer <dfnsonfsdu...@gmx.de> wrote: > Hmm, but this: > > result = cursor.fetchone() > yield result > > Works nicely -- only the fetchmany() makes the example break.
Okay, now it's sounding specific to sqlite. I'll bow out. :) > >> Would it spoil your performance improvements to do all the fetchmany >> calls before yielding anything? > > Well this would effectively then be a fetchall() call -- this is > problematic since the source data is LARGE (spekaing of gigabytes of > data here). That would be a "yes", then. Scratch that! >> Alternatively, can you separate the >> two by opening a separate database connection for the foo-reading (so >> it isn't affected by the commit)? > > At that point in the code I don't actually have a filename anymore, > merely the connection. But shouldn't the cursor actually be the > "correct" solution? I.e. in theory, should the example work at all or am > I thinking wrong? You say "db.commit()", not "cur2.commit()", so I don't see that a cursor would un-break what part-way commits is breaking. > Because if I'm approaching this from the wrong angle, I'll have no > choice but to change all that code to open separate connections to the > same file (something that currently are no provisions for). Is that an sqlite limitation, or just one of your code? I poked around at the sqlite3 docs, but didn't find any obvious "clone" option on the connection, nor a way to retrieve the file name. That would have been fairly convenient. Oh well. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
