The reason I started down this path is that the program is grinding to a halt (my PC as well) and the only way to get it going again is: >>>import gc >>>gc.collect() 2524104
Memory used by the process is around 500MB, I have posted about this problem before on this newsgroup but I didn't get a resolution. So I was trying to separate the logic into different files and finish each one with gc.collect(). Sorry I didn't post the original problem as well but I really just wanted the thing to work and not cloud the issue again. I have implemented your approach and the code is much cleaner (except for another problem that I have posted to sqlAlchemy). It doesn't solve my original problem with the memory, but I really need SOMETHING to work. So I will keep plodding away and then maybe move the XML processing to lxml and see if that resolves the memory problem. "John Machin" <sjmac...@lexicon.net> wrote in message news:3215638c-2408-4cf6-9321-f85f560e6...@u39g2000pru.googlegroups.com... On Apr 27, 3:31 pm, "Paul Hemans" <dar...@nowhere.com> wrote: > Thanks John, I understand where you are coming from and will try and > digest > it all. One problem though that I didn't mention in my original posting > was > that the replication may only require updating one or more fields, that is > a > problem with a generating a single SQL statement to cover all requests. OK so you need TWO programs -- one tailored for initial loading of a table, the other for subsequent updates. Still O(1) :-) OR you could still do it in one, by using the prepared whole-row SQL statement if appropriate, or building an SQL statement on the fly if it's an update. > I am > having a look at eval and exec WRONG WAY GO BACK > to see if they will give me a little more > flexibility in runtime generation of the code. And another thing that you didn't mention was that you are using SQLAlchemy -- perhaps you might like to ask your question on that package's forum ... including a few more facts than heretofore :-) HTH John -- http://mail.python.org/mailman/listinfo/python-list
