On 2013-03-29 21:19, Roy Smith wrote: > We're doing it all in one transaction, on purpose. We start with > an initial dump, then get updates about once a day. We want to > make sure that the updates either complete without errors, or back > out cleanly. If we ever had a partial daily update, the result > would be a mess.
Having had to do some similarly-sized bulk data loads (in my case, MSSqlServer at $JOB) couple other ideas occur to me: 1) I believe MySQL has a side-loading function (I'd have to go digging for it; a quick google suggests a "LOAD DATA INFILE" statement[1]) that allows you to load data from an external file such as an XML or CSV file 2) Load into a temp table in testable batches, then do some sort of batch insert into your main table. Again, a quick google suggest the "INSERT ... SELECT" syntax[2] -tkc [1] http://dev.mysql.com/doc/refman/5.1/en/load-data.html [2] http://dev.mysql.com/doc/refman/5.0/en/insert-select.html -- http://mail.python.org/mailman/listinfo/python-list
