next time post something close to reality, or reproducing it leads to nowhere. Speed and memory-wise (not leak, but still...), use a straight update() over update_record.
BTW, what I still don't understand is you comparing update_or_insert to row.update_record.... they do very different things. If you don't need the update_or_insert() functionality - as it seems by the query you posted - just go for update(), it's a lot faster. PS: don't measure times with loads like this on SQlite starting to tune your code (e.g. using cache.ram to store a 25k resultset) if you know you'll end up with another backend: you are wasting time (assuming you're passing to a serious db backend). PS2: this would be the perfect candidate (memory-wise) to have web2py's DAL using iterators instead of doing a fetchall() and stuffing all in the Rows object :P On Wednesday, May 22, 2013 12:34:18 AM UTC+2, Simon Ashley wrote: > > Thanks Niphlod > > Yep, sorry for the typos (@ 4am the brain doesn't function correctly). > Main point was to describe the 2 methods. > (*update_or_insert *and *row.update_record*) > Actual code would have been too heavy. > > Routines were tested with limitby=(0,1000) in the selects. > Caching only involved in the *non update* selects (i.e. the reference > table). > Improves performance by 50% or more. > > Will attempt to break it down into smaller chucks, to see if memory > releases. > > Routines would be eventually handled by a/the scheduler, and while this is > basically a one off, there will be others requiring the same or greater > heavy updates. > While will move off SQLite, using it at present for initial data prep. > Wanted to use the DAL as much as possible to make stuff as portable as > possible in the meantime. > > Also using multiple databases to mitigate locking issues. > > Will report back. > > -- --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
