On Thu, 2008-10-02 at 03:11 -0700, Mike Barton wrote: > I'm having trouble getting data out of my database through > activerecord when the amount data returned by the sql is very large. <snip> > http://gist.github.com/14333 >
Hi Mike, All kinds of things could be going wrong here. You really should be grabbing these in batches from the database, rather than all at once. Use the limit option to grab a thousand at a time or something. If you're using an ACID database, do the whole thing in a transaction to ensure records inserting mid batch don't mess things up. Here is an extract from a rake task I wrote to do batched conversion of Wordpress comments. I'm using some Rails models, but you can see what is going on: http://gist.github.com/14336 I'm sure there is a better way to do this tbh, but it works ok. Really though, if it's not a complex query you're running and it's a very large dataset, then consider using mysqldump - it'll be faster and use less ram than Ruby, and I'm sure it supports csv output. John. -- http://www.brightbox.co.uk - UK Ruby on Rails hosting http://johnleach.co.uk --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "NWRUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/nwrug-members?hl=en -~----------~----~----~----~------~----~------~--~---
