Or simpler, IMHO at least, use the MySQL adapter directly. Again,
untested though :
csv = FasterCSV.open( target_file, "w" )
res = ActiveRecord::Base.connection.execute( your_sql )
# Header
csv << res.fetch_fields.map { |f| f.name }
# Results
res.each do |row_data|
csv << row_data
end
csv.close
This is probably specific to the MySQL adapter, although I'm sure
there will be similar methods to fetch_fields in the other adapters.
Regards,
Carl.
On 2 Oct 2008, at 11:28, Francis Fish wrote:
> No time to respond in detail
>
> Break data into chunks (order by ID or something like that)
>
> something like this (untested - set limit to something bigger than
> 1000 -
> but this also stops you eating all the memory)
>
> def output_from_sql(sql_file,target_file)
>
> raise ArgumentError, "SQL file not found : #{sql_file}" unless
> File.exists?(sql_file)
>
> keys = data.first.keys
> FasterCSV.open(target_file,'w') do |csv|
> csv << keys
> last_key = 0
> while true
> data =
> ActiveRecord
> ::Base.connection.select_all(File.open(sql_file).read,:order =>
> 'id',:limit => 2000, :conditions => ['id > ?',last_key])
> data.each do |row|
> csv << keys.map {|key| row[key]}
> end
> break if data.empty?
> last_key = data.last.id
> end
> end
> end
>
> On Thu, Oct 2, 2008 at 11:11 AM, Mike Barton
> <[EMAIL PROTECTED]>wrote:
>
>>
>> Hi,
>>
>> I'm having trouble getting data out of my database through
>> activerecord when the amount data returned by the sql is very
>> large. I
>> could use plain mysql at the command line, but would prefer to keep
>> it
>> in Ruby.
>>
>> This gist shows the helper I'm using at the moment, works for
>> datasets
>> in the tens of thousands, but not in the millions. The error
>> message I
>> get is that activerecord timed out.
>>
>> http://gist.github.com/14333
>>
>> Any suggestions would be great.
>>
>> Thanks
>>
>> Mike
>>>
>>
>
>
> --
> Thanks and regards,
>
> Francis Fish
>
> >
--
Carl Drinkwater
Director,
29degrees Ltd
Bespoke Web Application Development
t: 0161 953 6669
m: 07515 351 589
f: 07092 217 531
w: http://29degrees.co.uk
-----------------------------------------------
Office: 23 New Mount Street, Manchester, M4 4DE
Registered Address: St. George's House, 215/219
Chester Road, Manchester, M14 4JE.
Company Number 05952987 | VAT Number 897293655
-----------------------------------------------
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"NWRUG" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/nwrug-members?hl=en
-~----------~----~----~----~------~----~------~--~---
