Rewrite or provide an alternative to ? -Thadeus
On Sat, Jan 30, 2010 at 2:51 PM, mdipierro <mdipie...@cs.depaul.edu> wrote: > I think we should consider rewriting the export function using a > tmpfile instead of StringIO and doing pagination. > > On Jan 30, 2:49 pm, Thadeus Burgess <thade...@thadeusb.com> wrote: >> Perhaps you need to write your own csv export. Save a csv file to /tmp >> instead and then when your finished creating your csv file then stream >> it for download. >> >> This might reduce the memory requirements? Though not sure what you >> can do about the fact web2py cache's your select into memory, maybe do >> it in chunks as you generate the csv file? >> >> -Thadeus >> >> On Sat, Jan 30, 2010 at 2:37 PM, mdipierro <mdipie...@cs.depaul.edu> wrote: >> > The problem is that all records are copied into memory before being >> > serialized + the entire csv is copied into memory. We need to find a >> > better way when exporting a large of records. >> >> > On Jan 30, 12:19 pm, Johann Spies <johann.sp...@gmail.com> wrote: >> >> It happened more than once that a csv-export using appadmin resulted >> >> in incomplete data both with Postgresql-backend and sqlite. >> >> >> This one was on sqlite: >> >> >> Selected records: 139413 >> >> csv-exported (first effort): 287 records. >> >> csv-exported (second effort) 216 records >> >> >> This after a lot of time and slowing down the computer to a near >> >> standstill. >> >> >> I suppose it might be easier and more reliable directly form the database. >> >> >> Regards >> >> Johann >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "web2py-users" group. >> > To post to this group, send email to web...@googlegroups.com. >> > To unsubscribe from this group, send email to >> > web2py+unsubscr...@googlegroups.com. >> > For more options, visit this group >> > athttp://groups.google.com/group/web2py?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "web2py-users" group. > To post to this group, send email to web...@googlegroups.com. > To unsubscribe from this group, send email to > web2py+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/web2py?hl=en. > > -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
