I think we should consider rewriting the export function using a tmpfile instead of StringIO and doing pagination.
On Jan 30, 2:49 pm, Thadeus Burgess <thade...@thadeusb.com> wrote: > Perhaps you need to write your own csv export. Save a csv file to /tmp > instead and then when your finished creating your csv file then stream > it for download. > > This might reduce the memory requirements? Though not sure what you > can do about the fact web2py cache's your select into memory, maybe do > it in chunks as you generate the csv file? > > -Thadeus > > On Sat, Jan 30, 2010 at 2:37 PM, mdipierro <mdipie...@cs.depaul.edu> wrote: > > The problem is that all records are copied into memory before being > > serialized + the entire csv is copied into memory. We need to find a > > better way when exporting a large of records. > > > On Jan 30, 12:19 pm, Johann Spies <johann.sp...@gmail.com> wrote: > >> It happened more than once that a csv-export using appadmin resulted > >> in incomplete data both with Postgresql-backend and sqlite. > > >> This one was on sqlite: > > >> Selected records: 139413 > >> csv-exported (first effort): 287 records. > >> csv-exported (second effort) 216 records > > >> This after a lot of time and slowing down the computer to a near > >> standstill. > > >> I suppose it might be easier and more reliable directly form the database. > > >> Regards > >> Johann > > > -- > > You received this message because you are subscribed to the Google Groups > > "web2py-users" group. > > To post to this group, send email to web...@googlegroups.com. > > To unsubscribe from this group, send email to > > web2py+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/web2py?hl=en. > > -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
