This does not solve the problem unfortunately because s.read() still
reads everything in memory.
I am thinking about an hypotetical action that does:
def export():
s=StringIO.StringIO()
db(query).select().export_to_csv_file(s)
return s.getvalue()
and replace it with
def export():
s=mkstemp()
k=0
while True:
rows = db(query).select(limitby=(k,k+1000))
k+=1000
if rows: rows.export_to_csv_file(s)
if len(rows)<1000: break
s.seek(0)
return response.stream(s)
The problem is that this may cause duplicate or missing records
depending on the database engine and how they handle multiple select()
because records may be inserted or deleted in other process between
subsequent queries.
On Jan 30, 6:57 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
> Here is at least the tempfile part of it. Not sure how you would do
> the pagination, since its exporting an already selected Rows object.
>
> Johann could you look and see if using a tempfile instead works? I
> attached the hg export diff.
>
> -Thadeus
>
> On Sat, Jan 30, 2010 at 3:32 PM, mdipierro <mdipie...@cs.depaul.edu> wrote:
> > I'd just rewrite it to use tempfile and use add an option to do
> > pagination (the counterindidation with pagination is that it may cause
> > data loss or duplication although very rarely).
>
> > On Jan 30, 3:03 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
> >> Rewrite or provide an alternative to ?
>
> >> -Thadeus
>
> >> On Sat, Jan 30, 2010 at 2:51 PM, mdipierro <mdipie...@cs.depaul.edu> wrote:
> >> > I think we should consider rewriting the export function using a
> >> > tmpfile instead of StringIO and doing pagination.
>
> >> > On Jan 30, 2:49 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
> >> >> Perhaps you need to write your own csv export. Save a csv file to /tmp
> >> >> instead and then when your finished creating your csv file then stream
> >> >> it for download.
>
> >> >> This might reduce the memory requirements? Though not sure what you
> >> >> can do about the fact web2py cache's your select into memory, maybe do
> >> >> it in chunks as you generate the csv file?
>
> >> >> -Thadeus
>
> >> >> On Sat, Jan 30, 2010 at 2:37 PM, mdipierro <mdipie...@cs.depaul.edu>
> >> >> wrote:
> >> >> > The problem is that all records are copied into memory before being
> >> >> > serialized + the entire csv is copied into memory. We need to find a
> >> >> > better way when exporting a large of records.
>
> >> >> > On Jan 30, 12:19 pm, Johann Spies <johann.sp...@gmail.com> wrote:
> >> >> >> It happened more than once that a csv-export using appadmin resulted
> >> >> >> in incomplete data both with Postgresql-backend and sqlite.
>
> >> >> >> This one was on sqlite:
>
> >> >> >> Selected records: 139413
> >> >> >> csv-exported (first effort): 287 records.
> >> >> >> csv-exported (second effort) 216 records
>
> >> >> >> This after a lot of time and slowing down the computer to a near
> >> >> >> standstill.
>
> >> >> >> I suppose it might be easier and more reliable directly form the
> >> >> >> database.
>
> >> >> >> Regards
> >> >> >> Johann
>
> >> >> > --
> >> >> > You received this message because you are subscribed to the Google
> >> >> > Groups "web2py-users" group.
> >> >> > To post to this group, send email to web...@googlegroups.com.
> >> >> > To unsubscribe from this group, send email to
> >> >> > web2py+unsubscr...@googlegroups.com.
> >> >> > For more options, visit this group
> >> >> > athttp://groups.google.com/group/web2py?hl=en.
>
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups "web2py-users" group.
> >> > To post to this group, send email to web...@googlegroups.com.
> >> > To unsubscribe from this group, send email to
> >> > web2py+unsubscr...@googlegroups.com.
> >> > For more options, visit this group
> >> > athttp://groups.google.com/group/web2py?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "web2py-users" group.
> > To post to this group, send email to web...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > web2py+unsubscr...@googlegroups.com.
> > For more options, visit this group
> > athttp://groups.google.com/group/web2py?hl=en.
>
>
>
> tmpfileexport.diff
> 1KViewDownload
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
