Thanks Dave. That was helpful. On Tuesday, September 3, 2019 at 3:29:02 AM UTC-4, Dave S wrote: > > > > On Monday, September 2, 2019 at 12:44:18 PM UTC-7, Ian W. Scott wrote: >> >> I'm trying to lower the memory use of an app and have some general >> questions about how memory is used in DAL selects: >> >> >> 1. Am I right that the memory used while performing the select isn't >> released right away, even if the select isn't assigned to a variable? >> 2. I'm aware of iterselect. Am I right that with iterselect the >> memory used is just enough to store one row of data (instead of the whole >> selected set)? >> 3. Does this mean that, generally, you want to perform as few >> separate selects as possible, unless you can use iterselect? >> 4. Selects seem to occupy a significant amount of memory, even when >> the result set is only one row. Is the memory use for the select >> determined >> by the table size or the result size? >> >> Here's an example of the kind of situation I'm working with. I'm using a >> list comprehension to loop through a list, performing a select in each loop: >> >> p_here = [p for p in cpaths if loc_id in db.steps[int(p['steps'][0 >> ])].locations] >> >> >> When I run a memory profiler, this line results in over 1MB of memory >> being occupied, and that memory isn't released for at least several >> minutes. The table "steps" has about 3000 rows, so it's not enormous. The >> result for each select is a single row and doesn't include a huge amount of >> data (a few strings, ints, etc.). The "cpaths" list might have 50 or so >> items. So is the memory issue emerging because (a) the memory use for each >> select is determined by the table size, and (b) memory is being occupied >> (and not released) separately for each iteration of the loop? Is there a >> way to rewrite this so that it uses less memory? >> > > Naive db abuser responding: > > You've already mentioned iterselect(),. but I'll put the link here for > other readers: > <URL: > http://web2py.com/books/default/chapter/29/06/the-database-abstraction-layer#Using-an-iterator-based-select-for-lower-memory-use > > > > And the general advice oft repeated hereabouts is to not do in Python > what you can do in the DB, so it may be worth trying to recast the > comprehension into elements of the select. > > Also, this may depend on the combination of database and driver. In my > case, pg8000 was appearenetly costing me memory growth with the scheduler. > When I finally figured out installing psycopg2 where web2py could find it, > this issue diminished. > > Good luck! > > /dps > >
-- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/db29c691-3192-4ba7-8afd-fb4352b34225%40googlegroups.com.
