[web2py] Re: web2py GAE and memory usage

2011-07-27 Thread Wikus van de Merwe
The exact behaviour of this instance memory control mechanism was not revealed (see http://code.google.com/p/googleappengine/issues/detail?id=1646 ). From your log message it looks like it was the first request that triggered the limit so the garbage collection can hardly be blamed here. And fr

[web2py] Re: web2py GAE and memory usage

2011-07-27 Thread howesc
thanks for the cursor tip - didn't think of that. i suspect my problem is some combination of the data being read in together with the references and de-referencing them - i probably end up with lots of memory usage. what i don't understand is, does GAE just flag a spike in ram usage and resta

[web2py] Re: web2py GAE and memory usage

2011-07-27 Thread Wikus van de Merwe
The thing is that when you read from datastore you always read all properties of an entity. Each entity can take up to 1MB, so I can imagine reading 500 of them could hit the instance limit. The solution would be to use GAE cursor and cycle your task. In pseudo code: def request_handler(): ma