Hi Moises, we're currently trying to deal with this issue too. Not in mapreduce, just regular handlers.
same here - "Python 2.7 and F2 instances with 256MB" Early on I found that fetching 1000 entities and looping through them to update a property would blow the instance. Reducing this to say 100 fixed the issue. (than when create another task do do the next 100) "After handling this request" - as far as I understand this isn't so bad unless you're blowing it on every request and starting a new instance is detrimental. "While handling this request" - this concerns us most as the request does not complete and breaks stuff.. and we see far too many of them. I've spent more than a little time trying to work out what is causing the blowouts but as far as I've been able to work out, memory usage and what causes it is near impossible (or just very, very hard). Are you using NDB? if so.. - you could try disabling the in-memory cache. As I see it, even though you only access one entity at a time, NDB's in memory cache will store them all until the request is completed. - you could try projection queries if you don't need the complete object (or possibly even if you do). Projection queries get their data from an index and the entities returned cannot be put() so I assume they are not cached at all. We're trialling some fixes with these atm. ** If anyone knows any of this is incorrect, please let me know.. I'm actually surprised there is not more discussion of these issues from what we have experienced so maybe we're doing something fundamentally wrong, but I don't believe so. oh, is appstats turned on? I believe the most noticeable improvement we've seen was when we turned it off.. regards, Jason On Fri, Oct 11, 2013 at 5:51 PM, Moises Belchin <[email protected]>wrote: > Hi Vinny, > > Thanks for the tips, but actually I'm not loading a file. I'm only using > mapreduce lib for read all the entities for one of my kinds, work with them > (I only read some properties to compose the csv line format) and then I > write to CSV file on cloud storage using mapreduce FileOutputWriter. > > Any idea why I'm getting this Criticals memory errors? > > Thanks all again. > > > Saludos. > Moisés Belchín. > > > 2013/10/10 Vinny P <[email protected]> > >> On Thu, Oct 10, 2013 at 5:47 AM, Moises Belchin <[email protected]> >> wrote: >> >>> I'm getting a lot of memory limit critical errors in my app when I use >>> mapreduce library. >>> I'm working with Python 2.7, app engine 1.8.5 and F2 instances with >>> 256MB. >>> >>> Has anyone got these critical errors? >>> >>> >> >> Hello Moises, >> >> How large is your input file? Are you loading the entire file into memory >> at once? If so, try moving to a BlobstoreLineInputReader which reads in one >> line at a time - it reduces the memory being used during file processing. >> >> >> >> ----------------- >> -Vinny P >> Technology & Media Advisor >> Chicago, IL >> >> App Engine Code Samples: http://www.learntogoogleit.com >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/google-appengine. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/google-appengine. > For more options, visit https://groups.google.com/groups/opt_out. > -- Jason Galea lecstor.com -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
