Hey Max, I'm eating crow now. Turns out this is *not* a bug in the datastore code. I discovered that one of my DAO parent classes was doing a lookup before it actually created the Entity, and this lookup was getting an EntityManager but not closing it. Turns out my own code was the leak.
DOH! Appologies for the false alarm and the waste of time.... david On Mar 15, 8:00 pm, David Fuelling <[email protected]> wrote: > Hey Max, > > Took a heap dump using two different scenarios. In test 1, I set the - > Xmx to 100M, and in test 2 I set the max to 512M, and am comparing the > difference between the number of objects across the two memory > scenarios. In my code, I'm running a for-loop that executes 5000 > times, and most of the objects present in JHat show up no more than > 500 times, and in equal amounts, in either memory scenario (leads me > to believe these are not the culprits). However, there are a handful > of datastore classes that show a 1:1 relationship between the amount > of memory I allow in each test. Following are the outputs from JHat: > > TEST #1 (100M Max) > 101489 instances of class com.google.storage.onestore.v3.OnestoreEntity > $PropertyValue > 101488 instances of class com.google.storage.onestore.v3.OnestoreEntity > $Property > 52846 instances of class com.google.storage.onestore.v3.OnestoreEntity > $Path$Element > 43623 instances of class com.google.storage.onestore.v3.OnestoreEntity > $Path > 29782 instances of class com.google.storage.onestore.v3.OnestoreEntity > $Reference > 15944 instances of class com.google.apphosting.api.DatastorePb$Query > 13839 instances of class com.google.storage.onestore.v3.OnestoreEntity > $EntityProto > 9237 instances of class > com.google.appengine.api.datastore.dev.LocalCompositeIndexManager > $IndexComponentsOnlyQuery > 6705 instances of class > com.google.appengine.api.datastore.dev.LocalDatastoreService > $LiveQuery > 4613 instances of class > com.google.appengine.api.datastore.dev.LocalDatastoreService$Profile > $EntityGroup > > TEST #2 (512M Max) > 579558 instances of class com.google.storage.onestore.v3.OnestoreEntity > $Property > 579558 instances of class com.google.storage.onestore.v3.OnestoreEntity > $PropertyValue > 25863 instances of class com.google.storage.onestore.v3.OnestoreEntity > $Path$Element > 24805 instances of class com.google.storage.onestore.v3.OnestoreEntity > $Path > 17683 instances of class com.google.storage.onestore.v3.OnestoreEntity > $Reference > 10564 instances of class com.google.apphosting.api.DatastorePb$Query > 8441 instances of class > com.google.appengine.api.datastore.dev.LocalDatastoreService > $LiveQuery > 6065 instances of class com.google.storage.onestore.v3.OnestoreEntity > $EntityProto > 5002 instances of class > com.google.appengine.api.datastore.dev.LocalDatastoreService$Profile > $EntityGroup > > From what I can tell, the OnestoreEntity$Property and OnestoreEntity > $PropertyValue classes are growing the most (and somewhat linearly) > with the number of Datastore operations I execute (capped only by the > amount of memory I allow the JVM). Now, my first inclination is to > explain my error away on the grounds that the local datastore is "in- > memory", and so if I don't allow enough JVM memory for my datastore, > this is expected behavior. > > However, I'm only persisting 5,000 User entities (and not even getting > that far). My "User" is not terribly complicated (6 properties, 4 > Lists, and a Preferences object). The lists and preferences are Lazy > and empty -- I'm not creating anything for these in my test -- just a > User....plus I'm only populating one of the field properties > (username) with a very short string. > > For me to run out of 512MB of memory after creating only 4k-ish users > means that each of my User entities is taking up around 125k each. > Accounting for other datastore components taking up memory, we could > say that my entities are only taking up 100k each, leaving the > datastore the other 25k, or 100MB of memory. That still seems like my > entities are taking up way too much memory (hence some sort of leak?), > though I guess I don't know what's going on with the OnestoreEntity > class to simulate the datastore, so I can't say for sure. > > Any thoughts here would be much appreciated. If I can provide any > more data for you, please let me know. > > Thanks! > > David > > On Mar 12, 9:37 pm, "Max Ross (Google)" <[email protected]> > wrote: > > > > > Thanks for the report David, this certainly seems suspicious. There is at > > least one memory leak I'm aware of but it's related to transactions so > > that's probably not what you're bumping into. Have you tried taking a heap > > dump to see what exactly is building up? -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
