One more followup on this, when you say "you just pay your cost at write time, possibly asynchronously" are you referring to cron jobs? As far as I can tell there's no other support for concurrent processing.
On May 4, 3:28 am, "Ikai L (Google)" <[email protected]> wrote: > Query performance is generally a function of how many objects have to be > returned in the simple case, which seems to be what you are doing here. How > many PlayerGameStates exist per User? > > Queries work like this: > > - iterate through the index to find all PlayerGameStates with the property > given to find the keys of all PlayerGameStates matching this property > - run the query for all objects > > Are PlayerGameStates child entities of a User? I have to double check, but I > believe this could reduce the App Engine datastore's ability to parallelize > the queries-by-key, though the effect of this should be negligible if you > are only fetching 5 objects. > > The reason I mentioned "denormalizing" is because the concept of > normalization doesn't make as much sense with App Engine's datastore. We > don't provide referential integrity, for instance, and transactions exist > around entity groups, not globally. You don't pay a penalty for storing data > that is sparse - some User entities can be huge, some can be tiny. You > should optimize for reads. Doing multiple writes and the risk of data > getting out of sync can suck, but as long as it is recoverable you should be > okay. Contrast the performance of the query above with storing > PlayerGameStates inside User. You'd do this: > > - Get user by key > > That's it! This is really fast, so you don't have to go through doing an > unnecessary query - you just pay your cost at write time, possibly > asynchronously. > > > > On Tue, May 4, 2010 at 12:14 PM, Philip Tucker <[email protected]> wrote: > > I don't think duplicating the data won't work because then I'll be > > doing 2 writes for every turn of the game, and I need that to be fast. > > I don't mind if fetching a user's game list is a little bit slow, but > > 5-20 seconds is too much. I can denormalize all the data such that > > Game 2 fields for each entry in PlayerGameState if that helps, but I'd > > still like to know why it's so slow now. > > > I do have appstat installed. For one particular misbehaving request, > > the query fetches 5 PlayerGameState records. It iterates through the > > first 4 taking ~20 ms each, then I see a RunQuery taking 8 seconds. > > > I don't see an easy way to export appstat data, but I can include more > > details if necessary. > > > Here's the query code in question. > > > Set<Key> myUserCache = Sets.newHashSet(); > > Query query = pm.newQuery(PlayerGameState.class); > > query.declareParameters( > > "com.google.appengine.api.datastore.Key userKeyParam"); > > query.setFilter("userKey == userKeyParam"); > > List<PlayerGameState> playerStates = > > (List<PlayerGameState>) query.execute(playerKey); > > List<Game> games = Lists.newArrayList(); > > for (PlayerGameState playerState : playerStates) { > > Game game = playerState.getGame(); > > myUserCache.add(game.getPlayerState().getUserKey()); > > myUserCache.add(game.getOpponentState().getUserKey()); > > games.add(game); > > } > > > On May 3, 1:16 am, "Ikai L (Google)" <[email protected]> wrote: > > > Indexes don't work the same way they do in relational databases. If you > > can > > > fetch a value with a query, that likely means an index exists on that > > > property. In a relational database, when an index is not present, the > > > database will do a full table scan. > > > > A fetch by Key is always fast. Have you considered storing multiple games > > > inside a single Game instance? Or simply storing the data you need both > > in > > > the Game instances? If you want to cut down on request latency, do as > > many > > > by key gets as possible. Don't try to approach the modeling from a > > > normalization perspective. How many entities are you retrieving? And have > > > you tried looking at what's happening via AppStats yet ( > >http://code.google.com/appengine/docs/java/tools/appstats.html)? > > > > On Fri, Apr 30, 2010 at 1:54 AM, Philip Tucker <[email protected]> > > wrote: > > > > I've got a Game object that includes two PlayerGameState objects. Both > > > > are persistence-capable. My query fetches all PlayerGameState objects > > > > for a particular user, then I get the associated game and other > > > > player. The initial fetch is always fast, and the entire operation is > > > > generally lass than a second. But periodically, maybe 5 of the time, > > > > it takes much longer, from 5-20 seconds. It's not fetching the > > > > PlayerGameState objects that's slow, it's accessing > > > > PlagerGameState.game and Game.playerStates. > > > > > I shouldn't need to add an index on the keys, should I? I'd like to > > > > fetch the Game objects directly but as far as I can tell we can't > > > > query on a joined table; ie, I can't do something like "select from > > > > Game, PlayerGameState join on PlayerGameState.game where > > > > PlayerGameState.userKey == userKeyParam". > > > > > @PersistenceCapable(identityType = IdentityType.APPLICATION) > > > > public class Game { > > > > �...@primarykey > > > > �...@persistent(valueStrategy = IdGeneratorStrategy.IDENTITY) > > > > private Key key; > > > > > �...@persistent(mappedBy = "game") > > > > �...@element(dependent = "true") > > > > private List<PlayerGameState> playerStates; > > > > > ... > > > > } > > > > > @PersistenceCapable(identityType = IdentityType.APPLICATION) > > > > public class PlayerGameState { > > > > �...@suppresswarnings("unused") > > > > �...@primarykey > > > > �...@persistent(valueStrategy = IdGeneratorStrategy.IDENTITY) > > > > private Key key; > > > > > �...@persistent > > > > private Key userKey; > > > > > �...@persistent > > > > private Game game; > > > > > ... > > > > } > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Google App Engine for Java" group. > > > > To post to this group, send email to > > > > [email protected]. > > > > To unsubscribe from this group, send email to > > > > [email protected]<google-appengine-java%[email protected]><google-appengine-java%2B > > [email protected]> > > > > . > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > > -- > > > Ikai Lan > > > Developer Relations, Google App Engine > > > Twitter:http://twitter.com/ikai > > > Delicious:http://delicious.com/ikailan > > > > ---------------- > > > Google App Engine links: > > > Blog:http://googleappengine.blogspot.com > > > Twitter:http://twitter.com/app_engine > > > Reddit:http://www.reddit.com/r/appengine > > > > -- > > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > > To post to this group, send email to > > [email protected]. > > > To unsubscribe from this group, send email to > > [email protected]<google-appengine-java%[email protected]> > > . > > > For more options, visit this group athttp:// > > groups.google.com/group/google-appengine-java?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to > > [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine-java%[email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine-java?hl=en. > > -- > Ikai Lan > Developer Relations, Google App Engine > Twitter:http://twitter.com/ikai > Delicious:http://delicious.com/ikailan > > ---------------- > Google App Engine links: > Blog:http://googleappengine.blogspot.com > Twitter:http://twitter.com/app_engine > Reddit:http://www.reddit.com/r/appengine > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group > athttp://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
