Hello, Philip, i was wondering how many entities of that kind do you have. (are there tens of thousands ?) And also, if you managed to find a solution (i'm also experiencing this pb)
On Aug 10, 4:45 pm, Alfred Fuller <[email protected]> wrote: > On Sat, Aug 7, 2010 at 6:18 PM, Philip Tucker <[email protected]> wrote: > > I've just started getting the following error on my server. This query > > had been working fine for months, and I haven't changed the indexes > > lately. Are there some indexes that just buckle under heavy load or > > larger data sets? This particular table contains only 242,604 entries. > > > The built-in indices are not efficient enough for this query and > > your data. Please add a composite index for this query.. An index is > > missing but we are unable to tell you which one due to a bug in the > > App Engine SDK. If your query only contains equality filters you most > > likely need a composite index on all the properties referenced in > > those filters. > > (full stack dump included below) > > > My query source code: > > > Query query = pm.newQuery(GameDataV2.class); > > Object[] params; > > query.declareParameters( > > "com.google.appengine.api.datastore.Key userKeyParam" > > + ", com.honkentuber.wordwise.GameState stateParam1" > > + ", com.honkentuber.wordwise.GameState stateParam2" > > + ", com.honkentuber.wordwise.GameState stateParam3"); > > params = new Object[4]; > > params[0] = userKey; > > params[1] = GameState.PENDING; > > params[2] = GameState.PLAYER_TURN; > > params[3] = GameState.OPPONENT_TURN; > > query.setFilter("(memberKeys == userKeyParam)" > > + " && ((state == stateParam1)" > > + " || (state == stateParam2)" > > + " || (state == stateParam3))"); > > > List<GameDataV2> gamesData = (List<GameDataV2>) > > query.executeWithArray(params); > > > I have the following 2 indexes defined: > > memberKeys ▲ , state ▲ , memberLastAccessMs ▲ > > memberKeys ▲ , state ▲ , memberScores ▼ > > > I'm not sure why I don't have an index with only memberKeys and state > > - does datastore-indexes autoGenerate not generate a new index if it > > already has a superset index? > > > I've changed the query to the following in my local development > > environment: > > > Query query = pm.newQuery(GameDataV2.class); > > query.setFilter("(memberKeys == :userKeyParam) > > && :statesParam.contains(state)"); > > List<GameDataV2> gamesData = (List<GameDataV2>) query.execute( > > userKey, > > Arrays.asList( > > GameState.PENDING.name(), > > GameState.PLAYER_TURN.name(), > > GameState.OPPONENT_TURN.name())); > > > Some questions about this: > > > 1) Running locally with autoGenerate="true", I'm not getting a new > > GameDataV2 index on memberKeys and states. Does contains() not require > > an index on that field? Is this just generating 2 separate queries > > internally? > > It's using zigzag merge join to solve your query. We do not generate an > index for this query because it is not strictly needed. The error message > you are getting will appear if you pull a lot of data or the specific data > you are querying is especially inefficient for zigzag merge join to solve. I > am removing this particular error soon, but that doesn't mean zigzag will > become more efficient when working with your data (if this is in fact the > problem). And example when efficiency is not the problem is having X > equality filters and pulling 10,000/X results (this will always produce this > error even when zigzag merge join is perfectly efficient). > > > 2) Is the contains() query more efficient than the nested ORs I had in > > my original query? Or will I still get the "not efficient enough for > > this query and your data" error? > > 3) Why do enumerated types not work with contains()? I had to > > add .name() to each entry in the list to get this to work. > > 4) Why do declared parameters not work with collections? I tried > > declaring statesParam as a both List<GameState> and List, and I got > > errors resolving the types. > > > com.google.appengine.api.datastore.DatastoreNeedIndexException: The > > built-in indices are not efficient enough for this query and your > > data. Please add a composite index for this query.. An index is > > missing but we are unable to tell you which one due to a bug in the > > App Engine SDK. If your query only contains equality filters you most > > likely need a composite index on all the properties referenced in > > those filters. > > at > > > com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java: > > 40) > > at > > > com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(DatastoreApiHelper.java: > > 67) > > at > > > com.google.appengine.api.datastore.PreparedQueryImpl.runQuery(PreparedQueryImpl.java: > > 127) > > at > > > com.google.appengine.api.datastore.PreparedQueryImpl.asIterator(PreparedQueryImpl.java: > > 87) > > at com.google.appengine.api.datastore.PreparedMultiQuery > > $DedupingMultiQueryIterator.getNextIterator(PreparedMultiQuery.java: > > 154) > > at com.google.appengine.api.datastore.PreparedMultiQuery > > $DedupingMultiQueryIterator.computeNext(PreparedMultiQuery.java:173) > > at com.google.appengine.api.datastore.PreparedMultiQuery > > $DedupingMultiQueryIterator.computeNext(PreparedMultiQuery.java:98) > > at > > > com.google.appengine.api.datastore.AbstractIterator.tryToComputeNext(AbstractIterator.java: > > 52) > > at > > > com.google.appengine.api.datastore.AbstractIterator.hasNext(AbstractIterator.java: > > 47) > > at com.google.appengine.api.datastore.BasePreparedQuery > > $UncompilablePreparedQuery$1.hasNext(BasePreparedQuery.java:78) > > at > > > org.datanucleus.store.appengine.query.RuntimeExceptionWrappingIterator.hasNext(RuntimeExceptionWrappingIterator.java: > > 44) > > at > > > org.datanucleus.store.appengine.query.LazyResult.resolveAll(LazyResult.java: > > 115) > > at > > org.datanucleus.store.appengine.query.LazyResult.size(LazyResult.java: > > 110) > > at > > > org.datanucleus.store.appengine.query.StreamingQueryResult.size(StreamingQueryResult.java: > > 124) > > at > > > org.datanucleus.store.query.AbstractQueryResult.toArray(AbstractQueryResult.java: > > 399) > > at java.util.ArrayList.addAll(Unknown Source) > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to > > [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine-java%[email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine-java?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
