On Sat, Aug 7, 2010 at 6:18 PM, Philip Tucker <[email protected]> wrote:
> I've just started getting the following error on my server. This query
> had been working fine for months, and I haven't changed the indexes
> lately. Are there some indexes that just buckle under heavy load or
> larger data sets? This particular table contains only 242,604 entries.
>
> The built-in indices are not efficient enough for this query and
> your data. Please add a composite index for this query.. An index is
> missing but we are unable to tell you which one due to a bug in the
> App Engine SDK. If your query only contains equality filters you most
> likely need a composite index on all the properties referenced in
> those filters.
> (full stack dump included below)
>
> My query source code:
>
> Query query = pm.newQuery(GameDataV2.class);
> Object[] params;
> query.declareParameters(
> "com.google.appengine.api.datastore.Key userKeyParam"
> + ", com.honkentuber.wordwise.GameState stateParam1"
> + ", com.honkentuber.wordwise.GameState stateParam2"
> + ", com.honkentuber.wordwise.GameState stateParam3");
> params = new Object[4];
> params[0] = userKey;
> params[1] = GameState.PENDING;
> params[2] = GameState.PLAYER_TURN;
> params[3] = GameState.OPPONENT_TURN;
> query.setFilter("(memberKeys == userKeyParam)"
> + " && ((state == stateParam1)"
> + " || (state == stateParam2)"
> + " || (state == stateParam3))");
>
> List<GameDataV2> gamesData = (List<GameDataV2>)
> query.executeWithArray(params);
>
> I have the following 2 indexes defined:
> memberKeys ▲ , state ▲ , memberLastAccessMs ▲
> memberKeys ▲ , state ▲ , memberScores ▼
>
> I'm not sure why I don't have an index with only memberKeys and state
> - does datastore-indexes autoGenerate not generate a new index if it
> already has a superset index?
>
> I've changed the query to the following in my local development
> environment:
>
> Query query = pm.newQuery(GameDataV2.class);
> query.setFilter("(memberKeys == :userKeyParam)
> && :statesParam.contains(state)");
> List<GameDataV2> gamesData = (List<GameDataV2>) query.execute(
> userKey,
> Arrays.asList(
> GameState.PENDING.name(),
> GameState.PLAYER_TURN.name(),
> GameState.OPPONENT_TURN.name()));
>
> Some questions about this:
>
> 1) Running locally with autoGenerate="true", I'm not getting a new
> GameDataV2 index on memberKeys and states. Does contains() not require
> an index on that field? Is this just generating 2 separate queries
> internally?
>
It's using zigzag merge join to solve your query. We do not generate an
index for this query because it is not strictly needed. The error message
you are getting will appear if you pull a lot of data or the specific data
you are querying is especially inefficient for zigzag merge join to solve. I
am removing this particular error soon, but that doesn't mean zigzag will
become more efficient when working with your data (if this is in fact the
problem). And example when efficiency is not the problem is having X
equality filters and pulling 10,000/X results (this will always produce this
error even when zigzag merge join is perfectly efficient).
> 2) Is the contains() query more efficient than the nested ORs I had in
> my original query? Or will I still get the "not efficient enough for
> this query and your data" error?
> 3) Why do enumerated types not work with contains()? I had to
> add .name() to each entry in the list to get this to work.
> 4) Why do declared parameters not work with collections? I tried
> declaring statesParam as a both List<GameState> and List, and I got
> errors resolving the types.
>
> com.google.appengine.api.datastore.DatastoreNeedIndexException: The
> built-in indices are not efficient enough for this query and your
> data. Please add a composite index for this query.. An index is
> missing but we are unable to tell you which one due to a bug in the
> App Engine SDK. If your query only contains equality filters you most
> likely need a composite index on all the properties referenced in
> those filters.
> at
>
> com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:
> 40)
> at
>
> com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(DatastoreApiHelper.java:
> 67)
> at
>
> com.google.appengine.api.datastore.PreparedQueryImpl.runQuery(PreparedQueryImpl.java:
> 127)
> at
>
> com.google.appengine.api.datastore.PreparedQueryImpl.asIterator(PreparedQueryImpl.java:
> 87)
> at com.google.appengine.api.datastore.PreparedMultiQuery
> $DedupingMultiQueryIterator.getNextIterator(PreparedMultiQuery.java:
> 154)
> at com.google.appengine.api.datastore.PreparedMultiQuery
> $DedupingMultiQueryIterator.computeNext(PreparedMultiQuery.java:173)
> at com.google.appengine.api.datastore.PreparedMultiQuery
> $DedupingMultiQueryIterator.computeNext(PreparedMultiQuery.java:98)
> at
>
> com.google.appengine.api.datastore.AbstractIterator.tryToComputeNext(AbstractIterator.java:
> 52)
> at
>
> com.google.appengine.api.datastore.AbstractIterator.hasNext(AbstractIterator.java:
> 47)
> at com.google.appengine.api.datastore.BasePreparedQuery
> $UncompilablePreparedQuery$1.hasNext(BasePreparedQuery.java:78)
> at
>
> org.datanucleus.store.appengine.query.RuntimeExceptionWrappingIterator.hasNext(RuntimeExceptionWrappingIterator.java:
> 44)
> at
>
> org.datanucleus.store.appengine.query.LazyResult.resolveAll(LazyResult.java:
> 115)
> at
> org.datanucleus.store.appengine.query.LazyResult.size(LazyResult.java:
> 110)
> at
>
> org.datanucleus.store.appengine.query.StreamingQueryResult.size(StreamingQueryResult.java:
> 124)
> at
>
> org.datanucleus.store.query.AbstractQueryResult.toArray(AbstractQueryResult.java:
> 399)
> at java.util.ArrayList.addAll(Unknown Source)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine for Java" group.
> To post to this group, send email to
> [email protected].
> To unsubscribe from this group, send email to
> [email protected]<google-appengine-java%[email protected]>
> .
> For more options, visit this group at
> http://groups.google.com/group/google-appengine-java?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
