We have 50,000,000+ search documents that we need to change (i.e., delete the old version, create a new version). These are sharded across many individual indexes.
I think the per-second rate limit for interacting with the search API is 50/s. We need to leave some room for live traffic, so if we clamp our re-indexing to 25/s, that means it will take 23 days to process all these documents. Does anyone have other ideas? Are there any tools coming out that would allow faster throughput? The ability to completely drop an index in one call might help: we could build up our new indexes, cut over to them, then drop the old ones in a single operation - this would at least cut the number of operations in half. Does anyone have any war stories about operating over search index documents at scale? Thanks, j -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
