That's a nice way of working around the rate limits Emlyn. In your _post_put function you can also compare a hash of the last document you put in the Search API index for the Datastore entity, and only send to the Search API if the current document's hash is different.
To make it fast and cost-effective, you can use Memcache to store the hash of the most recent document sent to the Search API for a particular Datastore entity. This method is most useful if your app does a lot of updates to Datastore entities which would not actually change the Search API document at all (e.g. when updating Datastore entity properties which are not indexed in the Search API). Nick On 22 June 2017 at 11:42, Emlyn <[email protected]> wrote: > Thanks George, I read that, that's great. I may have to go for a support > level in the near future. > > For anyone who's interested, I was able to fix my problems with the Search > API as follows: > > My code enqueues a huge amount of tasks very rapidly, which modify > datastore objects. In the _post_put handlers, these also update search > indices. > > I was having problems that the datastore updates were working, but the > rate was too high for the search indices; I was getting timeouts, quota > limitations, bad things :-) > > I've changed my _post_put handlers to do the updates to search indices in > separate tasks, on a queue I've configured just for this. It's rate limited > to below the maximum rate for the search api (ie: maximum search throughput > is 15000/minute, or 250/s, so I've set it lower than this). > > That's completely fixed the problem. And because I'm not hitting quota > limits, the real throughput is much higher than it was; I don't see this > queue hitting the limit and lagging behind the rest of the job. > > Now I get to sleep comfortably at night :-) Thanks for the help. > > > On 22 June 2017 at 03:53, 'George (Cloud Platform Support)' via Google App > Engine <[email protected]> wrote: > >> Hi Emlyn, >> >> News is not entirely bad: "Note: Although these limits are enforced by >> the minute, the Cloud Platform Console displays the daily totals for each. >> Customers with Silver, Gold, or Platinum support can request higher >> throughput limits by contacting their support representative." This is to >> be read just below the quotas paragraph we speak about. In short, you may >> consider asking for a quota increase. >> >> Information on how to reduce your app's needs for paid resources, and so >> keep within quotas, you may consult the "Managing App Resources" >> documentation page >> <https://cloud.google.com/appengine/docs/standard/python/console/managing-resources> >> . >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/google-appengine. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/google-appengine/623ea57b-1c0c-4454-a854-5cb5fa84bd8b% >> 40googlegroups.com >> <https://groups.google.com/d/msgid/google-appengine/623ea57b-1c0c-4454-a854-5cb5fa84bd8b%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Emlyn > > https://medium.com/the-infinite-machine - A publication about Google App > Engine > sutllang.com - My language sUTL > https://plus.google.com/u/0/100281903174934656260 - Google+ > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/google-appengine. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/google-appengine/CAMp1VPBy7WjB-Qw8N0mkdR0CEP% > 2BqEf%2Byz%2BxwKCn2GwT1EfGsuA%40mail.gmail.com > <https://groups.google.com/d/msgid/google-appengine/CAMp1VPBy7WjB-Qw8N0mkdR0CEP%2BqEf%2Byz%2BxwKCn2GwT1EfGsuA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAOj3zuCVz3iR9HDYZdAHHF_LygKGa7HwtZZiKe-Swb6mVpL9sA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
