Hello Joshua,
Thank you for your response. I agree with you on the point that caching won’t be of much help if repeat hits are quite limited. Besides the recommendations aforementioned, I suggest you to open tech support tickets with GCP support so we could be able to dig deep into your projects for any possible improvements. @Linus, thank you for sharing your experience. In such a scenario, setting the max number of instances for autoscaling could greatly help, which is part of the recommendations <https://cloud.google.com/appengine/docs/managing-costs> to control App Engine-related costs. On Thursday, August 27, 2020 at 11:32:26 AM UTC-4 Joshua Smith wrote: > On this particular topic of Cloud Datastore Read Operations being the cost > driver, unfortunately caching doesn’t help my case. > > This site has about 60,000 individual meetings listed, and I like having > *useful* *well-behaved* crawlers find all those meetings so people can use > search engines to find meetings that happened in various towns. But what > keeps happening is some new useless/poorly-behaved crawler will decide to > read all 60,000 of those meetings as fast as it possibly can. (Ignoring the > crawl-delay.) Caching in that situation would just add *more* cost, since > there are no repeat hits from a crawler. > > I periodically look at my access logs, find peaks, look at the requests, > and figure out what new bot needs to be added to my robots.txt disallow > list. And sometimes I have to add a firewall rule because that bot isn’t > obeying robots.txt at all. > > -Joshua > > On Aug 26, 2020, at 4:47 PM, 'yananc' via Google App Engine < > [email protected]> wrote: > > Back to the issue of ‘Cloud Datastore Read Operations’ being too high, a > possible solution is to leverage cache mechanism to avoid excessive > operations. You may find more information from the topic [2]. > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/f6b45f06-f6e1-48d2-8fb2-b33f91e01324n%40googlegroups.com.
