GCP support was not able to help me. I eventually managed to find the issue by using Audit Logs to view all GCS accesses, and turns out one of my apps was misbehaving and trying to read a file every second.
It still isn't clear to me why: 1. It gave me quota limit errors instead of just charging my billing account 2. Why the limit being hit wasn't shown on any page 3. why it was failing in some strange non-descriptive "internal error" way It seems like it was actually hitting some internal "rate limit" that not surfaced at all in GCP. Which is strange for a Cloud which is meant to be able to scale up without limits. On Monday, September 14, 2020 at 2:59:39 PM UTC-4 David (Cloud Platform Support) wrote: > Hello, > > This error can show up due to various matters usually related to billing > or a quota being exceeded. I would recommend you to contact GCP support as > they would be able to inspect your project and tell you the reason you are > seeing this error message. > On Tuesday, September 8, 2020 at 12:39:35 AM UTC-4 [email protected] wrote: > >> Nvm, this is still happening. The closest I can find is this: >> >> https://stackoverflow.com/questions/61052413/google-cloud-storage-quota-hit-how >> But I'm not anywhere near my limit. >> >> On Saturday, August 15, 2020 at 4:38:32 PM UTC-4 Ehsan Kia wrote: >> >>> For what it's worth, it got fixed the next day as the quotas reset, even >>> though no quota showed as past it's limit and even though I had billing >>> enabled. So if you run into the same issue, just deploy the right thing >>> again and wait a day. >>> >>> On Sat., Aug. 15, 2020, 4:11 p.m. Ehsan Kia, <[email protected]> wrote: >>> >>>> 3 days ago, I went to deploy a newer AppEngine project using `gcloud >>>> app deploy` but forgot that the default project was set to an older >>>> project. >>>> I have since reverted that by redeploying the older project, but my >>>> Google Cloud Storage seems to be completely broken. >>>> >>>> It does not let me access any of the files, either from the UI or >>>> gsutil. It throws "Internal server error", and on gsutil it gives me "The >>>> App Engine application does not have enough quota". >>>> i see 429 when trying to access from the API and 403 in other places. >>>> Trying to transfer the files or download the files or anything just >>>> continuously throws errors. >>>> I don't actually see any quotas that are full, and it doesn't matter >>>> anyways because I have billing enabled and nowhere close my budget limit. >>>> Not sure what "quota" this is referring to. >>>> Nothing seems to fix it, the whole storage bucket seems to be locked up >>>> and erroring. Anyone have any idea how I can recover this? >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/google-appengine/ba0c56bb-d7fa-4739-b83d-313a20ba5209o%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/google-appengine/ba0c56bb-d7fa-4739-b83d-313a20ba5209o%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/dfd607e2-1458-4249-9bcf-c7240f3974dan%40googlegroups.com.
