Damon (damondouglas) is working on a global throttling solution for Beam. There should be a design doc shared here in a week or two. However, you're right that the PR referenced should have addressed this. It may be due for an update. Do you have time to take a look at the code and see if it needs something added? Kerry
On Tue, Nov 15, 2022 at 5:53 AM Daniel Mateus Pires <dmate...@gmail.com> wrote: > hey we have a big job that fails after it hits BigQuery Storage API Read > quota limits, we’ve increased it once already but it keeps happening. > When a job fails we see a spike of 429 s on the API Monitoring page for > “ReadRows” and eventually we see "The worker lost contact with the service > in Dataflow.." > Shouldn’t this PR [1] on the Beam project have prevented that issue from > happening? > > Is there a way for us to throttle our requests and keep that from > happening on the Beam side? > > [1] https://github.com/apache/beam/pull/15445 >
