Hi Team, Thanks for your feedback. we will make further checking on these.
Meanwhile, could you help check the cause as below two failed request today, not for sure the error reason as mentioned Request ------- MethodName: google.ads.googleads.v12.services.GoogleAdsService/SearchStream Endpoint: googleads.googleapis.com:443 Headers: {developer-token=REDACTED, login-customer-id=5693729523, x-goog-api-client=gl-java/17.0.2 gccl/22.0.0 gapic/22.0.0 gax/2.12.2 grpc/1.44.0 pb/3.19.6} Body: customer_id: "1161207172" query: "SELECT customer.id, campaign.id, campaign_budget.amount_micros FROM campaign WHERE segments.date = \'2022-12-21\' ORDER BY campaign.id" Response -------- Headers: Metadata(content-type=application/grpc,request-id=UprDhyB4aNklsoPcyhKigQ,date=Wed, 21 Dec 2022 06:01:05 GMT,alt-svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43") Body: null Failure message: errors { error_code { internal_error: INTERNAL_ERROR } message: "An internal error has occurred." } request_id: "*UprDhyB4aNklsoPcyhKigQ*" Status: Status{code=INTERNAL, description=Internal error encountered., cause=null}. date:2022-12-21T15:01:05.574+09:00 thread:main track: level:ERROR logger:com.rakuten.rmp.connect.sync.domain.service.googleads.GoogleAdsReaderServiceImpl message:Cannot get Google API Ads - ex_account_id: 1161207172 com.google.ads.googleads.v12.errors.GoogleAdsException: errors { error_code { internal_error: INTERNAL_ERROR } message: "An internal error has occurred." } request_id: "UprDhyB4aNklsoPcyhKigQ" at com.google.ads.googleads.v12.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:42) at com.google.ads.googleads.v12.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:34) at com.google.ads.googleads.lib.stubs.exceptions.BaseGoogleAdsException$Factory.createGoogleAdsException(BaseGoogleAdsException.java:105) at com.google.ads.googleads.lib.callables.GoogleAdsExceptionTransformation.transform(GoogleAdsExceptionTransformation.java:39) at com.google.ads.googleads.lib.stubs.callables.ExceptionTransformingServerStreamingCallable$ExceptionTransformingStreamObserver.onError(ExceptionTransformingServerStreamingCallable.java:71) at com.google.api.gax.tracing.TracedResponseObserver.onError(TracedResponseObserver.java:103) at com.google.api.gax.rpc.RetryingServerStreamingCallable$1.onFailure(RetryingServerStreamingCallable.java:99) at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1100) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1277) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808) at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:200) at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135) at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:117) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1277) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808) at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95) at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77) at com.google.api.core.SettableApiFuture.setException(SettableApiFuture.java:52) at com.google.api.gax.rpc.ServerStreamingAttemptCallable.onAttemptError(ServerStreamingAttemptCallable.java:378) at com.google.api.gax.rpc.ServerStreamingAttemptCallable.access$600(ServerStreamingAttemptCallable.java:97) at com.google.api.gax.rpc.ServerStreamingAttemptCallable$2.onErrorImpl(ServerStreamingAttemptCallable.java:249) at com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:86) at com.google.api.gax.rpc.Watchdog$WatchdogStream.onErrorImpl(Watchdog.java:284) ~~~~~~~~~~~~~~~~~~~~~~ Request ------- MethodName: google.ads.googleads.v12.services.GoogleAdsService/SearchStream Endpoint: googleads.googleapis.com:443 Headers: {developer-token=REDACTED, login-customer-id=5693729523, x-goog-api-client=gl-java/17.0.2 gccl/22.0.0 gapic/22.0.0 gax/2.12.2 grpc/1.44.0 pb/3.19.6} Body: customer_id: "1154526700" query: "SELECT customer.descriptive_name, campaign.name, customer.id, campaign.id, ad_group.id, ad_group.name, ad_group_ad.ad.id, ad_group_ad.ad.name, metrics.cost_micros, metrics.impressions, metrics.average_cpm, metrics.clicks, metrics.average_cpc, metrics.ctr, metrics.video_views, metrics.video_view_rate, metrics.video_quartile_p25_rate, metrics.video_quartile_p50_rate, metrics.video_quartile_p75_rate, metrics.video_quartile_p100_rate, metrics.conversions, metrics.conversions_from_interactions_rate, metrics.cost_per_conversion, metrics.conversions_value FROM ad_group_ad WHERE segments.date = \'2022-12-20\' ORDER BY campaign.id" Response -------- Headers: Metadata(content-type=application/grpc,request-id=t-3-psKzVikj_CTAbb8_-g,date=Wed, 21 Dec 2022 06:01:06 GMT,alt-svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43") Body: null Failure message: errors { error_code { internal_error: INTERNAL_ERROR } message: "An internal error has occurred." } request_id: "*t-3-psKzVikj_CTAbb8_-g*" Status: Status{code=INTERNAL, description=Internal error encountered., cause=null}. date:2022-12-21T15:01:06.710+09:00 thread:main track: level:ERROR logger:com.rakuten.rmp.connect.sync.domain.service.googleads.GoogleAdsReaderServiceImpl message:Cannot get Google API Ads - ex_account_id: 1154526700 com.google.ads.googleads.v12.errors.GoogleAdsException: errors { error_code { internal_error: INTERNAL_ERROR } message: "An internal error has occurred." } request_id: "t-3-psKzVikj_CTAbb8_-g" at com.google.ads.googleads.v12.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:42) at com.google.ads.googleads.v12.errors.GoogleAdsException$Factory.createException(GoogleAdsException.java:34) at com.google.ads.googleads.lib.stubs.exceptions.BaseGoogleAdsException$Factory.createGoogleAdsException(BaseGoogleAdsException.java:105) at com.google.ads.googleads.lib.callables.GoogleAdsExceptionTransformation.transform(GoogleAdsExceptionTransformation.java:39) Thank you very much ~ Regards, Aaron. On Tuesday, December 20, 2022 at 1:52:57 PM UTC+8 adsapi wrote: > Hi Aaron, > > I work with Nirmita. Thanks for getting back to us. > > I understand that you have encountered an RESOURCE_EXHAUSTED > <https://developers.google.com/google-ads/api/docs/best-practices/common-errors#resource_exhausted> > > error, and error message ‘Too many requests. Retry in 900 seconds“. This > seems to be that API requests hitting the server aggressively. The server > throws this error once it detects that your request has exceeded the system > frequency limit. When you retry requests, you may continue using an > exponential backoff policy by delaying 5-10 seconds. For example, if you > first pause 5 seconds before the first retry, you could pause 10 seconds > after the second and 20 seconds after the third retry. Exponential backoff > helps ensure you are not calling the API too aggressively. > > Also, you mentioned that you set your batch job hourly to process sync > campaign data from Google Platform, but in the above email you provided > error logs for the GoogleAdsService/SearchStream API request. So, there > seems to be concurrent API requests making calls to Google Ads API at same > time, and this is why you encountered said error. > > That being said, we suggest setting a reasonable upper bound for the total > number of concurrent tasks that are going to make requests (across all > processes and machines), and adjusting upward to optimize your throughput > without exceeding the rate limit is recommended. Otherwise, you may > possibly experience the exceeding rate limits that the client app is > spawning an excessive number of parallel tasks since we don't limit the > number of parallel requests a client app can have, this can easily exceed > the Requests Per Second limit at the developer token level. However, > Throttles can be implemented at the consumer side by limiting the number of > concurrent consumers, or implement rate limiters or throttlers for either > the producers or consumers.For more information, you may refer to this > throttling QPS from the client side (check out *Throttling and rate > limiters* > <https://developers.google.com/google-ads/api/docs/best-practices/rate-limits?hl=en#Throttling%20and%20rate%20limiters> > ). > > You may apply above mentioned points while sending requests to Google Ads > API server, and let us know how this goes on. > > Regards, > [image: Google Logo] > Yasar > Google Ads API Team > > > ref:_00D1U1174p._5004Q2gQH50:ref > -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog: https://googleadsdeveloper.blogspot.com/ =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To post to this group, send email to adwords-api@googlegroups.com To unsubscribe from this group, send email to adwords-api+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/adwords-api?hl=en --- You received this message because you are subscribed to the Google Groups "Google Ads API and AdWords API Forum" group. To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/d61044eb-49e9-4293-8416-bf9bbbe1b7abn%40googlegroups.com.