Since your app is not in production yet, you could test the latency by changing your scaling from auto scaling to basic or manual scaling in app.yaml (see Known Issues <https://cloud.google.com/endpoints/docs/known-issues#poor_performance_on_automatic_scaling_instances>) and compare the latencies of both scaling settings.
I would also compare the traces with a request that was directly sent to your API's backend, without Cloud Endpoints. Perhaps the problem is in your app, not with Endpoints. I'm not 100% sure when the trace starts, but as far as I understand the documentation, the timeline of a trace starts when the (App Engine) application received the incoming request, not when Cloud Endpoints received the request (mentioned here <https://cloud.google.com/trace/docs/overview#trace> and here <https://cloud.google.com/trace/docs/viewing-details#trace_details>). If it's true that Cloud Endpoints latency is not shown in the trace, then I'd assume that you would see the same high latency if you send a request directly to the API's backend. However, by following the logs of a trace or by comparing the trace timestamps with those of the matching logs in App Engine and Cloud Endpoints, you should be able to verify whether Endpoints latency is shown in a trace or not. (the log in Endpoints <https://cloud.google.com/endpoints/docs/frameworks/ref-endpoints-logs> would start before the log in App Engine <https://cloud.google.com/appengine/docs/standard/java/logs/#understanding_request_log_fields> and end after the log in App Engine ended) If the trace is just for your App Engine app, I would try to figure out what actually happens in the long block of *untraced* time, i.e. before you see the first span, e.g. the first RPC to datastore. I don't know how it is with Java standard, but for Python standard, the first RPCs show up after maybe 10ms, 20ms - or if it's the first request of a new instance maybe after 400ms. You could use Cloud Logging and Cloud Debug to get a better understanding why it takes so much time until the first RPC call. I hope this helps. Best regards, Ani On Tuesday, June 12, 2018 at 9:53:40 AM UTC+2, Azeem Haider wrote: > > I'm using Automatic Scaling. > > Sorry, in my first post I did not upload stack traces. Check these below > > > <https://lh3.googleusercontent.com/--gXFV6gA5JI/Wx97MMqH8bI/AAAAAAAACnY/duFBWRRYrkw8PVoUteTJpXtfU5X2W7AJACLcBGAs/s1600/datastore-trace-report.PNG> > > > Here is RPCs. And can you please what RPCs and LATENCY time mean here ? > > > <https://lh3.googleusercontent.com/-fSJC0V-Sgnw/Wx97b2kYhDI/AAAAAAAACng/eO6ARGQKGb8KYpnUpiheFVvZmSCUKyZAwCLcBGAs/s1600/rpc.PNG> > > > Expanding trace for most user URI. > > > <https://lh3.googleusercontent.com/-XYB6E0iU7zA/Wx97pSp0fYI/AAAAAAAACnk/9N0JTDPkVZ86RSyvi_y1Gt4Bj2Nu4mTewCLcBGAs/s1600/most%2Bused.PNG> > > > Here is trace detail of user request that I mention in my question, if you > see it fetch from memory but still function take around 2 sec > > > <https://lh3.googleusercontent.com/-dvGOxd_ryaI/Wx97-GAcUUI/AAAAAAAACnw/tT79g8953bE9uBlpGrbQcdGoYlMaeIAigCLcBGAs/s1600/user1.PNG> > > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/0d95dfd3-6e5c-4272-88a4-79fa5f32964e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
