Hey Hemanth, There remain, after your post, some questions as to how you've implemented your system. The rate of inserts according to 47,000 / hour is approximately 13 / second. That seems very low - the maximum rate of inserts for streaming inserts is 100,000 rows per second, per table and 1,000,000 rows per second, per project. <https://cloud.google.com/bigquery/streaming-data-into-bigquery>
Correct me if I'm wrong, but this starts to look like this number (47,000 in 1 hour) represents the overall time of the entire pipeline from API call to sending the BigQuery insert(), not simply the rate at which inserts could theoretically take place. Ultimately, the best way to insert BigQuery rows is not with isolated HTTP requests, which have a lot of overhead costs, but with batch inserts. It might be worth looking into ways that you could aggregate the records in a layer after the task pipeline but before BigQuery which would allow you to send big batched inserts. Let me know your thoughts on this, and best wishes, Nick Cloud Platform Community Support On Friday, March 18, 2016 at 8:46:59 AM UTC-4, Hemanth Kumar wrote: > > > HI ALL > > Urgently required some help. > > I am facing a slowness problem while writing the data to the big query. > > My problem is I have 5000 task , and each task is interacting with the > Rest Api. The rest api is giving me the JSON response. I am parsing the > JSON response, and the same response I have to write to big query. > > Every JSON response will give me around 3000 List of JSON Array. > If I calculate the total data which will get inserted in the google big > query is around 15000000. > To insert 47000 record its taking 1 hour. If I have to insert lakhs of > record of data , my performance is getting hit. > > Please give a good suggestion to improve the time complexity for this > problem. How I will resolve this using Google App Engine, > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/21fa5145-ac51-422a-8513-c660bdcb0157%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
