There are really only two hard issues that will limit your write rate, assuming adequate budget:
1) Limit of one write per second per entity group. This one is usually really easy to work around; just break your app up into many entity groups. Most apps tend to naturally break down by user or by business or some other categorization that has a low per-unit write rate. As long as you are cognizant of query eventuality and XG transaction limits, you can scale up to any write rate. If each user is a separate entity group, it's like each user is running on it's own little database. The problems tend to appear when you need accurate runtime counts across changing data. At this point look into sharded counters. You shouldn't need to do this often. 2) Throughput limit on tablet splits for increasing index values You won't hit this until many hundreds of writes per second. The problem is when you have an index on a more-or-less monotonically increasing field like say a timestamp. When the index is updated, the writes will always be to the end of the table... and you'll get a "hot tablet" that will split (causing a delay), then another "hot tablet" since you're always writing to the end. The HRD helps in that it gives you a multiple of the total write rate, but you still get a limit. Ikai wrote about this (and drew some awesome cartoons) here: http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-increasing-values-are-bad/ If you design with these two issues in mind, you shouldn't have any problem doing thousands of writes per second... or whatever you can afford. Jeff 2012/1/29 Andrei-Ştefăniţă Cocorean <[email protected]>: > I think my initial message was a bit unclear. I meant that each client > sends on average 1 post/minute and the system has to support several > thousand clients. Hundreds of writes per second may not be that far > away in the future. > > Do you know some best practices in order to sustain the rates you > mentioned? I thought about performing writes from task queues to try > to reduce datastore contention by keeping a constant rate of > operations (regardless of how many clients post to the service at the > same time). > > Does the number of entities in a given group have any impact on the > performance of subsequent writes (transactional or not) to that group? > > > Regards, > Andrei > > > On Jan 28, 7:20 am, Robert Kluin <[email protected]> wrote: >> One of the things that amazes me so much about App Engine is that it can >> handle hundreds, or thousands, of writes per second. I've seen it do this >> numerous times. It takes some knowledge to sustain those rates, but it is >> very possible. >> >> Robert >> >> On Friday, January 27, 2012, Ikai Lan (Google) <[email protected]> wrote:> >> Yes, 1 post a minute is fine. Even 1 write a second is fine. >> > Where you will run into issues is when your application does something >> >> like ~hundreds of writes a second. There are very few systems that will do >> this well, and you will have to start looking into sharding your writes - >> but for now this is a premature optimization. >> >> > -- >> > Ikai Lan >> > Developer Programs Engineer, Google App Engine >> > plus.ikailan.com >> >> > On Thu, Jan 26, 2012 at 1:57 PM, Andrei-Ştefăniţă Cocorean < >> [email protected]> wrote: >> >> >> Hi, >> >> >> I want to develop a web service which will be used by a few thousand >> >> clients to save new pieces of data at a rate of aprox. 1 post/minute. I've >> been looking at what App Engine has to offer but I'm not sure if it's the >> right platform for this kind of system. At a glance it seems the datastore >> is optimized for efficient reads, but not so much for write intensive >> applications. >> >> >> I'd appreciate any help on deciding whether it's worth to try and build >> >> this service on App Engine. Also, if you implemented something similar I'd >> be interested in hearing what problems you encountered along the way. >> >> >> Regards, >> >> Andrei >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups "Google App Engine" group.>> To view this discussion on the web visit >> >> https://groups.google.com/d/msg/google-appengine/-/bW3pKqHKcSEJ.>> To post >> to this group, send email to [email protected]. >> >> To unsubscribe from this group, send email to >> >> [email protected].>> For more options, visit >> this group at >> >> http://groups.google.com/group/google-appengine?hl=en. >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> >> "Google App Engine" group.> To post to this group, send email to >> [email protected]. >> > To unsubscribe from this group, send email to >> >> [email protected].> For more options, visit this >> group at >> >> http://groups.google.com/group/google-appengine?hl=en. >> >> >> >> -- >> ------ >> Robert Kluin >> Ezox Systems, LLC > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
