Hi Andrei, I just wanted to comment on one little thing in addition to all the stuff Jeff said. I'd not suggest using the task-queue to do all writes by default, but it is a good idea. I would suggest that you try the write, then if you hit an exception (over quota, timeout, deadline, internal error, etc...) defer the write to a task. I've had extremely good luck with this approach on very high sustained write rate apps in the past. If you're wanting to return to the client as fast as possible, set a low deadline for the db put in the user request. That way if you hit a little latency flare-up, you have more control of the user experience.
Robert 2012/1/29 Andrei-Ştefăniţă Cocorean <[email protected]>: > I think my initial message was a bit unclear. I meant that each client > sends on average 1 post/minute and the system has to support several > thousand clients. Hundreds of writes per second may not be that far > away in the future. > > Do you know some best practices in order to sustain the rates you > mentioned? I thought about performing writes from task queues to try > to reduce datastore contention by keeping a constant rate of > operations (regardless of how many clients post to the service at the > same time). > > Does the number of entities in a given group have any impact on the > performance of subsequent writes (transactional or not) to that group? > > > Regards, > Andrei > > > On Jan 28, 7:20 am, Robert Kluin <[email protected]> wrote: >> One of the things that amazes me so much about App Engine is that it can >> handle hundreds, or thousands, of writes per second. I've seen it do this >> numerous times. It takes some knowledge to sustain those rates, but it is >> very possible. >> >> Robert >> >> On Friday, January 27, 2012, Ikai Lan (Google) <[email protected]> wrote:> >> Yes, 1 post a minute is fine. Even 1 write a second is fine. >> > Where you will run into issues is when your application does something >> >> like ~hundreds of writes a second. There are very few systems that will do >> this well, and you will have to start looking into sharding your writes - >> but for now this is a premature optimization. >> >> > -- >> > Ikai Lan >> > Developer Programs Engineer, Google App Engine >> > plus.ikailan.com >> >> > On Thu, Jan 26, 2012 at 1:57 PM, Andrei-Ştefăniţă Cocorean < >> [email protected]> wrote: >> >> >> Hi, >> >> >> I want to develop a web service which will be used by a few thousand >> >> clients to save new pieces of data at a rate of aprox. 1 post/minute. I've >> been looking at what App Engine has to offer but I'm not sure if it's the >> right platform for this kind of system. At a glance it seems the datastore >> is optimized for efficient reads, but not so much for write intensive >> applications. >> >> >> I'd appreciate any help on deciding whether it's worth to try and build >> >> this service on App Engine. Also, if you implemented something similar I'd >> be interested in hearing what problems you encountered along the way. >> >> >> Regards, >> >> Andrei >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups "Google App Engine" group.>> To view this discussion on the web visit >> >> https://groups.google.com/d/msg/google-appengine/-/bW3pKqHKcSEJ.>> To post >> to this group, send email to [email protected]. >> >> To unsubscribe from this group, send email to >> >> [email protected].>> For more options, visit >> this group at >> >> http://groups.google.com/group/google-appengine?hl=en. >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> >> "Google App Engine" group.> To post to this group, send email to >> [email protected]. >> > To unsubscribe from this group, send email to >> >> [email protected].> For more options, visit this >> group at >> >> http://groups.google.com/group/google-appengine?hl=en. >> >> >> >> -- >> ------ >> Robert Kluin >> Ezox Systems, LLC > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
