Just raise one transactional task that in turn spawns a series of (non-transactional) _named_ sub-tasks. Just remember to "pass" on a "TaskAlreadyExists" error.
See Brett Slatkin's 2010 IO talk on pipelines for additional discussion. Robert On Wed, Jul 7, 2010 at 12:49 PM, hawkett <[email protected]> wrote: > Thanks for the detail - pretty comfortable with the ins and outs of > the task queue - any ideas about the 5 task limit? > >> Also, don't confuse transactional tasks with the task queue. Transactional >> tasks are the those that will all happen simultaneously or will graciously >> fail without any partial commits. Due to the implementation details you >> probably want to avoid intentionally doing lots of transaction updates on >> the same few objects, since it is possible (if you are kicking lots of >> simultaneous jobs off with the task queue) to shoot yourself in the foot and >> have a very small success rate. > > I wasn't confused until I read this paragraph :) Transactional tasks > do not need to operate on the same entity group as the transaction in > which they are raised - this is one of their greatest strengths. They > are fast to raise, so you can actually include quite a lot of work in > a 'transaction' of this sort, all done in parallel. Generally if I > want to do something in the same entity group as the originating > transaction, I would do it right there in the transaction and not > raise a task. So at least for me (and I expect for many others), > transactional tasks are almost exclusively *not* operating on the same > entity group as the transaction in which they are raised, unless > perhaps they are being used in a continuation style bulk update, in > which case they are likely to be in series and not cause contention. > > Anyway, question still there :) Is the 5 task limit a restriction that > is expected to remain indefinitely? Can I get around the restriction > by raising more in a single batch API call - i.e. is it a restriction > on API calls or actual tasks? Cheers, > > Colin > > On Jul 7, 4:46 pm, Nate Bauernfeind <[email protected]> > wrote: >> I have noticed that batch datastore calls run within a single transaction. >> The maximum number of entities that can be added or deleted (and probably >> modified, though I have not tried) was 500. I'm betting you could probably >> wrap them around a single transaction. Though, from my experience, I >> wouldn't really recommend doing this (since trying to commit two batches of >> 500 to the datastore within the same call tended to time out for me). >> >> Also, don't confuse transactional tasks with the task queue. Transactional >> tasks are the those that will all happen simultaneously or will graciously >> fail without any partial commits. Due to the implementation details you >> probably want to avoid intentionally doing lots of transaction updates on >> the same few objects, since it is possible (if you are kicking lots of >> simultaneous jobs off with the task queue) to shoot yourself in the foot and >> have a very small success rate. >> >> Transactions work with the task queue in such a way that the task will only >> be added to the queue if no other piece of your commits fail. For example, >> you wouldn't really want your app to run the "new user" code if you couldn't >> create the new user account for that user because someone else registered >> that user name at the same time. I.E. Things on the task queue do not >> continue running within the same transaction if they were created within >> one. >> >> >> >> On Wed, Jul 7, 2010 at 4:22 AM, hawkett <[email protected]> wrote: >> > Hi, >> >> > This page - >> >http://code.google.com/appengine/docs/python/datastore/transactions.html >> > - says that we cannot raise more than 5 transactional tasks in a >> > single transaction, and I wanted to check if this was a limit that you >> > were hoping to raise, or if this is likely to be a long term >> > restriction? Does this restriction limit API calls to the task queue >> > or actual number of tasks - e.g. could I raise more than 5 by doing >> > them in batch with a single call to the task queue API? Cheers, >> >> > Colin >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]<google-appengine%2Bunsubscrib >> > [email protected]> >> > . >> > For more options, visit this group at >> >http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
