Excellent, thanks! One question though: isn't there an issue similar to the HRD 'Eventual Consistency' with the Task Queue? In other words, there is a variable latency between queue insert and lease availability that could potentially spike high enough so that the fan- in task misses some work.
If this is true, we still need some sort of cleanup mechanism for a robust implementation. I have several ideas for this, but wanted to make sure I wasn't missing something. On May 1, 7:48 pm, Brett Slatkin <[email protected]> wrote: > Indeed! > > > For each unit of work: > > -Write work to Pull queue with tag=BatchID > > -Write named 'fan-in' task to Push Queue for execution <batch period ms> > > in the future, containing the BatchID > > Example:http://code.google.com/p/8-bits/source/browse/trunk/backend/main.py#168 > > > When the named task executes: > > -Lease all available tasks from the Pull queue using the tag=BatchID > > -Aggregate the work > > -Apply the update to the Datastore > > > Is a feasible/appropriate/recommended use of the Pull Queue tag mechanism? > > Example:http://code.google.com/p/8-bits/source/browse/trunk/backend/main.py#202 > > I haven't load tested this yet, but I think it should work? > > -Brett -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
