Thanks for the reply Rishi. I think I will try the pull queue idea since I was going to move to a model like that after switching to backend instances. I will also try to see how fast my process can finish using a lower amount of instances with this in place, each task takes about 4 seconds and we'd like to finish running within 30 minutes, so 10 instances should be fine.
If the datastore CPU time is being billed against what I see as CPU Time, then this answers my question. The new pricing scheme estimates millions of writes everytime I run my task, I hope I can find a way to reduce this as it will end up costing more than what the current pricing scheme will cost me. Thanks again! -Prateek On Oct 4, 11:27 am, Rishi Arora <[email protected]> wrote: > I don't think backend CPU time counts against your front-end instance CPU > hours quota. Backends are purely billed based on uptime - and I think this > is true for both current and new pricing (starting November 1). But you > mentioned something about datastore CPU? It is likely that the most of your > billing is because of this. > > Few more questions: Is the reason for 20 backend instances that you want to > execute all your 20 deferred tasks parallely? You mentioned you have ~4500 > tasks. How long does each one take, and how often does each one need to > execute in a day? Lets assume that neither of these tasks are sensitive to > latency, and can be executed at any time during the entire day. If each > task takes 10 seconds on average, and needs to execute, for example, 6 times > a day... that's a total of 4500 * 10 * 6 / 3600 = 75 instance hours. You > should try to "schedule" your backends yourself so that you only pay for 75 > instance hours. > > If you allow 20 instances to get created, then at some point all these 20 > instances will complete their work, and then idle for 15 minutes (or at > least billed for 15 minutes of idle time after the last task completes > processing), before they're shutdown. You'll be wasting 5 instance hours > each time this happens. I think your focus should be to minimize your > instance hours by minimizing the number of parallel instances you allow > running. In my calculation above you only need a total of 75 instance > hours, and so you should set "instances" to 3 or 4 in backends.yaml. > > A yet another way of doing this in a more controlled fashion is by using > "pull" queues instead of enqueueing tasks on the regular "push" type task > queues. You can enqueue all your 4500 tasks on a single "pull" queue, and > all your backends will constantly run pulling tasks out of pull queues and > executing them, until the pull queue is empty. Then they can be woken up > again by a cron job to go check the pull queue again. > > Lastly, any cost you're incurring because of Datastore CPU hours in the > current pricing model, or because of Datastore writes in the new pricing > model - those can't be avoided. You will incur those costs regardless of > the context of execution of your tasks - front-end or backend. > > Hope this helps. > > > > > > > > On Tue, Oct 4, 2011 at 9:33 AM, someone1 <[email protected]> wrote: > > Hello, > > > Thank you for the replies. > > > The tasks are indeed being run on my backend as I am being charged for > > backend usage (I don't use backends in any other way otherwise). I > > also see the _ah/deferred logs on my backend ID but not on my app. The > > CPU time being used seems to correspond to the Datastore CPU time, are > > the two currently linked? Even under the new pricing scheme, would I > > be billed for my main app up-time when executing deferred tasks on my > > backend? What part of my main app is considered under use when > > executing tasks on the backend? > > > The max number of instances allowed on any backend is 20. My backend > > is setup as dynamic B1 with 20 instances. I let Google determine how > > many instances needed to be up and running when I queue up my tasks > > (usually all 20 run for 20-30 minutes each). > > > Again, I'd just like to know what part of my main app is being used > > during the backend operation that is eating up my CPU. I really > > haven't coded anything else on my app except for this data mining > > portion which should all be run on the backend now. > > > Thanks, > > Prateek > > > On Oct 4, 8:38 am, Rishi Arora <[email protected]> wrote: > > > I think deferred tasks is an excellent use case for backends. That's how > > I > > > use my backend as well. Can you confirm from your logs that your tasks > > are > > > indeed being processed on the backend? In the drop down for app > > versions, > > > there's a special "version" which is named after your backend. Select > > that > > > to check your logs specific to the backend. Also, I'm assuming the > > reason > > > you're blowing through your budget is because you're spanning out > > multiple, > > > possibly hundreds of instances. Can you find out how many instances get > > > spawned for your deferred tasks? Can you find out how many backend > > > instances are being spawned, if the backend is indeed being used for your > > > tasks? Finally, when you configured your backend, what did you set as > > your > > > "instances" parameter in backends.yaml? I don't know what the default > > is, > > > but it is likely "unlimited". In your case, a instance of 1 or 2 sounds > > > sufficient, but you'll have to play around with that, based on how much > > > queueing occurs for your tasks. > > > > On Tue, Oct 4, 2011 at 12:06 AM, Gerald Tan <[email protected]> > > wrote: > > > > I believe CPU time will no longer be billable after the new pricing is > > out > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Google App Engine" group. > > > > To view this discussion on the web visit > > > >https://groups.google.com/d/msg/google-appengine/-/Crry-7yTG4QJ. > > > > > To post to this group, send email to [email protected] > > . > > > > To unsubscribe from this group, send email to > > > > [email protected]. > > > > For more options, visit this group at > > > >http://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
