Thanks for the clarification, Greg. It sounds that the main motivation for these changes is the realization that idle instances floating around cost more than originally anticipated. The new pricing model incentivizes developers to optimize their apps to reduce idle time. I understand why this makes sense.
The issue, for me at least, is that I assumed (or wished) that GAE will optimize everything in such a way that I don't need to optimize my own code. If I do a urlfetch that takes 5 seconds, GAE will somehow utilize that idle waiting time efficiently for other things. And I also don't need to use threads because running work serially or in parallel costs the same because GAE works at such a big scale that it can optimize things better than I can. That's the vision I wished for, and it's what I feel I'm losing. But I understand if the technology is not there to make this a reality yet. I can work with that and adjust my "wishful" expectations accordingly. I did back-of-the-envelope calculations based on the numbers you posted, and it seems that I'm looking at an increase of $10-15K to my monthly bill. That's a significant enough number to get me worried. The big chunk of the cost for me is in datastore ops, so optimizing to reduce idle time won't help me that much. I'm looking for things to optimize and indexes to delete. It's not clear yet whether the new prices are too expensive or the current prices are too cheap. It would great if you guys provide a well-balanced price comparison to help us understand the value we're getting. Waleed On Sat, Jun 25, 2011 at 12:24 AM, Gregory D'alesandre <[email protected]>wrote: > On Sat, Jun 25, 2011 at 12:24 AM, Waleed Abdulla <[email protected]> wrote: > >> A: Many customers have optimized for low CPU usage to keep bills low, >>> but in turn are often using a large amount of memory (by having high latency >>> applications). >> >> >> How does that work? Can you share an example? >> >> Also, instead of pushing the burden of supporting threading on to the >> developers, shouldn't GAE optimize resources behind the scenes? You're >> running many instances on every physical computer, so when one instance is >> idel, others should be using the CPU and other resources. If keeping an idle >> instance in memory costs too much, then adjust the pricing by adding a cost >> per idle second, or something along these lines. >> > > That is in essence what we've done, added a charge having an instance > residing in memory, the difference is that we will not be billing the CPU > separately anymore. > > >> >> My point is: GAE has a lot of limitations compared to other solutions, but >> it has one HUGE advantage: it's simple and scales automatically. That >> advantage is so big that it trumps all the other limitations (at least for >> me). Now that I have to manage my instances and rewrite my code to handle >> threading, I can't help but feel that I'm losing that one big advantage. >> > > Sorry that hear you feel that way Waleed. But maybe there is a > misunderstanding, you don't have to manage instances at all (in fact the > only options we give you allow you to tune how we manage your instances). > It still scales automatically. In terms of rewriting code to handle > threading, we suggest it because it means you can do more with less memory > and it will save you money but if you'd rather not and are willing to pay > for it, you don't have to. > > Hope that helps! > > Greg > > >> >> >> Waleed >> >> >> >> On Fri, Jun 24, 2011 at 4:55 PM, Waleed Abdulla <[email protected]> wrote: >> >>> Gregory, Thanks for the update. >>> >>> I think the scheduler min & max pending latency should be per url (i.e. >>> in app.yaml) rather than being global at the app level. Most apps have >>> requests that require a quick response (UI and APIs), and others that don't >>> (cron, some tasks, backend work, ..etc). I'd probably want to set my max >>> pending latency to 50ms for UI requests and 5 seconds for some non-urgent >>> tasks. >>> >>> Waleed >>> >>> >>> On Fri, Jun 24, 2011 at 4:04 PM, Jeff Schnitzer <[email protected]>wrote: >>> >>>> Thanks, this clarifies much! Questions below: >>>> >>>> On Thu, Jun 23, 2011 at 11:49 PM, Gregory D'alesandre <[email protected] >>>> > wrote: >>>> >>>>> Datastore APIs Q: Which operations are being charged for? >>>>> A: There are 3 categories of Datastore operations: >>>>> - Write operations (Entity Put, Entity Delete, Index Write), each of >>>>> these operations will cost $0.10 per 100k operations >>>>> - Read operations (Query, Entity Fetch), each of these operations will >>>>> cost $0.07 per 100k operations >>>>> - Small operations (Key Fetch, Id Allocation), each of these operations >>>>> will cost $0.01 per 100k operations >>>>> >>>>> Q: Under the new scheme, is it more economical to do a keys-only >>>>> query that fetches 1000 keys, and then do a get on the 500 of them that I >>>>> need, or just do a regular (non keys-only) query for all 1000 directly? >>>>> A: The first is more economical. Fetching 1000 keys + fetching 500 >>>>> entities = $0.0001 + 0.00035 = $0.00045; fetching 1000 entities = $0.0007. >>>>> >>>> >>>> This makes sense, and encourages more use of memcache. to hold entities. >>>> One question that I've been wondering a while - presuming no caching, does >>>> this query-keys+batch-get approach produce higher latency than a simple >>>> query, and if so, by how much? >>>> >>>> Also, is there any way we can get the transaction timestamp out on >>>> datastore writes? This would *dramatically* improve the robustness of code >>>> that tries to keep memcache in sync with the datastore during contention. >>>> I've spoken with Alfred and Max about this, but I don't know if it's a >>>> priority. This could potentially reduce datastore bills by orders of >>>> magnitude. >>>> >>>> Thanks, >>>> Jeff >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Google App Engine" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group at >>>> http://groups.google.com/group/google-appengine?hl=en. >>>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
