I meant to skip count. On Saturday, 20 October 2012 15:28:56 UTC-5, Massimo Di Pierro wrote: > > How about adding a gae only parameter to the gae adapter_args that tells > it to skip fetch? > > On Saturday, 20 October 2012 11:25:51 UTC-5, howesc wrote: >> >> It appears that the most efficient way to delete on app engine is to: >> - build a query object, like we are doing now >> - call run with keys_only=True ( >> https://developers.google.com/appengine/docs/python/datastore/queryclass#Query_run) >> >> which returns an iterator. >> - pass that iterator to the datastore delete method ( >> https://developers.google.com/appengine/docs/python/datastore/functions#delete >> ) >> >> this avoids the cost of loading the rows into memory, decreases the >> likelihood of timeout, and has the cost of 1 datastore small operation per >> row. but it does prevent us from getting a count of rows deleted. >> >> the way we do it now: >> - run count() on the query. this has a cost (time and money) of >> iterating over all the rows that match the query on GAE (1 datastore small >> operation per row) >> - run fetch(limit=1000) and call delete() successively until no more >> rows. this has the cost of running a full query (at least 1 datastore read >> operation per row) and loading the result set into memory and then deleting >> the results. >> >> in my case i'm timing out on the count() call so i don't even start the >> delete. from an efficiency standpoint i'd rather have more rows deleted >> for less cost then get a count....but this may not be acceptable for all. >> at a minimum i think we should switch to use keys_only=True for the fetch, >> and skip the leading count() call and just sum the number of times we call >> fetch. we may also consider catching the datastore timeout error and >> trying to handle a partial delete more gracefully (or continue to let the >> user catch the error). >> >> what is the "right" approach for web2py? if the approach with count is >> correct, could i propose a gae bulk_delete method that does not return >> count but uses my first method? >> >> thanks for the input! >> >> cfh >> >> On Saturday, October 20, 2012 7:58:56 AM UTC-7, Massimo Di Pierro wrote: >>> >>> Delete should return the number of deleted records. What is your >>> proposal? >>> >>> On Wednesday, 17 October 2012 17:30:22 UTC-5, howesc wrote: >>>> >>>> Hi all, >>>> >>>> I'm trying to clean up old expired sessions.....but i waited a long >>>> time to get to this and now my GAE delete is just timing out. Reading the >>>> GAE docs, there appears to be some improvements that we can make to the >>>> query delete method on GAE that will make it faster and cheaper. what we >>>> lose then is the count of the number of rows deleted. >>>> >>>> my question is, does having a db(db.table.something==True).delete() >>>> that does not return a count break the web2py API contract, or break >>>> anyone's applications? >>>> >>>> thanks, >>>> >>>> christian >>>> >>>
--
