Re: Interesting list() un-optimization

2013-03-10 Thread Roy Smith
In article <513d18d6$0$6512$c3e8da3$54964...@news.astraweb.com>, Steven D'Aprano wrote: > On Sun, 10 Mar 2013 18:34:58 -0400, Roy Smith wrote: > > > Yeah, that's what I was afraid of. The "obvious" solution of: > > > > class QuerySet(mongoengine.queryset.QuerySet): > > def __init__(self,

Re: Interesting list() un-optimization

2013-03-10 Thread Steven D'Aprano
On Sun, 10 Mar 2013 18:34:58 -0400, Roy Smith wrote: > Yeah, that's what I was afraid of. The "obvious" solution of: > > class QuerySet(mongoengine.queryset.QuerySet): > def __init__(self, document, collection): > super(QuerySet, self).__init__(document, collection) [...] > d

Re: Interesting list() un-optimization

2013-03-10 Thread Roy Smith
In article , Terry Reedy wrote: > > It turns out, we don't actually use QuerySet in our models. We've > > defined our own QuerySet subclass which adds a few convenience methods. > > Adding > > > > def __len__(self): > > raise NotImplemented > > > > to our subclass should do the jo

Re: Interesting list() un-optimization

2013-03-10 Thread Terry Reedy
On 3/10/2013 9:05 AM, Roy Smith wrote: In article , Roy Smith wrote: The problem is, QuerySets have a __len__() method. Calling it is a lot faster than iterating over the whole query set and counting the items, but it does result in an additional database query, which is a lot slower than t

Re: Interesting list() un-optimization

2013-03-10 Thread Roy Smith
In article , Roy Smith wrote: > The problem is, QuerySets have a __len__() method. Calling it is a lot > faster than iterating over the whole query set and counting the items, > but it does result in an additional database query, which is a lot > slower than the list resizing! Writing the c

Re: Interesting list() un-optimization

2013-03-08 Thread Roy Smith
In article <513a26fa$0$30001$c3e8da3$54964...@news.astraweb.com>, Steven D'Aprano wrote: > On Wed, 06 Mar 2013 22:20:11 -0500, Roy Smith wrote: > > > I stumbled upon an interesting bit of trivia concerning lists and list > > comprehensions today. > > > > We use mongoengine as a database model

Re: Interesting list() un-optimization

2013-03-08 Thread Steven D'Aprano
On Wed, 06 Mar 2013 22:20:11 -0500, Roy Smith wrote: > I stumbled upon an interesting bit of trivia concerning lists and list > comprehensions today. > > We use mongoengine as a database model layer. A mongoengine query > returns an iterable object called a QuerySet. The "obvious" way to > crea

Re: Interesting list() un-optimization

2013-03-07 Thread Terry Reedy
On 3/7/2013 3:41 PM, Wolfgang Maier wrote: Iterators do not generally have __len__ methods. len(iter(range(10))) Traceback (most recent call last): File "", line 1, in TypeError: object of type 'range_iterator' has no len() But iterators have a length hint method that are used for some

Re: Interesting list() un-optimization

2013-03-07 Thread Wolfgang Maier
> >>> Iterators do not generally have __len__ methods. > >>> > >>> >>> len(iter(range(10))) > >>> Traceback (most recent call last): > >>> File "", line 1, in > >>> TypeError: object of type 'range_iterator' has no len() > >> > >> But iterators have a length hint method that are used for some >

Re: Interesting list() un-optimization

2013-03-07 Thread Terry Reedy
On 3/7/2013 11:20 AM, Christian Heimes wrote: But iterators have a length hint method that are used for some optimizations and preallocations, too. This is easy when the base iterable has a length method, as do range objects. i = iter(range(10)) i.__length_hint__() 10 And the length_hin

Re: Interesting list() un-optimization

2013-03-07 Thread Terry Reedy
On 3/7/2013 11:00 AM, Ian Kelly wrote: But on this point, you are correct. The mongoengine QuerySet.__iter__ method is defined as: def __iter__(self): self.rewind() return self This is unfortunate design. Not only does it mean that the iterator's __len__ method cannot

Re: Interesting list() un-optimization

2013-03-07 Thread Ian Kelly
On Thu, Mar 7, 2013 at 12:19 PM, Stefan Behnel wrote: >> Didn't know about that, thanks. Presumably a proper iter(QuerySet()) >> object could implement __length_hint__ in an efficient manner rather >> than by just calling the __len__ of the underlying QuerySet, > > And how exactly would it do tha

Re: Interesting list() un-optimization

2013-03-07 Thread Stefan Behnel
Ian Kelly, 07.03.2013 18:31: > On Thu, Mar 7, 2013 at 9:20 AM, Christian Heimes wrote: >> Am 07.03.2013 17:00, schrieb Ian Kelly: >>> On Thu, Mar 7, 2013 at 4:22 AM, Wolfgang Maier wrote: Well, it skips the costly len() call because your iter(Foo()) returns iter(range()) under the hood an

Re: Interesting list() un-optimization

2013-03-07 Thread Ian Kelly
On Thu, Mar 7, 2013 at 9:20 AM, Christian Heimes wrote: > Am 07.03.2013 17:00, schrieb Ian Kelly: >> On Thu, Mar 7, 2013 at 4:22 AM, Wolfgang Maier >> wrote: >>> Well, it skips the costly len() call because your iter(Foo()) returns >>> iter(range()) under the hood and list() uses that object's __

Re: Interesting list() un-optimization

2013-03-07 Thread Christian Heimes
Am 07.03.2013 17:00, schrieb Ian Kelly: > On Thu, Mar 7, 2013 at 4:22 AM, Wolfgang Maier > wrote: >> Well, it skips the costly len() call because your iter(Foo()) returns >> iter(range()) under the hood and list() uses that object's __len__() method. > > Iterators do not generally have __len__ me

Re: Interesting list() un-optimization

2013-03-07 Thread Ian Kelly
On Thu, Mar 7, 2013 at 4:22 AM, Wolfgang Maier wrote: > Well, it skips the costly len() call because your iter(Foo()) returns > iter(range()) under the hood and list() uses that object's __len__() method. Iterators do not generally have __len__ methods. >>> len(iter(range(10))) Traceback (most r

Re: Interesting list() un-optimization

2013-03-07 Thread Wolfgang Maier
Tim Chase tim.thechases.com> writes: > On 2013-03-06 22:20, Roy Smith wrote: > > I stumbled upon an interesting bit of trivia concerning lists and > > list comprehensions today. > > A little testing > shows that this can be rewritten as > > my_objects = list(iter(my_query_set)) > > which se

Re: Interesting list() un-optimization

2013-03-06 Thread Kev Dwyer
Roy Smith wrote: > I stumbled upon an interesting bit of trivia concerning lists and list > comprehensions today. > > We use mongoengine as a database model layer. A mongoengine query > returns an iterable object called a QuerySet. The "obvious" way to > create a list of the query results would

Re: Interesting list() un-optimization

2013-03-06 Thread Tim Chase
On 2013-03-06 22:20, Roy Smith wrote: > I stumbled upon an interesting bit of trivia concerning lists and > list comprehensions today. I agree with Dave Angel that this is interesting. A little testing shows that this can be rewritten as my_objects = list(iter(my_query_set)) which seems to th

Re: Interesting list() un-optimization

2013-03-06 Thread Dave Angel
On 03/06/2013 10:20 PM, Roy Smith wrote: I stumbled upon an interesting bit of trivia concerning lists and list comprehensions today. We use mongoengine as a database model layer. A mongoengine query returns an iterable object called a QuerySet. The "obvious" way to create a list of the query

Interesting list() un-optimization

2013-03-06 Thread Roy Smith
I stumbled upon an interesting bit of trivia concerning lists and list comprehensions today. We use mongoengine as a database model layer. A mongoengine query returns an iterable object called a QuerySet. The "obvious" way to create a list of the query results would be: my_objects = list