Re: QuerySet: "if obj in queryset" can be very slow

2011-11-03 Thread Tom Evans
On Wed, Nov 2, 2011 at 6:26 PM, Ian Clelland wrote: > Just looking at the source to QuerySet (finally), and it looks like the > __contains__ method actually does something different than this: It > evaluates the whole QuerySet in bulk at the database level, and starts > creating model instances ba

Re: QuerySet: "if obj in queryset" can be very slow

2011-11-02 Thread Thomas Guettler
Hi, I created a ticket: https://code.djangoproject.com/ticket/17156 {{{ The documentation should explain what happens in this code: if obj in queryset: If queryset is huge, the above code can be very slow. If you want to run the code in the database, you should use this: if queryset

Re: QuerySet: "if obj in queryset" can be very slow

2011-11-02 Thread Ian Clelland
On Wed, Nov 2, 2011 at 10:46 AM, Tom Evans wrote: > OK, take this example. I have a django model table with 70 million > rows in it. Doing any kind of query on this table is slow, and > typically the query is date restrained - which mysql will use as the > optimum key, meaning any further filteri

Re: QuerySet: "if obj in queryset" can be very slow

2011-11-02 Thread Ian Clelland
On Wed, Nov 2, 2011 at 10:46 AM, Tom Evans wrote: > On Wed, Nov 2, 2011 at 5:30 PM, Ian Clelland wrote: > > On Wed, Nov 2, 2011 at 8:25 AM, Thomas Guettler wrote: > >> > >> # This is my current solution > >> if get_special_objects().filter(pk=obj.pk).count(): > >># yes, it is special > >> >

Re: QuerySet: "if obj in queryset" can be very slow

2011-11-02 Thread Furbee
Thanks Tom, that's a great explanation! Furbeenator On Wed, Nov 2, 2011 at 10:46 AM, Tom Evans wrote: > On Wed, Nov 2, 2011 at 5:30 PM, Ian Clelland wrote: > > On Wed, Nov 2, 2011 at 8:25 AM, Thomas Guettler wrote: > >> > >> # This is my current solution > >> if get_special_objects().filter(p

Re: QuerySet: "if obj in queryset" can be very slow

2011-11-02 Thread Tom Evans
On Wed, Nov 2, 2011 at 5:30 PM, Ian Clelland wrote: > On Wed, Nov 2, 2011 at 8:25 AM, Thomas Guettler wrote: >> >> # This is my current solution >> if get_special_objects().filter(pk=obj.pk).count(): >>    # yes, it is special >> > > I can't speak to the "why" of this situation; it seems to me th

Re: QuerySet: "if obj in queryset" can be very slow

2011-11-02 Thread Ian Clelland
On Wed, Nov 2, 2011 at 8:25 AM, Thomas Guettler wrote: > # This is my current solution > if get_special_objects().filter(pk=obj.pk).count(): ># yes, it is special > > I can't speak to the "why" of this situation; it seems to me that this could always be converted into a more efficient databas

Re: QuerySet: "if obj in queryset" can be very slow

2011-11-02 Thread Tom Evans
On Wed, Nov 2, 2011 at 3:25 PM, Thomas Guettler wrote: > Here is a better example: > > def get_special_objects(): >    # Some are special. But the result can still be huge! >    return MyModel.objects.filter() > > obj=MyModel.objects.get() > > # is this object "special?" > if obj in get_sp

Re: QuerySet: "if obj in queryset" can be very slow

2011-11-02 Thread Thomas Guettler
Here is a better example: def get_special_objects(): # Some are special. But the result can still be huge! return MyModel.objects.filter() obj=MyModel.objects.get() # is this object "special?" if obj in get_special_objects(): # This is very slow if there are many rows in the res

Re: QuerySet: "if obj in queryset" can be very slow

2011-11-02 Thread Jirka Vejrazka
> queryset=MyModel.objects.all() > obj=MyModel.objects.get(id=319526) Thomas, if the second line works for you (i.e. does not raise MyModel.DoesNotExist exception), then obj is in queryset by definition. What are you trying to achieve? Jirka -- You received this message because you are

Re: QuerySet: "if obj in queryset" can be very slow

2011-11-02 Thread Flavia Missi
That's because your queryset is being evaluated when you compare, maybe if you explain your problem, we can give you a better solution than the use of `in`. Take a look at the docs about when querysets are evaluated: https://docs.djangoproject.com/en/dev/ref/models/querysets/#when-querysets-are-e

QuerySet: "if obj in queryset" can be very slow

2011-11-02 Thread Thomas Guettler
Hi, I just discovered, that "if obj in queryset" can be very slow: queryset=MyModel.objects.all() obj=MyModel.objects.get(id=319526) #if obj in queryset: # This is very slow, if queryset is big: All lines from queryset get created to Python objects #print 'in' if queryset.filter(id=obj.id)