It turns out the speed problem is REALLY bad. I have a table with
virtualfields of 14,000 rows. When I run raw sql:
a = db.executesql('select * from people;')
...the query returns in 121ms. But when I run it through the DAL on
only a subset of the data:
a = db(db.people.id > 0).select(limitby=(0,1000))
...it returns in 141096.431ms. That's... 141 seconds. So 1000x longer
on .1 of the database.
My virtualfields are all lazy functions. I'm looking into what's
causing it and will report back when I find out. It seems it might
have something to do with the lazy decorator func because when I hit C-
c the code is often stuck there... inside import copy or something.
def lazy(f):
def g(self,f=f):
import copy
self=copy.copy(self)
return lambda *a,**b: f(self,*a,**b)
return g
Anyway, I'll send an update when I have more info.
On Aug 2, 3:03 pm, Michael Toomim <too...@gmail.com> wrote:
> That's way better syntax! Great idea!
>
> On Aug 2, 2011, at 2:31 AM, Massimo Di Pierro wrote:
>
>
>
>
>
>
>
> > We need to work on the speed. This can perhaps help the syntax:
>
> > db=DAL()
> > db.define_table('a',Field('b','integer'))
> > for i in range(10):
> > db.a.insert(b=i)
>
> > def lazy(f):
> > def g(self,f=f):
> > import copy
> > self=copy.copy(self)
> > return lambda *a,**b: f(self,*a,**b)
> > return g
>
> > class Scale:
> > @lazy
> > def c(self,scale=1):
> > return self.a.b*scale
>
> > db.a.virtualfields.append(Scale())
> > for row in db(db.a).select():
> > print row.b, row.c(1), row.c(2), row.c(3)
>
> > On Aug 1, 3:10 pm, Michael Toomim <too...@gmail.com> wrote:
> >> Maybe it helps for me to explain my use-case. I mainly use virtual fields
> >> as lazy methods, to help traverse related tables. I was actually surprised
> >> that lazy evaluation wasn't the default. I noticed a few implications of
> >> this:
> >> - Large queries are slowed byvirtualfields, even if they won't be
> >> needed, esp if they query db
> >> - My definitions forvirtualfieldsaren't as clean as they could be,
> >> because I have many nested "lazy" funcs in the class definition
> >> - We can't serialize all objects intosessionvariables
>
> >> So really I'm just using this because it's a nicer notation to call
> >> row.otherthing() instead of getotherthing(row). Maybe I really want some
> >> different feature here?
>
> >> On Aug 1, 2011, at 5:40 AM, Anthony Bastardi wrote:
>
> >>> Note, after looking at this some more, Massimo recalled that the reason
> >>> auth_user virtual fields were excluded from auth.user (and therefore from
> >>> saving in thesession) is because some virtual fields are objects that
> >>> cannot be pickled and therefore cannot be serialized to store in
> >>> thesession. So, we're thinking of either creating an option to store
> >>> auth_user virutual fields in auth.user, or maybe testing to make sure the
> >>> virtual fields can be pickled, and excluding them if not.
>
> >>> Anthony
>
> >>> On Mon, Aug 1, 2011 at 5:30 AM, Michael Toomim <too...@cs.washington.edu>
> >>> wrote:
> >>> Awesome! I did not know there was an issue submission system.
>
> >>> On Jul 30, 2011, at 7:02 AM, Anthony wrote:
>
> >>>> An issue has been submitted, and this should be corrected soon.
>
> >>>> Anthony
>
> >>>> On Friday, July 29, 2011 9:57:30 PM UTC-4, Anthony wrote:
> >>>> auth.user is Storage(table_user._filter_fields(user, id=True)). The
> >>>> _filter_fields method of the auth_user table only selects actual table
> >>>> fields, not virtual fields, so auth.user will not include any virtual
> >>>> fields. Perhaps this should be changed.
>
> >>>> Anthony
>
> >>>> On Friday, July 29, 2011 9:05:39 PM UTC-4, Michael Toomim wrote:
> >>>> I think I found a bug invirtualfields. I have the following
> >>>> controller:
>
> >>>> def posts():
> >>>> user =session.auth.user
> >>>> n = user.name # returns None
>
> >>>> Where "person" is defined as a virtualfield on user:
>
> >>>> class Users():
> >>>> def name(self):
> >>>> return self.users.first_name + ' ' + self.users.last_name
> >>>> db.users.virtualfields.append(Users())
>
> >>>> The problem is that user.name returns None, because apparently the
> >>>> virtualfield isn't loaded into thesessionvariable of user.
>
> >>>> I made this work with the following modification to the controller:
>
> >>>> def posts():
> >>>> user = db.users[session.auth.user.id]
> >>>> n = user.name # returns the user name correctly!
>
> >>>> I just had to refetch the user from the database.
