First of all thank you for your through tests into the DAL.
I worked into this specific issue and I fixed the discrepancy. It
deserves an explanation anyway. Here is a minimal example to reproduce
the problem:
code
db.define_table('b', Field('b'))
db.define_table('c', Field('d'))
print db()._select(db.b.ALL, db.c.ALL)
output with sql.py
SELECT b.id, b.b, c.id, c.d FROM c, b;
output with dal.py (before fix in trunk)
SELECT b.id, b.b, c.id, c.d FROM b;
While only the latter fails, they are both wrong because the original
query is wrong. It is doing an inner join but it is missing a link
between table. sql.py assumes you want a cartesian product but that is
not a good idea because the number of records blows up.
dal.py did assumes anything (after the fix behaves as sql.py).
In your code you use this as
if len(db().select(db.person.ALL, db.dog_type.ALL, db.dog.ALL))==0
and if you have 3000 records (1000 in each table) this returns 10^9
rows. This runs out out RAM. I think you simply want
if db(db.person.id)count()+db(db.dog_type.id).count()
+db(db.dog.ALL).count():
On Apr 8, 2:57 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
> Massimo, I am attaching a application that can replicate this issue.
>
> Works fine on sql.py
>
> Breaks on dal.py
>
> This application is designed after my big application, the represent
> functions HAVE to stay the same, there cannot be ANY code edits to
> them!
>
> Make sure to run with sql.py first to confirm that it IS working
> correctly. THEN copy dal.py to sql.py, restart web2py, and then
> confirm that it no longer works.
>
> -Thadeus
>
> On Thu, Apr 8, 2010 at 10:12 AM, Thadeus Burgess <thade...@thadeusb.com>
> wrote:
> > Agreed. Lets narrow this down, I will see if I can replicate the issue
> > on a smaller scale.
>
> > -Thadeus
>
> > On Wed, Apr 7, 2010 at 11:08 PM, Massimo Di Pierro
> > <mdipie...@cs.depaul.edu> wrote:
> >> I guess my question is " why does it not throw an exception in sql.py?" It
> >> should since
>
> >>>> db.dog.owner.represent = lambda value: "%s" % db.person[value].name
>
> >> when called with value==None should always result in the error you see in
> >> dal.py.
>
> >> Can you help me debug this? My problem is not why it does not work with
> >> dal.py (that is the part I understand). My problem is how is it that it
> >> works with sql.py?
>
> >> On Apr 7, 2010, at 10:58 PM, Thadeus Burgess wrote:
>
> >>> I am not using different datasets, or using different queries.
>
> >>> So here we go, let me explain in every minuet detail this process.
>
> >>>>>> cd ~
> >>>>>> hg clonehttps://web2py.googlecode.com/hgweb2py
> >>>>>> cd web2py
> >>>>>> ln -s ~/path/to/my/application applications/pms
> >>>>>> python web2py.py -a <recycle>
>
> >>> *go to 127.0.0.1:8000/pms/default/index
> >>> *everything works perfectly, I see the page as it should be with
> >>> records in place, providing the proper "names" (this is only three
> >>> records, I KNOW they have values)
>
> >>>>>> kill -SIGTERM <web2py process id>
> >>>>>> cd gluon/
> >>>>>> mv sql.py sql.bak.py
> >>>>>> ln -s dal.py sql.py
> >>>>>> cd ..
> >>>>>> python web2py.py -a <recycle>
>
> >>> *go to 127.0.0.1:8000/pms/default/index
> >>> *receive stacktrace.
>
> >>>>>> kill -SIGTERM <web2py process id>
> >>>>>> cd gluon/
> >>>>>> rm sql.py
> >>>>>> mv sql.bak.py sql.py
> >>>>>> cd ..
> >>>>>> python web2py.py -a <recycle>
>
> >>> *go to 127.0.0.1:8000/pms/default/index
> >>> *behold, everything works perfectly.
>
> >>> Even IF, and I say IF my dataset has a bad reference to a None record,
> >>> and it executes just fine with sql.py, it should execute with dal.py
> >>> exactly the same, it should not be throwing the exception.
>
> >>> -Thadeus
>
> >>> On Wed, Apr 7, 2010 at 10:10 PM, Massimo Di Pierro
> >>> <mdipie...@cs.depaul.edu> wrote:
>
> >>>> I assume it is
>
> >>>>> db.dog.owner.represent = lambda value: "%s" % db.person[value].name
>
> >>>> The stacktrace you get is because one of your records has owner==None
> >>>> hence
> >>>> db.person[value] is also None and None has no .name. It is not a bug. You
> >>>> are doing different queries or using different datasets.
>
> >>>> Anyway if you do
>
> >>>>> db.define_table('person', Field('name'), format='%(name)s')
> >>>>> db.define_table('dog', Field('nickname'), Field('owner', db.person))
>
> >>>> the requires is set automatically and should take care of this exception.
>
> >>>> On Apr 7, 2010, at 10:01 PM, Thadeus Burgess wrote:
>
> >>>>> I am defining this function.
>
> >>>>> db.define_table('person', Field('name'))
> >>>>> db.define_table('dog', Field('nickname'), Field('owner', db.person))
>
> >>>>> db.dog.owner.represent = lambda value: "%s" % db.owner[value].name
>
> >>>>> This works in sql.py
>
> >>>>> This does not work in dal.py
>
> >>>>> You said you made changes, I tested, works just fine on sql.py, and
> >>>>> has been for quite a while, however I get that stacktrace when I use
> >>>>> dal.py.
>
> >>>>> -Thadeus
>
> >>>>> On Wed, Apr 7, 2010 at 8:34 PM, mdipierro <mdipie...@cs.depaul.edu>
> >>>>> wrote:
>
> >>>>>> I guess I do not understand. Who defined this represent function? is
> >>>>>> it not in your code? Are you saying this is a bug in sql.py?
>
> >>>>>> On Apr 7, 7:48 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
>
> >>>>>>> No.
>
> >>>>>>> For dal.py to be accepted it has to work exactly like sql.py
>
> >>>>>>> Therefore I cannot make ANY code changes if we want a new DAL.
>
> >>>>>>> -Thadeus
>
> >>>>>>> On Wed, Apr 7, 2010 at 5:31 PM, mdipierro <mdipie...@cs.depaul.edu>
> >>>>>>> wrote:
>
> >>>>>>>> This
>
> >>>>>>>> db.TableA.id_TableB.represent = lambda value: "%s" %
> >>>>>>>> db.TableB[value].name
>
> >>>>>>>> should be
>
> >>>>>>>> db.TableA.id_TableB.represent = lambda value: (db.TableB[value] or
> >>>>>>>> {}).get('name','anonymous')
>
> >>>>>>>> On Apr 7, 1:36 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
>
> >>>>>>>>> Traceback (most recent call last):
> >>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/restricted.py", line
> >>>>>>>>> 173, in restricted
> >>>>>>>>> exec ccode in environment
> >>>>>>>>> File
>
> >>>>>>>>> "/home/tburgess/Applications/web2py/applications/pms/views/default/index.html",
> >>>>>>>>> line 62, in <module>
> >>>>>>>>> <thead>
> >>>>>>>>> File
>
> >>>>>>>>> "/home/tburgess/Applications/web2py/applications/pms/models/common.py",
> >>>>>>>>> line 197, in <lambda>
> >>>>>>>>> db.TableA.id_TableB.represent = lambda value: "%s" %
> >>>>>>>>> db.TableB[value].name
> >>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/sql.py", line 2402,
> >>>>>>>>> in __getitem__
> >>>>>>>>> return self._db(self.id == key).select().first()
> >>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/sql.py", line 3276,
> >>>>>>>>> in
> >>>>>>>>> select
> >>>>>>>>> return self._db._adapter.select(self._query,*fields,**attributes)
> >>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/sql.py", line 507, in
> >>>>>>>>> select
> >>>>>>>>> query = self.SELECT(query,*fields, **attributes)
> >>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/sql.py", line 433, in
> >>>>>>>>> SELECT
> >>>>>>>>> for field in self.db[table]:
> >>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/sql.py", line 2102,
> >>>>>>>>> in __getitem__
> >>>>>>>>> return dict.__getitem__(self, str(key))
> >>>>>>>>> KeyError: 'None'
>
> >>>>>>>>> -Thadeus
>
> >>>>>>>>> On Mon, Apr 5, 2010 at 6:00 PM, mdipierro <mdipie...@cs.depaul.edu>
> >>>>>>>>> wrote:
>
> >>>>>>>>>> uploading fix. please check it. Thanks for testing this.
>
> >>>>>>>>>> On Apr 5, 3:59 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
>
> >>>>>>>>>>> Traceback (most recent call last):
> >>>>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/restricted.py",
> >>>>>>>>>>> line
> >>>>>>>>>>> 173, in restricted
> >>>>>>>>>>> exec ccode in environment
> >>>>>>>>>>> File
>
> >>>>>>>>>>> "/home/tburgess/Applications/web2py/applications/pms/controllers/default.py",
> >>>>>>>>>>> line 278, in <module>
> >>>>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/globals.py", line
> >>>>>>>>>>> 96,
> >>>>>>>>>>> in <lambda>
> >>>>>>>>>>> self._caller = lambda f: f()
> >>>>>>>>>>> File
>
> >>>>>>>>>>> "/home/tburgess/Applications/web2py/applications/pms/controllers/default.py",
> >>>>>>>>>>> line 76, in duplicates
> >>>>>>>>>>> orderby=db.potentials.A
> >>>>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/sql.py", line 3258,
> >>>>>>>>>>> in select
> >>>>>>>>>>> return self._db._adapter.select(self._query,*fields,**attributes)
> >>>>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/sql.py", line 512,
> >>>>>>>>>>> in select
> >>>>>>>>>>> return self.parse(rows,self._colnames)
> >>>>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/sql.py", line 706,
> >>>>>>>>>>> in parse
> >>>>>>>>>>> (rid._table, rid._record) = (db[referee], None)
> >>>>>>>>>>> NameError: global name 'db' is not defined
>
> >>>>>>>>>>> -Thadeus
>
> >>>>>>>>>>> On Mon, Apr 5, 2010 at 3:01 PM, mdipierro
> >>>>>>>>>>> <mdipie...@cs.depaul.edu>
> >>>>>>>>>>> wrote:
>
> >>>>>>>>>>>> try now
>
> >>>>>>>>>>>> On Apr 5, 2:30 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
>
> >>>>>>>>>>>>> Now I get this for reference field
>
> >>>>>>>>>>>>> Traceback (most recent call last):
> >>>>>>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/restricted.py",
> >>>>>>>>>>>>> line
> >>>>>>>>>>>>> 173, in restricted
> >>>>>>>>>>>>> exec ccode in environment
> >>>>>>>>>>>>> File
>
> >>>>>>>>>>>>> "/home/tburgess/Applications/web2py/applications/pms/views/default/index.html",
> >>>>>>>>>>>>> line 84, in <module>
> >>>>>>>>>>>>> {{=db.table[column].represent(d.B[column])}}
> >>>>>>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/sql.py", line
> >>>>>>>>>>>>> 2210,
> >>>>>>>>>>>>> in __getattr__
> >>>>>>>>>>>>> self.__allocate()
> >>>>>>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/sql.py", line
> >>>>>>>>>>>>> 2203,
> >>>>>>>>>>>>> in __allocate
> >>>>>>>>>>>>> self._record = self._table[int(self)]
> >>>>>>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/sql.py", line
> >>>>>>>>>>>>> 2384,
> >>>>>>>>>>>>> in __getitem__
> >>>>>>>>>>>>> return self._db(self.id == key).select().first()
> >>>>>>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/sql.py", line
> >>>>>>>>>>>>> 3258, in select
> >>>>>>>>>>>>> return
> >>>>>>>>>>>>> self._db._adapter.select(self._query,*fields,**attributes)
> >>>>>>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/sql.py", line
> >>>>>>>>>>>>> 512,
> >>>>>>>>>>>>> in select
> >>>>>>>>>>>>> return self.parse(rows,self._colnames)
> >>>>>>>>>>>>> File "/home/tburgess/Applications/web2py/gluon/sql.py", line
> >>>>>>>>>>>>> 707,
> >>>>>>>>>>>>> in parse
> >>>>>>>>>>>>> colset[fieldname] = rid = Reference(value)
> >>>>>>>>>>>>> TypeError: int() argument must be a string or a number, not
> >>>>>>>>>>>>> 'NoneType'
>
> >>>>>>>>>>>>> -Thadeus
>
> >>>>>>>>>>>>> On Mon, Apr 5, 2010 at 2:20 PM, mdipierro
> >>>>>>>>>>>>> <mdipie...@cs.depaul.edu> wrote:
>
> >>>>>>>>>>>>>> I fixed the former. I did not fix the latter. It is a known
> >>>>>>>>>>>>>> problem
> >>>>>>>>>>>>>> with the
>
> ...
>
> read more »
>
> web2py.app.test_dalpy.w2p
> 110KViewDownload
--
To unsubscribe, reply using "remove me" as the subject.
