[web2py] left outer join problem on db2
Hi.
I've come across another subtle problem with web2py's DAL running over IBM
DB2 database.
DB2 requires the left table to immidiatedly preceed the 'left outer join'
keyword.
IOW:
select * from a, b left outer join c on c.a_id=a.id
is incorrect. Correct form is:
select * from b, a left outer join c on c.a_id=a.id
it looks moronish enough, but alas, we have to live with it.
I have modified my local copy of web2py to satisfy this requirement.
Please note that this sample can not be copypasted into web2py's sql.py as
is because of full_table_name wrapper that I added to solve another problem
- schemas support. But replacing 'full_table_name(db, t)' with just 't'
should work just fine.
join = attributes['left']
command = self._db._translator['left join']
if not isinstance(join, (tuple, list)):
join = [join]
joint = [t._tablename for t in join if not isinstance(t,
SQLJoin)]
joinon = [t for t in join if isinstance(t, SQLJoin)]
tables_to_merge={}
[tables_to_merge.update(dict.fromkeys(parse_tablenames(str(t.query for t
in joinon]
joinont = [t.table._tablename for t in joinon]
[tables_to_merge.pop(t) for t in joinont if t in
tables_to_merge]
important_tablenames = joint + joinont + tables_to_merge.keys()
excluded = [t for t in tablenames if not t in
important_tablenames ]
sql_t = ', '.join([full_table_name(self._db, t) for t in
excluded + tables_to_merge.keys()])
if joint:
sql_t += ' %s %s' % (command, ',
'.join([full_table_name(self._db, t) for t in joint]))
for t in joinon:
sql_t += ' %s %s' % (command, str(t))
Regards
Alexey
[web2py] bug in attributes check in _select
Hello. There is a bug in these lines:
if [key for key in attributes if not key
in valid_attributes]:
raise SyntaxError, 'invalid select attribute: %s' % key
At the moment of raise the 'key' variable holds last ispected value which is
not neccessarily incorrect.
The correct way will be something like:
offenders = [key for key in attributes if not key in valid_attributes]
if offenders: raise SyntaxError, 'invalid select attributes: %s'%((',
'.join(offenders)))
If same pattern was used elsewhere in the code then this bug is duplicated
in other places.
Regards
Alexey
Re: [web2py] this must be asked a 1000 times, but I couldn't find the answer
You are free to write it the way you want. You don't you do that?
{{=DIV(HR(),H1('README'),UL(LI(T('blah'}}
I think, though, that using wrappers gives lower perfomance so you'll be
consuming more CPU.
Otherwise - no objections to your method.
On Wed, Aug 4, 2010 at 12:39 PM, Stef Mientki wrote:
> please forgive my ignorance,
> but I wonder why views are not written in Python, instead of html ( see
> comparison below of te first
> part of default\index.html.
>
> The advantage of using Python:
> - the user can stick to 1 language
> - no redundant information in the closing tags, like ,
> - most editors checks the closing brackets
> - normal indentation, no problems with missing "pass"
> The disavantages of using Python:
> - ???
>
> thanks,
> Stef Mientki
>
> # ***
> extend ( 'layout.html' )
>
> if 'message' in globals () :
> H2 ( message )
> else:
> DISPLAY ( BEAUTIFY ( response._vars ) )
>
>
> HR ()
> H1 ( 'README' )
> UL (
> LI ( T ( 'You are successfully running web2py' ) ),
> LI ( T ( 'This is a copy of the scaffolding application' ) ),
> LI ( T ( 'You can modify this application and adapt it to your needs' ) ),
> LI ( T ( A ( B ( T ( "click here for the administrative interface")),
> _HREF ( URL ( 'admin', 'default', 'index' ) ) ) ) ),
> LI ( A ( T ( "click here for online examples"),
> _HREF ( URL ( 'examples', 'default', 'index' ) ) ) ),
> LI ( HREF ( "http://web2py.com";, 'web2py.com' ) ),
> LI ( HREF ( "http://web2py.com/book";, T ( 'documentation' ) ) ) )
>
> # ***
> {{extend 'layout.html'}}
>
> {{if 'message' in
> globals():}}{{=H2(message)}}{{else:}}{{=BEAUTIFY(response._vars)}}{{pass}}
>
>
> README
>
> {{=T('You are successfully running web2py')}}
> {{=T('This is a copy of the scaffolding application')}}
> {{=T('You can modify this application and adapt it to your
> needs')}}
> {{=A(B(T("click here for the administrative interface")),
> _href=URL('admin','default','index'))}}
> {{=A(T("click here for online examples"),
> _href=URL('examples','default','index'))}}
> http://web2py.com";>web2py.com
> http://web2py.com/book";>{{=T('documentation')}}
>
>
>
Re: [web2py] Will a compiled app run on GAE?
I think your version of python must at least match version on Google's server. Did you check that? Regards Alexey On Thu, Aug 5, 2010 at 7:48 PM, mat -- wrote: > After clicking the `compile` button in appadmin, my web2py app no longer > runs on GAE. > > Is this expected behavior? > > Can only uncompiled app run on GAE? > > -- > Mat >
Re: [web2py] new dal
I_can_test_with_Oracle_if_new_DAL_supports_@(connector)_and_.(schema)_in_tablenames, I_had_to_modify_old_DAL_for_this. Sorry_for_underscores._Space_looks_broken. On Fri, Feb 5, 2010 at 7:14 PM, mdipierro wrote: > > I encourage eveybody running from source to try the new dal: > > cd gluon > cp sql.py sql.py.bak > cp dal.py sal.py > > and restart web2py. > Let me know what works or does not work. > > Massimo > > -- > You received this message because you are subscribed to the Google Groups > "web2py-users" group. > To post to this group, send email to web...@googlegroups.com. > To unsubscribe from this group, send email to > web2py+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/web2py?hl=en. > -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Bug in sql.Set ?
Hi.
Sorry, didn't check if that bug still exists in latest version.
Here is the problem: in Oracle substracting one datetime column from the
other gives the floating number (days).
However the web2py considers the result to be another datetime, so this Set
produces invalid condition string:
(Pdb) print (table.time_stop-table.time_start)>300/24.0/3600
(PINGER_RESULTS.time_stop-PINGER_RESULTS.time_start)>to_date('0.003472','-mm-dd
hh24:mi:ss')
I worked around it with:
31 diff = (table.time_stop-table.time_start)
32 diff.type = 'double'
33 import pdb;pdb.set_trace()
(Pdb) print diff > 3600.0/24/3600
(PINGER_RESULTS.time_stop-PINGER_RESULTS.time_start)>0.041664
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
Re: [web2py] Re: Bug in sql.Set ?
Hmmm. I thought that the code snippets I provided is enough. NP, here is
more:
db.define_table('PINGER_RESULTS',
Field('ip','string',length=16),
Field('time_start','datetime'),
Field('time_stop' ,'datetime'), # -> time_stop
Field('active','integer'),# 0 = archive, 1 = active
)
It's a simple app - it pings set of IP addresses. If some address stops
pinging - it creates a record in db.
If it comes back - record is closed w/.
There are a lot of tiny alerts like 'server was inaccessible for 30
seconds'. When browsing archive I'd like to filter these out
So I tried to display only these which were longer than 5 minutes.
So PINGER_RESULTS.time_stop-PINGER_RESULTS.time_start must be more than 300
seconds.
Indeed, in Oracle shell this query works as I need it:
select * from PINGER_RESULTS where (time_stop-time_start)>0.0034722;
-- Oracle measured datetime difference as floating number - days.
On Thu, Feb 11, 2010 at 9:14 PM, mdipierro wrote:
> can you provide an example of usage?
>
> On Feb 11, 12:48 am, Alexey Nezhdanov wrote:
> > Hi.
> > Sorry, didn't check if that bug still exists in latest version.
> > Here is the problem: in Oracle substracting one datetime column from the
> > other gives the floating number (days).
> > However the web2py considers the result to be another datetime, so this
> Set
> > produces invalid condition string:
> >
> > (Pdb) print (table.time_stop-table.time_start)>300/24.0/3600
> >
> (PINGER_RESULTS.time_stop-PINGER_RESULTS.time_start)>to_date('0.003472','-mm-dd
> > hh24:mi:ss')
> >
> > I worked around it with:
> > 31 diff = (table.time_stop-table.time_start)
> > 32 diff.type = 'double'
> > 33 import pdb;pdb.set_trace()
> >
> > (Pdb) print diff > 3600.0/24/3600
> > (PINGER_RESULTS.time_stop-PINGER_RESULTS.time_start)>0.041664
>
> --
> You received this message because you are subscribed to the Google Groups
> "web2py-users" group.
> To post to this group, send email to web...@googlegroups.com.
> To unsubscribe from this group, send email to
> web2py+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/web2py?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
[web2py] Problem with using 'exists' keyword in complex SQL query
Hello.
I'm trying to use 'exists' keyword from the DB2 database. I know that web2py
doesn't support that natively so I tried to pass the additional condition in
as a string.
However I struck a problem that the same table used in the same query 3
times (query greatly simplified here but it still has the problem):
select NAS.ip from NAS n, SERVICE s where s.intval=n.id and exists (select
s1.id from service s1,service s2 where s.par_id=s1.par_id and
s1.par_id=s2.par_id s1.dict=123 and s2.dict=456);
so I came out with the web2py Set like this:
q = (db2.SERVICE.intval==db2.NAS.id)&("exists (select s1.id from service
s1,service s2 where SERVICE.par_id=s1.par_id and s1.par_id=s2.par_id
s1.dict=123 and s2.dict=456)")
result=db2(q).select(db2.NAS.ip)
However it fails with the KeyError: 's1' due to parsing the string and
finding s1.something and s2.something and collecting s1 and s2 as
tablenames.
Leaving aside questioning the need for _parsing_generated_string_ I came out
with the following fix to sql.py:
def parse_tablenames(text):
text = regex_quotes.sub('', text)
while 1:
i = text.find('IN (SELECT ')
+if i == -1:
+i = text.find('EXISTS (SELECT ')
if i == -1:
break
...
and slightly modifying the Set by making two words uppercase:
q = (db2.SERVICE.intval==db2.NAS.id)&("EXISTS (SELECT s1.id from service
s1,service s2 where SERVICE.par_id=s1.par_id and s1.par_id=s2.par_id
s1.dict=123 and s2.dict=456)")
--
Regards
Alexey
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
[web2py] bug in current sql.py: aliases w/ left join
Hello. Here is everything that is needed to reproduce the bug. All code is
in models/db.py.
web2py is the latest released version (1.76.5)
>
/home/snake/python/web2py/1.76.5/web2py/applications/ttt/models/db.py(77)()->None
-> import pdb;pdb.set_trace()
(Pdb) l 70
65 db.define_table('mytable1',
66 Field('key','string')
67 )
68
69 db.define_table('mytable2',
70 Field('key','string'),
71 Field('name','string')
72 )
73
74 t1=db.mytable1.with_alias('t1')
75 t2=db.mytable2.with_alias('t2')
(Pdb)
76
77 -> import pdb;pdb.set_trace()
[EOF]
(Pdb) print db(t1.id==123)._select(t1.id,t2.name,
left=t1.on(t1.key==t2.key))
SELECT t1.id, t2.name FROM t2 LEFT JOIN mytable1 AS t1 ON t1.key=t2.key
WHERE t1.id=123;
(Pdb) print db(t1.id==123).select(t1.id,t2.name, left=t1.on(t1.key==t2.key))
*** OperationalError: no such table: t2
(Pdb)
-- Regards
Alexey Nezhdanov
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
[web2py] Can't use placeholders when calling stored procedure in DB2
Problem:
(Pdb) print db2db.executesql("call webxdsl.abonsearch (123, 'port_id',
'41110', '1', ?, ?);")
*** Error: ('07001', '[07001] [IBM][CLI Driver] CLI0100E Wrong number of
parameters. SQLSTATE=07001 (-9) (SQLExecDirectW)')
(Pdb) print db2db._execute("call webxdsl.abonsearch (?)",(1,2))
*** TypeError: () takes exactly 1 argument (2 given)
Solution
--- sql.old.py 2010-01-18 17:00:36.0 +0300
+++ sql.py 2010-03-23 12:08:51.374244734 +0300
@@ -1072,7 +1072,7 @@
self._dbname, cnxn = self._uri.split(':', 1)
self._pool_connection(lambda : pyodbc.connect(cnxn))
self._cursor = self._connection.cursor()
-self._execute = lambda a: self._cursor.execute(a[:-1])
+self._execute = lambda a, b=(): self._cursor.execute(a[:-1],b)
elif is_jdbc and self._uri[:9] == 'sqlite://':
self._dbname='sqlite'
if uri[9] != '/':
Result:
(Pdb) print db2db.executesql("call webxdsl.abonsearch (123, 'port_id',
'41110', '1', ?, ?);",(1,2))
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
[web2py] BUG in with_alias processing
Ok, I bumped into same problem again.
But this time I actually DO use same table twice.
So, the task: there is a table that MAY refer to itself.
In my application monitors network tree for disconnections (is just pings
hosts). Some alerts 'come out of the blue' and some are caused by these -
i.e. we have cascade effects:
monitoring_app--A--B--C
if A is the server and B fails - then there will be two alerts: 'B
inaccessible' and 'C inaccessible' but record for node C will actually refer
to record
for node B because C probably is ok, we just can't check it.
I need to fetch all 'out of the blue' alert records AND number of associated
alerts for each.
correct SQL will be something like
select a.id, count(b.id) from TEST a, TEST b where b.top_id=a.id and a.id=0
group by a.id
However web2py losts table names in the process causing DB error.
(stripped everything down so below is all that is needed to reproduce the
problem. groupby is not needed so stripped out too)
===models/db.py=
db = DAL('sqlite://storage.sqlite')
db.define_table('TEST',Field('top_id','integer'))
==controllers/default.py=
def index():
a=db.TEST.with_alias('a')
b=db.TEST.with_alias('b')
print db((a.id==0)&(b.top_id==a.id))._select(a.ALL, b.id.count())
print db((a.id==0)&(b.top_id==a.id)). select(a.ALL, b.id.count())
return dict(message=T('Hello World'))
==
The question is - is there a workaround (except obvious - two TWO sql
queries) or may be I am just not using aliases in the correct way again?
Regards
Alexey.
--
To unsubscribe, reply using "remove me" as the subject.
Re: [web2py] Self Reference or Null
Question not clear.
What does mean 'map' here?
On Thu, Apr 15, 2010 at 4:45 PM, Leandro - ProfessionalIT <
lsever...@gmail.com> wrote:
> Friends,
>
> I have a situation where I can have a dog that I don't know your
> father and mother. How to resolve this ? for example, a dog that I buy
> in a shop.
>
> db.define_table('dog',
>Field('name'),
>Field('father_id','reference dog'),
>Field('mother_id','reference dog'))
>
> How to map this table ? becaus in my situation I can have a null
> father_id and mother_id.
>
> Thanks
>
> -- Leandro.
>
--
To unsubscribe, reply using "remove me" as the subject.
Re: [web2py] Re: Self Reference or Null
Hmm. Model looks ok. Why? On Thu, Apr 15, 2010 at 8:35 PM, Leandro - ProfessionalIT < lsever...@gmail.com> wrote: > Hi Alexey, > > sorry, map = define table in models.py. > > -- Leandro. > > > -- > To unsubscribe, reply using "remove me" as the subject. >
Re: [web2py] Re: BUG in with_alias processing
Oh, Massimo, actually I was wrong here again. Sorry for second time
reporting same issue while still not being a bug.
However, I still catched one, related specifically to Oracle.
As it happened, using LEFT JOIN here is more correct as I need empty groups
too. But Oracle chockes with error ORA-00905 if there is a keyword 'AS' in
sql. So the following patch made it work for me:
--- sql.py 2010-03-11 18:18:38.0 +0300
+++ sql.py 2010-04-16 10:04:34.087553445 +0400
@@ -1586,7 +1590,10 @@
def __str__(self):
if self.get('_ot', None):
-return '%s AS %s' % (self._ot, self._tablename)
+if self._db._dbname == 'oracle':
+return '%s %s' % (self._ot, self._tablename)
+else:
+return '%s AS %s' % (self._ot, self._tablename)
return self._tablename
def with_alias(self, alias):
Regards
Alexey
On Thu, Apr 15, 2010 at 5:40 PM, mdipierro wrote:
> Alias was not designed to let you rename tables in arbitrary cases. It
> was designed to prevent naming conflicts when you left join the same
> table multiple times. It works in this second case. So this should
> work
>
> db(db.TEST.id>0). select(TEST.ALL,
> b.id.count(),left=b.on(b.top_id==TEST.id))
>
> I agree this is not completely equivalent to your query.
>
> Massimo
>
> On Apr 15, 3:07 am, Alexey Nezhdanov wrote:
> > Ok, I bumped into same problem again.
> > But this time I actually DO use same table twice.
> >
> > So, the task: there is a table that MAY refer to itself.
> > In my application monitors network tree for disconnections (is just pings
> > hosts). Some alerts 'come out of the blue' and some are caused by these -
> > i.e. we have cascade effects:
> > monitoring_app--A--B--C
> > if A is the server and B fails - then there will be two alerts: 'B
> > inaccessible' and 'C inaccessible' but record for node C will actually
> refer
> > to record
> > for node B because C probably is ok, we just can't check it.
> >
> > I need to fetch all 'out of the blue' alert records AND number of
> associated
> > alerts for each.
> > correct SQL will be something like
> > select a.id, count(b.id) from TEST a, TEST b where b.top_id=a.id and
> a.id=0
> > group by a.id
> > However web2py losts table names in the process causing DB error.
> > (stripped everything down so below is all that is needed to reproduce the
> > problem. groupby is not needed so stripped out too)
> > ===models/db.py=
> > db = DAL('sqlite://storage.sqlite')
> > db.define_table('TEST',Field('top_id','integer'))
> > ==controllers/default.py=
> > def index():
> > a=db.TEST.with_alias('a')
> > b=db.TEST.with_alias('b')
> > print db((a.id==0)&(b.top_id==a.id))._select(a.ALL, b.id.count())
> > print db((a.id==0)&(b.top_id==a.id)). select(a.ALL, b.id.count())
> > return dict(message=T('Hello World'))
> > ==
> >
> > The question is - is there a workaround (except obvious - two TWO sql
> > queries) or may be I am just not using aliases in the correct way again?
> >
> > Regards
> > Alexey.
>
>
> --
> To unsubscribe, reply using "remove me" as the subject.
>
Re: [web2py] Re: BUG in with_alias processing
Hmm. I do not know about if there is a documentation. Probable there is some
but I just do not know about this.
I used just 'try and fail' approach:
sn...@nezhdanov:~/VTC/pinger/elixir$ ./console.py
select PINGER_RESULTS.id from PINGER_RESULTS left outer join PINGER_RESULTS
pr1 on PINGER_RESULTS.id=pr1.top_problem_id where PINGER_RESULTS.id>300;
[('ID', , 127, 22, 0, -127, 0)]
select PINGER_RESULTS.id from PINGER_RESULTS left outer join PINGER_RESULTS
as pr1 on PINGER_RESULTS.id=pr1.top_problem_id where
PINGER_RESULTS.id>300;
ORA-00905: отсутствует ключевое слово
Note that the error is opposite. 00905 is 'missing keyword' where as
_removing_ helps situation (though may be there is something more that can
be added).
On Fri, Apr 16, 2010 at 10:24 AM, mdipierro wrote:
> No problem. Better report and check that not check and leave a bug in
> there.
>
> Can you explain the fix below? Oracle does not understand AS? Is there
> documentation about this somewhere I can reference as a comment to the
> fix?
>
> Massimo
>
> On Apr 16, 1:11 am, Alexey Nezhdanov wrote:
> > Oh, Massimo, actually I was wrong here again. Sorry for second time
> > reporting same issue while still not being a bug.
> > However, I still catched one, related specifically to Oracle.
> > As it happened, using LEFT JOIN here is more correct as I need empty
> groups
> > too. But Oracle chockes with error ORA-00905 if there is a keyword 'AS'
> in
> > sql. So the following patch made it work for me:
> > --- sql.py 2010-03-11 18:18:38.0 +0300
> > +++ sql.py 2010-04-16 10:04:34.087553445 +0400
> > @@ -1586,7 +1590,10 @@
> >
> > def __str__(self):
> > if self.get('_ot', None):
> > -return '%s AS %s' % (self._ot, self._tablename)
> > +if self._db._dbname == 'oracle':
> > +return '%s %s' % (self._ot, self._tablename)
> > +else:
> > +return '%s AS %s' % (self._ot, self._tablename)
> > return self._tablename
> >
> > def with_alias(self, alias):
> >
> > Regards
> > Alexey
> >
> > On Thu, Apr 15, 2010 at 5:40 PM, mdipierro
> wrote:
> > > Alias was not designed to let you rename tables in arbitrary cases. It
> > > was designed to prevent naming conflicts when you left join the same
> > > table multiple times. It works in this second case. So this should
> > > work
> >
> > > db(db.TEST.id>0). select(TEST.ALL,
> > > b.id.count(),left=b.on(b.top_id==TEST.id))
> >
> > > I agree this is not completely equivalent to your query.
> >
> > > Massimo
> >
> > > On Apr 15, 3:07 am, Alexey Nezhdanov wrote:
> > > > Ok, I bumped into same problem again.
> > > > But this time I actually DO use same table twice.
> >
> > > > So, the task: there is a table that MAY refer to itself.
> > > > In my application monitors network tree for disconnections (is just
> pings
> > > > hosts). Some alerts 'come out of the blue' and some are caused by
> these -
> > > > i.e. we have cascade effects:
> > > > monitoring_app--A--B--C
> > > > if A is the server and B fails - then there will be two alerts: 'B
> > > > inaccessible' and 'C inaccessible' but record for node C will
> actually
> > > refer
> > > > to record
> > > > for node B because C probably is ok, we just can't check it.
> >
> > > > I need to fetch all 'out of the blue' alert records AND number of
> > > associated
> > > > alerts for each.
> > > > correct SQL will be something like
> > > > select a.id, count(b.id) from TEST a, TEST b where b.top_id=a.id and
> > > a.id=0
> > > > group by a.id
> > > > However web2py losts table names in the process causing DB error.
> > > > (stripped everything down so below is all that is needed to reproduce
> the
> > > > problem. groupby is not needed so stripped out too)
> > > > ===models/db.py=
> > > > db = DAL('sqlite://storage.sqlite')
> > > > db.define_table('TEST',Field('top_id','integer'))
> > > > ==controllers/default.py=
> > > > def index():
> > > > a=db.TEST.with_alias('a')
> > > > b=db.TEST.with_alias('b')
> > > > print db((a.id==0)&(b.top_id==a.id))._select(a.ALL,
> b.id.count())
> > > > print db((a.id==0)&(b.top_id==a.id)). select(a.ALL,
> b.id.count())
> > > > return dict(message=T('Hello World'))
> > > > ==
> >
> > > > The question is - is there a workaround (except obvious - two TWO sql
> > > > queries) or may be I am just not using aliases in the correct way
> again?
> >
> > > > Regards
> > > > Alexey.
> >
> > > --
> > > To unsubscribe, reply using "remove me" as the subject.
>
[web2py] can I hook my code to db.connect() ?
Hi.
Sometimes I have to execute this line prior to making a query:
oradb.executesql("ALTER SESSION set NLS_DATE_FORMAT = '-MM-DD
HH24:MI:SS';")
That is because Oracle by default uses different date format that causes my
queries to fail.
The problem is that doing that in model is incorrect - this should be
executed just once for each db connection.
Doing that just prior to query is inconvenient and still incorrect - I am
probably reusing same connection.
Can I somehow tell web2py to execute this sql right after calling connect()?
Regards
Alexey
--
Subscription settings: http://groups.google.com/group/web2py/subscribe?hl=en
Re: [web2py] Re: can I hook my code to db.connect() ?
No, I'm all for it. Actually it sounds more like bugfix since atm web2py
can't make datetime queries on Oracle.
On Mon, Apr 19, 2010 at 6:16 PM, mdipierro wrote:
> There is no hook for this, but should this not be always the default?
> If so I am happy to add it to trunk. Any counterindication?
>
> On Apr 19, 12:32 am, Alexey Nezhdanov wrote:
> > Hi.
> > Sometimes I have to execute this line prior to making a query:
> > oradb.executesql("ALTER SESSION set NLS_DATE_FORMAT = '-MM-DD
> > HH24:MI:SS';")
> >
> > That is because Oracle by default uses different date format that causes
> my
> > queries to fail.
> > The problem is that doing that in model is incorrect - this should be
> > executed just once for each db connection.
> > Doing that just prior to query is inconvenient and still incorrect - I am
> > probably reusing same connection.
> > Can I somehow tell web2py to execute this sql right after calling
> connect()?
> >
> > Regards
> > Alexey
> >
> > --
> > Subscription settings:
> http://groups.google.com/group/web2py/subscribe?hl=en
>
Re: [web2py] Re: can I hook my code to db.connect() ?
Oh, well.
May be that's an Oracle bug then. I use my custom 'console.py' script here,
but output should be clear enough:
BTW - time_start column was created by web2py with this line:
Field('time_start','datetime'),
sn...@nezhdanov:~/VTC/pinger/elixir$ ./console.py
select time_start from pinger_results where id=1;
[('TIME_START', , 23, 7, 0, 0, 1)]
select time_start from pinger_results where time_start='2010-04-20
15:16:17';
ORA-01861: литерал не соответствует формату строки
ALTER SESSION set NLS_DATE_FORMAT = '-MM-DD HH24:MI:SS'
ok
select time_start from pinger_results where time_start='2010-04-20
15:16:17';
[('TIME_START', , 23, 7, 0, 0, 1)]
==
no comments
Alexey
On Mon, Apr 19, 2010 at 8:27 PM, mdipierro wrote:
> taking a second look at the source code
>
>self._execute("ALTER SESSION SET NLS_DATE_FORMAT = '-
> MM-DD';")
>self._execute("ALTER SESSION SET NLS_TIMESTAMP_FORMAT =
> '-MM-DD HH24:MI:SS';")
>
> Now you propose adding:
>
>self._execute("ALTER SESSION set NLS_DATE_FORMAT = '-
> MM-DD HH24:MI:SS';")
>
> But date does not have HH224, MI, SS, only timestamp does.
>
> On Apr 19, 11:19 am, Alexey Nezhdanov wrote:
> > No, I'm all for it. Actually it sounds more like bugfix since atm web2py
> > can't make datetime queries on Oracle.
> >
> > On Mon, Apr 19, 2010 at 6:16 PM, mdipierro
> wrote:
> > > There is no hook for this, but should this not be always the default?
> > > If so I am happy to add it to trunk. Any counterindication?
> >
> > > On Apr 19, 12:32 am, Alexey Nezhdanov wrote:
> > > > Hi.
> > > > Sometimes I have to execute this line prior to making a query:
> > > > oradb.executesql("ALTER SESSION set NLS_DATE_FORMAT = '-MM-DD
> > > > HH24:MI:SS';")
> >
> > > > That is because Oracle by default uses different date format that
> causes
> > > my
> > > > queries to fail.
> > > > The problem is that doing that in model is incorrect - this should be
> > > > executed just once for each db connection.
> > > > Doing that just prior to query is inconvenient and still incorrect -
> I am
> > > > probably reusing same connection.
> > > > Can I somehow tell web2py to execute this sql right after calling
> > > connect()?
> >
> > > > Regards
> > > > Alexey
> >
> > > > --
> > > > Subscription settings:
> > >http://groups.google.com/group/web2py/subscribe?hl=en
>
Re: [web2py] Re: can I hook my code to db.connect() ?
Ah, one more little comment though. In two cases select yielded an empty
list - that is intentional.
On Tue, Apr 20, 2010 at 9:37 AM, Alexey Nezhdanov wrote:
> Oh, well.
> May be that's an Oracle bug then. I use my custom 'console.py' script here,
> but output should be clear enough:
>
> BTW - time_start column was created by web2py with this line:
> Field('time_start','datetime'),
>
> sn...@nezhdanov:~/VTC/pinger/elixir$ ./console.py
> select time_start from pinger_results where id=1;
> [('TIME_START', , 23, 7, 0, 0, 1)]
> select time_start from pinger_results where time_start='2010-04-20
> 15:16:17';
> ORA-01861: литерал не соответствует формату строки
>
> ALTER SESSION set NLS_DATE_FORMAT = '-MM-DD HH24:MI:SS'
> ok
> select time_start from pinger_results where time_start='2010-04-20
> 15:16:17';
> [('TIME_START', , 23, 7, 0, 0, 1)]
> ==
> no comments
>
> Alexey
>
>
> On Mon, Apr 19, 2010 at 8:27 PM, mdipierro wrote:
>
>> taking a second look at the source code
>>
>>self._execute("ALTER SESSION SET NLS_DATE_FORMAT = '-
>> MM-DD';")
>>self._execute("ALTER SESSION SET NLS_TIMESTAMP_FORMAT =
>> '-MM-DD HH24:MI:SS';")
>>
>> Now you propose adding:
>>
>>self._execute("ALTER SESSION set NLS_DATE_FORMAT = '-
>> MM-DD HH24:MI:SS';")
>>
>> But date does not have HH224, MI, SS, only timestamp does.
>>
>> On Apr 19, 11:19 am, Alexey Nezhdanov wrote:
>> > No, I'm all for it. Actually it sounds more like bugfix since atm web2py
>> > can't make datetime queries on Oracle.
>> >
>> > On Mon, Apr 19, 2010 at 6:16 PM, mdipierro
>> wrote:
>> > > There is no hook for this, but should this not be always the default?
>> > > If so I am happy to add it to trunk. Any counterindication?
>> >
>> > > On Apr 19, 12:32 am, Alexey Nezhdanov wrote:
>> > > > Hi.
>> > > > Sometimes I have to execute this line prior to making a query:
>> > > > oradb.executesql("ALTER SESSION set NLS_DATE_FORMAT =
>> '-MM-DD
>> > > > HH24:MI:SS';")
>> >
>> > > > That is because Oracle by default uses different date format that
>> causes
>> > > my
>> > > > queries to fail.
>> > > > The problem is that doing that in model is incorrect - this should
>> be
>> > > > executed just once for each db connection.
>> > > > Doing that just prior to query is inconvenient and still incorrect -
>> I am
>> > > > probably reusing same connection.
>> > > > Can I somehow tell web2py to execute this sql right after calling
>> > > connect()?
>> >
>> > > > Regards
>> > > > Alexey
>> >
>> > > > --
>> > > > Subscription settings:
>> > >http://groups.google.com/group/web2py/subscribe?hl=en
>>
>
>
Re: [web2py] Re: can I hook my code to db.connect() ?
Sure.
In short: everything works ok.
In detail - see below.
(Pdb) l 1
1 # -*- coding: utf-8 -*-
2
3 exec('from applications.%s.modules import cfg' %
request.application)
4 db = DAL('oracle://%s/%...@%s'%(cfg.oralogin,cfg.orapassword,cfg.oradb),
pool_size=10)
5
6 db.define_table('mytable',
7 Field('myfield','string'),
8 Field('mydt','datetime'),
9 )
10
11 -> import pdb;pdb.set_trace()
(Pdb) db.mytable.insert(myfield='aaa',mydt='2010-04-22 00:00:00')
1
(Pdb) db.mytable.insert(myfield='bbb',mydt='2010-04-22 10:00:00')
2
(Pdb) db.commit()
(Pdb) print db(db.mytable.mydt>'2010-04-22 09:00:00').select()
mytable.id,mytable.myfield,mytable.mydt
2,bbb,2010-04-22 10:00:00
(Pdb) print db(db.id>0).select()
*** KeyError: 'id'
(Pdb) print db(db.mytable.id>0).select()
mytable.id,mytable.myfield,mytable.mydt
1,aaa,2010-04-22 00:00:00
2,bbb,2010-04-22 10:00:00
(Pdb)
On Tue, Apr 20, 2010 at 6:20 PM, mdipierro wrote:
> I think I understand better the problem. Oracle uses the same time for
> DATE and DATETIME, i.e. DATE.
> I made in trunk the change you suggested. Can you please check this
> does not break insert and select of Field('somename','date')?
>
> On Apr 20, 12:37 am, Alexey Nezhdanov wrote:
> > Oh, well.
> > May be that's an Oracle bug then. I use my custom 'console.py' script
> here,
> > but output should be clear enough:
> >
> > BTW - time_start column was created by web2py with this line:
> > Field('time_start','datetime'),
> >
> > sn...@nezhdanov:~/VTC/pinger/elixir$ ./console.py
> > select time_start from pinger_results where id=1;
> > [('TIME_START', , 23, 7, 0, 0, 1)]
> > select time_start from pinger_results where time_start='2010-04-20
> > 15:16:17';
> > ORA-01861: литерал не соответствует формату строки
> > ALTER SESSION set NLS_DATE_FORMAT = '-MM-DD HH24:MI:SS'
> > ok
> > select time_start from pinger_results where time_start='2010-04-20
> > 15:16:17';
> > [('TIME_START', , 23, 7, 0, 0, 1)]
> > ==
> > no comments
> >
> > Alexey
> >
> > On Mon, Apr 19, 2010 at 8:27 PM, mdipierro
> wrote:
> > > taking a second look at the source code
> >
> > > self._execute("ALTER SESSION SET NLS_DATE_FORMAT = '-
> > > MM-DD';")
> > >self._execute("ALTER SESSION SET NLS_TIMESTAMP_FORMAT =
> > > '-MM-DD HH24:MI:SS';")
> >
> > > Now you propose adding:
> >
> > >self._execute("ALTER SESSION set NLS_DATE_FORMAT = '-
> > > MM-DD HH24:MI:SS';")
> >
> > > But date does not have HH224, MI, SS, only timestamp does.
> >
> > > On Apr 19, 11:19 am, Alexey Nezhdanov wrote:
> > > > No, I'm all for it. Actually it sounds more like bugfix since atm
> web2py
> > > > can't make datetime queries on Oracle.
> >
> > > > On Mon, Apr 19, 2010 at 6:16 PM, mdipierro
> > > wrote:
> > > > > There is no hook for this, but should this not be always the
> default?
> > > > > If so I am happy to add it to trunk. Any counterindication?
> >
> > > > > On Apr 19, 12:32 am, Alexey Nezhdanov wrote:
> > > > > > Hi.
> > > > > > Sometimes I have to execute this line prior to making a query:
> > > > > > oradb.executesql("ALTER SESSION set NLS_DATE_FORMAT =
> '-MM-DD
> > > > > > HH24:MI:SS';")
> >
> > > > > > That is because Oracle by default uses different date format that
> > > causes
> > > > > my
> > > > > > queries to fail.
> > > > > > The problem is that doing that in model is incorrect - this
> should be
> > > > > > executed just once for each db connection.
> > > > > > Doing that just prior to query is inconvenient and still
> incorrect -
> > > I am
> > > > > > probably reusing same connection.
> > > > > > Can I somehow tell web2py to execute this sql right after calling
> > > > > connect()?
> >
> > > > > > Regards
> > > > > > Alexey
> >
> > > > > > --
> > > > > > Subscription settings:
> > > > >http://groups.google.com/group/web2py/subscribe?hl=en
>
Re: [web2py] Re: can I hook my code to db.connect() ?
Aargh...
You asked to check _date_ type of field. But it works ok too:
-> import pdb;pdb.set_trace()
(Pdb) l 1
1 # -*- coding: utf-8 -*-
2
3 exec('from applications.%s.modules import cfg' %
request.application)
4 db = DAL('oracle://%s/%...@%s'%(cfg.oralogin,cfg.orapassword,cfg.oradb),
pool_size=10)
5
6 db.define_table('mytable',
7 Field('myfield','string'),
8 Field('mydt','datetime'),
9 Field('mydt2','date'),
10 )
11
(Pdb)
12 -> import pdb;pdb.set_trace()
[EOF]
(Pdb) db.mytable.insert(myfield='ccc',mydt2='2010-04-22')
3
(Pdb) db.mytable.insert(myfield='ddd',mydt2='2010-04-23')
4
(Pdb) db.commit()
(Pdb) db(db.mytable.id>0).select()
(Pdb) print db(db.mytable.id>0).select()
mytable.id,mytable.myfield,mytable.mydt,mytable.mydt2
1,aaa,2010-04-22 00:00:00,
2,bbb,2010-04-22 10:00:00,
3,ccc,,2010-04-22
4,ddd,,2010-04-23
(Pdb) print db(db.mytable.mydt2>'2010-04-22').select()
mytable.id,mytable.myfield,mytable.mydt,mytable.mydt2
4,ddd,,2010-04-23
(Pdb)
Regards
Alexey
On Thu, Apr 22, 2010 at 2:43 PM, Alexey Nezhdanov wrote:
> Sure.
> In short: everything works ok.
>
> In detail - see below.
>
>
> (Pdb) l 1
> 1 # -*- coding: utf-8 -*-
> 2
> 3 exec('from applications.%s.modules import cfg' %
> request.application)
> 4 db =
> DAL('oracle://%s/%...@%s'%(cfg.oralogin,cfg.orapassword,cfg.oradb),
> pool_size=10)
> 5
> 6 db.define_table('mytable',
> 7 Field('myfield','string'),
> 8 Field('mydt','datetime'),
> 9 )
> 10
> 11 -> import pdb;pdb.set_trace()
> (Pdb) db.mytable.insert(myfield='aaa',mydt='2010-04-22 00:00:00')
> 1
> (Pdb) db.mytable.insert(myfield='bbb',mydt='2010-04-22 10:00:00')
> 2
> (Pdb) db.commit()
> (Pdb) print db(db.mytable.mydt>'2010-04-22 09:00:00').select()
> mytable.id,mytable.myfield,mytable.mydt
> 2,bbb,2010-04-22 10:00:00
>
> (Pdb) print db(db.id>0).select()
> *** KeyError: 'id'
> (Pdb) print db(db.mytable.id>0).select()
> mytable.id,mytable.myfield,mytable.mydt
> 1,aaa,2010-04-22 00:00:00
> 2,bbb,2010-04-22 10:00:00
>
> (Pdb)
>
>
>
> On Tue, Apr 20, 2010 at 6:20 PM, mdipierro wrote:
>
>> I think I understand better the problem. Oracle uses the same time for
>> DATE and DATETIME, i.e. DATE.
>> I made in trunk the change you suggested. Can you please check this
>> does not break insert and select of Field('somename','date')?
>>
>> On Apr 20, 12:37 am, Alexey Nezhdanov wrote:
>> > Oh, well.
>> > May be that's an Oracle bug then. I use my custom 'console.py' script
>> here,
>> > but output should be clear enough:
>> >
>> > BTW - time_start column was created by web2py with this line:
>> > Field('time_start','datetime'),
>> >
>> > sn...@nezhdanov:~/VTC/pinger/elixir$ ./console.py
>> > select time_start from pinger_results where id=1;
>> > [('TIME_START', , 23, 7, 0, 0, 1)]
>> > select time_start from pinger_results where time_start='2010-04-20
>> > 15:16:17';
>> > ORA-01861: литерал не соответствует формату строки
>> > ALTER SESSION set NLS_DATE_FORMAT = '-MM-DD HH24:MI:SS'
>> > ok
>> > select time_start from pinger_results where time_start='2010-04-20
>> > 15:16:17';
>> > [('TIME_START', , 23, 7, 0, 0, 1)]
>> > ==
>> > no comments
>> >
>> > Alexey
>> >
>> > On Mon, Apr 19, 2010 at 8:27 PM, mdipierro
>> wrote:
>> > > taking a second look at the source code
>> >
>> > >self._execute("ALTER SESSION SET NLS_DATE_FORMAT = '-
>> > > MM-DD';")
>> > >self._execute("ALTER SESSION SET NLS_TIMESTAMP_FORMAT =
>> > > '-MM-DD HH24:MI:SS';")
>> >
>> > > Now you propose adding:
>> >
>> > >self._execute("ALTER SESSION set NLS_DATE_FORMAT = '-
>> > > MM-DD HH24:MI:SS';")
>> >
>> > > But date does not have HH224, MI, SS, only timestamp does.
>> >
>> > > On Apr 19, 11:19 am, Alexey Nezhdanov wrote:
>> > > > No, I'm all for it. Actually it sounds more like bugfix since atm
>> web2py
>> >
[web2py] updated ru-ru.py file
Hi.
While writting an application bumped into partial translation of russian
language.
Updated file attached. It is from welcome.py application, no my app specific
strings present.
Regards
Alexey
# coding: utf8
{
'"update" is an optional expression like "field1=\'newvalue\'". You cannot update or delete the results of a JOIN': '"Изменить" - необязательное выражение вида "field1=\'новое значение\'". Результаты операции JOIN нельзя изменить или удалить.',
'%Y-%m-%d': '%Y-%m-%d',
'%Y-%m-%d %H:%M:%S': '%Y-%m-%d %H:%M:%S',
'%s rows deleted': '%s строк удалено',
'%s rows updated': '%s строк изменено',
'Available databases and tables': 'Базы данных и таблицы',
'Cannot be empty': 'Пустое значение недопустимо',
'Change Password': 'Смените пароль',
'Check to delete': 'Удалить',
'Check to delete:': 'Удалить:',
'Client IP': 'Client IP',
'Current request': 'Текущий запрос',
'Current response': 'Текущий ответ',
'Current session': 'Текущая сессия',
'Delete:': 'Удалить:',
'Description': 'Описание',
'E-mail': 'E-mail',
'Edit Profile': 'Редактировать профиль',
'Edit current record': 'Редактировать текущую запись',
'First name': 'Имя',
'Group ID': 'Group ID',
'Hello World': 'Заработало!',
'Import/Export': 'Импорт/экспорт',
'Internal State': 'Внутренне состояние',
'Invalid Query': 'Неверный запрос',
'Invalid email': 'Неверный email',
'Invalid login': 'Неверный логин',
'Invalid password': 'Неверный пароль',
'Last name': 'Фамилия',
'Logged in': 'Вход выполнен',
'Logged out': 'Выход выполнен',
'Login': 'Вход',
'Logout': 'Выход',
'Lost Password': 'Забыли пароль?',
'Name': 'Name',
'New Record': 'Новая запись',
'New password': 'Новый пароль',
'No databases in this application': 'В приложении нет баз данных',
'Old password': 'Старый пароль',
'Origin': 'Происхождение',
'Password': 'Пароль',
"Password fields don't match": 'Пароли не совпадают',
'Query:': 'Запрос:',
'Record ID': 'ID записи',
'Register': 'Зарегистрироваться',
'Registration key': 'Ключ регистрации',
'Remember me (for 30 days)': 'Запомнить меня (на 30 дней)',
'Reset Password key': 'Сбросить ключ пароля',
'Role': 'Роль',
'Rows in table': 'Строк в таблице',
'Rows selected': 'Выделено строк',
'Submit': 'Отправить',
'Sure you want to delete this object?': 'Подтвердите удаление объекта',
'Table name': 'Имя таблицы',
'The "query" is a condition like "db.table1.field1==\'value\'". Something like "db.table1.field1==db.table2.field2" results in a SQL JOIN.': '"Запрос" - это условие вида "db.table1.field1==\'значение\'". Выражение вида "db.table1.field1==db.table2.field2" формирует SQL JOIN.',
'Timestamp': 'Отметка времени',
'Update:': 'Изменить:',
'Use (...)&(...) for AND, (...)|(...) for OR, and ~(...) for NOT to build more complex queries.': 'Для построение сложных запросов используйте операторы "И": (...)&(...), "ИЛИ": (...)|(...), "НЕ": ~(...).',
'User %(id)s Logged-in': 'Пользователь %(id)s вошёл',
'User %(id)s Logged-out': 'Пользователь %(id)s вышел',
'User %(id)s Password changed': 'Пользователь %(id)s сменил пароль',
'User %(id)s Profile updated': 'Пользователь %(id)s обновил профиль',
'User %(id)s Registered': 'Пользователь %(id)s зарегистрировался',
'User ID': 'ID пользователя',
'Verify Password': 'Повторите пароль',
'Welcome to web2py': 'Добро пожаловать в web2py',
'click here for online examples': 'примеры он-лайн',
'click here for the administrative interface': 'административный интерфейс',
'customize me!': 'настройте внешний вид!',
'data uploaded': 'данные загружены',
'database': 'база данных',
'database %s select': 'выбор базы данных %s',
'db': 'БД',
'design': 'дизайн',
'done!': 'готово!',
'export as csv file': 'экспорт в csv-файл',
'insert new': 'добавить',
'insert new %s': 'добавить %s',
'invalid request': 'неверный запрос',
'login': 'вход',
'logout': 'выход',
'new record inserted': 'новая запись добавлена',
'next 100 rows': 'следующие 100 строк',
'or import from csv file': 'или импорт из csv-файла',
'password': 'пароль',
'previous 100 rows': 'предыдущие 100 строк',
'profile': 'профиль',
'record does not exist': 'запись не найдена',
'record id': 'id записи',
'selected': 'выбрано',
'state': 'состояние',
'table': 'таблица',
'unable to parse csv file': 'нечитаемый csv-файл',
}
Re: [web2py] Web2py with barcode Reader
The question is not THAT clear, but I guess you can use external library to connect to barcode reader. However there is at least one problem - using the barcode reader is not a browser operation. So you will need either an ajax app that will poll the server for the presence of an active barcode or write a server-side standalone daemon and query it. Regards Alexey On Wed, Aug 25, 2010 at 1:33 PM, Neveen Adel wrote: > Hello, > > Is there a way to connect my web2py application to barcode reader and > then get the data after passing > barcode card into the barcode? > > Thanks in Advance > > Neveen
Re: [web2py] Re: fascinating
Yes, once I used it to print an 'enhanced stack backtrace'. Each time the
site engine was crashing (not web2py, long ago) the stack trace was emailed
to me along with all variables values that existed at all levels of the
trace.
On Fri, Aug 27, 2010 at 6:53 AM, GoldenTiger wrote:
> do you refer to module inspect? it's great for introspecting on live
> objects and their source code
>
> On 27 ago, 03:44, mdipierro wrote:
> > I discovered this:
> >
> > def f(name):
> > import inspect
> > c_frame = inspect.getouterframes(inspect.currentframe(), 1)[1][0]
> > c_args, c_varargs, c_varkw, c_locals =
> > inspect.getargvalues(c_frame)
> > d = dict(c_locals)
> > return d[name]
> >
> > def g():
> > a=5
> > print f('a')
> >
> > g() # prints 5
>
Re: [web2py] Re: Has anyone used standalone web2py DAL in a project ?
I use web2py's DAL in my project (daemon). I modified db.py so that all calls to create_table happen with migrate=False (because I reuse the same db as in web app). So all I need from DAL is .select, .update, .delete and .insert. Also I take care to call these functions only from single thread. That's all and it works for me. Ah, yes, the files: in the current folder I have symlinks to gluon and to db.py Also in db.py I have to explicitly check for DAL presence and if missing - import it from gluon. On Sat, Aug 28, 2010 at 6:51 AM, Kevin wrote: > Of course, it is always an option (and never a difficulty) to wrap > functional programming concepts in an object oriented wrapper -- > that's more a benefit of the functional style than then anything OOP > gives to you though. > > On Aug 27, 7:13 pm, Thadeus Burgess wrote: > > >.< DON'T DO IT! > > > > The DAL is ment for functional programming... so be careful if you > > have to use it in a class based system. Plus you will have to have > > your stand alone application manage the .table files too. Just keep it > > in mind. > > > > In any case... > > > > You will need. > > > > gluon/LICENSE > > gluon/sql.py > > gluon/portalocker.py > > gluon/utils.py > > > > Also, you will NOT need validators.. so edit gluon/sql.py and comment > > out anything to do with the following > > > > from serializers import json - remove this, it conflicts with the real > > version of simplejson installed from pypi. > > from http import HTTP -- doubt you will be making any http exceptions > > import validators -- unless you are somehow using SQLFORM, you dont > > need this. Be sure to remove all traces of validators from the rest of > > sql.py Just do a search for "validators" and remove that code. > > > > Yay, you now have a stand alone DAL... see how easy that was to > > de-couple things?! =) > > > > -- > > Thadeus > > > > On Fri, Aug 27, 2010 at 6:34 PM, Sujan Shakya > wrote: > > > Hi All, > > > > > I'm going to have to use a MySQL orm in a project in near future. > > > Do you think web2py DAL is good for that purpose regarding scalability > > > and performance ? > > > And what files do I need to import or execute to use only web2py DAL ? > > > > > Thanks. > > > > >
Re: [web2py] web2py Africa
These times fans don't need to be around the place where the site located. More than this - it says that it runs on AppEngine :) On Sat, Sep 4, 2010 at 1:23 AM, hcvst wrote: > Hi, > http://www.domain-name-registration.co.za powered by w2p, > South African (co.za) domain name registration. > Any w2p fans in/around Joburg? > HC
[web2py] CRUD RBAC problem
Hi. I think that I found some inconsistency in the topic. When you do, say, auth.add_permission(group_id) - it assumes the permission name 'any', table name empty and record_id 0. Which in turn feels like "full admin rights" - any action on any table on any record. In fact, that gives no permissions whatsoever. I've came out with the following patch to make it work for me, but since that is the very core of RBAC, I'm not sure if that is the right solution or if I am looking in the correct direction at all. --- tools.old.py2010-09-08 08:40:22.266751051 +0400 +++ tools.py2010-09-08 08:41:25.894746181 +0400 @@ -2420,10 +2420,9 @@ == record_id).select(permission.group_id) groups_required = set([row.group_id for row in rows]) if record_id: -rows = self.db(permission.name -== name)(permission.table_name - == str(table_name))(permission.record_id - == 0).select(permission.group_id) +rows = self.db(permission.name.belongs((name,'any'))& + permission.table_name.belongs((str(table_name),''))& + permission.record_id.belongs((record_id,0))).select(permission.group_id) groups_required = groups_required.union(set([row.group_id for row in rows])) if groups.intersection(groups_required): Regards Alexey
Re: [web2py] Re: CRUD RBAC problem
Updated version of the patch then. Includes case where there is no such row
('create' action).
However, I'm not sure if these changes are GAE-compartible. Not sure if
bigtable likes .belongs on multiple columns.
Can anybody test?
Regards
Alexey.
--- tools.py_ 2010-09-08 08:40:22.266751051 +0400
+++ tools.py2010-09-08 09:44:30.050746520 +0400
@@ -2415,17 +2415,10 @@
== user_id).select(membership.group_id)
groups = set([row.group_id for row in rows])
permission = self.settings.table_permission
-rows = self.db(permission.name == name)(permission.table_name
- == str(table_name))(permission.record_id
- == record_id).select(permission.group_id)
+rows = self.db(permission.name.belongs((name,'any'))&
+ permission.table_name.belongs((str(table_name),''))&
+
permission.record_id.belongs((record_id,0))).select(permission.group_id)
groups_required = set([row.group_id for row in rows])
-if record_id:
-rows = self.db(permission.name
-== name)(permission.table_name
- == str(table_name))(permission.record_id
- == 0).select(permission.group_id)
-groups_required = groups_required.union(set([row.group_id
-for row in rows]))
if groups.intersection(groups_required):
r = True
else:
On Wed, Sep 8, 2010 at 5:14 PM, mdipierro wrote:
> I think this should be considered a bug and I agree with the change.
> Anybody opposed?
>
> Massimo
>
> On Sep 7, 11:46 pm, Alexey Nezhdanov wrote:
> > Hi. I think that I found some inconsistency in the topic.
> > When you do, say,
> > auth.add_permission(group_id) - it assumes the permission name 'any',
> table
> > name empty and record_id 0.
> > Which in turn feels like "full admin rights" - any action on any table on
> > any record.
> > In fact, that gives no permissions whatsoever.
> >
> > I've came out with the following patch to make it work for me, but since
> > that is the very core of RBAC, I'm not sure if that is the right solution
> or
> > if I am looking in the correct direction at all.
> >
> > --- tools.old.py2010-09-08 08:40:22.266751051 +0400
> > +++ tools.py2010-09-08 08:41:25.894746181 +0400
> > @@ -2420,10 +2420,9 @@
> > == record_id).select(permission.group_id)
> > groups_required = set([row.group_id for row in rows])
> > if record_id:
> > -rows = self.db(permission.name
> > -== name)(permission.table_name
> > - == str(table_name))(permission.record_id
> > - == 0).select(permission.group_id)
> > +rows = self.db(permission.name.belongs((name,'any'))&
> > +
> > permission.table_name.belongs((str(table_name),''))&
> > +
> > permission.record_id.belongs((record_id,0))).select(permission.group_id)
> > groups_required = groups_required.union(set([row.group_id
> > for row in rows]))
> > if groups.intersection(groups_required):
> >
> > Regards
> > Alexey
>
Re: [web2py] Re: CRUD RBAC problem
Sure
On Thu, Sep 9, 2010 at 6:11 AM, mdipierro wrote:
> No, it does not. It must be done in two queries. Can you send me the
> patch by email? thanks.
>
> On Sep 8, 8:44 pm, Alexey Nezhdanov wrote:
> > Updated version of the patch then. Includes case where there is no such
> row
> > ('create' action).
> > However, I'm not sure if these changes are GAE-compartible. Not sure if
> > bigtable likes .belongs on multiple columns.
> > Can anybody test?
> >
> > Regards
> > Alexey.
> >
> > --- tools.py_ 2010-09-08 08:40:22.266751051 +0400
> > +++ tools.py2010-09-08 09:44:30.050746520 +0400
> > @@ -2415,17 +2415,10 @@
> > == user_id).select(membership.group_id)
> > groups = set([row.group_id for row in rows])
> > permission = self.settings.table_permission
> > -rows = self.db(permission.name == name)(permission.table_name
> > - == str(table_name))(permission.record_id
> > - == record_id).select(permission.group_id)
> > +rows = self.db(permission.name.belongs((name,'any'))&
> > +
> permission.table_name.belongs((str(table_name),''))&
> > +
> > permission.record_id.belongs((record_id,0))).select(permission.group_id)
> > groups_required = set([row.group_id for row in rows])
> > -if record_id:
> > -rows = self.db(permission.name
> > -== name)(permission.table_name
> > - == str(table_name))(permission.record_id
> > - == 0).select(permission.group_id)
> > -groups_required = groups_required.union(set([row.group_id
> > -for row in rows]))
> > if groups.intersection(groups_required):
> > r = True
> > else:
> >
> > On Wed, Sep 8, 2010 at 5:14 PM, mdipierro
> wrote:
> > > I think this should be considered a bug and I agree with the change.
> > > Anybody opposed?
> >
> > > Massimo
> >
> > > On Sep 7, 11:46 pm, Alexey Nezhdanov wrote:
> > > > Hi. I think that I found some inconsistency in the topic.
> > > > When you do, say,
> > > > auth.add_permission(group_id) - it assumes the permission name 'any',
> > > table
> > > > name empty and record_id 0.
> > > > Which in turn feels like "full admin rights" - any action on any
> table on
> > > > any record.
> > > > In fact, that gives no permissions whatsoever.
> >
> > > > I've came out with the following patch to make it work for me, but
> since
> > > > that is the very core of RBAC, I'm not sure if that is the right
> solution
> > > or
> > > > if I am looking in the correct direction at all.
> >
> > > > --- tools.old.py2010-09-08 08:40:22.266751051 +0400
> > > > +++ tools.py2010-09-08 08:41:25.894746181 +0400
> > > > @@ -2420,10 +2420,9 @@
> > > > == record_id).select(permission.group_id)
> > > > groups_required = set([row.group_id for row in rows])
> > > > if record_id:
> > > > -rows = self.db(permission.name
> > > > -== name)(permission.table_name
> > > > - == str(table_name))(permission.record_id
> > > > - == 0).select(permission.group_id)
> > > > +rows = self.db(permission.name.belongs((name,'any'))&
> > > > +
> > > > permission.table_name.belongs((str(table_name),''))&
> > > > +
> > > >
> permission.record_id.belongs((record_id,0))).select(permission.group_id)
> > > > groups_required =
> groups_required.union(set([row.group_id
> > > > for row in rows]))
> > > > if groups.intersection(groups_required):
> >
> > > > Regards
> > > > Alexey
> >
> >
>
--- tools.py_ 2010-09-08 08:40:22.266751051 +0400
+++ tools.py 2010-09-08 09:44:30.050746520 +0400
@@ -2415,17 +2415,10 @@
== user_id).select(membership.group_id)
groups = set([row.group_id for row in rows])
permission = self.settings.table_permission
-rows = self.db(permission.name == name)(permission.table_name
- == str(table_name))(permission.record_id
- == record_id).select(permission.group_id)
+rows = self.db(permission.name.belongs((name,'any'))&
+ permission.table_name.belongs((str(table_name),''))&
+ permission.record_id.belongs((record_id,0))).select(permission.group_id)
groups_required = set([row.group_id for row in rows])
-if record_id:
-rows = self.db(permission.name
-== name)(permission.table_name
- == str(table_name))(permission.record_id
- == 0).select(permission.group_id)
-groups_required = groups_required.union(set([row.group_id
-for row in rows]))
if groups.intersection(groups_required):
r = True
else:
[web2py] Instructions to setup web2py under shared wsgi hosting
Hello. Recently I had to setup web2py on a shared hosting. However that hosting has wsgi support. Also that was not a new site but rather an old site, ported to web2py I didn't find any good documentation on how to setup web2py in this case so had to google and experiment. So, 1) The problem: 1) I do not have any control over main apache config file. Only .htaccess 2) Old site must continue to function until web2py's clone is developed enough. web2py must function in parallel 2) Solution: 1) put web2py into root site's folder. So that you can browse, download source files and so on. 2) put .htaccess into web2py's root folder. Here it's contents: SetHandler wsgi-script Options +ExecCGI 3) put .htaccess into site's root folder. Here it's contents: RewriteEngine On RewriteRule ^((static|auth|admin|mycontroller1|mycontroller2).*)$ /web2py/wsgihandler.py/myappname/$1 [QSA,PT,L] Done. Now old site works as it did before, but several folders (static, auth, admin, my controllers) are redirected to web2py. BTW, Massimo, it may be worth adding this .htaccess to web2py's default distribution. Comment all lines out and put some comments (may be just copypaste my ones). Regards Alexey.
[web2py] SQLFORM fix
Hi. I managed to crash SQLFORM(db.table).accepts(request.vars,session) call by adding a 'delete_this_record=on' to the list of request variables. Proposed fix: --- sqlhtml.bak.py 2010-10-13 09:52:01.202884906 +0400 +++ sqlhtml.py 2010-10-13 09:52:06.662884519 +0400 @@ -949,7 +949,7 @@ raise SyntaxError, 'user is tampering with form\'s record_id: ' \ '%s != %s' % (record_id, self.record_id) -if requested_delete: +if requested_delete and self.custom.deletable: if keyed: qry = reduce(lambda x,y: x & y, [self.table[k]==record_id[k] for k in self.table._primarykey]) Regards Alexey.
[web2py:23317] web2py perfomance
Hello again. Recently I measured the perfomance of web2py regarding to 'milliseconds per request'. Got some unexpected results. The most slow part of the application is the model. It takes 40-60% of total time. Measurement was done simply by putting import time;print time.time(),'model start' at the beginning of db.py and similar line at the end of it. Here is what it produces on my laptop (Turion64, 1.6GHz, 1.5G RAM): 1244187446.32 model start 1244187446.62 model stop 0.3 second just to set up the model! I can live with 0.05 for it, may be even 0.1, but 0.3 for _each_ GET or POST request is a bit too much, don't you think? That is for not too complex model - 17 tables, averaging 8.6 SQLFields per one. On another web2py project it takes 0.38...0.42 second each time :( I tried compiling my app and measuring again: 1244187625.31 model start 1244187625.69 model stop Not any better. In fact, it's even worse, but since results vary from run to run I suspect that it is just the same perfomance. Massimo, as I know you've been working on new model for some time already. Is there any hope of having a faster model? I suspect more lazy evaluation should do the magic, but I didn't do any research yet. Frankly speaking when I first discovered the fact that web2py always _executes_ model, controller, view, I thought that it may be a perfomance hog. Until I actually did that check I thought that it will execute db.py each time it changes on-disk and then just keep built structures somewhere around, probably pickled. May be it is still possible to use that approach to some extent? Or may be I am just completely missing the point. Please comment. -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:23363] Re: web2py perfomance
On Friday 05 June 2009 17:07:55 mdipierro wrote: > In a production environment you would be using mysql or postgresql. In > this case you should be using > > SQLDB(...,pool_size=10) > dn.define_table(,migrate=False) > > the connection pooling and migrations off make a big difference. > Perhaps you can run some tests and quantify this. migrate=False makes cuts the model init time in half - now I'm getting about 0.15-0.17s each time. Testing MySQL, stand by... Hmmm. 0.21...0.25s with mysql and migrations off... and pool_size=10. > When using sqlite you cannot use pooling and that means web2py has to > open the db every time. > > Massimo > > On Jun 5, 2:58 am, Alexey Nezhdanov wrote: > > Hello again. > > Recently I measured the perfomance of web2py regarding to 'milliseconds > > per request'. Got some unexpected results. The most slow part of the > > application is the model. It takes 40-60% of total time. Measurement was > > done simply by putting > > import time;print time.time(),'model start' > > at the beginning of db.py and similar line at the end of it. Here is what > > it produces on my laptop (Turion64, 1.6GHz, 1.5G RAM): > > > > 1244187446.32 model start > > 1244187446.62 model stop > > 0.3 second just to set up the model! I can live with 0.05 for it, may be > > even 0.1, but 0.3 for _each_ GET or POST request is a bit too much, don't > > you think? > > That is for not too complex model - 17 tables, averaging 8.6 SQLFields > > per one. On another web2py project it takes 0.38...0.42 second each time > > :( > > > > I tried compiling my app and measuring again: > > 1244187625.31 model start > > 1244187625.69 model stop > > Not any better. In fact, it's even worse, but since results vary from run > > to run I suspect that it is just the same perfomance. > > > > Massimo, as I know you've been working on new model for some time > > already. Is there any hope of having a faster model? I suspect more lazy > > evaluation should do the magic, but I didn't do any research yet. > > > > Frankly speaking when I first discovered the fact that web2py always > > _executes_ model, controller, view, I thought that it may be a perfomance > > hog. Until I actually did that check I thought that it will execute db.py > > each time it changes on-disk and then just keep built structures > > somewhere around, probably pickled. May be it is still possible to use > > that approach to some extent? > > > > Or may be I am just completely missing the point. Please comment. > > > > -- > > Sincerely yours > > Alexey Nezhdanov > > -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:23385] Re: web2py perfomance
On Friday 05 June 2009 21:03:20 mdipierro wrote: > Can you tell us more about the setup, os, hardward etc. is mysql on > the same machine? Kubuntu 8.04. Turion64 1.6GHz, 1.6G RAM. MySQL is on the same box. SiS motherboard w/ nForce chipset. Laptop 3 years old (and it was about 1 year old model when was bought). > How much is the the SQLDB() vs the define_tables? Do you have many > tables? how long? 16 tables, 152 SQLFields. single SQLDB (currently MySQL, but I'll switch it back to SQLite) > One trick is to add is statements in the model so that only those > tables needed are defined, depending on request.controller and > request.action. yes, I thought of that. But that makes it inflexible. That's why I suggested lazy tables init. And regarding 'turion is not very fast'. I don't really have any load on this box. So 0.5 seconds per GET is VERY slow. 8-years old Celeron 800 should be behaving something like 0.05 seconds per request (of course with ad-hoc programming, no DAL). This is not the empty complaint. We can't really afford saying 'throw in more CPU'. If web2py targets GAE - then it absolutely must be CPU-friendly. GAE can help with adding more nodes but it charges for processor time anyways. And actually the same goes about dedicated hosting too. If someone targets only a few visitors per day - it's ok. But not if we want tens and hundreds pageloads per second. > On Jun 5, 11:29 am, Alexey Nezhdanov wrote: > > On Friday 05 June 2009 17:07:55 mdipierro wrote:> In a production > > environment you would be using mysql or postgresql. In > > > > > this case you should be using > > > > > > SQLDB(...,pool_size=10) > > > dn.define_table(,migrate=False) > > > > > > the connection pooling and migrations off make a big difference. > > > Perhaps you can run some tests and quantify this. > > > > migrate=False makes cuts the model init time in half - now I'm getting > > about 0.15-0.17s each time. Testing MySQL, stand by... > > > > Hmmm. > > 0.21...0.25s with mysql and migrations off... and pool_size=10. > > > > > When using sqlite you cannot use pooling and that means web2py has to > > > open the db every time. > > > > > > Massimo > > > > > > On Jun 5, 2:58 am, Alexey Nezhdanov wrote: > > > > Hello again. > > > > Recently I measured the perfomance of web2py regarding to > > > > 'milliseconds per request'. Got some unexpected results. The most > > > > slow part of the application is the model. It takes 40-60% of total > > > > time. Measurement was done simply by putting > > > > import time;print time.time(),'model start' > > > > at the beginning of db.py and similar line at the end of it. Here is > > > > what it produces on my laptop (Turion64, 1.6GHz, 1.5G RAM): > > > > > > > > 1244187446.32 model start > > > > 1244187446.62 model stop > > > > 0.3 second just to set up the model! I can live with 0.05 for it, may > > > > be even 0.1, but 0.3 for _each_ GET or POST request is a bit too > > > > much, don't you think? > > > > That is for not too complex model - 17 tables, averaging 8.6 > > > > SQLFields per one. On another web2py project it takes 0.38...0.42 > > > > second each time > > > > > > > > :( > > > > > > > > I tried compiling my app and measuring again: > > > > 1244187625.31 model start > > > > 1244187625.69 model stop > > > > Not any better. In fact, it's even worse, but since results vary from > > > > run to run I suspect that it is just the same perfomance. > > > > > > > > Massimo, as I know you've been working on new model for some time > > > > already. Is there any hope of having a faster model? I suspect more > > > > lazy evaluation should do the magic, but I didn't do any research > > > > yet. > > > > > > > > Frankly speaking when I first discovered the fact that web2py always > > > > _executes_ model, controller, view, I thought that it may be a > > > > perfomance hog. Until I actually did that check I thought that it > > > > will execute db.py each time it changes on-disk and then just keep > > > > built structures somewhere around, probably pickled. May be it is > > > > still possible to use that approach to some extent? > > > > > > > > Or may be I am just completely missing the point. Please comment. > > > > > > > > -- > > > > Sincerely yours > > > > Alexey Nezhdanov > > > > -- > > Sincerely yours > > Alexey Nezhdanov > > -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:23402] Re: web2py perfomance
ON Saturday 06 June 2009 00:25:47 mdipierro wrote:
> One other trick you can try is replace
>
> db.define_table('table',SQLField('field'),...)
> db.table.field.requires=
>
> with
>
> db.define_table('table',SQLfield('field',requires=...),...)
>
> and so for all the other attributes.
That will make minor difference. I do not have too many 'requires' and mod of
what I have are set up through function call.
> Did you bytecode compile the app?
> Does it make a difference?
I just run some automated tests. Here is average time over 100 runs each:
sqlite+nomigrate+py 0.123
sqlite+nomigrate+pyc 0.122
mysql+nomigrate+py 0.123
mysql+nomigrate+pyc 0.123
I think I'll try this approach:
1) define each table as a function which yelds a table.
2) modify sql.py so that db object will test the type of table.
if it has __exec__ method - execute it and replace it with return result.
This way my tables will be lazily defined when controller actually needs them.
> Massimo
>
> On Jun 5, 3:05 pm, Alexey Nezhdanov wrote:
> > On Friday 05 June 2009 21:03:20 mdipierro wrote:> Can you tell us more
> > about the setup, os, hardward etc. is mysql on
> >
> > > the same machine?
> >
> > Kubuntu 8.04. Turion64 1.6GHz, 1.6G RAM. MySQL is on the same box. SiS
> > motherboard w/ nForce chipset. Laptop 3 years old (and it was about 1
> > year old model when was bought).
> >
> > > How much is the the SQLDB() vs the define_tables? Do you have many
> > > tables? how long?
> >
> > 16 tables, 152 SQLFields. single SQLDB (currently MySQL, but I'll switch
> > it back to SQLite)
> >
> > > One trick is to add is statements in the model so that only those
> > > tables needed are defined, depending on request.controller and
> > > request.action.
> >
> > yes, I thought of that. But that makes it inflexible. That's why I
> > suggested lazy tables init.
> >
> > And regarding 'turion is not very fast'. I don't really have any load on
> > this box. So 0.5 seconds per GET is VERY slow. 8-years old Celeron 800
> > should be behaving something like 0.05 seconds per request (of course
> > with ad-hoc programming, no DAL).
> >
> > This is not the empty complaint. We can't really afford saying 'throw in
> > more CPU'. If web2py targets GAE - then it absolutely must be
> > CPU-friendly. GAE can help with adding more nodes but it charges for
> > processor time anyways. And actually the same goes about dedicated
> > hosting too. If someone targets only a few visitors per day - it's ok.
> > But not if we want tens and hundreds pageloads per second.
> >
> > > On Jun 5, 11:29 am, Alexey Nezhdanov wrote:
> > > > On Friday 05 June 2009 17:07:55 mdipierro wrote:> In a production
> > > > environment you would be using mysql or postgresql. In
> > > >
> > > > > this case you should be using
> > > > >
> > > > > SQLDB(...,pool_size=10)
> > > > > dn.define_table(,migrate=False)
> > > > >
> > > > > the connection pooling and migrations off make a big difference.
> > > > > Perhaps you can run some tests and quantify this.
> > > >
> > > > migrate=False makes cuts the model init time in half - now I'm
> > > > getting about 0.15-0.17s each time. Testing MySQL, stand by...
> > > >
> > > > Hmmm.
> > > > 0.21...0.25s with mysql and migrations off... and pool_size=10.
> > > >
> > > > > When using sqlite you cannot use pooling and that means web2py has
> > > > > to open the db every time.
> > > > >
> > > > > Massimo
> > > > >
> > > > > On Jun 5, 2:58 am, Alexey Nezhdanov wrote:
> > > > > > Hello again.
> > > > > > Recently I measured the perfomance of web2py regarding to
> > > > > > 'milliseconds per request'. Got some unexpected results. The most
> > > > > > slow part of the application is the model. It takes 40-60% of
> > > > > > total time. Measurement was done simply by putting
> > > > > > import time;print time.time(),'model start'
> > > > > > at the beginning of db.py and similar line at the end of it. Here
> > > > > > is what it produces on my laptop (Turion64, 1.6GHz, 1.5G RAM):
> > > > > >
> > > > > > 1244187446.32 model start
> > > > > > 1244187446.62
[web2py:23406] Re: web2py perfomance
Switched to lazy table definitions.
Model init time was cut down to 0.046s.
Some of excess time is eliminated, some (my guess is 30%) is moved into
controller execution. At any rate - this is faster than before.
Next step would be full-scale profiling but not yet.
Here is excerpt from my SQLStorage:
--
def __getitem__(self, key):
value = dict.__getitem__(self, str(key))
if not callable(value) or key[0]=='_' or isinstance(value,
SQLCallableList): return value
value.__call__()# That must redefine table in-place
return dict.__getitem__(self, str(key))
and here is excerpt from my db.py:
--
def define_table_system_participant():
db.define_table('system_participant',
SQLField('firm_id','integer'),
migrate=migrate,
)
db.system_participant=define_table_system_participant
--
On Saturday 06 June 2009 07:53:18 Alexey Nezhdanov wrote:
> ON Saturday 06 June 2009 00:25:47 mdipierro wrote:
> > One other trick you can try is replace
> >
> > db.define_table('table',SQLField('field'),...)
> > db.table.field.requires=
> >
> > with
> >
> > db.define_table('table',SQLfield('field',requires=...),...)
> >
> > and so for all the other attributes.
>
> That will make minor difference. I do not have too many 'requires' and mod
> of what I have are set up through function call.
>
> > Did you bytecode compile the app?
> > Does it make a difference?
>
> I just run some automated tests. Here is average time over 100 runs each:
>
> sqlite+nomigrate+py 0.123
> sqlite+nomigrate+pyc 0.122
> mysql+nomigrate+py 0.123
> mysql+nomigrate+pyc 0.123
>
> I think I'll try this approach:
> 1) define each table as a function which yelds a table.
> 2) modify sql.py so that db object will test the type of table.
> if it has __exec__ method - execute it and replace it with return result.
>
> This way my tables will be lazily defined when controller actually needs
> them.
>
> > Massimo
> >
> > On Jun 5, 3:05 pm, Alexey Nezhdanov wrote:
> > > On Friday 05 June 2009 21:03:20 mdipierro wrote:> Can you tell us more
> > > about the setup, os, hardward etc. is mysql on
> > >
> > > > the same machine?
> > >
> > > Kubuntu 8.04. Turion64 1.6GHz, 1.6G RAM. MySQL is on the same box. SiS
> > > motherboard w/ nForce chipset. Laptop 3 years old (and it was about 1
> > > year old model when was bought).
> > >
> > > > How much is the the SQLDB() vs the define_tables? Do you have many
> > > > tables? how long?
> > >
> > > 16 tables, 152 SQLFields. single SQLDB (currently MySQL, but I'll
> > > switch it back to SQLite)
> > >
> > > > One trick is to add is statements in the model so that only those
> > > > tables needed are defined, depending on request.controller and
> > > > request.action.
> > >
> > > yes, I thought of that. But that makes it inflexible. That's why I
> > > suggested lazy tables init.
> > >
> > > And regarding 'turion is not very fast'. I don't really have any load
> > > on this box. So 0.5 seconds per GET is VERY slow. 8-years old Celeron
> > > 800 should be behaving something like 0.05 seconds per request (of
> > > course with ad-hoc programming, no DAL).
> > >
> > > This is not the empty complaint. We can't really afford saying 'throw
> > > in more CPU'. If web2py targets GAE - then it absolutely must be
> > > CPU-friendly. GAE can help with adding more nodes but it charges for
> > > processor time anyways. And actually the same goes about dedicated
> > > hosting too. If someone targets only a few visitors per day - it's ok.
> > > But not if we want tens and hundreds pageloads per second.
> > >
> > > > On Jun 5, 11:29 am, Alexey Nezhdanov wrote:
> > > > > On Friday 05 June 2009 17:07:55 mdipierro wrote:> In a production
> > > > > environment you would be using mysql or postgresql. In
> > > > >
> > > > > > this case you should be using
> > > > > >
> > > > > > SQLDB(...,pool_size=10)
> > > > > > dn.define_table(,migrate=False)
> > > > > >
> > > > > > the connection pooling and migrations off make a big difference.
> > > > > > Perhaps you can run some tests and quantify this.
> > > > >
> > > > > migrate=False makes cu
[web2py:23477] Re: web2py perfomance
=
1) When initialising, SQLField uses dir() call to find any clashes
with reserved names. That takes 11% of TOTAL execution
time, including controller and view. Very easy to optimise out -
for instance use 'migrate' as a flag that we running on production
environment so skip this check.
2) is_integer is a fast call, but with 1.1k (!) calls _per_singe_ GET request
makes it consume almost 8% of absolute processor time. I didn't research
though why it is used at all so can't comment if and how it could be optimised.
3) sqlhtml_validators is an obvious bug. It creates 9 different validators on
each call
and then throws out 8 of them! 8/9 of work is wasted. More than that - I suspect
that dictionary building may be moved into module namespace, moving this
function
far-far down in this list.
4) lazy tables init is on, but even with that SQLField.__init__ takes 30% (!!!)
of time. That expense happens mostly in the kids, but the function itself
takes a good share too. Definitely worths looking closer.
5) another __init__, now the SQLTable's. Same strings attached as to (4).
> Massimo
>
> On Jun 6, 1:14 am, Alexey Nezhdanov wrote:
> > Switched to lazy table definitions.
> > Model init time was cut down to 0.046s.
> > Some of excess time is eliminated, some (my guess is 30%) is moved into
> > controller execution. At any rate - this is faster than before.
> >
> > Next step would be full-scale profiling but not yet.
> >
> > Here is excerpt from my SQLStorage:
> > --
> > def __getitem__(self, key):
> > value = dict.__getitem__(self, str(key))
> > if not callable(value) or key[0]=='_' or isinstance(value,
> > SQLCallableList): return value
> > value.__call__() # That must redefine table in-place
> > return dict.__getitem__(self, str(key))
> >
> > and here is excerpt from my db.py:
> > --
> > def define_table_system_participant():
> > db.define_table('system_participant',
> > SQLField('firm_id','integer'),
> > migrate=migrate,
> > )
> > db.system_participant=define_table_system_participant
> > --
> >
> > On Saturday 06 June 2009 07:53:18 Alexey Nezhdanov wrote:
> > > ON Saturday 06 June 2009 00:25:47 mdipierro wrote:
> > > > One other trick you can try is replace
> > > >
> > > > db.define_table('table',SQLField('field'),...)
> > > > db.table.field.requires=
> > > >
> > > > with
> > > >
> > > > db.define_table('table',SQLfield('field',requires=...),...)
> > > >
> > > > and so for all the other attributes.
> > >
> > > That will make minor difference. I do not have too many 'requires' and
> > > mod of what I have are set up through function call.
> > >
> > > > Did you bytecode compile the app?
> > > > Does it make a difference?
> > >
> > > I just run some automated tests. Here is average time over 100 runs
> > > each:
> > >
> > > sqlite+nomigrate+py 0.123
> > > sqlite+nomigrate+pyc 0.122
> > > mysql+nomigrate+py 0.123
> > > mysql+nomigrate+pyc 0.123
> > >
> > > I think I'll try this approach:
> > > 1) define each table as a function which yelds a table.
> > > 2) modify sql.py so that db object will test the type of table.
> > > if it has __exec__ method - execute it and replace it with return
> > > result.
> > >
> > > This way my tables will be lazily defined when controller actually
> > > needs them.
> > >
> > > > Massimo
> > > >
> > > > On Jun 5, 3:05 pm, Alexey Nezhdanov wrote:
> > > > > On Friday 05 June 2009 21:03:20 mdipierro wrote:> Can you tell us
> > > > > more about the setup, os, hardward etc. is mysql on
> > > > >
> > > > > > the same machine?
> > > > >
> > > > > Kubuntu 8.04. Turion64 1.6GHz, 1.6G RAM. MySQL is on the same box.
> > > > > SiS motherboard w/ nForce chipset. Laptop 3 years old (and it was
> > > > > about 1 year old model when was bought).
> > > > >
> > > > > > How much is the the SQLDB() vs the define_tables? Do you have
> > > > > > many tables? how long?
> > > > >
> > > > > 16 tables, 152 SQLFields. single SQLDB (currently MySQL, but I'll
> > > > > switch it back to SQLite)
> > > > >
&g
[web2py:23480] Re: web2py perfomance
On Sunday 07 June 2009 11:13:49 Alexey Nezhdanov wrote:
> 1) When initialising, SQLField uses dir() call to find any clashes
> with reserved names.
I was a bit incorrect here. It checks for already defined fields too.
Anyways - I optimised it by commenting it out :-P
In practice I'd recommend similar approach:
1) don't check for clashes in the production environment (SQLDB init
parameter?)
2) (optional) maintain cache of names for each table. Don't use dir().
> 2) is_integer is a fast call, but with 1.1k (!) calls ...
Replaced it with my own version:
integer_pat=re.compile('[0-9]+$')
is_integer=lambda x: integer_pat.match(x)
it's about 2.3 times faster. C version would be even better.
> 3) sqlhtml_validators is an obvious bug... 8/9 of work is wasted.
> ... I suspect that dictionary building may be moved into module namespace
did that with some tweaks (same validators get reused). Dangerous approach
may introduce hidden dataflow, but that could be prevented by somehow making
these validators read-only.
Got another 9.5% model time speedup. (0.096s vs 0.106s). Though this figures
get more and more meaningless as model init time reduces it's share in total
page generation time. Probably I'll just switch to time measuring of
urllib.urlopen().
...OK, switched. Here are stats. Each line is in the format
model_init_time/urlopen_measured_time - average over 100 calls
These timings are not comparable with my previous emails because I did some
additional arrangements that should make these times more than usual, but
more stable across tests. Also application is compiled this time.
1) no optimisations
0.1219/0.1649
2) skip CREATE TABLE in SQLTable init
0.0865/0.1261
3) + three steps, described above in this mail.
0.0439/0.0855
4) + lazy tables init in model
This one is application specific. For instance in my application it causes
4 tables out of 16 to be initialised in model and 2 more - in controller. This
is for front page. Other pages will have different figures. Yet, here are
timings:
0.0194/0.0622
5) And finally, for mere comparison, timings of (4) again, but this time with
non-compiled source.
0.0208/0.1216
Some conclusions:
1) always compile your app for production. It saves you about 50% CPU.
2) With very little effort it was possible to speedup web2py app further 2.65
times, dropping page generation times from tenths of second to hundredths.
I hope at least some of that work will find it's way into mainline. I'm
attaching all patches that I used to this mail. These patches are against
quite old web2py - Version 1.61.4 (2009-04-21 10:02:50) (I didn't like how
newer versions looked like so I stick to initially installed version).
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
--- sql.py.old 2009-05-23 14:28:15.286720712 +0400
+++ sql.py 2009-06-07 12:38:16.509986136 +0400
@@ -1053,6 +1058,8 @@
return False
return True
+integer_pat=re.compile('[0-9]+$')
+is_integer=lambda x: integer_pat.match(x) #0m3.688s
class SQLTable(dict):
--- sql.py.old 2009-05-23 14:28:15.286720712 +0400
+++ sql.py 2009-06-07 12:38:56.777987959 +0400
@@ -487,6 +487,10 @@
"""
def __getitem__(self, key):
+value = dict.__getitem__(self, str(key))
+if not callable(value) or key[0]=='_' or not isinstance(value, types.FunctionType): return value
+print 'calling',value
+value.__call__()# That must redefine table in-place
return dict.__getitem__(self, str(key))
def __setitem__(self, key, value):
--- sql.py.old 2009-05-23 14:28:15.286720712 +0400
+++ sql.py 2009-06-07 11:54:40.977982790 +0400
@@ -331,12 +331,9 @@
}
-def sqlhtml_validators(field_type, length):
-v = {
+preset_validators = {
'boolean': [],
-'string': validators.IS_LENGTH(length),
'text': validators.IS_LENGTH(2 ** 16),
-'password': validators.IS_LENGTH(length),
'blob': [],
'upload': [],
'double': validators.IS_FLOAT_IN_RANGE(-1e100, 1e100),
@@ -346,10 +343,19 @@
'datetime': validators.IS_DATETIME(),
'reference': validators.IS_INT_IN_RANGE(0, 1e100),
}
-try:
-return v[field_type[:9]]
-except KeyError:
-return []
+preset_validators2 = {
+'string': {},
+'password': {},
+}
+d
[web2py:23482] Re: web2py perfomance
On Sunday 07 June 2009 14:36:24 Iceberg wrote: > > Hmm. Tried this: > > if args['migrate']: > > sql_locker.acquire() > > try: > > query = t._create(migrate=args['migrate']) > > except BaseException, e: > > sql_locker.release() > > raise e > > sql_locker.release() > Just a remind. The try...except clause above should better change into > this one. That's reminder to Massimo :) In that code sample I have only one line (if:) everything other was there before me. > try: > query=t._create(...) > finally: > sql_locker.release() > Only in this way, any exception (if any) inside t._create(...) can > show its correct break point, rather than the line "raise e". yes, and even better would be with sql_locker: query= just two lines - instead of 5 with the same breakpoint benefit. On the other hand - it requires doing from __future__ import with_statement and that WILL break python 2.4 compartibility once again. -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:23490] Re: web2py perfomance
On Sunday 07 June 2009 15:49:31 Iceberg wrote:
> On Jun7, 6:35pm, Alexey Nezhdanov wrote:
> > > 2) is_integer is a fast call, but with 1.1k (!) calls ...
> >
> > Replaced it with my own version:
> > integer_pat=re.compile('[0-9]+$')
> > is_integer=lambda x: integer_pat.match(x)
> > it's about 2.3 times faster. C version would be even better.
>
> If so, perhaps this is even better:
>
> integer_pat=re.compile('[0-9]+$')
> is_integer=integer_pat.match
OH! You win almost 1.5 times.
2.340s vs mine 0m3.328s for 20k operations.
> Because lambda is considered as much slower than built-in functions,
> AFAIK.
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:23491] Re: web2py perfomance
On Sunday 07 June 2009 16:10:10 mdipierro wrote: > BTW... why you did not like the newer versions 1.63.5? I'm all for simplistic designs. I didn't like these things: 1) session.flash as semi-transparent window over content. 2) popup menus. I prefer clicking on them 3) I wasn't able to find site admin link from the first time. Found it only on next day or may be ever the next day after that. Before I had to navigate through menu of admin for current application. P.S. Let me run some tests now. Thank you for incorporating these changes. I knew that they couldn't be there as is - because I knew that I possible breaking other people's apps. But these changes were too effective to not try. -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:23493] Re: web2py perfomance
On Sunday 07 June 2009 16:09:40 mdipierro wrote: > Can you run your test again please and then we'll work on this more? I don't have bzr installed (and not really want to) and didn't find how to download tree tip out of launchpad. So I just downloaded single sql.py and put it into gluon/. That should be fine since my changes are at any rate only for sql.py. Under the same conditions I got this figures for upstream sql.py: 0.0646/0.1044 As I understand - that should be compared to my >3) + three steps, described above in this mail. >0.0439/0.0855 So about 18% slower than in my case but still 1.58 times faster than yesterday. I'll run profiling now and provide it as attachment. ... attached. Also for a reference I attach sql.py which is a stock version from BZR with only one line commented out (json support) to let it live with older web2py. ... oh...I looked into profile file and then into sql.py. You omitted the validators patch completely. Why? W/o it all that testing is loss of time. :( I need to go now. I'll provide backwards-compartible validators patch when I return (something like 5 hours from now). -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~--- nik.profile.txt.gz Description: GNU Zip compressed data sql.py.gz Description: GNU Zip compressed data
[web2py:23494] Re: web2py perfomance
You are right. Somehow I thought that if I upgrade - I'll get new design to my (already existing app). :) Will upgrade later today. On Sunday 07 June 2009 16:55:25 mdipierro wrote: > I see. But that is just the scaffoling app. Keep the old welcome.w2p > if you like or make your own scaffolding app (replace welcome.w2p with > any packed app). There are many bug fixes in the library and new > features that you would otherwise miss. > > Massimo > > On Jun 7, 7:33 am, Alexey Nezhdanov wrote: > > On Sunday 07 June 2009 16:10:10 mdipierro wrote:> BTW... why you did not > > like the newer versions 1.63.5? > > > > I'm all for simplistic designs. > > I didn't like these things: > > 1) session.flash as semi-transparent window over content. > > 2) popup menus. I prefer clicking on them > > 3) I wasn't able to find site admin link from the first time. Found it > > only on next day or may be ever the next day after that. Before I had to > > navigate through menu of admin for current application. > > > > P.S. Let me run some tests now. > > Thank you for incorporating these changes. I knew that they couldn't be > > there as is - because I knew that I possible breaking other people's > > apps. But these changes were too effective to not try. > > > > -- > > Sincerely yours > > Alexey Nezhdanov > > -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:23520] Re: web2py perfomance
On Sunday 07 June 2009 18:41:30 mdipierro wrote:
> I implemented a backward compatible speedup of the default validators.
> I also realized that those many calls to is_integer could be avoided.
> I think I fixed it.
0.0429/0.0844
Somehow consistency is not kept across reboots :(
... about a hour later: and actually now I get about 10% noise so it become
very hard to make any good measurements. I'll try tomorrow on better
hardware. In the meanwhile, I'll try to profile it once again.
...
ok, there are still two perfomance hogs left: SQLTable.__init__ (about 11%)
and SQLField.__init__ (about 6,5%). These percentages are 'self' times - i.e.
it was spent directly in lines 1099-1149, 1693-1752.
Python profiling doesn't allow to attribute CPU consumption to individual
lines so for now I don't have any more precise pointers. But that could be
split into functional blocks and narrowed further down.
In any case - we collected almost all low-hanging fruits. Any further
optimisation will give gains of range 2-5%, not times anymore.
Except lazy tables which still may give tens of percents. BTW - Massimo, you
said something about SQLLazyTable class. You should implement SQLLazyField
too then. Or may be just wait a day or two until I do deeper profiling.
> Please check out the latest trunk.
>
> Massimo
>
P.S. Here is small fix to is_integer. We forgot about negative numbers:
Also this version is tiny bit faster
is_integer=re.compile('-?[\d]+).match
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:23585] Re: web2py perfomance
Hi again.
Here are another testing results.
First of all, let me remind you about is_integer bug. It doesn't allow
negative numbers. Correct code is:
is_integer=re.compile('-?[\d]+).match
Now. I've run more timings, more profiling and more code tweaks.
This time it was Core2 Duo @2.20GHz with 1G RAM. Ubuntu 8.04.
Testing strategy: 20 sets of 1000 GET requests to the main page. Same
application as before. For each set I have average GET request time.
For a group of 20 sets I get minimum average, global average and
maximum average. Also I have average model init time for 20k requests
(no splitting into each set).
1) Upstream sql.py
model 0.01646
min 0.02979
avg 0.03267
max 0.03597
2) Optimised SQLTable.__init__ and SQLField.__init__ (patches
attached, backwards compartible, though you may want to remove my
testing marks)
model 0.01380
min 0.02697
avg 0.03003
max 0.03189
So - about 8% speedup (by avg time) here.
3) Now lazy tables (patch attached).
Since I expect that you will not find any objections to my patches for
(2) here are cumulative results, i.e. optimised __init__s + lazy
tables
...
hmmm. It doesn't improves results significantly anymore. It looks like
these patches are interchangeable. I've run my test three times,
results are:
model 0.01009
min 0.02555
avg 0.0297105
max 0.033360
model 0.01012
min 0.02369
avg 0.02983
max 0.03340
model 0.00966
min 0.02415
avg 0.02850
max 0.03208
So it makes some improvement about 1%-5% but somehow it also makes
testing results more disperse. Do not know if it worths including.
Massimo, your opinion?
4) Also I've tried to further speedup validator generation w/o losing
backwards compartibility, but improvement was about 1% and below the
margin of error (even less noticeable than with lazy tables).
On Sun, Jun 7, 2009 at 11:45 PM, Alexey Nezhdanov wrote:
> On Sunday 07 June 2009 18:41:30 mdipierro wrote:
>> I implemented a backward compatible speedup of the default validators.
>> I also realized that those many calls to is_integer could be avoided.
>> I think I fixed it.
> 0.0429/0.0844
> Somehow consistency is not kept across reboots :(
> ... about a hour later: and actually now I get about 10% noise so it become
> very hard to make any good measurements. I'll try tomorrow on better
> hardware. In the meanwhile, I'll try to profile it once again.
> ...
> ok, there are still two perfomance hogs left: SQLTable.__init__ (about 11%)
> and SQLField.__init__ (about 6,5%). These percentages are 'self' times - i.e.
> it was spent directly in lines 1099-1149, 1693-1752.
> Python profiling doesn't allow to attribute CPU consumption to individual
> lines so for now I don't have any more precise pointers. But that could be
> split into functional blocks and narrowed further down.
>
> In any case - we collected almost all low-hanging fruits. Any further
> optimisation will give gains of range 2-5%, not times anymore.
>
> Except lazy tables which still may give tens of percents. BTW - Massimo, you
> said something about SQLLazyTable class. You should implement SQLLazyField
> too then. Or may be just wait a day or two until I do deeper profiling.
>
>> Please check out the latest trunk.
>>
>> Massimo
>>
>
> P.S. Here is small fix to is_integer. We forgot about negative numbers:
> Also this version is tiny bit faster
> is_integer=re.compile('-?[\d]+).match
>
> --
> Sincerely yours
> Alexey Nezhdanov
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
--- /home/snake/Desktop/sql.py 2009-06-08 09:36:27.225210464 +0400
+++ sql.py 2009-06-08 11:32:18.285211851 +0400
@@ -1112,39 +1112,46 @@
@raise SyntaxError: when a supplied field is of incorrect type.
"""
-new_fields = []
-for field in fields:
-if isinstance(field, SQLField):
-new_fields.append(field)
-elif isinstance(field, SQLTable):
-new_fields += [copy.copy(field[f]) for f in
- field.fields if f != 'id']
-else:
-raise SyntaxError, \
-'define_table argument is not a SQLField'
-fields = new_fields
-self._db = db
-self._tablename = tablename
-self.fields = SQLCallableList()
-self._referenced_by = []
-fields = list(fields)
-fields.insert(0, SQLField('id', 'id
[web2py:23592] Re: web2py perfomance
On Monday 08 June 2009 17:57:36 mdipierro wrote:
> Thanks Alexey, since these tests are on a different architecture, can
> you give the total relative speedup on the same architecture for
> executing your original model (code in trunk vs original code)?
You see, I switched to another architecture because testing it on my laptop
become unreliable and obtaining more-or-less stable values required a lot of
time. Powersave and CPU frequency scaling features kick in and introduce a
great deal of noise. So I switched to powerful desktop to run tests at high
speed in greater volume to gain at least one more digit of preciseness.
And may be you missed it - I actually provided comparison numbers. See below.
If you want me to measure also old code (2 days old) on same box - I'll do
that.
> > First of all, let me remind you about is_integer bug. It doesn't allow
> > negative numbers. Correct code is:
> > is_integer=re.compile('-?[\d]+).match
>
> This is not a bug. Since ID values cannot be negative.
Then it is a bug in name. It should be is_id_int or is_non_negative_int.
Here are the figures for the code in trunk
> > 1) Upstream sql.py
> > model 0.01646
> > min 0.02979
> > avg 0.03267
> > max 0.03597
Here are the figures for the first two patches. They give about 8% speedup.
> > 2) Optimised SQLTable.__init__ and SQLField.__init__ (patches
> > attached, backwards compartible, though you may want to remove my
> > testing marks)
> > model 0.01380
> > min 0.02697
> > avg 0.03003
> > max 0.03189
> > So - about 8% speedup (by avg time) here.
Here are the figures for the last patch. Uncertain 1-5% speedup.
> > 3) Now lazy tables (patch attached).
> > Since I expect that you will not find any objections to my patches for
> > (2) here are cumulative results, i.e. optimised __init__s + lazy
> > tables
> > ...
> > hmmm. It doesn't improves results significantly anymore. It looks like
> > these patches are interchangeable. I've run my test three times,
> > results are:
> >
> > model 0.01009
> > min 0.02555
> > avg 0.0297105
> > max 0.033360
> >
> > model 0.01012
> > min 0.02369
> > avg 0.02983
> > max 0.03340
> >
> > model 0.00966
> > min 0.02415
> > avg 0.02850
> > max 0.03208
> >
> > So it makes some improvement about 1%-5% but somehow it also makes
> > testing results more disperse. Do not know if it worths including.
> > Massimo, your opinion?
> >
> > 4) Also I've tried to further speedup validator generation w/o losing
> > backwards compartibility, but improvement was about 1% and below the
> > margin of error (even less noticeable than with lazy tables).
> >
> > On Sun, Jun 7, 2009 at 11:45 PM, Alexey Nezhdanov
wrote:
> > > On Sunday 07 June 2009 18:41:30 mdipierro wrote:
> > >> I implemented a backward compatible speedup of the default validators.
> > >> I also realized that those many calls to is_integer could be avoided.
> > >> I think I fixed it.
> > >
> > > 0.0429/0.0844
> > > Somehow consistency is not kept across reboots :(
> > > ... about a hour later: and actually now I get about 10% noise so it
> > > become very hard to make any good measurements. I'll try tomorrow on
> > > better hardware. In the meanwhile, I'll try to profile it once again.
> > > ...
> > > ok, there are still two perfomance hogs left: SQLTable.__init__ (about
> > > 11%) and SQLField.__init__ (about 6,5%). These percentages are 'self'
> > > times - i.e. it was spent directly in lines 1099-1149, 1693-1752.
> > > Python profiling doesn't allow to attribute CPU consumption to
> > > individual lines so for now I don't have any more precise pointers. But
> > > that could be split into functional blocks and narrowed further down.
> > >
> > > In any case - we collected almost all low-hanging fruits. Any further
> > > optimisation will give gains of range 2-5%, not times anymore.
> > >
> > > Except lazy tables which still may give tens of percents. BTW -
> > > Massimo, you said something about SQLLazyTable class. You should
> > > implement SQLLazyField too then. Or may be just wait a day or two until
> > > I do deeper profiling.
> > >
> > >> Please check out the latest trunk.
> > >>
> > >> Massimo
> > >
> > > P.S. Here is small fix to is_integer. We forgot about negative numbers:
> > > Also this version is tiny bit faster
> > > is_integer=re.compile('-?[\d]+).match
> > >
> > > --
> > > Sincerely yours
> > > Alexey Nezhdanov
> >
> > x_table.diff
> > 6KViewDownload
> >
> > x_field.diff
> > 2KViewDownload
> >
> > lazy_define_table2.diff
> > 2KViewDownload
>
>
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:23598] Re: web2py perfomance
On Monday 08 June 2009 19:18:26 mdipierro wrote:
> I understand what you say below but I think it is important to measure
> the total speedup from the original/stable sql.py and the one in trunk
> on the same machine and same complex model that you have.
NP, it is running right now. Quite slow, will finish in ~10minutes.
Preliminary results (10 sets:)
min 0.05579
avg 0.055944
max 0.05611
model time unknown
That is against revision 875.
> Massimo
>
> On Jun 8, 9:54 am, Alexey Nezhdanov wrote:
> > On Monday 08 June 2009 17:57:36 mdipierro wrote:> Thanks Alexey, since
> > these tests are on a different architecture, can
> >
> > > you give the total relative speedup on the same architecture for
> > > executing your original model (code in trunk vs original code)?
> >
> > You see, I switched to another architecture because testing it on my
> > laptop become unreliable and obtaining more-or-less stable values
> > required a lot of time. Powersave and CPU frequency scaling features kick
> > in and introduce a great deal of noise. So I switched to powerful desktop
> > to run tests at high speed in greater volume to gain at least one more
> > digit of preciseness.
> >
> > And may be you missed it - I actually provided comparison numbers. See
> > below. If you want me to measure also old code (2 days old) on same box -
> > I'll do that.
> >
> > > > First of all, let me remind you about is_integer bug. It doesn't
> > > > allow negative numbers. Correct code is:
> > > > is_integer=re.compile('-?[\d]+).match
> > >
> > > This is not a bug. Since ID values cannot be negative.
> >
> > Then it is a bug in name. It should be is_id_int or is_non_negative_int.
> >
> > Here are the figures for the code in trunk
> >
> > > > 1) Upstream sql.py
> > > > model 0.01646
> > > > min 0.02979
> > > > avg 0.03267
> > > > max 0.03597
> >
> > Here are the figures for the first two patches. They give about 8%
> > speedup.
> >
> > > > 2) Optimised SQLTable.__init__ and SQLField.__init__ (patches
> > > > attached, backwards compartible, though you may want to remove my
> > > > testing marks)
> > > > model 0.01380
> > > > min 0.02697
> > > > avg 0.03003
> > > > max 0.03189
> > > > So - about 8% speedup (by avg time) here.
> >
> > Here are the figures for the last patch. Uncertain 1-5% speedup.
> >
> > > > 3) Now lazy tables (patch attached).
> > > > Since I expect that you will not find any objections to my patches
> > > > for (2) here are cumulative results, i.e. optimised __init__s + lazy
> > > > tables
> > > > ...
> > > > hmmm. It doesn't improves results significantly anymore. It looks
> > > > like these patches are interchangeable. I've run my test three times,
> > > > results are:
> > > >
> > > > model 0.01009
> > > > min 0.02555
> > > > avg 0.0297105
> > > > max 0.033360
> > > >
> > > > model 0.01012
> > > > min 0.02369
> > > > avg 0.02983
> > > > max 0.03340
> > > >
> > > > model 0.00966
> > > > min 0.02415
> > > > avg 0.02850
> > > > max 0.03208
> > > >
> > > > So it makes some improvement about 1%-5% but somehow it also makes
> > > > testing results more disperse. Do not know if it worths including.
> > > > Massimo, your opinion?
> > > >
> > > > 4) Also I've tried to further speedup validator generation w/o losing
> > > > backwards compartibility, but improvement was about 1% and below the
> > > > margin of error (even less noticeable than with lazy tables).
> > > >
> > > > On Sun, Jun 7, 2009 at 11:45 PM, Alexey Nezhdanov
> >
> > wrote:
> > > > > On Sunday 07 June 2009 18:41:30 mdipierro wrote:
> > > > >> I implemented a backward compatible speedup of the default
> > > > >> validators. I also realized that those many calls to is_integer
> > > > >> could be avoided. I think I fixed it.
> > > > >
> > > > > 0.0429/0.0844
> > > > > Somehow consistency is not kept across reboots :(
> > > > > ... about a hour later: and actually now I get about 10% noise so
> > > > > it become very hard to make any good measurements. I'll try
> > > > >
[web2py:23599] Re: web2py perfomance
On Monday 08 June 2009 19:27:29 Alexey Nezhdanov wrote: > On Monday 08 June 2009 19:18:26 mdipierro wrote: > > I understand what you say below but I think it is important to measure > > the total speedup from the original/stable sql.py and the one in trunk > > on the same machine and same complex model that you have. > > NP, it is running right now. Quite slow, will finish in ~10minutes. Revision 875: model 0.04167 min 0.05579 avg 0.0559455 max 0.05647 Revision 882: model 0.01646 min 0.02979 avg 0.03267 max 0.03597 882+optimised inits: model 0.01380 min 0.02697 avg 0.03003 max 0.03189 882+optimised inits + lazy tables: model 0.01009 min 0.02555 avg 0.0297105 max 0.033360 model 0.01012 min 0.02369 avg 0.02983 max 0.03340 model 0.00966 min 0.02415 avg 0.02850 max 0.03208 -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:23602] Re: web2py perfomance
В сообщении от Monday 08 June 2009 19:54:35 mdipierro написал(а): > So for everybody let me summarize this: > > Thanks to Alexey, > the web2py in trunk can execute models 2.5x faster than the current > stable/production version (requires migrate=False and bytecode > compiled models). This is a general result since each define_table > speeds up the same. For his particular app, this results on a speedup > of 70% when serving pages (this number depends on the app). > > There is also the possibility of an additional 70% speedup of the > models by implementing additional tricks. We'll keep working on this. A pity really that we didn't yet hit 100% barrier. If you don't mind - I'll continue working on it. And btw - I just realised that my 'optimised inits' patch is not optimal. I can do it better. Stand by. > Massimo -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:23609] Re: web2py perfomance
Somehow I got confused in so many similar-named sql.py's floating around. So I decided to re-run all tests to make sure that now I'm really using revisions that I claim that I use. Got very strange results. Long story short - I got too much noise into my results. It seems that trunk currently only about 40% faster than stable, not 70%. As it appears - it is very important if I have any print statements in my application. I had couple in the model, then I commented them out. Somehow that decreased(!!!) perfomance by about 1/3. I'll not be doing any more work today. Have to work out new plan. Probably we won't hit 100% barrier :( I'm sorry guys. So much excitement... -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:23651] Re: web2py perfomance
Ok. Now the confusion is resolved. 1) Speed improvements of 70% and up that I reported yesterday are really exist. I just reproduced a 3.47 times model speedup and 2.15 overall speedup for my app (r875 vs r822+inits). BUT this app is atypical. I have added some time measuring code there so it prints out two lines per each model init. So when I am testing perfomance - screen very quickly scrolls up 2) Simply commenting out two print statements gives me only 1.67 overall speedup given equal other conditions. I think that processor receive additional interrupts from videocard that in turn results in more often checks of tasks queue. 3) I declare all my previous testing results spoiled by noise generated by print statement and inappropriate kernel scheduler setting. I've set up yet another test box with these parameters: Intel Core2 Duo 2.66GHz, 2G RAM, Ubuntu 9.04, 'server' flavour kernel 2.6.28-11.42. Initially I considered to install a 'realtime' kernel, but it appeared to be unstable on that hardware (and afterall - it's for sound/video processing and 'server' type is more likely to be installed on servers). Will report new testing results (and finally I hope to write 'optimised inits ver.2' patch) later today. On Tue, Jun 9, 2009 at 5:14 AM, mdipierro wrote: > > Please try launchpad 893. I think it should be faster on GAE. > We can do better with lazy tables but at least the validators and > calls to getitem are eliminated. > > Massimo > > On Jun 8, 1:05 pm, Markus Gritsch wrote: >> On Mon, Jun 8, 2009 at 5:54 PM, mdipierro wrote: >> >> > the web2py in trunk can execute models 2.5x faster than the current >> > stable/production version (requires migrate=False and bytecode >> > compiled models). >> >> Will this speedup also has an effect on GAE? IMO one uploads no .pyc files? >> >> Markus > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:23655] Re: web2py perfomance
new testing: SERVER KERNEL --prints r8750.04898 r822ini 0.03070 1.60x --silent r8750.04914 r822ini 0.03049 1.61x So I get much more consistent results on this hardware. While this is obviously not the best perfomance (my weaker box, with less RAM, troubled with video output 1280x1024, software 90deg rotation - performs BETTER), that does not matter. What does - is that I can now be sure that these results are noise-free so I can safely compare timings from various patches. Proceeding with writing 'inits v2'. On Tue, Jun 9, 2009 at 10:43 AM, Alexey Nezhdanov wrote: > Ok. Now the confusion is resolved. > 1) Speed improvements of 70% and up that I reported yesterday are > really exist. I just reproduced a 3.47 times model speedup and 2.15 > overall speedup for my app (r875 vs r822+inits). > BUT this app is atypical. I have added some time measuring code there > so it prints out two lines per each model init. So when I am testing > perfomance - screen very quickly scrolls up > > 2) Simply commenting out two print statements gives me only 1.67 > overall speedup given equal other conditions. I think that processor > receive additional interrupts from videocard that in turn results in > more often checks of tasks queue. > > 3) I declare all my previous testing results spoiled by noise > generated by print statement and inappropriate kernel scheduler > setting. > I've set up yet another test box with these parameters: > > Intel Core2 Duo 2.66GHz, 2G RAM, Ubuntu 9.04, 'server' flavour kernel > 2.6.28-11.42. Initially I considered to install a 'realtime' kernel, > but it appeared to be unstable on that hardware (and afterall - it's > for sound/video processing and 'server' type is more likely to be > installed on servers). > > Will report new testing results (and finally I hope to write > 'optimised inits ver.2' patch) later today. > > On Tue, Jun 9, 2009 at 5:14 AM, mdipierro wrote: >> >> Please try launchpad 893. I think it should be faster on GAE. >> We can do better with lazy tables but at least the validators and >> calls to getitem are eliminated. >> >> Massimo >> >> On Jun 8, 1:05 pm, Markus Gritsch wrote: >>> On Mon, Jun 8, 2009 at 5:54 PM, mdipierro wrote: >>> >>> > the web2py in trunk can execute models 2.5x faster than the current >>> > stable/production version (requires migrate=False and bytecode >>> > compiled models). >>> >>> Will this speedup also has an effect on GAE? IMO one uploads no .pyc files? >>> >>> Markus >> >> >> > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:23662] Re: web2py perfomance
You probably right. But I can't give away the project I'm testing it on and don't yet 'ready' to write a separate one exclusively for testing. I'm still fighting with getting stable results even on this box :( On Tue, Jun 9, 2009 at 11:17 AM, Yarko Tymciurak wrote: > it might help to have tests in a state that you can ask others to run them; > a dozen or so other random boxes will help you gain the confidence you > seek I think.... > > On Tue, Jun 9, 2009 at 2:12 AM, Alexey Nezhdanov wrote: >> >> new testing: >> >> SERVER KERNEL >> --prints >> r875 0.04898 >> r822ini 0.03070 1.60x >> --silent >> r875 0.04914 >> r822ini 0.03049 1.61x >> >> So I get much more consistent results on this hardware. >> While this is obviously not the best perfomance (my weaker box, >> with less RAM, troubled with video output 1280x1024, >> software 90deg rotation - performs BETTER), that does not matter. >> What does - is that I can now be sure that these results are >> noise-free so I can safely compare timings from various patches. >> Proceeding with writing 'inits v2'. >> >> >> On Tue, Jun 9, 2009 at 10:43 AM, Alexey Nezhdanov >> wrote: >> > Ok. Now the confusion is resolved. >> > 1) Speed improvements of 70% and up that I reported yesterday are >> > really exist. I just reproduced a 3.47 times model speedup and 2.15 >> > overall speedup for my app (r875 vs r822+inits). >> > BUT this app is atypical. I have added some time measuring code there >> > so it prints out two lines per each model init. So when I am testing >> > perfomance - screen very quickly scrolls up >> > >> > 2) Simply commenting out two print statements gives me only 1.67 >> > overall speedup given equal other conditions. I think that processor >> > receive additional interrupts from videocard that in turn results in >> > more often checks of tasks queue. >> > >> > 3) I declare all my previous testing results spoiled by noise >> > generated by print statement and inappropriate kernel scheduler >> > setting. >> > I've set up yet another test box with these parameters: >> > >> > Intel Core2 Duo 2.66GHz, 2G RAM, Ubuntu 9.04, 'server' flavour kernel >> > 2.6.28-11.42. Initially I considered to install a 'realtime' kernel, >> > but it appeared to be unstable on that hardware (and afterall - it's >> > for sound/video processing and 'server' type is more likely to be >> > installed on servers). >> > >> > Will report new testing results (and finally I hope to write >> > 'optimised inits ver.2' patch) later today. >> > >> > On Tue, Jun 9, 2009 at 5:14 AM, mdipierro >> > wrote: >> >> >> >> Please try launchpad 893. I think it should be faster on GAE. >> >> We can do better with lazy tables but at least the validators and >> >> calls to getitem are eliminated. >> >> >> >> Massimo >> >> >> >> On Jun 8, 1:05 pm, Markus Gritsch wrote: >> >>> On Mon, Jun 8, 2009 at 5:54 PM, mdipierro >> >>> wrote: >> >>> >> >>> > the web2py in trunk can execute models 2.5x faster than the current >> >>> > stable/production version (requires migrate=False and bytecode >> >>> > compiled models). >> >>> >> >>> Will this speedup also has an effect on GAE? IMO one uploads no .pyc >> >>> files? >> >>> >> >>> Markus >> >> >> >> >> >> > >> >> > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:23665] Re: web2py perfomance
Wrote 'optimised inits v2' patch. Tested everything. Once again I changed my testing pattern a bit. 1) I dumped my own timing tool in favor of much more standard 'ab' (apache benchmark). 2) I decidedly will publish all these results in two main. I just finished first round of testing. Here are results (reported values are average request time in milliseconds. If anyone interested - I'm attaching a full output from my console). I will do all that testing again and publish same results again so that the amount of noise in all this will be public and clearly visible (here I'm fighting with my own temptation to rerun the tests if timings seem to be 'too away' to me). First four lines share my app's db.py. Last two require a modified db.py, but modification is cosmetical - each define_table is put into separate function and I do db.tablename=create_table_tablename after each function. Model init time was not measured. No more words. Dry figures only. Each line represents result of 10k requests. r87546.025ms r88221.048ms inits v.1 18.918ms inits v.2 18.122ms inits1+lazyT16.032ms inits2+lazyT14.391ms P.S. I'm surprised about last line... Noise? On Tue, Jun 9, 2009 at 11:12 AM, Alexey Nezhdanov wrote: > new testing: > > SERVER KERNEL > --prints > r875 0.04898 > r822ini 0.03070 1.60x > --silent > r875 0.04914 > r822ini 0.03049 1.61x > > So I get much more consistent results on this hardware. > While this is obviously not the best perfomance (my weaker box, > with less RAM, troubled with video output 1280x1024, > software 90deg rotation - performs BETTER), that does not matter. > What does - is that I can now be sure that these results are > noise-free so I can safely compare timings from various patches. > Proceeding with writing 'inits v2'. > > > On Tue, Jun 9, 2009 at 10:43 AM, Alexey Nezhdanov wrote: >> Ok. Now the confusion is resolved. >> 1) Speed improvements of 70% and up that I reported yesterday are >> really exist. I just reproduced a 3.47 times model speedup and 2.15 >> overall speedup for my app (r875 vs r822+inits). >> BUT this app is atypical. I have added some time measuring code there >> so it prints out two lines per each model init. So when I am testing >> perfomance - screen very quickly scrolls up >> >> 2) Simply commenting out two print statements gives me only 1.67 >> overall speedup given equal other conditions. I think that processor >> receive additional interrupts from videocard that in turn results in >> more often checks of tasks queue. >> >> 3) I declare all my previous testing results spoiled by noise >> generated by print statement and inappropriate kernel scheduler >> setting. >> I've set up yet another test box with these parameters: >> >> Intel Core2 Duo 2.66GHz, 2G RAM, Ubuntu 9.04, 'server' flavour kernel >> 2.6.28-11.42. Initially I considered to install a 'realtime' kernel, >> but it appeared to be unstable on that hardware (and afterall - it's >> for sound/video processing and 'server' type is more likely to be >> installed on servers). >> >> Will report new testing results (and finally I hope to write >> 'optimised inits ver.2' patch) later today. >> >> On Tue, Jun 9, 2009 at 5:14 AM, mdipierro wrote: >>> >>> Please try launchpad 893. I think it should be faster on GAE. >>> We can do better with lazy tables but at least the validators and >>> calls to getitem are eliminated. >>> >>> Massimo >>> >>> On Jun 8, 1:05 pm, Markus Gritsch wrote: >>>> On Mon, Jun 8, 2009 at 5:54 PM, mdipierro wrote: >>>> >>>> > the web2py in trunk can execute models 2.5x faster than the current >>>> > stable/production version (requires migrate=False and bytecode >>>> > compiled models). >>>> >>>> Will this speedup also has an effect on GAE? IMO one uploads no .pyc >>>> files? >>>> >>>> Markus >>> >>> >>> >> > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~--- ab-runs.gz Description: GNU Zip compressed data
[web2py:23671] Re: web2py perfomance
Results of second set of test runs: r87545.523ms r88220.614ms inits v.1 18.464ms inits v.2 18.677ms inits1+lazyT15.377ms inits2+lazyT15.280ms Observed noise was 0.502ms for slowest execution (90:1 signal-to-noise) and 0.889ms for fastest execution (16:1 signal-to-noise). Given than, I think it is safe enough to say that "this particular application on this particular hardware/OS combination was sped up 2.88...3.31 times". As in last case - for completeness I'm attaching full console output I think I know what I'll do. Currently I do not want to release my project as any kind of open-source. But I can 1) Strip out any code I do not use in this perfomance testing 2) rename all sensitive fieldnames/variables to something like field1, field2, var3, var4. Release resulting package. It would be useless as it is, but it will have same perfomance parameters so could be used for testing. On Tue, Jun 9, 2009 at 1:35 PM, Alexey Nezhdanov wrote: > Wrote 'optimised inits v2' patch. Tested everything. > > Once again I changed my testing pattern a bit. > > 1) I dumped my own timing tool in favor of much more standard > 'ab' (apache benchmark). > > 2) I decidedly will publish all these results in two main. > I just finished first round of testing. Here are results (reported > values are average request time in milliseconds. If anyone interested > - I'm attaching a full output from my console). > I will do all that testing again and publish same results again so > that the amount of noise in all this will be public and clearly > visible (here I'm fighting with my own temptation to rerun the tests > if timings seem to be 'too away' to me). > > First four lines share my app's db.py. Last two require a modified > db.py, but modification is cosmetical - each define_table is put into > separate function and I do db.tablename=create_table_tablename after > each function. Model init time was not measured. > > No more words. Dry figures only. Each line represents result of 10k requests. > r875 46.025ms > r882 21.048ms > inits v.1 18.918ms > inits v.2 18.122ms > inits1+lazyT 16.032ms > inits2+lazyT 14.391ms > > P.S. I'm surprised about last line... Noise? > > On Tue, Jun 9, 2009 at 11:12 AM, Alexey Nezhdanov wrote: >> new testing: >> >> SERVER KERNEL >> --prints >> r875 0.04898 >> r822ini 0.03070 1.60x >> --silent >> r875 0.04914 >> r822ini 0.03049 1.61x >> >> So I get much more consistent results on this hardware. >> While this is obviously not the best perfomance (my weaker box, >> with less RAM, troubled with video output 1280x1024, >> software 90deg rotation - performs BETTER), that does not matter. >> What does - is that I can now be sure that these results are >> noise-free so I can safely compare timings from various patches. >> Proceeding with writing 'inits v2'. >> >> >> On Tue, Jun 9, 2009 at 10:43 AM, Alexey Nezhdanov wrote: >>> Ok. Now the confusion is resolved. >>> 1) Speed improvements of 70% and up that I reported yesterday are >>> really exist. I just reproduced a 3.47 times model speedup and 2.15 >>> overall speedup for my app (r875 vs r822+inits). >>> BUT this app is atypical. I have added some time measuring code there >>> so it prints out two lines per each model init. So when I am testing >>> perfomance - screen very quickly scrolls up >>> >>> 2) Simply commenting out two print statements gives me only 1.67 >>> overall speedup given equal other conditions. I think that processor >>> receive additional interrupts from videocard that in turn results in >>> more often checks of tasks queue. >>> >>> 3) I declare all my previous testing results spoiled by noise >>> generated by print statement and inappropriate kernel scheduler >>> setting. >>> I've set up yet another test box with these parameters: >>> >>> Intel Core2 Duo 2.66GHz, 2G RAM, Ubuntu 9.04, 'server' flavour kernel >>> 2.6.28-11.42. Initially I considered to install a 'realtime' kernel, >>> but it appeared to be unstable on that hardware (and afterall - it's >>> for sound/video processing and 'server' type is more likely to be >>> installed on servers). >>> >>> Will report new testing results (and finally I hope to write >>> 'optimised inits ver.2' patch) later today. >>> >>> On Tue, Jun 9, 2009 at 5:14 AM, mdipierro wrote: &
[web2py:23672] Re: web2py perfomance
Forgot about patches. Here they are. 'optimised inits v.2' 'lazy tables' (this one is modified to be applicable to trunk version of sql.py) On Tue, Jun 9, 2009 at 3:10 PM, Alexey Nezhdanov wrote: > Results of second set of test runs: > > r87545.523ms > r88220.614ms > inits v.1 18.464ms > inits v.2 18.677ms > inits1+lazyT15.377ms > inits2+lazyT15.280ms > > Observed noise was 0.502ms for slowest execution (90:1 > signal-to-noise) and 0.889ms for fastest execution (16:1 > signal-to-noise). > Given than, I think it is safe enough to say that > "this particular application on this particular hardware/OS > combination was sped up 2.88...3.31 times". > > As in last case - for completeness I'm attaching full console output > > I think I know what I'll do. Currently I do not want to release my > project as any kind of open-source. But I can > 1) Strip out any code I do not use in this perfomance testing > 2) rename all sensitive fieldnames/variables to something like field1, > field2, var3, var4. > Release resulting package. It would be useless as it is, but it will > have same perfomance parameters so could be used for testing. > > On Tue, Jun 9, 2009 at 1:35 PM, Alexey Nezhdanov wrote: > > Wrote 'optimised inits v2' patch. Tested everything. > > > > Once again I changed my testing pattern a bit. > > > > 1) I dumped my own timing tool in favor of much more standard > > 'ab' (apache benchmark). > > > > 2) I decidedly will publish all these results in two main. > > I just finished first round of testing. Here are results (reported > > values are average request time in milliseconds. If anyone interested > > - I'm attaching a full output from my console). > > I will do all that testing again and publish same results again so > > that the amount of noise in all this will be public and clearly > > visible (here I'm fighting with my own temptation to rerun the tests > > if timings seem to be 'too away' to me). > > > > First four lines share my app's db.py. Last two require a modified > > db.py, but modification is cosmetical - each define_table is put into > > separate function and I do db.tablename=create_table_tablename after > > each function. Model init time was not measured. > > > > No more words. Dry figures only. Each line represents result of 10k > requests. > > r87546.025ms > > r88221.048ms > > inits v.1 18.918ms > > inits v.2 18.122ms > > inits1+lazyT16.032ms > > inits2+lazyT14.391ms > > > > P.S. I'm surprised about last line... Noise? > > > > On Tue, Jun 9, 2009 at 11:12 AM, Alexey Nezhdanov > wrote: > >> new testing: > >> > >> SERVER KERNEL > >> --prints > >> r8750.04898 > >> r822ini 0.03070 1.60x > >> --silent > >> r8750.04914 > >> r822ini 0.03049 1.61x > >> > >> So I get much more consistent results on this hardware. > >> While this is obviously not the best perfomance (my weaker box, > >> with less RAM, troubled with video output 1280x1024, > >> software 90deg rotation - performs BETTER), that does not matter. > >> What does - is that I can now be sure that these results are > >> noise-free so I can safely compare timings from various patches. > >> Proceeding with writing 'inits v2'. > >> > >> > >> On Tue, Jun 9, 2009 at 10:43 AM, Alexey Nezhdanov > wrote: > >>> Ok. Now the confusion is resolved. > >>> 1) Speed improvements of 70% and up that I reported yesterday are > >>> really exist. I just reproduced a 3.47 times model speedup and 2.15 > >>> overall speedup for my app (r875 vs r822+inits). > >>> BUT this app is atypical. I have added some time measuring code there > >>> so it prints out two lines per each model init. So when I am testing > >>> perfomance - screen very quickly scrolls up > >>> > >>> 2) Simply commenting out two print statements gives me only 1.67 > >>> overall speedup given equal other conditions. I think that processor > >>> receive additional interrupts from videocard that in turn results in > >>> more often checks of tasks queue. > >>> > >>> 3) I declare all my previous testing results spoiled by noise > >>> generated by print statement and inappropriate kernel scheduler > >>> setting. > >>> I've set up ye
[web2py:23679] Re: web2py perfomance
Finally, I started migration (before I was just slapping sql.py, checked out
of the trunk over my ancient 1.61.4 install).
Migration is not easy, I've been bending source for my needs so now I have
to port all this to 1.63.5. Massimo, do you have any objections to this
patch? I proposed it some time ago, but you were too busy at the time.
The purpose is - if we want custom forms instead of SQLFORM then we have to
write our own view.html. But on the other hand - that is double work in the
regard that we need to write down each time while we already generated all this in controller.
I'd really like to be able to write in my controller:
{{=form.custom.label.my_input}}
{{=form.custom.widget.my_input}}
See attached patch.
On Tue, Jun 9, 2009 at 3:14 PM, Alexey Nezhdanov wrote:
> Forgot about patches. Here they are.
> 'optimised inits v.2'
> 'lazy tables' (this one is modified to be applicable to trunk version of
> sql.py)
>
>
> On Tue, Jun 9, 2009 at 3:10 PM, Alexey Nezhdanov wrote:
>
>> Results of second set of test runs:
>>
>> r87545.523ms
>> r88220.614ms
>> inits v.1 18.464ms
>> inits v.2 18.677ms
>> inits1+lazyT15.377ms
>> inits2+lazyT15.280ms
>>
>> Observed noise was 0.502ms for slowest execution (90:1
>> signal-to-noise) and 0.889ms for fastest execution (16:1
>> signal-to-noise).
>> Given than, I think it is safe enough to say that
>> "this particular application on this particular hardware/OS
>> combination was sped up 2.88...3.31 times".
>>
>> As in last case - for completeness I'm attaching full console output
>>
>> I think I know what I'll do. Currently I do not want to release my
>> project as any kind of open-source. But I can
>> 1) Strip out any code I do not use in this perfomance testing
>> 2) rename all sensitive fieldnames/variables to something like field1,
>> field2, var3, var4.
>> Release resulting package. It would be useless as it is, but it will
>> have same perfomance parameters so could be used for testing.
>>
>> On Tue, Jun 9, 2009 at 1:35 PM, Alexey Nezhdanov
>> wrote:
>> > Wrote 'optimised inits v2' patch. Tested everything.
>> >
>> > Once again I changed my testing pattern a bit.
>> >
>> > 1) I dumped my own timing tool in favor of much more standard
>> > 'ab' (apache benchmark).
>> >
>> > 2) I decidedly will publish all these results in two main.
>> > I just finished first round of testing. Here are results (reported
>> > values are average request time in milliseconds. If anyone interested
>> > - I'm attaching a full output from my console).
>> > I will do all that testing again and publish same results again so
>> > that the amount of noise in all this will be public and clearly
>> > visible (here I'm fighting with my own temptation to rerun the tests
>> > if timings seem to be 'too away' to me).
>> >
>> > First four lines share my app's db.py. Last two require a modified
>> > db.py, but modification is cosmetical - each define_table is put into
>> > separate function and I do db.tablename=create_table_tablename after
>> > each function. Model init time was not measured.
>> >
>> > No more words. Dry figures only. Each line represents result of 10k
>> requests.
>> > r87546.025ms
>> > r88221.048ms
>> > inits v.1 18.918ms
>> > inits v.2 18.122ms
>> > inits1+lazyT16.032ms
>> > inits2+lazyT14.391ms
>> >
>> > P.S. I'm surprised about last line... Noise?
>> >
>> > On Tue, Jun 9, 2009 at 11:12 AM, Alexey Nezhdanov
>> wrote:
>> >> new testing:
>> >>
>> >> SERVER KERNEL
>> >> --prints
>> >> r8750.04898
>> >> r822ini 0.03070 1.60x
>> >> --silent
>> >> r8750.04914
>> >> r822ini 0.03049 1.61x
>> >>
>> >> So I get much more consistent results on this hardware.
>> >> While this is obviously not the best perfomance (my weaker box,
>> >> with less RAM, troubled with video output 1280x1024,
>> >> software 90deg rotation - performs BETTER), that does not matter.
>> >> What does - is that I can now be sure that these results are
>> >> noise-free so I can safely compare timings from various patches.
>> >> Proceeding with writing 'inits v2'.
>> >>
>> >>
>> >>
[web2py:23680] Re: web2py perfomance
Wanted to attach test data as well but automatically clicked 'send' and
oops... it gone.
Here are couple of test runs:
1.63.544.572
On Tue, Jun 9, 2009 at 4:14 PM, Alexey Nezhdanov wrote:
> Finally, I started migration (before I was just slapping sql.py, checked
> out of the trunk over my ancient 1.61.4 install).
> Migration is not easy, I've been bending source for my needs so now I have
> to port all this to 1.63.5. Massimo, do you have any objections to this
> patch? I proposed it some time ago, but you were too busy at the time.
> The purpose is - if we want custom forms instead of SQLFORM then we have to
> write our own view.html. But on the other hand - that is double work in the
> regard that we need to write down name='nnn' /> each time while we already generated all this in controller.
> I'd really like to be able to write in my controller:
>
> {{=form.custom.label.my_input}}
> {{=form.custom.widget.my_input}}
>
> See attached patch.
>
>
> On Tue, Jun 9, 2009 at 3:14 PM, Alexey Nezhdanov wrote:
>
>> Forgot about patches. Here they are.
>> 'optimised inits v.2'
>> 'lazy tables' (this one is modified to be applicable to trunk version of
>> sql.py)
>>
>>
>> On Tue, Jun 9, 2009 at 3:10 PM, Alexey Nezhdanov wrote:
>>
>>> Results of second set of test runs:
>>>
>>> r87545.523ms
>>> r88220.614ms
>>> inits v.1 18.464ms
>>> inits v.2 18.677ms
>>> inits1+lazyT15.377ms
>>> inits2+lazyT15.280ms
>>>
>>> Observed noise was 0.502ms for slowest execution (90:1
>>> signal-to-noise) and 0.889ms for fastest execution (16:1
>>> signal-to-noise).
>>> Given than, I think it is safe enough to say that
>>> "this particular application on this particular hardware/OS
>>> combination was sped up 2.88...3.31 times".
>>>
>>> As in last case - for completeness I'm attaching full console output
>>>
>>> I think I know what I'll do. Currently I do not want to release my
>>> project as any kind of open-source. But I can
>>> 1) Strip out any code I do not use in this perfomance testing
>>> 2) rename all sensitive fieldnames/variables to something like field1,
>>> field2, var3, var4.
>>> Release resulting package. It would be useless as it is, but it will
>>> have same perfomance parameters so could be used for testing.
>>>
>>> On Tue, Jun 9, 2009 at 1:35 PM, Alexey Nezhdanov
>>> wrote:
>>> > Wrote 'optimised inits v2' patch. Tested everything.
>>> >
>>> > Once again I changed my testing pattern a bit.
>>> >
>>> > 1) I dumped my own timing tool in favor of much more standard
>>> > 'ab' (apache benchmark).
>>> >
>>> > 2) I decidedly will publish all these results in two main.
>>> > I just finished first round of testing. Here are results (reported
>>> > values are average request time in milliseconds. If anyone interested
>>> > - I'm attaching a full output from my console).
>>> > I will do all that testing again and publish same results again so
>>> > that the amount of noise in all this will be public and clearly
>>> > visible (here I'm fighting with my own temptation to rerun the tests
>>> > if timings seem to be 'too away' to me).
>>> >
>>> > First four lines share my app's db.py. Last two require a modified
>>> > db.py, but modification is cosmetical - each define_table is put into
>>> > separate function and I do db.tablename=create_table_tablename after
>>> > each function. Model init time was not measured.
>>> >
>>> > No more words. Dry figures only. Each line represents result of 10k
>>> requests.
>>> > r87546.025ms
>>> > r88221.048ms
>>> > inits v.1 18.918ms
>>> > inits v.2 18.122ms
>>> > inits1+lazyT16.032ms
>>> > inits2+lazyT14.391ms
>>> >
>>> > P.S. I'm surprised about last line... Noise?
>>> >
>>> > On Tue, Jun 9, 2009 at 11:12 AM, Alexey Nezhdanov
>>> wrote:
>>> >> new testing:
>>> >>
>>> >> SERVER KERNEL
>>> >> --prints
>>> >> r8750.04898
>>> >> r822ini 0.03070 1.60x
>>> >> --silent
>>> >> r8750.04914
>>> >> r82
[web2py:23682] Re: web2py perfomance
:( using web interface is inconvenient. Ignore last mail, resending:
-
Wanted to attach test data as well but automatically clicked 'send' and
oops... it gone.
Here are couple of test runs:
1.63.5 44.572ms
1.63.5 44.898ms
detected noise 0.326ms (136:1 signal-to-noise).
full 'ab' output attached
On Tue, Jun 9, 2009 at 4:25 PM, Alexey Nezhdanov wrote:
> Wanted to attach test data as well but automatically clicked 'send' and
> oops... it gone.
> Here are couple of test runs:
> 1.63.544.572
>
> On Tue, Jun 9, 2009 at 4:14 PM, Alexey Nezhdanov wrote:
>
>> Finally, I started migration (before I was just slapping sql.py, checked
>> out of the trunk over my ancient 1.61.4 install).
>> Migration is not easy, I've been bending source for my needs so now I have
>> to port all this to 1.63.5. Massimo, do you have any objections to this
>> patch? I proposed it some time ago, but you were too busy at the time.
>> The purpose is - if we want custom forms instead of SQLFORM then we have
>> to write our own view.html. But on the other hand - that is double work in
>> the regard that we need to write down > id='zzz' name='nnn' /> each time while we already generated all this in
>> controller.
>> I'd really like to be able to write in my controller:
>>
>> {{=form.custom.label.my_input}}
>> {{=form.custom.widget.my_input}}
>>
>> See attached patch.
>>
>>
>> On Tue, Jun 9, 2009 at 3:14 PM, Alexey Nezhdanov wrote:
>>
>>> Forgot about patches. Here they are.
>>> 'optimised inits v.2'
>>> 'lazy tables' (this one is modified to be applicable to trunk version of
>>> sql.py)
>>>
>>>
>>> On Tue, Jun 9, 2009 at 3:10 PM, Alexey Nezhdanov wrote:
>>>
>>>> Results of second set of test runs:
>>>>
>>>> r87545.523ms
>>>> r88220.614ms
>>>> inits v.1 18.464ms
>>>> inits v.2 18.677ms
>>>> inits1+lazyT15.377ms
>>>> inits2+lazyT15.280ms
>>>>
>>>> Observed noise was 0.502ms for slowest execution (90:1
>>>> signal-to-noise) and 0.889ms for fastest execution (16:1
>>>> signal-to-noise).
>>>> Given than, I think it is safe enough to say that
>>>> "this particular application on this particular hardware/OS
>>>> combination was sped up 2.88...3.31 times".
>>>>
>>>> As in last case - for completeness I'm attaching full console output
>>>>
>>>> I think I know what I'll do. Currently I do not want to release my
>>>> project as any kind of open-source. But I can
>>>> 1) Strip out any code I do not use in this perfomance testing
>>>> 2) rename all sensitive fieldnames/variables to something like field1,
>>>> field2, var3, var4.
>>>> Release resulting package. It would be useless as it is, but it will
>>>> have same perfomance parameters so could be used for testing.
>>>>
>>>> On Tue, Jun 9, 2009 at 1:35 PM, Alexey Nezhdanov
>>>> wrote:
>>>> > Wrote 'optimised inits v2' patch. Tested everything.
>>>> >
>>>> > Once again I changed my testing pattern a bit.
>>>> >
>>>> > 1) I dumped my own timing tool in favor of much more standard
>>>> > 'ab' (apache benchmark).
>>>> >
>>>> > 2) I decidedly will publish all these results in two main.
>>>> > I just finished first round of testing. Here are results (reported
>>>> > values are average request time in milliseconds. If anyone interested
>>>> > - I'm attaching a full output from my console).
>>>> > I will do all that testing again and publish same results again so
>>>> > that the amount of noise in all this will be public and clearly
>>>> > visible (here I'm fighting with my own temptation to rerun the tests
>>>> > if timings seem to be 'too away' to me).
>>>> >
>>>> > First four lines share my app's db.py. Last two require a modified
>>>> > db.py, but modification is cosmetical - each define_table is put into
>>>> > separate function and I do db.tablename=create_table_tablename after
>>>> > each function. Model init time was not measured.
>>>> >
>>>> > No more words. Dry figures only. Each line represen
[web2py:23810] Re: web2py perfomance
I get these results (best timing):
try-variant: 10.24s
regex: 9.58s
So on my laptop try:except: function loses about 5% to regex - probably it
depends on hardware/OS.
Still there is a serious reason why to stick with regexp. try: int(x) is not
really checking for an int - it checks for a _number_. So floats will pass
that test.
And as Massimo pointed out - we don't really check for an int - we check for
valid id. Function should be remamed, yes.
Also there is another problem with your testing - you get too much noise this
way. 5-10 ms _per_loop_ is too slow. I get about 9.993 seconds for 40M (!)
operations effectively - 0.25usec per loop - about 10k times faster than
yours. You can check it yourself, my script is attached.
On Tuesday 09 June 2009 23:43:05 AchipA wrote:
> Regardless of speed, if you *really* want to check for integers,
> you'll need a far more complex regex (how do you handle sign ? int
> size ?). int() does all that for you without too much effort.
>
> But, for the record, to satisfy my own curiosity, too - one quickie
> profiling coming up (python 2.5.2, x86_64 linux):
>
> blinky:~# python -m timeit "s=[str(i) for i in range(1)]"
> 100 loops, best of 3: 4.64 msec per loop
> blinky:~# python -m timeit "s=[str(i) for i in range(1)]" "def
> is_integer(i):" " try:" "int(i)" "return True" " except:
> return False" "for i in s:" " is_integer(i)"
> 100 loops, best of 3: 9.32 msec per loop
> blinky:~# python -m timeit "s=[str(i) for i in range(1)]" "import
> re" "integer_pat=re.compile('[0-9]+$')" "is_integer=integer_pat.match"
> "for i in s:" " is_integer(i)"
> 100 loops, best of 3: 10.5 msec per loop
>
> So, technically it's 4.68 vs 5.86msec which is a ~20% speed increase
> over the simplified regex even with the user function overhead. defs
> might be slow, but regexes are no rockets either (C or not).
>
> On Jun 9, 8:05 pm, Iceberg wrote:
> > I don't know but you'd better do some profiling if you really want to
> > find out. IMHO, try...except might be fast, but wrapping it inside a
> > user-defined function could be another story, because defining a
> > function is expensive in python, so we shall call native function
> > (implemented by C) whenever possible.
> >
> > On Jun10, 0:48am, mdipierro wrote:
> > > good point. Anyway, the function is no longer called as often as
> > > Alexey originally pointed out, so it would make a negligible
> > > difference. I will change it though.
> > >
> > > Massimo
> > >
> > > On Jun 9, 11:24 am, AchipA wrote:
> > > > Any particular reason not doing is_integer via a 'try: int(i) except:
> > > > return False' statement ? It should be faster than regexes.
> > > >
> > > > On Jun 7, 1:49 pm, Iceberg wrote:
> > > > > On Jun7, 6:35pm, Alexey Nezhdanov wrote:
> > > > > > > 2) is_integer is a fast call, but with 1.1k (!) calls ...
> > > > > >
> > > > > > Replaced it with my own version:
> > > > > > integer_pat=re.compile('[0-9]+$')
> > > > > > is_integer=lambda x: integer_pat.match(x)
> > > > > > it's about 2.3 times faster. C version would be even better.
> > > > >
> > > > > If so, perhaps this is even better:
> > > > >
> > > > > integer_pat=re.compile('[0-9]+$')
> > > > > is_integer=integer_pat.match
> > > > >
> > > > > Because lambda is considered as much slower than built-in
> > > > > functions, AFAIK.
>
>
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
a.py
Description: application/python
b.py
Description: application/python
sh
Description: application/shellscript
[web2py:23811] Re: web2py perfomance
On Wednesday 10 June 2009 09:01:04 Alexey Nezhdanov wrote: > about 10k times faster than yours. Ah... no. I never used timit module before. You get values of the same order. -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:23818] Re: web2py perfomance
On Wednesday 10 June 2009 12:40:44 AchipA wrote:
> On Jun 10, 7:01 am, Alexey Nezhdanov wrote:
> > I get these results (best timing):
> > try-variant: 10.24s
> > regex: 9.58s
> > So on my laptop try:except: function loses about 5% to regex - probably
> > it depends on hardware/OS.
>
> Interesting, which python/platform are you using ?
Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Ubuntu 8.04 installed on ASUS A6Q00Km (Turion 64 1.6GHz, SiS motherboard, 1.5G
RAM).
> > And as Massimo pointed out - we don't really check for an int - we check
> > for valid id. Function should be remamed, yes.
>
> Right, it's more like is_convertible_to_int. Now that we are more
> clear about what we're looking for, it sounds exactly like
> string.isdigit() to me.
yes. I thought that there should be such function but failed to find it. Thank
you for pointing this out.
> b...@black:/tmp$ python -m timeit "s=[str(i) for i in range(1)]"
> "import re" "integer_pat=re.compile('[+-]?[0-9]{1,15}$')"
> "is_integer=integer_pat.match"
> 100 loops, best of 3: 3.64 msec per loop
Hmm? No loop...
> b...@black:/tmp$ python -m timeit "s=[str(i) for i in range(1)]"
> "import re" "integer_pat=re.compile('[+-]?[0-9]{1,15}$')"
> "is_integer=integer_pat.match" "for i in s:" " is_integer(i)"
> 100 loops, best of 3: 9.87 msec per loop
roughly the same time here.
> b...@black:/tmp$ python -m timeit "s=[str(i) for i in range(1)]"
> "for i in s:" " i.isdigit()"
> 100 loops, best of 3: 4.63 msec per loop
> Which gives a 6.2x (!) faster result.
I'm sorry, how do you get 6.2x?
9.87/ 4.63 =2.13 here...
And again - you still doing a lot of excess work. First, you populating the
list, then you iterate over it. You don't need to do either. Look at my
scripts attached.
Anyways, my testing results are as follows:
I get:
9.84s this time for regex-based 4M calls,
7.70s for function based on 'isdigit'
7.68s for lambda based on 'isdigit'
6.24s for direct 'isdigit' call.
So, Massimo, I think it worths replacing globally each
is_integer(str(x))
to
str(x).isdigit()
If you still prefer having is_integer function then here is it:
is_integer = lambda x: str(x).isdigit()
In this case please rename it to something like is_id.
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
a.py
Description: application/python
b.py
Description: application/python
e.py
Description: application/python
c.py
Description: application/python
sh
Description: application/shellscript
d.py
Description: application/python
[web2py:23821] Re: web2py perfomance
On Wednesday 10 June 2009 16:31:31 AchipA wrote:
> > > > So on my laptop try:except: function loses about 5% to regex -
> > > > probably it depends on hardware/OS.
> > >
> > > Interesting, which python/platform are you using ?
> >
> > Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52)
> > [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
>
> Hm, is it perhaps 32bit ? That could be related to int performance...
Yes. 64-bit OS was too buggy so after several attempts I gave up and continued
using 32bit.
> > > b...@black:/tmp$ python -m timeit "s=[str(i) for i in range(1)]"
> > > "import re" "integer_pat=re.compile('[+-]?[0-9]{1,15}$')"
> > > "is_integer=integer_pat.match"
> > > 100 loops, best of 3: 3.64 msec per loop
> >
> > Hmm? No loop...
>
> Timeit loops, it is looping over your code. So, the above means timeit
> ran the code 100 times.
>
> > I'm sorry, how do you get 6.2x?
> > 9.87/ 4.63 =2.13 here...
>
> Subtract the init (3.6msec) time from both. just wanted to avoid the
> extra cost of the import. However, even when doing it 'more right' I'm
> getting a significantly larger number (although unlike my firrst
> example this does not take into account the varying length of
> strings).
I see. I'm not sure if we are allowed to do that (substraction) though.
> blinky:~# python -m timeit -s "import re" -s "integer_pat=re.compile
> ('[0-9]+$')" -s "is_integer=integer_pat.match" "is_integer('123')"
> 100 loops, best of 3: 0.574 usec per loop
> blinky:~# python -m timeit "'123'.isdigit()"
> 1000 loops, best of 3: 0.141 usec per loop
>
> The problem of your methodology (timing a total run) is that you're
> averaging out results and polluting the results with load noise from
> the OS and other apps. Timeit (among other neat tricks) counts the
> timing of the *BEST* iterations. If one was slower than the other,
> that means that the other got interrupted somewhere (disk, app,
> multitaskting, etc).
>
> See http://docs.python.org/library/timeit.html , it's quite useful.
I was wondering during my previous testing if I should use only 'minimal'
time. Thank you for the link, will read.
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:23826] Re: web2py perfomance
Ok.
Basically I'd like you to:
1) apply 'inits2.diff' - it optimises SQLTable.__init__ and SQLField.__init__
2) remove is_integer altogether and replace it everywhere with
str(id).isdigit()
3) apply custom_widget.diff. This one is not perfomance related so please
regard it as usual feature request.
I do not know if these are applicable against latest trunk and I has something
to do tonight so I'll test all this either later today or perhaps tomorrow
morning.
On Wednesday 10 June 2009 19:36:40 mdipierro wrote:
> Alexey,
>
> I am lost with all the attachments. If there is something you want me
> to include please send me by email one patch that applies to the
> lastest trunk. I would not yet include lazy table evals just yet since
> I need to think more about it. I think there may be an easier way of
> doing but since I am rewriting the DAL anyway I think it is best to
> wait.
>
> Massimo
>
> On Jun 10, 7:48 am, Alexey Nezhdanov wrote:
> > On Wednesday 10 June 2009 16:31:31 AchipA wrote:> > > > So on my laptop
> > try:except: function loses about 5% to regex -
> >
> > > > > > probably it depends on hardware/OS.
> > > > >
> > > > > Interesting, which python/platform are you using ?
> > > >
> > > > Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52)
> > > > [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
> > >
> > > Hm, is it perhaps 32bit ? That could be related to int performance...
> >
> > Yes. 64-bit OS was too buggy so after several attempts I gave up and
> > continued using 32bit.
> >
> > > > > b...@black:/tmp$ python -m timeit "s=[str(i) for i in
> > > > > range(1)]" "import re"
> > > > > "integer_pat=re.compile('[+-]?[0-9]{1,15}$')"
> > > > > "is_integer=integer_pat.match"
> > > > > 100 loops, best of 3: 3.64 msec per loop
> > > >
> > > > Hmm? No loop...
> > >
> > > Timeit loops, it is looping over your code. So, the above means timeit
> > > ran the code 100 times.
> > >
> > > > I'm sorry, how do you get 6.2x?
> > > > 9.87/ 4.63 =2.13 here...
> > >
> > > Subtract the init (3.6msec) time from both. just wanted to avoid the
> > > extra cost of the import. However, even when doing it 'more right' I'm
> > > getting a significantly larger number (although unlike my firrst
> > > example this does not take into account the varying length of
> > > strings).
> >
> > I see. I'm not sure if we are allowed to do that (substraction) though.
> >
> > > blinky:~# python -m timeit -s "import re" -s "integer_pat=re.compile
> > > ('[0-9]+$')" -s "is_integer=integer_pat.match" "is_integer('123')"
> > > 100 loops, best of 3: 0.574 usec per loop
> > > blinky:~# python -m timeit "'123'.isdigit()"
> > > 1000 loops, best of 3: 0.141 usec per loop
> > >
> > > The problem of your methodology (timing a total run) is that you're
> > > averaging out results and polluting the results with load noise from
> > > the OS and other apps. Timeit (among other neat tricks) counts the
> > > timing of the *BEST* iterations. If one was slower than the other,
> > > that means that the other got interrupted somewhere (disk, app,
> > > multitaskting, etc).
> > >
> > > Seehttp://docs.python.org/library/timeit.html, it's quite useful.
> >
> > I was wondering during my previous testing if I should use only 'minimal'
> > time. Thank you for the link, will read.
> >
> > --
> > Sincerely yours
> > Alexey Nezhdanov
>
>
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:23837] Re: web2py perfomance
В сообщении от Wednesday 10 June 2009 22:45:02 mdipierro написал(а):
> Most of what you proposed is already in trunk (althought in s lightly
> different way). I added what was missing.
>
> Mind that it is not form.custom.widget.submit but form.custom.submit.
Ok, thank you.
Will have a look at the trunk tomorrow morning.
> Massimo
>
> On Jun 10, 11:10 am, Alexey Nezhdanov wrote:
> > Ok.
> >
> > Basically I'd like you to:
> > 1) apply 'inits2.diff' - it optimises SQLTable.__init__ and
> > SQLField.__init__ 2) remove is_integer altogether and replace it
> > everywhere with
> > str(id).isdigit()
> > 3) apply custom_widget.diff. This one is not perfomance related so please
> > regard it as usual feature request.
> >
> > I do not know if these are applicable against latest trunk and I has
> > something to do tonight so I'll test all this either later today or
> > perhaps tomorrow morning.
> >
> > On Wednesday 10 June 2009 19:36:40 mdipierro wrote:
> > > Alexey,
> > >
> > > I am lost with all the attachments. If there is something you want me
> > > to include please send me by email one patch that applies to the
> > > lastest trunk. I would not yet include lazy table evals just yet since
> > > I need to think more about it. I think there may be an easier way of
> > > doing but since I am rewriting the DAL anyway I think it is best to
> > > wait.
> > >
> > > Massimo
> > >
> > > On Jun 10, 7:48 am, Alexey Nezhdanov wrote:
> > > > On Wednesday 10 June 2009 16:31:31 AchipA wrote:> > > > So on my
> > > > laptop try:except: function loses about 5% to regex -
> > > >
> > > > > > > > probably it depends on hardware/OS.
> > > > > > >
> > > > > > > Interesting, which python/platform are you using ?
> > > > > >
> > > > > > Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52)
> > > > > > [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
> > > > >
> > > > > Hm, is it perhaps 32bit ? That could be related to int
> > > > > performance...
> > > >
> > > > Yes. 64-bit OS was too buggy so after several attempts I gave up and
> > > > continued using 32bit.
> > > >
> > > > > > > b...@black:/tmp$ python -m timeit "s=[str(i) for i in
> > > > > > > range(1)]" "import re"
> > > > > > > "integer_pat=re.compile('[+-]?[0-9]{1,15}$')"
> > > > > > > "is_integer=integer_pat.match"
> > > > > > > 100 loops, best of 3: 3.64 msec per loop
> > > > > >
> > > > > > Hmm? No loop...
> > > > >
> > > > > Timeit loops, it is looping over your code. So, the above means
> > > > > timeit ran the code 100 times.
> > > > >
> > > > > > I'm sorry, how do you get 6.2x?
> > > > > > 9.87/ 4.63 =2.13 here...
> > > > >
> > > > > Subtract the init (3.6msec) time from both. just wanted to avoid
> > > > > the extra cost of the import. However, even when doing it 'more
> > > > > right' I'm getting a significantly larger number (although unlike
> > > > > my firrst example this does not take into account the varying
> > > > > length of strings).
> > > >
> > > > I see. I'm not sure if we are allowed to do that (substraction)
> > > > though.
> > > >
> > > > > blinky:~# python -m timeit -s "import re" -s
> > > > > "integer_pat=re.compile ('[0-9]+$')" -s
> > > > > "is_integer=integer_pat.match" "is_integer('123')" 100 loops,
> > > > > best of 3: 0.574 usec per loop
> > > > > blinky:~# python -m timeit "'123'.isdigit()"
> > > > > 1000 loops, best of 3: 0.141 usec per loop
> > > > >
> > > > > The problem of your methodology (timing a total run) is that you're
> > > > > averaging out results and polluting the results with load noise
> > > > > from the OS and other apps. Timeit (among other neat tricks) counts
> > > > > the timing of the *BEST* iterations. If one was slower than the
> > > > > other, that means that the other got interrupted somewhere (disk,
> > > > > app, multitaskting, etc).
> > > > >
> > > > > Seehttp://docs.python.org/library/timeit.html, it's quite useful.
> > > >
> > > > I was wondering during my previous testing if I should use only
> > > > 'minimal' time. Thank you for the link, will read.
> > > >
> > > > --
> > > > Sincerely yours
> > > > Alexey Nezhdanov
> >
> > --
> > Sincerely yours
> > Alexey Nezhdanov
>
>
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:23902] not authorized is 401, not 400
Hello.
It is more proper to use 401 status code than 400 for 'not authorized' case.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.2
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
--- tools.py 2009-06-11 17:10:31.216533525 +0400
+++ tools.401.py 2009-06-11 17:10:41.856530678 +0400
@@ -1136,17 +1136,17 @@
session = self.environment.session
auth = session.auth
if not self.is_logged_in():
-raise HTTP(400, "Not Authorized")
+raise HTTP(401, "Not Authorized")
if user_id == DEFAULT and self.environment.request.args:
user_id = self.environment.request.args[1]
if user_id and user_id != self.user.id and user_id != '0':
if not self.has_permission('impersonate',
self.settings.table_user_name,
user_id):
-raise HTTP(400, "Not Authorized")
+raise HTTP(401, "Not Authorized")
user = self.settings.table_user[request.args[1]]
if not user:
-raise HTTP(400, "Not Authorized")
+raise HTTP(401, "Not Authorized")
auth.impersonator = cPickle.dumps(session)
auth.user.update(self.settings.table_user._filter_fields(user, True))
self.user = auth.user
@@ -2032,7 +2032,7 @@
request = self.environment['request']
if len(request.args) < 1:
-raise HTTP(400, "Not Authorized")
+raise HTTP(401, "Not Authorized")
arg0 = request.args[0]
if arg0 == 'run':
return self.serve_run(request.args[1:])
[web2py:23903] Re: not authorized is 401, not 400
:( I had to read code closer. There are actually three different keys so three
different codes required. Modified patch attached.
On Thursday 11 June 2009 17:15:23 Alexey Nezhdanov wrote:
> Hello.
>
> It is more proper to use 401 status code than 400 for 'not authorized'
> case.
>
> http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.2
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
--- tools.py 2009-06-11 17:10:31.216533525 +0400
+++ tools.401.py 2009-06-11 17:16:54.300531107 +0400
@@ -1136,17 +1136,17 @@
session = self.environment.session
auth = session.auth
if not self.is_logged_in():
-raise HTTP(400, "Not Authorized")
+raise HTTP(401, "Not Authorized")
if user_id == DEFAULT and self.environment.request.args:
user_id = self.environment.request.args[1]
if user_id and user_id != self.user.id and user_id != '0':
if not self.has_permission('impersonate',
self.settings.table_user_name,
user_id):
-raise HTTP(400, "Not Authorized")
+raise HTTP(403, "Forbidden")
user = self.settings.table_user[request.args[1]]
if not user:
-raise HTTP(400, "Not Authorized")
+raise HTTP(401, "Not Authorized")
auth.impersonator = cPickle.dumps(session)
auth.user.update(self.settings.table_user._filter_fields(user, True))
self.user = auth.user
@@ -2032,7 +2032,7 @@
request = self.environment['request']
if len(request.args) < 1:
-raise HTTP(400, "Not Authorized")
+raise HTTP(400, "Bad request")
arg0 = request.args[0]
if arg0 == 'run':
return self.serve_run(request.args[1:])
[web2py:23937] Re: web2py 1.64.0 posted
On Thursday 11 June 2009 20:16:54 mdipierro wrote: > Ouch! I apologize to Alexey. NP. I saw that mistake but decided to be modest :) > On Jun 11, 10:48 am, Hans Donner wrote: > > Error codes are from our performance man! > > > > On Thu, Jun 11, 2009 at 5:27 PM, mdipierro wrote: > > > Please check it out! > > > > > > - Models are much faster ~ 2.5x times (thanks Alexey) > > > - fully works on Jython without tweaks (sqlite and postgresql with > > > zxjdbc) > > > - custom forms and better error codes (thanks Hans) > > > - better LDAP support (thanks Fran and Mr. Freeze) > > > > > > Massimo > > -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:23973] Re: YACI, or how to speed up model loading
or it's dependencies - see below.
> >
> > > Let's suppose we change that.
> > > Where do you keep common model code?
> > > common = used by all ctls, except those with a name
> >
> > Now, for the kicker - in the model we could do an equivalent of
> > {{extend 'layout.html'}} (=sort of import ?) to include any
> > dependencies. This would also pull in any 'common' models (just like
> > we do with layout.html). Note the subtle difference - I don't want a
> > single EXTRA model to be included - I want to avoid loading
> > unnecessary models.
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:23978] Re: Self-reference table error after upgrading to the latest svn version of web2py
You can reference the table itself like this.
Try:
db.define_table('table_example',
SQLField('some_int_field','integer'),
SQLField('parent_id_field', 'integer'))
and then treat parent_id_field just as it's name suggests
On Friday 12 June 2009 10:27:09 cpt1002 wrote:
> Hi,
>
> Has anyone gotten an error using the "reference table" call with
> similar model code:
>
> db.define_table('table_example',
> SQLField('some_int_field','integer'),
> SQLField('parent_id_field', 'reference table_example'))
>
>
> The error is:
>
> Traceback (most recent call last):
> File "/home/web2py/gluon/restricted.py", line 107, in restricted
> exec ccode in environment
> File "/home/web2py/applications/cpt1002/models/db.py", line 82, in
>
> SQLField('parent','reference category'))
> File "/home/web2py/gluon/sql.py", line 953, in define_table
> else:
> File "/home/web2py/gluon/sql.py", line 1143, in __init__
> new_fields = [ SQLField('id', 'id') ]
> SyntaxError: SQLTable: table does not exist
>
>
>
> This worked before until I updated just this evening to Version 1.64.1
>
> Any help would be appreciated. Thanks!
>
>
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:23982] Re: web2py 1.64.1 and Auth funnies
On Friday 12 June 2009 14:25:29 Joe Barnhart wrote: > I upgraded my development site to 1.64.1 but I find I'm not > "authorized" to see most of my site now. I think there may have been > some changes to the way auth works. Hello Joe. I can't reproduce your problem on my site (which uses a slightly modified Auth module). Does it denies access each time you request the page? I.e. if you see 'access denied' and then just refresh page - does it show same message again? There should be a way to reproduce the error consistently then it would be easier to fix. May be you can intercept actual bytestream 'on the wire' with the tool like tcpflow/ethereal? > For example, if I select a link that goes to a controller function > that has the decorator @auth.reqires_login() it sometimes displays the > "access denied" message instead of the form I'm expecting. But not > all of the functions work this way. Some let me operate normally. (I > am properly logged in during these operations.) > > I'll play around more tomorrow evening and see if I can see any rhyme > or reason for the rejections. -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:24058] Re: web2py and Debian
On Friday 12 June 2009 20:49:28 mdipierro wrote: > I would go with bare web2py unless somebody wants to work on improving > and clening up the sysadmin appliance. Umm... You propose to drop /admin/ from debian web2py package??? > Thanks for your help -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:24063] Proposal for auth
I'm writing unit tests for my app and come across this problem:
if I try to access the page which has @auth.requires_login()
I get error 303 - i.e. redirect to the page with login/password form.
While this works visually for browsers, it is actually wrong for testing AND
for search engines. Status codes are important for robots and here status is
set incorrectly (should be 401).
So why we not set this to 401 and make redirect with other means
(javascript/meta tags)? We know that pages that are larger than certain size
are displayed ok in IE so this should not be a problem.
Here is sample page with redirect:
"""
<!--
window.location='%(nexturl)s';
// -->
"""%{'nexturl':URL(r=request,c='auth',f='login')}
If this is a good idea - I'll write a patch.
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:24097] Re: Proposal for auth
On Saturday 13 June 2009 16:58:33 mdipierro wrote:
> I will take this patch!
Here it is, attached. Tested with latest stable, then ported to trunk version
and tested again.
However there is one problem with it. On error code 401 browser actually
_renders_ the page causing it to be blanked and redrawn as opposed to 303
which just waits a bit, then instantly replaces page content.
I do not know if there is a way to suppress that page blanking.
Otherwise it seems to be working as intended.
> On Jun 13, 6:19 am, Alexey Nezhdanov wrote:
> > I'm writing unit tests for my app and come across this problem:
> > if I try to access the page which has @auth.requires_login()
> > I get error 303 - i.e. redirect to the page with login/password form.
> > While this works visually for browsers, it is actually wrong for testing
> > AND for search engines. Status codes are important for robots and here
> > status is set incorrectly (should be 401).
> > So why we not set this to 401 and make redirect with other means
> > (javascript/meta tags)? We know that pages that are larger than certain
> > size are displayed ok in IE so this should not be a problem.
> > Here is sample page with redirect:
> >
> > """
> >
> >
> >
> > <!--
> > window.location='%(nexturl)s';
> > // -->
> >
> >
> >
> > """%{'nexturl':URL(r=request,c='auth',f='login')}
> >
> > If this is a good idea - I'll write a patch.
> >
> > --
> > Sincerely yours
> > Alexey Nezhdanov
>
>
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
diff -ur gluon/http.py gluon-401/http.py
--- gluon/http.py 2009-06-13 19:55:47.0 +0400
+++ gluon-401/http.py 2009-06-13 19:56:19.109469004 +0400
@@ -96,8 +96,26 @@
return [body]
def redirect(location, how=303):
-location = location.replace('\r', '%0D').replace('\n', '%0A')
-raise HTTP(how, 'You are being redirected here'
+if 300 <= how < 399:
+location = location.replace('\r', '%0D').replace('\n', '%0A')
+raise HTTP(how, 'You are being redirected here'
% location, Location=location)
+else:
+raise HTTP(how, """
+
+
+
+ <!--
+ window.location='%(location)s';
+ // -->
+
+
+You are being redirected here
+
+
+""" % {'location' : location}) # location could be as short as '/'
diff -ur gluon/tools.py gluon-401/tools.py
--- gluon/tools.py 2009-06-13 19:55:55.0 +0400
+++ gluon-401/tools.py 2009-06-13 20:00:31.217467547 +0400
@@ -718,7 +718,7 @@
if not user:
## invalid login
session.flash = self.messages.invalid_login
-redirect(self.url(args=request.args))
+redirect(self.url(args=request.args), how = 401)
user = Storage(table_user._filter_fields(user, id=True))
session.auth = Storage(user=user, last_visit=request.now,
@@ -1034,7 +1034,7 @@
"""
if not self.is_logged_in():
-redirect(self.settings.login_url)
+redirect(self.settings.login_url, how = 401)
db = self.db
user = self.settings.table_user
usern = self.settings.table_user_name
@@ -1092,7 +1092,7 @@
"""
if not self.is_logged_in():
-redirect(self.settings.login_url)
+redirect(self.settings.login_url, how = 401)
password = self.settings.password_field
self.settings.table_user[password].writable = False
request = self.environment.request
@@ -1173,7 +1173,7 @@
"""
if not self.is_logged_in():
-redirect(self.settings.login_url)
+redirect(self.settings.login_url, how = 401)
memberships = self.db(self.settings.table_membership.user_id
== self.user.id).select()
table = TABLE()
@@ -1207,7 +1207,7 @@
if not self.basic() and not self.is_logged_in():
args = self.env
[web2py:24143] Re: Proposal for auth
Ok, Massimo, do you still think this is a good idea?
Should I make this optional for auth module with default set to [SE-friendly]
401?
On Saturday 13 June 2009 20:08:36 Alexey Nezhdanov wrote:
> On Saturday 13 June 2009 16:58:33 mdipierro wrote:
> > I will take this patch!
>
> Here it is, attached. Tested with latest stable, then ported to trunk
> version and tested again.
> However there is one problem with it. On error code 401 browser actually
> _renders_ the page causing it to be blanked and redrawn as opposed to 303
> which just waits a bit, then instantly replaces page content.
> I do not know if there is a way to suppress that page blanking.
> Otherwise it seems to be working as intended.
>
> > On Jun 13, 6:19 am, Alexey Nezhdanov wrote:
> > > I'm writing unit tests for my app and come across this problem:
> > > if I try to access the page which has @auth.requires_login()
> > > I get error 303 - i.e. redirect to the page with login/password form.
> > > While this works visually for browsers, it is actually wrong for
> > > testing AND for search engines. Status codes are important for robots
> > > and here status is set incorrectly (should be 401).
> > > So why we not set this to 401 and make redirect with other means
> > > (javascript/meta tags)? We know that pages that are larger than certain
> > > size are displayed ok in IE so this should not be a problem.
> > > Here is sample page with redirect:
> > >
> > > """
> > >
> > >
> > >
> > > <!--
> > > window.location='%(nexturl)s';
> > > // -->
> > >
> > >
> > >
> > > """%{'nexturl':URL(r=request,c='auth',f='login')}
> > >
> > > If this is a good idea - I'll write a patch.
> > >
> > > --
> > > Sincerely yours
> > > Alexey Nezhdanov
> >
> >
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:24185] Re: Proposal for auth
No, I didn't meant IE. I meant that since redirect page doesn't have anybody
for a user AND it is not a 301/303 page - it actually rendered.
That is visible at least in Firefox.
Also if I just add some random content (like 'you are being redirected') it
will be visible too (site -> 'you are being redirected' -> site again). If
you think that it is a good idea - I'll do.
On Monday 15 June 2009 01:05:25 mdipierro wrote:
> You can replace the page blanking by adding some content longer than
> 512 chars
>
> On Jun 13, 11:08 am, Alexey Nezhdanov wrote:
> > On Saturday 13 June 2009 16:58:33 mdipierro wrote:> I will take this
> > patch!
> >
> > Here it is, attached. Tested with latest stable, then ported to trunk
> > version and tested again.
> > However there is one problem with it. On error code 401 browser actually
> > _renders_ the page causing it to be blanked and redrawn as opposed to 303
> > which just waits a bit, then instantly replaces page content.
> > I do not know if there is a way to suppress that page blanking.
> > Otherwise it seems to be working as intended.
> >
> > > On Jun 13, 6:19 am, Alexey Nezhdanov wrote:
> > > > I'm writing unit tests for my app and come across this problem:
> > > > if I try to access the page which has @auth.requires_login()
> > > > I get error 303 - i.e. redirect to the page with login/password form.
> > > > While this works visually for browsers, it is actually wrong for
> > > > testing AND for search engines. Status codes are important for robots
> > > > and here status is set incorrectly (should be 401).
> > > > So why we not set this to 401 and make redirect with other means
> > > > (javascript/meta tags)? We know that pages that are larger than
> > > > certain size are displayed ok in IE so this should not be a problem.
> > > > Here is sample page with redirect:
> > > >
> > > > """
> > > >
> > > >
> > > >
> > > > <!--
> > > > window.location='%(nexturl)s';
> > > > // -->
> > > >
> > > >
> > > >
> > > > """%{'nexturl':URL(r=request,c='auth',f='login')}
> > > >
> > > > If this is a good idea - I'll write a patch.
> > > >
> > > > --
> > > > Sincerely yours
> > > > Alexey Nezhdanov
> >
> > --
> > Sincerely yours
> > Alexey Nezhdanov
> >
> > noauth-401-403.diff
> > 5KViewDownload
>
>
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:24193] Re: Proposal for auth
Sure. For a start I'll provide the list that I have.
Pros:
1) Search-engine friendly. SEs see proper '401 Not authorised' response
instead of '303 Found' when trying to access login-only pages
2) Unit tests - binding to 401/403 codes is more explicit. Receiving them
gives better understanding on what happens.
3) RFC compliance
Cons:
1) Browser must render the page to trigger the redirect. So at least in some
browsers (for me it is Fx 2.0) this causes a momentary page clear before new
page is drawn (old page -> blank -> new page).
So far I can see only one workaround: put there some content so that page
will not look so blank - in this case only 'content' part of the scren
should clear leaving menus/logos/other decorators static. That can be done
only in application I think.
On Mon, Jun 15, 2009 at 9:52 AM, mdipierro wrote:
>
> I would like to hear more opinions about this before changing it.
> Pros? Cons?
>
> Massimo
>
> On Jun 14, 10:28 pm, Alexey Nezhdanov wrote:
> > No, I didn't meant IE. I meant that since redirect page doesn't have
> anybody
> > for a user AND it is not a 301/303 page - it actually rendered.
> > That is visible at least in Firefox.
> > Also if I just add some random content (like 'you are being redirected')
> it
> > will be visible too (site -> 'you are being redirected' -> site again).
> If
> > you think that it is a good idea - I'll do.
> >
> > On Monday 15 June 2009 01:05:25 mdipierro wrote:
> >
> >
> >
> > > You can replace the page blanking by adding some content longer than
> > > 512 chars
> >
> > > On Jun 13, 11:08 am, Alexey Nezhdanov wrote:
> > > > On Saturday 13 June 2009 16:58:33 mdipierro wrote:> I will take this
> > > > patch!
> >
> > > > Here it is, attached. Tested with latest stable, then ported to trunk
> > > > version and tested again.
> > > > However there is one problem with it. On error code 401 browser
> actually
> > > > _renders_ the page causing it to be blanked and redrawn as opposed to
> 303
> > > > which just waits a bit, then instantly replaces page content.
> > > > I do not know if there is a way to suppress that page blanking.
> > > > Otherwise it seems to be working as intended.
> >
> > > > > On Jun 13, 6:19 am, Alexey Nezhdanov wrote:
> > > > > > I'm writing unit tests for my app and come across this problem:
> > > > > > if I try to access the page which has @auth.requires_login()
> > > > > > I get error 303 - i.e. redirect to the page with login/password
> form.
> > > > > > While this works visually for browsers, it is actually wrong for
> > > > > > testing AND for search engines. Status codes are important for
> robots
> > > > > > and here status is set incorrectly (should be 401).
> > > > > > So why we not set this to 401 and make redirect with other means
> > > > > > (javascript/meta tags)? We know that pages that are larger than
> > > > > > certain size are displayed ok in IE so this should not be a
> problem.
> > > > > > Here is sample page with redirect:
> >
> > > > > > """
> > > > > >
> > > > > >
> > > > > > />
> > > > > > <!--
> > > > > > window.location='%(nexturl)s';
> > > > > > // -->
> > > > > >
> > > > > >
> > > > > >
> > > > > > """%{'nexturl':URL(r=request,c='auth',f='login')}
> >
> > > > > > If this is a good idea - I'll write a patch.
> >
> > > > > > --
> > > > > > Sincerely yours
> > > > > > Alexey Nezhdanov
> >
> > > > --
> > > > Sincerely yours
> > > > Alexey Nezhdanov
> >
> > > > noauth-401-403.diff
> > > > 5KViewDownload
> >
> > --
> > Sincerely yours
> > Alexey Nezhdanov
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:24196] Re: Proposal for auth
That's a question to Massimo. Currently auth code just provides an URL that
should be displayed instead. Such change requires rendering new URL w/o
actually re-requesting needed page from the browser.
On Mon, Jun 15, 2009 at 10:16 AM, Hans Donner wrote:
>
> can't the login page be included in the error page?
> This way you have both the error and no need for redirect as a user
> can directly login.
>
> On Mon, Jun 15, 2009 at 8:08 AM, Alexey Nezhdanov
> wrote:
> > Sure. For a start I'll provide the list that I have.
> >
> > Pros:
> > 1) Search-engine friendly. SEs see proper '401 Not authorised' response
> > instead of '303 Found' when trying to access login-only pages
> > 2) Unit tests - binding to 401/403 codes is more explicit. Receiving them
> > gives better understanding on what happens.
> > 3) RFC compliance
> >
> > Cons:
> > 1) Browser must render the page to trigger the redirect. So at least in
> some
> > browsers (for me it is Fx 2.0) this causes a momentary page clear before
> new
> > page is drawn (old page -> blank -> new page).
> >
> > So far I can see only one workaround: put there some content so that page
> > will not look so blank - in this case only 'content' part of the scren
> > should clear leaving menus/logos/other decorators static. That can be
> done
> > only in application I think.
> >
> >
> > On Mon, Jun 15, 2009 at 9:52 AM, mdipierro
> wrote:
> >>
> >> I would like to hear more opinions about this before changing it.
> >> Pros? Cons?
> >>
> >> Massimo
> >>
> >> On Jun 14, 10:28 pm, Alexey Nezhdanov wrote:
> >> > No, I didn't meant IE. I meant that since redirect page doesn't have
> >> > anybody
> >> > for a user AND it is not a 301/303 page - it actually rendered.
> >> > That is visible at least in Firefox.
> >> > Also if I just add some random content (like 'you are being
> redirected')
> >> > it
> >> > will be visible too (site -> 'you are being redirected' -> site
> again).
> >> > If
> >> > you think that it is a good idea - I'll do.
> >> >
> >> > On Monday 15 June 2009 01:05:25 mdipierro wrote:
> >> >
> >> >
> >> >
> >> > > You can replace the page blanking by adding some content longer than
> >> > > 512 chars
> >> >
> >> > > On Jun 13, 11:08 am, Alexey Nezhdanov wrote:
> >> > > > On Saturday 13 June 2009 16:58:33 mdipierro wrote:> I will take
> this
> >> > > > patch!
> >> >
> >> > > > Here it is, attached. Tested with latest stable, then ported to
> >> > > > trunk
> >> > > > version and tested again.
> >> > > > However there is one problem with it. On error code 401 browser
> >> > > > actually
> >> > > > _renders_ the page causing it to be blanked and redrawn as opposed
> >> > > > to 303
> >> > > > which just waits a bit, then instantly replaces page content.
> >> > > > I do not know if there is a way to suppress that page blanking.
> >> > > > Otherwise it seems to be working as intended.
> >> >
> >> > > > > On Jun 13, 6:19 am, Alexey Nezhdanov wrote:
> >> > > > > > I'm writing unit tests for my app and come across this
> problem:
> >> > > > > > if I try to access the page which has @auth.requires_login()
> >> > > > > > I get error 303 - i.e. redirect to the page with
> login/password
> >> > > > > > form.
> >> > > > > > While this works visually for browsers, it is actually wrong
> for
> >> > > > > > testing AND for search engines. Status codes are important for
> >> > > > > > robots
> >> > > > > > and here status is set incorrectly (should be 401).
> >> > > > > > So why we not set this to 401 and make redirect with other
> means
> >> > > > > > (javascript/meta tags)? We know that pages that are larger
> than
> >> > > > > > certain size are displayed ok in IE so this should not be a
> >> > > > > > problem.
> >> > > > > > Here is sample page with redirect:
> >> >
> >> > > > > > """
> >> > > > > >
> >> > > > > >
> >> > > > > > content="0;url=%(nexturl)s/"
> >> > > > > > />
> >> > > > > > <!--
> >> > > > > > window.location='%(nexturl)s';
> >> > > > > > // -->
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > """%{'nexturl':URL(r=request,c='auth',f='login')}
> >> >
> >> > > > > > If this is a good idea - I'll write a patch.
> >> >
> >> > > > > > --
> >> > > > > > Sincerely yours
> >> > > > > > Alexey Nezhdanov
> >> >
> >> > > > --
> >> > > > Sincerely yours
> >> > > > Alexey Nezhdanov
> >> >
> >> > > > noauth-401-403.diff
> >> > > > 5KViewDownload
> >> >
> >> > --
> >> > Sincerely yours
> >> > Alexey Nezhdanov
> >>
> >
> >
> > >
> >
>
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:24298] Re: Views without controllers
On Tuesday 16 June 2009 21:39:50 blackthorne wrote:
> hi
>
> On a small page I am working on, very often I have views that do not
> require controllers or models. All I want is to make a link to a view
> file.
>
> It doesn't seem very efficient to call a controller that returns an
> empty dict() and does nothing but a response.view redefinition
> everytime I need to create a link to a new page that just has contents
> in the view side.
If you have any arguments/variables passed to the page then you need
controller to validate them. If you don't have any - why don't you make just
a static file?
I know why though. You just need to have some content inserted into a site
template. static file will not do that.
Massimo - is it possible to have a controllerless static view which is still
have view {{insert}} capabilities ?
> Thank you.
>
>
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:24439] Re: Access Javacript variable's value in Web2py Code From view
On Thursday 18 June 2009 19:38:43 Yannick wrote:
> Hello Mate,
> I have a little problem here and not sure if it is even possible to do
> so...
> I was wondering if you know how to access javascript value in the
> web2py python code from the view
>
> Here I'm trying to add a parameter to my URL and the value of that
> param is a Javascript variable
>
> ##
>
>
> var param = 'value'
>
> {{URL(r=request,c='static',f='action', args=[param])}}
>
> .
>
> ##
>
> This code gave me an error because the variable "param" is not defined
> which is normal...
> Do you know a way to get a param's value and put it as an argument to
> my URL ??
URL() is executed when generating page on server.
and param=... gets assigned _later_ when executing _already_generated_page_.
You can try though:
{{param='value'}}
var param={{=param}}
> Thanks for your help...
>
> Cheers,
> Yannick P.
>
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:26725] Re: Stumped Beyond Belief
BTW, this is perfect example of helpless backtrace. Consider how much better it would be if it look like this: == SyntaxError: invalid field name: 'tablename.fieldname' == I submitted couple of similar patches some time ago, do not know if they were included: http://groups.google.com/group/web2py/browse_thread/thread/8568a5ea5800908/be4e8cfefd4c5b12?lnk=gst&q=alexey+nezhdanov#be4e8cfefd4c5b12 -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:27137] Re: Stumped Beyond Belief
On Friday 17 July 2009 19:11:11 JohnMc wrote: > BTW, this is perfect example of helpless backtrace. > Consider how much better it would be if it look like this: > == > SyntaxError: invalid field name: 'tablename.fieldname' > > -- Alexey > > That might have helped some. But to tell you the truth I went field by > field twice looking for just such an occurrence. EXACTLY! You were looking for an error looking _field_by_field_. Even if you have just 50 of them - that's a lot of work. Instead of concentrating on the SINGLE field. > My own mind filled in > the blank. That level of human error no amount of syntax checking will > solve. I'm sure it can be helped. Human errors are well known and backtraces are the perfect place to help fixing them. Anyways, Massimo - I didn't look into the code for a while already so I do not know how the things are at the moment. Do you want me to go over all 'raise' statements and submit a patch that will add more useful info just like in this example? > Just sayin'. > > On Jul 17, 2:12 am, Alexey Nezhdanov wrote: > > BTW, this is perfect example of helpless backtrace. > > Consider how much better it would be if it look like this: > > == > > SyntaxError: invalid field name: 'tablename.fieldname' > > == > > > > I submitted couple of similar patches some time ago, do not know if they > > were > > included:http://groups.google.com/group/web2py/browse_thread/thread/8568a > >5ea58... > > > > -- > > Sincerely yours > > Alexey Nezhdanov > > -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:27229] Re: Stumped Beyond Belief
On Wednesday 22 July 2009 12:18:52 mdipierro wrote:
> Some of them have been fixed but some may still need fixes. Help will
> be appreciated.
Ok, here it is.
I went over all files in gluon/ and changed all raises that were providing
uhelpful (IMHO of course) tracebacks.
Patch is against current trunk
> Massimo
>
> On Jul 22, 3:09 am, Alexey Nezhdanov wrote:
> > On Friday 17 July 2009 19:11:11 JohnMc wrote:> BTW, this is perfect
> > example of helpless backtrace.
> >
> > > Consider how much better it would be if it look like this:
> > > ==
> > > SyntaxError: invalid field name: 'tablename.fieldname'
> > >
> > > -- Alexey
> > >
> > > That might have helped some. But to tell you the truth I went field by
> > > field twice looking for just such an occurrence.
> >
> > EXACTLY! You were looking for an error looking _field_by_field_. Even if
> > you have just 50 of them - that's a lot of work. Instead of concentrating
> > on the SINGLE field.> My own mind filled in
> >
> > > the blank. That level of human error no amount of syntax checking will
> > > solve.
> >
> > I'm sure it can be helped. Human errors are well known and backtraces are
> > the perfect place to help fixing them.
> >
> > Anyways, Massimo - I didn't look into the code for a while already so I
> > do not know how the things are at the moment. Do you want me to go over
> > all 'raise' statements and submit a patch that will add more useful info
> > just like in this example?
> >
> > > Just sayin'.
> > >
> > > On Jul 17, 2:12 am, Alexey Nezhdanov wrote:
> > > > BTW, this is perfect example of helpless backtrace.
> > > > Consider how much better it would be if it look like this:
> > > > ==
> > > > SyntaxError: invalid field name: 'tablename.fieldname'
> > > > ==
> > > >
> > > > I submitted couple of similar patches some time ago, do not know if
> > > > they were
> > > > included:http://groups.google.com/group/web2py/browse_thread/thread/8
> > > >568a 5ea58...
> > > >
> > > > --
> > > > Sincerely yours
> > > > Alexey Nezhdanov
> >
> > --
> > Sincerely yours
> > Alexey Nezhdanov
>
>
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
=== modified file 'gluon/contrib/feedparser.py'
--- gluon/contrib/feedparser.py 2009-05-15 16:51:05 +
+++ gluon/contrib/feedparser.py 2009-07-23 06:45:03 +
@@ -3268,7 +3268,7 @@
if _debug:
sys.stderr.write('date handler function must return 9-tuple\n'
)
-raise ValueError
+raise ValueError('date handler function must return 9-tuple')
map(int, date9tuple)
return date9tuple
except Exception, e:
=== modified file 'gluon/contrib/gql.py'
--- gluon/contrib/gql.py 2009-07-18 02:52:22 +
+++ gluon/contrib/gql.py 2009-07-23 06:49:18 +
@@ -193,7 +193,7 @@
elif isinstance(field, SQLTable):
new_fields += [copy.copy(field[f]) for f in field.fields if f != 'id']
else:
-raise SyntaxError, 'define_table argument is not a SQLField'
+raise SyntaxError, 'define_table argument \'%s\'is not a SQLField'%field
fields = new_fields
self._db = db
self._tablename = tablename
@@ -594,7 +594,7 @@
def __call__(self, where):
if isinstance(self.where, QueryException) or isinstance(where,
QueryException):
-raise SyntaxError
+raise SyntaxError("Neither self.where nor where can't be a QueryException instance")
if self.where:
return SQLSet(self._db, self.where & where)
else:
=== modified file 'gluon/contrib/memdb.py'
--- gluon/contrib/memdb.py 2009-07-18 02:52:22 +
+++ gluon/contrib/memdb.py 2009-07-23 07:35:42 +
@@ -50,7 +50,7 @@
def cleanup(text):
if re.compile('[^0-9a-zA-Z_]').findall(text):
raise SyntaxError, \
-'only [0
[web2py:27490] Re: A call for resumés...
Hello. My resume is attached. Usually I work mostly on backend software, not websites, but for last 3-6 months I gradually shift to web development. Currently I work on a web2py project for transport company startup. Already implemented features include: * Registration (custom, not web2py) * City lookup by name in database. Alternatives to select one in case of multiple matches or possibility to select a point on map (openstreetmap.org). (jQuery) * database management (custom create/read/update/delete w/ filters). * database search (select all freights going from point A to point B. Same for lorries. Points are specified as lat/lon + radius). * access granting - registered user can create additional logins and give/revoke permissions to modify db tables) I can give you link to this site but since it is in Russian, probably it will be of very little use to you. I submitted several patches to web2py, most noticeable - 3x speedup of model, producing at least 2x speedup of front page of the site just mentioned: http://groups.google.com/group/web2py/browse_thread/thread/218bb58c96883578 I am located in Penza, Russia (GMT+3) and work as a freelancer. On Sunday 26 July 2009 12:20:31 Jason Brower wrote: > I have a growing demand for developers. If anyone is interested in > working in Finland on projects using web2py and jquery please send me > your resumé to my email. > Previous or viewable work is recommended. > The project needs lots of small contract work in the not so distant > future, and then some perminent work if all plays out well. (The > contract work doesn't need to be in Finland. :D) > Best Regards, > Jason Brower > > > > -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~--- CAREER OBJECTIVE To obtain a position in Software Development (full time employment or remote work) that utilizes my years of system administration and programming. I am a professional architect and software developer with more than fourteen years of experience in almost all aspects of software development. My strength is designing and developing complex applications that require a broad range of experience with different platforms and technologies. My objective is to design and develop new products and technologies with a group of capable co-workers in an organization, where I can potentially further develop my knowledge and experience on software design and development. PROFESSIONAL SUMMARY I work since 1997 as system administrator and programmer. I have a huge expirience in designing and administrating linux-based networks and services and secure publishing services to the internet. I have a good expirience in C/C++/python computer languages. I have an experience in packaging software for general distribution and for Debian OS. SUMMARY OF EXPERIENCE Technologies, languages and applications Proficient in general purpose programming and script languages and programming techniques: BASIC (3 years), Pascal (3 years), Assembler (i386) (4 years), C++ (13 years), python (6 years), XML/XSLT (1 year). Huge expirience in system Administration: DOS (5 years), Windows 95/98/2k/XP (13 years), linux2.2/2.4/2.6 (7 years). Highly expirienced in source control software: CVS(3 years), SVN(1 year), git(3 years). Deep knowledges in mathematics, systems design, protocols (http, smtp, smpp, pop3, xmpp), internet security. Experienced in reverse engineering, writing network protocols for inter-modules communications. Have some experience at submitting patches to Linus Torvalds: http://kernel.org/git/?p=cogito/cogito.git;a=commit;h=e64e1b79d7c50a234e97d59aadc7a4911de91efe http://kernel.org/git/?p=cogito/cogito.git;a=commit;h=7d80694af1a68ee1915e36bad9b26fa9fe054f06 http://kernel.org/git/?p=cogito/cogito.git;a=commit;h=667bb59b2d5b0a2e7fca5970d6f757790a6edd74 http://kernel.org/git/?p=cogito/cogito.git;a=commit;h=bab5583a49bff27c53f9ab22489b7faae21f607d PROFESSIONAL CAREER HISTORY Freelance Programmer January 2006 - Present System administrator, 'Rescue man' August 2006 - Present Penza cellular company "Penza-GSM", Penza, Russian Federation - administrating linux servers: ntp, ftp, mysql, lotus domino, intranet www, internet www, mail (+antivirus+spam filtering), pop3, IPSec gateway, internet gateway, firewall, file server, MS domain server (samba), dial-in modem pool, maintaining internal services. - servicing and programming offices PBX modules. - securing existing systems, secure installing of new systems, designing firewall rules, organising
[web2py:27545] typo on web2py.com
Hello Massimo.
'role based access control' is linked to
http://web2py.com/{=URL(r=request,f='tools',anchor='authorization')}}
I.e. one left { is missing
--
Sincerely yours
Alexey Nezhdanov
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~--~~~~--~~--~--~---
[web2py:27677] Re: Default CRYPT() is unsecure
s because it's following a > > >> known generation rule (being formed by a string repeated 3 times). > > >> That makes the diversity of possible passwords much smaller which > > >> isn't a good password policy. > > >> > > >> Besides, the complexity is the same (which is close to say that the > > >> computational speed is also close) and an attacker would need the > > >> same > > >> number of attempts to break the password by brute force because all > > >> he > > >> had to for each attempt X, the attempt XXX. He doesn't need to try X > > >> or XX. > > >> > > >>> On Jul 30, 8:38 pm, Jonathan Lundell wrote: > > >>>> On Jul 30, 2009, at 8:30 PM, Bottiger wrote: > > >>>>> I know you have the mantra of not breaking backwards > > >>>>> compatibility, > > >>>>> but it is a pretty bad idea to have unsalted MD5 passwords. > > >>>>> > > >>>>> For example, let's say your password is "massimo". The MD5 hash of > > >>>>> that happens to be "8cac5ac44b51f182143a43c4cdb6c4ac". > > >>>>> > > >>>>> Even forgetting rainbow tables, you can simply do a search for > > >>>>> it on > > >>>>> Google and you have 10+ pages telling you that it is the hash for > > >>>>> "massimo". > > >>>> > > >>>> How about a new validator that does the right thing, and > > >>>> deprecating > > >>>> CRYPT? > > >>>> > > >>>> I'd prefer some less-predictable salt than the suggestion below, > > >>>> though. How about the old Unix passwd trick of choosing a some > > >>>> random > > >>>> salt, and appending the salt in plaintext to the hash? > > >>>> > > >>>>>http://www.google.com/search?q=8cac5ac44b51f182143a43c4cdb6c4ac > > >>>>> > > >>>>> On Jul 30, 8:10 pm, mdipierro wrote: > > >>>>>> We cannot break backward compatibility. People should specify a > > >>>>>> key > > >>>>>> and use the HMAC+SHA512 anyway. > > >>>>>> > > >>>>>> Massimo > > >>>>>> > > >>>>>> On Jul 30, 9:49 pm, Bottiger wrote: > > >>>>>>> The CRYPT validator is unsecure because it uses unsalted MD5. > > >>>>>>> > > >>>>>>> There are public rainbow tables that have unsalted MD5 passwords > > >>>>>>> of up > > >>>>>>> to 10 characters long including symbols. > > >>>>>>> > > >>>>>>> I highly recommend that if no "key" is specified, that CRYPT > > >>>>>>> will > > >>>>>>> automatically salt the password based on a substring of the > > >>>>>>> password > > >>>>>>> itself. For example: > > >>>>>>> > > >>>>>>> password = "secretpass" > > >>>>>>> hash = md5(password+password[-1]) > > >>>>>>> > > >>>>>>> This will of course break backward compatibility, but this is > > >>>>>>> a real > > >>>>>>> security vulnerability. > > -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:27754] Re: Can I use HTTP PUT method?
Probably this: http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.6 I quickly skimmed through it and it seems that syntax is the same as with POST with the difference that transferred contents must be saved under specified path instead of being processed by some back-end. On Saturday 01 August 2009 14:49:36 mdipierro wrote: > This could be changed but I do not know if put requests contain data > in body (as post) or url (as get). Can somebody point me to a relevant > rfc? > > On Jul 31, 9:04 pm, Bottiger wrote: > > You will need to do request.body.read() for PUT. Web2Py does not parse > > the body of PUT requests. > > > > Example: > > > > Request: > > > > PUT /test/default/ HTTP/1.1 > > Content-Length: 21 > > > > blah blah hello hello > > > > Output: > > > > Body > > "blah blah hello hello" > > > > On Jul 31, 12:28 am, 诚子 wrote: > > > Hi, I wan't use PUT method and DELETE method,to build a RESTful > > > service,how to get an PUT method and DELETE data? > > -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:28337] Re: Static Uploads Time Out
On Monday 10 August 2009 10:56:23 mdipierro wrote: > I cannot exclude it is a problem with wsgiserver and ssl. Perhaps the > decrytpion is done in ram and that is a problem. I will look into > this. I had very similar problems back when developing xmpppy library. On some particular hardware/OS/python combinations SSL connections just stuck and won't proceed until you kick it (for instance Ctrl-C). Then it behaves as if nothing serious have happened and happily continue sending and receiving packets. Though may be Ctrl-C was xmpppy-specific. I never fixed that, libraries just gradually get updated world-wide. > On Aug 9, 4:20 pm, Derek wrote: > > Internet explorer nor Chrome work, so browser doesn't make a > > different. > > > > When I turn off SSL, it works fine. But, that's annoying. One of the > > things I'm most excited about with web2py is being able to edit > > interactive webpages while I'm on my network at a Starbucks. Any > > suggestions on how to get this to work with SSL? Is it a bug? > > > > Thanks, > > > > Derek > > > > On Aug 9, 4:33 pm, mdipierro wrote: > > > And have you tried without ssl? > > > > > > On Aug 9, 2:48 pm, Yarko Tymciurak wrote: > > > > On Sun, Aug 9, 2009 at 1:03 PM, Derek wrote: > > > > > I tried setting the -o option to 600, since I assumed the timeout > > > > > was in seconds. Now it just sits there for 10 minutes, then gives > > > > > me the timeout error. > > > > > > > > > > I'm using the Windows version of web2py with whatever webserver it > > > > > includes (tinyhttpd or something?). I pass the -c server.crt and -k > > > > > server.key options on the command-line. > > > > > > > > It is the cherrypy server... > > > > > > > > What browser are you using to do the transfer? Have you tried > > > > different browsers? > > > > > > > > > 126KB should transfer over my local wifi connection in seconds, so > > > > > filesize nor timeout should really be a factor here. Plus, web > > > > > images can easily be 126KB or larger... so what gives? > > > > > > > > > > Derek > > > > > > > > > > On Aug 9, 12:34 pm, mdipierro wrote: > > > > > > Are you using ssh with wsgiserver or are you using apache with > > > > > > mod_ssl? > > > > > > > > > > > > Massimo > > > > > > > > > > > > On Aug 9, 10:22 am, Derek wrote: > > > > > > > Hi all, > > > > > > > > > > > > > > I'm trying to upload a 126KB static file, and it keeps giving > > > > > > > me a timeout. If there were to be any file size limit, I really > > > > > > > would have expected it to be much larger. Other smaller files > > > > > > > are uploading fine. > > > > > > > > > > > > > > I'm using the binary Windows distribution, and running it over > > > > > > > SSL on my local network. > > > > > > > > > > > > > > Any help would be greatly appreciated. > > > > > > > > > > > > > > Derek > > -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:29671] Re: Comma creates an invalid request?
Never tried but you probably also can try %2c On Mon, Aug 31, 2009 at 10:57 AM, SergeyPo wrote: > > Yes, in my application comma creates "invalid request' error too. I > had to write a special javascript routine to ensure that there is no > commas in requests (they are produced by flot library when you drag an > area in the chart). I recommend to use /geocoder/index/london/ontario > and retrieve values as request.args[:] > > On Aug 31, 6:20 am, Derek wrote: > > Hi, > > > > I'm doing some geocoding. I have a geocoder.py file with an index > > function. > > > > A url like: > > > > /geocoder/index/london > > > > uses the value 'london' (stored in request.args[0]) just fine. But if > > I add a comma, as in: > > > > /geocoder/index/london,_ontario > > > > I get an "Invalid request" error. It's not even a web2py ticket. > > > > Any thoughts? > > > > Thanks, > > > > Derek > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:29672] Does double submission still breaks with recent web2py code?
Hello. Massimo, I have a question regarding double form submission. There is a phrase in the book (page 188): = Moreover, when multiple forms are present on the same page, the mechanism for preventing double submission breaks... = That was a good reason for me to look into the code. I didn't find anything that should break there. So I typed in your double form example and played with it (catched the stream with tcpflow and used it to imitate double submission) - it correctly recognises second submission too for both forms (if I pass in session object into accepts()). Can you please explain what kind of breakage happens? -- Regards Alexey --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:29679] Re: developer questions
1. Yes, if you want to port your application to python then you definitely have to learn the language. 2. Since this maillist is most 'web2py developer rich' place probably it is safe to ask here if Massimo doesn't mind. On Mon, Aug 31, 2009 at 4:41 PM, Markus wrote: > > Hello > > I run a successful ecommerce portal. We use PHP. > > I am planning to use Web2py and GAE to remove the 'scaling' worries. > > Where should i start, learn python before starting web2py? > > Please advise. > > Also where can i find web2py developers to work on my project? > > Prakash > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:29688] Re: Does double submission still breaks with recent web2py code?
Aha. So the documentation is obsolette in this case. Recent web2py is smart enough to deal with multiple forms & double submits detection on the same page. Thank you for clarification (and for the book of course!). On Mon, Aug 31, 2009 at 4:58 PM, mdipierro wrote: > > That line should have been corrected. > > You cannot have two forms in the same page with the same _formname. > That is all. > > On Aug 31, 2:34 am, Alexey Nezhdanov wrote: > > Hello. > > Massimo, I have a question regarding double form submission. > > There is a phrase in the book (page 188): > > = > > Moreover, when multiple forms are present on the same page, the mechanism > > for preventing double submission breaks... > > = > > That was a good reason for me to look into the code. I didn't find > anything > > that should break there. > > So I typed in your double form example and played with it (catched the > > stream with tcpflow and used it to imitate double submission) - it > correctly > > recognises second submission too for both forms (if I pass in session > object > > into accepts()). > > Can you please explain what kind of breakage happens? > > -- > > Regards > > Alexey > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:29691] Re: web2py book, 2nd Ed
> > P.S. Some corrections have been proposed. I will implement those > corrections asap. > Oh, I've spotted several typos, but right now can recall just one :( page 187, onvalidation. In model you declare field 'd' but in external validator you set variable 'c'. I'll make a list of any other ones that I will find and submit to you separately. > > Massimo > > On Aug 28, 9:29 pm, Franklin Naval wrote: > > Congrats! > > Bought! > > -Franklin > > QuadraForte LLChttp://www.quadraforte.com > > > > On Sat, Aug 29, 2009 at 10:12 AM, cadrentes wrote: > > > > > It is well worth buying a second time. Thanks Massimo! I'm using > > > web2py at work and it is really wowing the boss! > > > > > On Aug 26, 3:34 am, mdipierro wrote: > > > > The new web2py book is available on lulu.com > > > > > > http://www.lulu.com/content/e-book/web2py/4968879 > > > > > > Lots of new stuff with100 more pages (341 pages in total). Covers > > > > Auth, Crud, Services, interaction with Pyjamas, PyAMF, and better > > > > deployment recipes. > > > > > > Same price as before ($12.5) and same ID (which means if you bought > > > > the old one you should be able to get the new one for free). > > > > > > I am also looking into posting it on scribd (free viewing but no > > > > download) and it will be published printed by Wiley soon. > > > > > > Thanks to all those who collaborated by sending corrections. I hope I > > > > acknowledged everybody properly in the introduction. If I forgot you > > > > let me know. > > > > > > Massimo > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:29692] Re: web2py book, 2nd Ed
And yet another one (sorry, didn't start compiling a list yet) Page 197 - screenshot of db record edition page. Accortding to preceeding text record must contain an image already so the page must show 'delete checkbox' and 'download link'. But screenshot doesn't have all this neither it is in html dump that is included under it. You have to regenerate both. On Mon, Aug 31, 2009 at 5:29 PM, Alexey Nezhdanov wrote: > P.S. Some corrections have been proposed. I will implement those >> corrections asap. >> > Oh, I've spotted several typos, but right now can recall just one :( > page 187, onvalidation. In model you declare field 'd' but in external > validator you set variable 'c'. > > I'll make a list of any other ones that I will find and submit to you > separately. > > >> >> Massimo >> >> On Aug 28, 9:29 pm, Franklin Naval wrote: >> > Congrats! >> > Bought! >> > -Franklin >> > QuadraForte LLChttp://www.quadraforte.com >> > >> > On Sat, Aug 29, 2009 at 10:12 AM, cadrentes >> wrote: >> > >> > > It is well worth buying a second time. Thanks Massimo! I'm using >> > > web2py at work and it is really wowing the boss! >> > >> > > On Aug 26, 3:34 am, mdipierro wrote: >> > > > The new web2py book is available on lulu.com >> > >> > > > http://www.lulu.com/content/e-book/web2py/4968879 >> > >> > > > Lots of new stuff with100 more pages (341 pages in total). Covers >> > > > Auth, Crud, Services, interaction with Pyjamas, PyAMF, and better >> > > > deployment recipes. >> > >> > > > Same price as before ($12.5) and same ID (which means if you bought >> > > > the old one you should be able to get the new one for free). >> > >> > > > I am also looking into posting it on scribd (free viewing but no >> > > > download) and it will be published printed by Wiley soon. >> > >> > > > Thanks to all those who collaborated by sending corrections. I hope >> I >> > > > acknowledged everybody properly in the introduction. If I forgot you >> > > > let me know. >> > >> > > > Massimo >> >> >> > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:31404] Re: [web2py:/] Re: file descriptors leak
You don't need the thread lock over the whole function: try: return MemcacheClient.__mc_instance else: lock() if not hasattr(MemcacheClient, '__mc_instance'): MemcacheClient.__mc_instance = _MemcacheClient(*a, **b) unlock() return MemcacheClient.__mc_instance On Sun, Sep 13, 2009 at 8:03 PM, mdipierro wrote: > > we can encapsulate the body of the function in thread lock. > > On Sep 13, 10:53 am, zahariash wrote: > > Ups, in my patch if statement isn't thread safe... > > > > On 13 Wrz, 17:35, zahariash wrote: > > > > > I think it is. > > > > > On 13 Wrz, 17:31, mdipierro wrote: > > > > > > I think is excellent as long as Memcache is thread safe and I assume > > > > it is. > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:35503] RSS encoding
Hello. I noticed that generic.rss produces non utf-8 text. Even worse - it produces iso-8859-1 text WITHOUT AN OPTION TO CHANGE THAT! So I made the following change to gluon/contrib/rss2.py : def dumps(rss, encoding='utf-8'): s = cStringIO.StringIO() rss.write_xml(s, encoding) return s.getvalue() -- Regards Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
[web2py:37639] How to connect to Oracle table with dot and 'at' in the name?
Hello.
I've googled this group for my problem, but didn't find the answer.
It seems that problem was mentioned once [1] [2], but I do not know if any
solution was done.
I need to connect to existing database so things like 'schemas are not
portable' are not really acceptable - I should be able to connect, period.
What is worse - there are actually two servers - DB2 and Oracle. I do not
have direct access to DB2 database and have to connect through Oracle.
So the sql query looks like this:
SELECT ip FROM rad@inet
so here is scheme RAD, table NAS at the 'INET' DB2 database.
web2py refuses me to do this:
db.define_table('rad@inet',
Field('ip','string',length=16),
migrate=False
)
So what should I do? Is there any way out?
Thanks
[1]
http://groups.google.com/group/web2py/search?group=web2py&q=oracle+table+dot&qt_g=Search+this+group
[2]
http://groups.google.com/group/web2py/browse_thread/thread/3c27af05333b34b5/a121f8bbccfeeb7f?lnk=gst&q=oracle+table+dot#a121f8bbccfeeb7f
--
Regards
Alexey Nezhdanov
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
Re: [web2py:37667] Re: How to connect to Oracle table with dot and 'at' in the name?
Sorry for double posting. While this would be an ideal - it is clearly not
supported atm. But may be I can somehow emulate it by, say, initialising
table with the usual way and then doing something like
db.inet.var1.var2='rad@inet' ?
If there is any coding needed - that's ok too, I can submit a patch (just
like before).
On Tue, Dec 22, 2009 at 7:15 PM, Alexey Nezhdanov wrote:
> I do not know actually as I am not the Oracle expert (or even regular
> user). Will find out tomorrow.
> Suppose - I can't - can you give any clue how to make an alias in web2py?
> The ideal would be something like this:
>
> db.define_table('inet', internal_name='rad@inet',
>
> Field('ip','string',length=16),
> migrate=False
> )
>
>
> On Tue, Dec 22, 2009 at 5:47 PM, mdipierro wrote:
>
>> No. "." and "@" are not allowed in table names. Can you alias in
>> Orcale?
>>
>> On Dec 22, 12:15 am, Alexey Nezhdanov wrote:
>> > Hello.
>> > I've googled this group for my problem, but didn't find the answer.
>> > It seems that problem was mentioned once [1] [2], but I do not know if
>> any
>> > solution was done.
>> >
>> > I need to connect to existing database so things like 'schemas are not
>> > portable' are not really acceptable - I should be able to connect,
>> period.
>> > What is worse - there are actually two servers - DB2 and Oracle. I do
>> not
>> > have direct access to DB2 database and have to connect through Oracle.
>> > So the sql query looks like this:
>> >
>> > SELECT ip FROM rad@inet
>> >
>> > so here is scheme RAD, table NAS at the 'INET' DB2 database.
>> >
>> > web2py refuses me to do this:
>> >
>> > db.define_table('rad@inet',
>> > Field('ip','string',length=16),
>> > migrate=False
>> > )
>> >
>> > So what should I do? Is there any way out?
>> > Thanks
>> >
>> > [1]
>> http://groups.google.com/group/web2py/search?group=web2py&q=oracle+ta...
>> > [2]
>> http://groups.google.com/group/web2py/browse_thread/thread/3c27af0533...
>> >
>> > --
>> > Regards
>> > Alexey Nezhdanov
>>
>> --
>> Regards
>>
> Alexey
>
>
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
Re: [web2py:37666] Re: How to connect to Oracle table with dot and 'at' in the name?
I do not know actually as I am not the Oracle expert (or even regular user).
Will find out tomorrow.
Suppose - I can't - can you give any clue how to make an alias in web2py?
The ideal would be something like this:
db.define_table('inet', internal_name='rad@inet',
Field('ip','string',length=16),
migrate=False
)
On Tue, Dec 22, 2009 at 5:47 PM, mdipierro wrote:
> No. "." and "@" are not allowed in table names. Can you alias in
> Orcale?
>
> On Dec 22, 12:15 am, Alexey Nezhdanov wrote:
> > Hello.
> > I've googled this group for my problem, but didn't find the answer.
> > It seems that problem was mentioned once [1] [2], but I do not know if
> any
> > solution was done.
> >
> > I need to connect to existing database so things like 'schemas are not
> > portable' are not really acceptable - I should be able to connect,
> period.
> > What is worse - there are actually two servers - DB2 and Oracle. I do not
> > have direct access to DB2 database and have to connect through Oracle.
> > So the sql query looks like this:
> >
> > SELECT ip FROM rad@inet
> >
> > so here is scheme RAD, table NAS at the 'INET' DB2 database.
> >
> > web2py refuses me to do this:
> >
> > db.define_table('rad@inet',
> > Field('ip','string',length=16),
> > migrate=False
> > )
> >
> > So what should I do? Is there any way out?
> > Thanks
> >
> > [1]
> http://groups.google.com/group/web2py/search?group=web2py&q=oracle+ta...
> > [2]
> http://groups.google.com/group/web2py/browse_thread/thread/3c27af0533...
> >
> > --
> > Regards
> > Alexey Nezhdanov
>
> --
> Regards
>
Alexey
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
[web2py:21575] Can I create an 'assotiated' input field?
Hello. I've started using web2py recently for writing a GAE-enabled site. As you know, BigTable does not have support for date/datetime fields so I decided just to use 'integer' and store time.time() there. But there is a problem with SQLFORM: I do not want user to see the actual seconds since 1.01.1970 but rather a proper calendar or, in my case, a three fields - day/month/year. So I ended up implementing an 'assotiated field' - the actual db field go in as hidden one and my 's hook into _validate chain and change the SQLFORM.vars on the fly. This is ugly but I do not know how to do it better. May be someone on this list encountered such problem before? Please share your experience. Here is my code that I'd like to get rid of: MyWidget=my_tools.DateWidget() db.freight.time_ready_start.widget=MyWidget.widget MyWidget.timespan=db.freight.time_ready_span # FIXME -- Sincerely yours Alexey Nezhdanov --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~--~~~~--~~--~--~---
