Turns out the current DAL spends lots of time in building the SQL
representation of a table (CREATE TABLE) even if the table does not
need to be created. I will fix this and it will speed it up a lot
without need for lazy evaluations.
Massimo
On Jun 6, 1:14 am, Alexey Nezhdanov <snak...@gmail.com> wrote:
> Switched to lazy table definitions.
> Model init time was cut down to 0.046s.
> Some of excess time is eliminated, some (my guess is 30%) is moved into
> controller execution. At any rate - this is faster than before.
>
> Next step would be full-scale profiling but not yet.
>
> Here is excerpt from my SQLStorage:
> --------------
> def __getitem__(self, key):
> value = dict.__getitem__(self, str(key))
> if not callable(value) or key[0]=='_' or isinstance(value,
> SQLCallableList): return value
> value.__call__() # That must redefine table in-place
> return dict.__getitem__(self, str(key))
> ------------
> and here is excerpt from my db.py:
> ----------
> def define_table_system_participant():
> db.define_table('system_participant',
> SQLField('firm_id','integer'),
> migrate=migrate,
> )
> db.system_participant=define_table_system_participant
> ----------
>
> On Saturday 06 June 2009 07:53:18 Alexey Nezhdanov wrote:
>
>
>
> > ON Saturday 06 June 2009 00:25:47 mdipierro wrote:
> > > One other trick you can try is replace
>
> > > db.define_table('table',SQLField('field'),...)
> > > db.table.field.requires=....
>
> > > with
>
> > > db.define_table('table',SQLfield('field',requires=...),...)
>
> > > and so for all the other attributes.
>
> > That will make minor difference. I do not have too many 'requires' and mod
> > of what I have are set up through function call.
>
> > > Did you bytecode compile the app?
> > > Does it make a difference?
>
> > I just run some automated tests. Here is average time over 100 runs each:
>
> > sqlite+nomigrate+py 0.123
> > sqlite+nomigrate+pyc 0.122
> > mysql+nomigrate+py 0.123
> > mysql+nomigrate+pyc 0.123
>
> > I think I'll try this approach:
> > 1) define each table as a function which yelds a table.
> > 2) modify sql.py so that db object will test the type of table.
> > if it has __exec__ method - execute it and replace it with return result.
>
> > This way my tables will be lazily defined when controller actually needs
> > them.
>
> > > Massimo
>
> > > On Jun 5, 3:05 pm, Alexey Nezhdanov <snak...@gmail.com> wrote:
> > > > On Friday 05 June 2009 21:03:20 mdipierro wrote:> Can you tell us more
> > > > about the setup, os, hardward etc. is mysql on
>
> > > > > the same machine?
>
> > > > Kubuntu 8.04. Turion64 1.6GHz, 1.6G RAM. MySQL is on the same box. SiS
> > > > motherboard w/ nForce chipset. Laptop 3 years old (and it was about 1
> > > > year old model when was bought).
>
> > > > > How much is the the SQLDB() vs the define_tables? Do you have many
> > > > > tables? how long?
>
> > > > 16 tables, 152 SQLFields. single SQLDB (currently MySQL, but I'll
> > > > switch it back to SQLite)
>
> > > > > One trick is to add is statements in the model so that only those
> > > > > tables needed are defined, depending on request.controller and
> > > > > request.action.
>
> > > > yes, I thought of that. But that makes it inflexible. That's why I
> > > > suggested lazy tables init.
>
> > > > And regarding 'turion is not very fast'. I don't really have any load
> > > > on this box. So 0.5 seconds per GET is VERY slow. 8-years old Celeron
> > > > 800 should be behaving something like 0.05 seconds per request (of
> > > > course with ad-hoc programming, no DAL).
>
> > > > This is not the empty complaint. We can't really afford saying 'throw
> > > > in more CPU'. If web2py targets GAE - then it absolutely must be
> > > > CPU-friendly. GAE can help with adding more nodes but it charges for
> > > > processor time anyways. And actually the same goes about dedicated
> > > > hosting too. If someone targets only a few visitors per day - it's ok.
> > > > But not if we want tens and hundreds pageloads per second.
>
> > > > > On Jun 5, 11:29 am, Alexey Nezhdanov <snak...@gmail.com> wrote:
> > > > > > On Friday 05 June 2009 17:07:55 mdipierro wrote:> In a production
> > > > > > environment you would be using mysql or postgresql. In
>
> > > > > > > this case you should be using
>
> > > > > > > SQLDB(...,pool_size=10)
> > > > > > > dn.define_table(....,migrate=False)
>
> > > > > > > the connection pooling and migrations off make a big difference.
> > > > > > > Perhaps you can run some tests and quantify this.
>
> > > > > > migrate=False makes cuts the model init time in half - now I'm
> > > > > > getting about 0.15-0.17s each time. Testing MySQL, stand by...
>
> > > > > > Hmmm.
> > > > > > 0.21...0.25s with mysql and migrations off... and pool_size=10.
>
> > > > > > > When using sqlite you cannot use pooling and that means web2py
> > > > > > > has to open the db every time.
>
> > > > > > > Massimo
>
> > > > > > > On Jun 5, 2:58 am, Alexey Nezhdanov <snak...@gmail.com> wrote:
> > > > > > > > Hello again.
> > > > > > > > Recently I measured the perfomance of web2py regarding to
> > > > > > > > 'milliseconds per request'. Got some unexpected results. The
> > > > > > > > most slow part of the application is the model. It takes 40-60%
> > > > > > > > of total time. Measurement was done simply by putting
> > > > > > > > import time;print time.time(),'model start'
> > > > > > > > at the beginning of db.py and similar line at the end of it.
> > > > > > > > Here is what it produces on my laptop (Turion64, 1.6GHz, 1.5G
> > > > > > > > RAM):
>
> > > > > > > > 1244187446.32 model start
> > > > > > > > 1244187446.62 model stop
> > > > > > > > 0.3 second just to set up the model! I can live with 0.05 for
> > > > > > > > it, may be even 0.1, but 0.3 for _each_ GET or POST request is
> > > > > > > > a bit too much, don't you think?
> > > > > > > > That is for not too complex model - 17 tables, averaging 8.6
> > > > > > > > SQLFields per one. On another web2py project it takes
> > > > > > > > 0.38...0.42 second each time
>
> > > > > > > > :(
>
> > > > > > > > I tried compiling my app and measuring again:
> > > > > > > > 1244187625.31 model start
> > > > > > > > 1244187625.69 model stop
> > > > > > > > Not any better. In fact, it's even worse, but since results
> > > > > > > > vary from run to run I suspect that it is just the same
> > > > > > > > perfomance.
>
> > > > > > > > Massimo, as I know you've been working on new model for some
> > > > > > > > time already. Is there any hope of having a faster model? I
> > > > > > > > suspect more lazy evaluation should do the magic, but I didn't
> > > > > > > > do any research yet.
>
> > > > > > > > Frankly speaking when I first discovered the fact that web2py
> > > > > > > > always _executes_ model, controller, view, I thought that it
> > > > > > > > may be a perfomance hog. Until I actually did that check I
> > > > > > > > thought that it will execute db.py each time it changes on-disk
> > > > > > > > and then just keep built structures somewhere around, probably
> > > > > > > > pickled. May be it is still possible to use that approach to
> > > > > > > > some extent?
>
> > > > > > > > Or may be I am just completely missing the point. Please
> > > > > > > > comment.
>
> > > > > > > > --
> > > > > > > > Sincerely yours
> > > > > > > > Alexey Nezhdanov
>
> > > > > > --
> > > > > > Sincerely yours
> > > > > > Alexey Nezhdanov
>
> > > > --
> > > > Sincerely yours
> > > > Alexey Nezhdanov
>
> --
> Sincerely yours
> Alexey Nezhdanov
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~----------~----~----~----~------~----~------~--~---
