Hi all - just had time this weekend to look further into this.
Thank you to suggestions from Iceberg, Yarko and Massimo. I have tried
putting table definitions into a module, and also run some timing
tests against a Postgres database
As a note, for my previous tests coming out at 12ms page response,
those same previous test now come out at 10ms on my computer .. I
think just in picking up latest Ubuntu patches etc. something must
have changed in my configuration, anyway all the figures below are
just a rough guide.
Here are the changes I made:
db and mail object creation now in modules\application.py.
These objects are created once then shared.
Setting 'ENABLE_APPLICATION_SCOPE = False' in application.py switches
back to standard behaviour (separate db and mail object on every page
request)
auth,crud,service objects are dependent on the current page request,
so I left these in models\db.py.
One exception is auth.define_tables - this is run in modules
\application.py. Usually runs once only unless
ENABLE_APPLICATION_SCOPE is False, then auth.define_tables runs on
every page request.
Postgres version is 8.4.0-2 installed on same machine web2py is
running on. Computer set-up same as for previous test (except for
Ubuntu patches mentioned above), the test is to request the index page
created by default for a new web2py application.
Running 1000 page requests with httperf tool:
postgres, compiled, migrate=false, application scope objects disabled,
pool_size parameter not set:
58.2ms average page response time
postgres, compiled, migrate=false, application scope objects disabled,
pool_size=100:
10.9ms average page response time
postgres, compiled, migrate=false, application scope objects
enabled,pool_size=100:
5.2ms average page response time
sqllite, compiled, migrate=false, application scope objects disabled:
10.1ms average page response time
sqllite, compiled, migrate=false, application scope objects enabled:
5.4ms average page response time
---------
My summary:
Sharing objects like the database model definition at the website
application scope does give a speed boost, but it is not as big a
difference as I thought - I have been pleasantly surprised that the
existing per-webpage request evaluation of models was already pretty
fast as it is.
The difference in Postgres with timings when pool_size is not set at
all is significant : This almost warrants defaulting the pool size for
this database to 100 anyway, or at least outputting to the log a
friendly warning to suggest setting a pool size parameter.
I can see some downsides when attempting to use objects at application
scope:
- I'm not yet sure objects like DAL and Mail will be happy being
shared at the application level. Could well be locking / threading
issues, would need to investigate further
- In general in Python, module source code changes do not get picked
up automatically. During development I would work around this with the
reload(...) statement
- My example shared module probably needs additional try..catch
statements to handle if it fails to run, so every webpage request will
keep on trying to create the shared objects until they have been
successfully created.
- There may be good reason to refresh these shared objects
periodically anyway, maybe tie the shared object creation to a cron
job.
If I could be confident there are no threading issues then using
shared objects could be an option. But until then I am now happier
sticking with the standard setup - putting maybe quite complex model
structures in the regular model folder, these will be evaluated on
every webpage request, but the site will still run plenty fast - good
enough for me anyway!
Here is my updated db.py and module source code at the end of this
message. Just add both to a blank application created by the web2py
admin.
Cheers,
- Alex
--
db.py model file, pulling in shared 'application scope' objects:
# coding: utf8
from gluon.tools import *
import applications.perftest.modules.application as app
#Get objects shared between page requests
app.Helper.get_application_scope(globals())
#Rest of this file sets up per-page model state:
if not globals().has_key('auth'):
auth=Auth(globals(),db)
auth.settings.hmac_key='6944f00d-1758-41e4-8fdb-625b0be17e1a'
auth.settings.mailer=mail # for user email
verification
auth.settings.registration_requires_verification = True
auth.settings.registration_requires_approval = True
auth.messages.verify_email = \
'Click on the link http://.../user/verify_email/%(key)s to verify
your email'
crud=Crud(globals(),db)
crud.settings.auth=auth
#Connect gae session if required
if request.env.web2py_runtime_gae: session.connect(request, response,
db=db)
#Set up service for this request
from gluon.tools import *
service=Service(globals())
----------------------------------------
application.py module file:
# coding: utf8
from gluon.storage import Storage
from gluon.sql import SQLDB, SQLField, DAL, Field
from gluon.tools import *
import thread
ENABLE_APPLICATION_SCOPE = True
class Helper(object):
singleton = None
def __init__(self, environment):
request = environment['request']
self.storage = Storage()
self.storage.db = self.get_db(request)
self.storage.mail = self.get_mail()
@staticmethod
def get_application_scope(globals):
if ENABLE_APPLICATION_SCOPE:
if Helper.singleton is None:
locker = thread.allocate_lock()
locker.acquire()
Helper.singleton = Helper(globals)
Helper.singleton.init_auth(globals);
locker.release()
globals.update(Helper.singleton.storage)
else:
request_scope = Helper(globals)
request_scope.storage.auth = request_scope.init_auth
(globals)
globals.update(request_scope.storage)
def get_db(self, request):
if request.env.web2py_runtime_gae:
db = DAL('gae')
else:
#db = DAL('sqlite://storage.sqlite')
db=DAL('postgres://testuser:t...@localhost:5432/web2py',
pool_size=100)
for i in range(1, 5):
db.define_table('perftest' + str(i),
SQLField('string1','string'),
SQLField('text1','text'),
SQLField('blob1','blob'),
SQLField('password1','password'),
SQLField('upload1','upload'),
SQLField('boolean1','boolean'),
SQLField('integer1','integer'),
SQLField('double1','double'),
SQLField('date1','date'),
SQLField('time1','time'),
SQLField('datetime1','datetime'),migrate=False)
return db
def get_mail(self):
mail=Mail() # mailer
mail.settings.server='smtp.gmail.com:587' # your SMTP
server
mail.settings.sender='y...@gmail.com' # your email
mail.settings.login='username:password' # your
credentials or None
return mail
#Just used to initialise auth database tables
def init_auth(self, globals):
auth=Auth(globals, self.storage.db)
auth.define_tables(migrate=False)
return auth
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~----------~----~----~----~------~----~------~--~---
