+1 for trying things differently :) On 8/19/13 12:14 AM, "Jay Pipes" <jaypi...@gmail.com> wrote:
>On 08/18/2013 10:33 PM, Joe Gordon wrote: >> An alternative I think would be better would be to scrap the >>use of >> the SQLAlchemy ORM; keep using the DB engine abstraction >>support. >> >> +1, I am hoping this will provide noticeable performance benefits while >> being agnostic of what DB back-end is being used. With the way we use >> SQLALchemy being 25x slower then MySQL we have lots of room for >> improvement (see http://paste.openstack.org/show/44143/ from >> https://bugs.launchpad.net/nova/+bug/1212418). > >@require_admin_context >def compute_node_get_all(context): > return model_query(context, models.ComputeNode).\ > options(joinedload('service')).\ > options(joinedload('stats')).\ > all() > >Well, yeah... I suppose if you are attempting to create 115K objects in >memory in Python (Need to collate each ComputeNode model object and each >of its relation objects for Service and Stats) you are going to run into >some performance problems. :) > >Would be interesting to see what the performance difference would be if >you instead had dicts instead of model objects and did something like >this instead (code not tested, just off top of head...): > ># Assume a method to_dict() that takes a Model ># and returns a dict with appropriate empty dicts for ># relationship fields. > >qr = session.query(ComputeNode).join(Service).join(Stats) > >results = {} > >for record in qr: > node_id = record.ComputeNode.id > service_id = record.Service.id > stat_id = record.ComputeNodeStat.id > if node_id not in results.keys(): > results[node_id] = to_dict(record.ComputeNode) > if service_id not in results[node_id]['services'].keys(): > results[node_id]['services'][service_id] = to_dict(record.Service) > if stat_id not in results[node_id]['stats'].keys(): > results[node_id]['stats'][stat_id] = to_dict(record.ComputeNodeStat) > >return results > >Whether it would be any faster than SQLAlchemy's joinedload... > >Besides that, though, probably is a good idea to look at even the >existence of DB calls that potentially do that kind of massive query >returning as A Bad Thing... > >Best, >-jay > >_______________________________________________ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
