yes. guppy seems to have some nicer string formatting for this dump as well,
but i was unable to figure out how to get this string format to write to a
file, it seems like the tool is very geared towards interactive console use.
We should pick a nice memory formatter we like, there’s a bunch of them, and
then add it to our standard toolset.
On Sep 9, 2014, at 10:35 AM, Doug Hellmann <d...@doughellmann.com> wrote:
>
> On Sep 8, 2014, at 8:12 PM, Mike Bayer <mba...@redhat.com> wrote:
>
>> Hi All -
>>
>> Joe had me do some quick memory profiling on nova, just an FYI if anyone
>> wants to play with this technique, I place a little bit of memory profiling
>> code using Guppy into nova/api/__init__.py, or anywhere in your favorite app
>> that will definitely get imported when the thing first runs:
>>
>> from guppy import hpy
>> import signal
>> import datetime
>>
>> def handler(signum, frame):
>> print "guppy memory dump"
>>
>> fname = "/tmp/memory_%s.txt" %
>> datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
>> prof = hpy().heap()
>> with open(fname, 'w') as handle:
>> prof.dump(handle)
>> del prof
>>
>> signal.signal(signal.SIGUSR2, handler)
>
> This looks like something we could build into our standard service startup
> code. Maybe in
> http://git.openstack.org/cgit/openstack/oslo-incubator/tree/openstack/common/service.py
> for example?
>
> Doug
>
>>
>>
>>
>> Then, run nova-api, run some API calls, then you hit the nova-api process
>> with a SIGUSR2 signal, and it will dump a profile into /tmp/ like this:
>>
>> http://paste.openstack.org/show/108536/
>>
>> Now obviously everyone is like, oh boy memory lets go beat up SQLAlchemy
>> again…..which is fine I can take it. In that particular profile, there’s a
>> bunch of SQLAlchemy stuff, but that is all structural to the classes that
>> are mapped in Nova API, e.g. 52 classes with a total of 656 attributes
>> mapped. That stuff sets up once and doesn’t change. If Nova used less
>> ORM, e.g. didn’t map everything, that would be less. But in that profile
>> there’s no “data” lying around.
>>
>> But even if you don’t have that many objects resident, your Python process
>> might still be using up a ton of memory. The reason for this is that the
>> cPython interpreter has a model where it will grab all the memory it needs
>> to do something, a time consuming process by the way, but then it really
>> doesn’t ever release it (see
>> http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm
>> for the “classic” answer on this, things may have improved/modernized in
>> 2.7 but I think this is still the general idea).
>>
>> So in terms of SQLAlchemy, a good way to suck up a ton of memory all at once
>> that probably won’t get released is to do this:
>>
>> 1. fetching a full ORM object with all of its data
>>
>> 2. fetching lots of them all at once
>>
>>
>> So to avoid doing that, the answer isn’t necessarily that simple. The
>> quick wins to loading full objects are to …not load the whole thing! E.g.
>> assuming we can get Openstack onto 0.9 in requirements.txt, we can start
>> using load_only():
>>
>> session.query(MyObject).options(load_only(“id”, “name”, “ip”))
>>
>> or with any version, just load those columns - we should be using this as
>> much as possible for any query that is row/time intensive and doesn’t need
>> full ORM behaviors (like relationships, persistence):
>>
>> session.query(MyObject.id, MyObject.name, MyObject.ip)
>>
>> Another quick win, if we *really* need an ORM object, not a row, and we have
>> to fetch a ton of them in one big result, is to fetch them using yield_per():
>>
>> for obj in session.query(MyObject).yield_per(100):
>> # work with obj and then make sure to lose all references to it
>>
>> yield_per() will dish out objects drawing from batches of the number you
>> give it. But it has two huge caveats: one is that it isn’t compatible with
>> most forms of eager loading, except for many-to-one joined loads. The other
>> is that the DBAPI, e.g. like the MySQL driver, does *not* stream the rows;
>> virtually all DBAPIs by default load a result set fully before you ever see
>> the first row. psycopg2 is one of the only DBAPIs that even offers a
>> special mode to work around this (server side cursors).
>>
>> Which means its even *better* to paginate result sets, so that you only ask
>> the database for a chunk at a time, only storing at most a subset of objects
>> in memory at once. Pagination itself is tricky, if you are using a naive
>> LIMIT/OFFSET approach, it takes awhile if you are working with a large
>> OFFSET. It’s better to SELECT into windows of data, where you can specify a
>> start and end criteria (against an indexed column) for each window, like a
>> timestamp.
>>
>> Then of course, using Core only is another level of fastness/low memory.
>> Though querying for individual columns with ORM is not far off, and I’ve
>> also made some major improvements to that in 1.0 so that query(*cols) is
>> pretty competitive with straight Core (and Core is…well I’d say becoming
>> visible in raw DBAPI’s rear view mirror, at least….).
>>
>> What I’d suggest here is that we start to be mindful of memory/performance
>> patterns and start to work out naive ORM use into more savvy patterns; being
>> aware of what columns are needed, what rows, how many SQL queries we really
>> need to emit, what the “worst case” number of rows will be for sections that
>> really need to scale. By far the hardest part is recognizing and
>> reimplementing when something might have to deal with an arbitrarily large
>> number of rows, which means organizing that code to deal with a “streaming”
>> pattern where you never have all the rows in memory at once - on other
>> projects I’ve had tasks that would normally take about a day, but in order
>> to organize it to “scale”, took weeks - such as being able to write out a 1G
>> XML file from a database (yes, actual use case - not only do you have to
>> stream your database data, but you also have to stream out your DOM nodes
>> for which I had to write some fancy SAX extensions).
>>
>> I know that using the ORM makes SQL development “easy”, and so many anti-ORM
>> articles insist that this lulls us all into not worrying about what is
>> actually going on (as much as SQLAlchemy eschews that way of working)…but I
>> remain optimistic that it *is* possible to use tools that save a vast amount
>> of effort, code verbosity and inconsistency that results from doing
>> everything “by hand”, while at the same time not losing our ability to
>> understand how we’re talking to the database. It’s a cake and eat it too,
>> situation, I know.
>>
>> This is already what I’m here to contribute on, I’ve been working out some
>> new SQLAlchemy patterns that hopefully will help, but in the coming weeks I
>> may try to find time to spot some more of these particular things within
>> current Nova code without getting too much into a total rewrite as of yet.
>>
>>
>>
>>
>>
>> On Sep 8, 2014, at 6:24 PM, Joe Gordon <joe.gord...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> We have recently started seeing assorted memory issues in the gate
>>> including the oom-killer [0] and libvirt throwing memory errors [1].
>>> Luckily we run ps and dstat on every devstack run so we have some insight
>>> into why we are running out of memory. Based on the output from job taken
>>> at random [2][3] a typical run consists of:
>>>
>>> * 68 openstack api processes alone
>>> * the following services are running 8 processes (number of CPUs on test
>>> nodes)
>>> * nova-api (we actually run 24 of these, 8 compute, 8 EC2, 8 metadata)
>>> * nova-conductor
>>> * cinder-api
>>> * glance-api
>>> * trove-api
>>> * glance-registry
>>> * trove-conductor
>>> * together nova-api, nova-conductor, cinder-api alone take over 45 %MEM
>>> (note: some of that is memory usage is counted multiple times as RSS
>>> includes shared libraries)
>>> * based on dstat numbers, it looks like we don't use that much memory
>>> before tempest runs, and after tempest runs we use a lot of memory.
>>>
>>> Based on this information I have two categories of questions:
>>>
>>> 1) Should we explicitly set the number of workers that services use in
>>> devstack? Why have so many workers in a small all-in-one environment? What
>>> is the right balance here?
>>>
>>> 2) Should we be worried that some OpenStack services such as nova-api,
>>> nova-conductor and cinder-api take up so much memory? Does there memory
>>> usage keep growing over time, does anyone have any numbers to answer this?
>>> Why do these processes take up so much memory?
>>>
>>> best,
>>> Joe
>>>
>>>
>>> [0]
>>> http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwib29tLWtpbGxlclwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiIxNzI4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDEwMjExMjA5NzY3fQ==
>>> [1] https://bugs.launchpad.net/nova/+bug/1366931
>>> [2] http://paste.openstack.org/show/108458/
>>> [3]
>>> http://logs.openstack.org/83/119183/4/check/check-tempest-dsvm-full/ea576e7/logs/screen-dstat.txt.gz
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
