On 12/18/2013 05:57 PM, Tim Simpson wrote:
python:
consumes 12.5MB of virt memory and 4.3MB of resident memory.
Very few Python only processes will take up such a small amount of virtual 
memory unless the authors are disciplined about not pulling in any dependencies 
at all and writing code in a way that isn't necessarily idiomatic. For an 
example of a project where code is simply written the obvious way, take the 
Trove reference guest, which uses just shy of 200MB of virtual memory and 39MB 
of resident. In defense of the reference guest,  there are some things we can 
do there to make that figure better, I'm certain. I just want to use it as an 
example of how large a Python process can get when the authors proceed doing 
things the way they normally would.

C:
4MB of virt memory and 328k of resident memory
C++:
12.5MB of virt memory and 784k of resident memory
Much of the space you're seeing is from the C++ standard library. Building a 
process normally, I get similar results. However it is also possible to 
statically link the standard libraries and knock off roughly half of that, to 
6.42MB virtual and 400kb resident.

Additionally, the C++ standard library can be omitted if necessary. At this 
point, you might argue that you'd just be writing C code, but even then you'd 
have the advantages of template metaprogramming and other features not present 
in plain C. Even without those features, there's no shame in writing C style 
code assuming you *have* to- C++ was designed to be compatible with C to take 
advantages of its strengths. The only loss would be some of the C99 stuff like 
named initializers.

Additionally, in a vast number of contexts the virtual memory "used" for the 
standard library is not going to matter as other processes will be including that code 
anyway.

Going back to the Trove C++ Agent, it takes 4MB of resident and 28MB of virtual 
memory. This is with some fairly non-trivial dependencies, such as Curl, libz, 
the MySQL and Rabbit client libraries. No special effort was expended making 
sure we kept the process small as in C++ things naturally stay tiny.

C++ is full of fail in a variety of ways and offers no useful advantage for 
something as small as an agent ;-)
If you haven't recently, I recommend you read up on modern C++. The language, 
and how it's written and explained, has changed a lot over the past ten years.
I am intimately familiar with C++. I t believe you missed the constraint "for something as small as an agent" in my response.

Regards
-steve

Thanks,

Tim


-----Original Message-----
From: Steven Dake [mailto:sd...@redhat.com]
Sent: Wednesday, December 18, 2013 4:15 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [trove] My thoughts on the Unified Guest Agent

On 12/18/2013 12:27 PM, Tim Simpson wrote:
Please provide proof of that assumption or at least a general hypothesis that 
we can test.
I can't prove that the new agent will be larger as it doesn't exist yet.

Since nothing was agreed upon anyway, I don't know how you came to that 
conclusion.  I would suggest that any agent framework be held to an extremely 
high standard for footprint for this very reason.
Sorry, I formed a conclusion based on what I'd read so far. There has been talk 
to add Salt to this Unified Agent along with several other things. So I think 
its a valid concern to state that making this thing small is not as high on the 
list of priorities as adding extra functionality.

The C++ agent is just over 3 megabytes of real memory and takes up less than 30 
megabytes  of virtual memory. I don't think an agent has to be *that* small. 
However it won't get near that small unless making it tiny is made a priority, 
and I'm skeptical that's possible while also deciding an agent will be capable 
of interacting with all major OpenStack projects as well as Salt.

Nobody has suggested writing an agent that does everything.
Steven Dake just said:

"A unified agent addresses the downstream viewpoint well, which is 'There is only 
one agent to package and maintain, and it supports all the integrated OpenStack Program 
projects'."

So it sounds like some people are saying there will only be one. Or that it is 
at least an idea.

If Trove's communication method is in fact superior to all others, then perhaps 
we should discuss using that in the unified agent framework.
My point is every project should communicate to an agent in its own interface, 
which can be swapped out for whatever implementations people need.

   In fact I've specifically been arguing to keep it focused on facilitating 
guest<->service communication and limiting its in-guest capabilities to 
narrowly focused tasks.
I like this idea better than creating one agent to rule them all, but I would 
like to avoid forcing a single method of communicating between agents.

Also I'd certainly be interested in hearing about whether or not you think the 
C++ agent could made generic enough for any project to use.
I certainly believe much of the code could be reused for other projects. Right 
now it communicates over RabbitMQ, Oslo RPC style, so I'm not sure how much it 
will fall in line with what the Unified Agent group wants. However, I would 
love to talk more about this. So far my experience has been that no one wants 
to pursue using / developing an agent that was written in C++.

Thanks,

Tim
________________________________________
From: Clint Byrum [cl...@fewbar.com]
Sent: Wednesday, December 18, 2013 11:36 AM
To: openstack-dev
Subject: Re: [openstack-dev] [trove] My thoughts on the Unified Guest
Agent

Excerpts from Tim Simpson's message of 2013-12-18 07:34:14 -0800:
I've been following the Unified Agent mailing list thread for awhile
now and, as someone who has written a fair amount of code for both of
the two existing Trove agents, thought I should give my opinion about
it. I like the idea of a unified agent, but believe that forcing
Trove to adopt this agent for use as its by default will stifle
innovation and harm the project.

"Them's fightin words". ;)

That is a very strong position to take. So I am going to hold your
statements of facts and assumptions to a very high standard below.

There are reasons Trove has more than one agent currently. While
everyone knows about the "Reference Agent" written in Python,
Rackspace uses a different agent written in C++ because it takes up
less memory. The concerns which led to the C++ agent would not be
addressed by a unified agent, which if anything would be larger than
the Reference Agent is currently.

"Would be larger..." - Please provide proof of that assumption or at
least a general hypothesis that we can test. Since nothing was agreed
upon anyway, I don't know how you came to that conclusion. I would
suggest that any agent framework be held to an extremely high standard
for footprint for this very reason.

I also believe a unified agent represents the wrong approach
philosophically. An agent by design needs to be lightweight, capable
of doing exactly what it needs to and no more. This is especially
true for a project like Trove whose goal is to not to provide overly
general PAAS capabilities but simply installation and maintenance of
different datastores. Currently, the Trove daemons handle most logic
and leave the agents themselves to do relatively little. This takes
some effort as many of the first iterations of Trove features have
too much logic put into the guest agents. However through
perseverance the subsequent designs are usually cleaner and simpler
to follow. A community approved, "do everything" agent would endorse
the wrong balance and lead to developers piling up logic on the guest
side. Over time, features would become dependent on the Unified
Agent, making it impossible to run or even contemplate light-weight agents.

Nobody has suggested writing an agent that does everything. A
framework for agents to build on is what has been suggested. In fact
I've specifically been arguing to keep it focused on facilitating
guest<->service communication and limiting its in-guest capabilities
to narrowly focused tasks.

Trove's interface to agents today is fairly loose and could stand to
be made stricter. However, it is flexible and works well enough.
Essentially, the duck typed interface of the trove.guestagent.api.API
class is used to send messages, and Trove conductor is used to
receive them at which point it updates the database. Because both of
these components can be swapped out if necessary, the code could
support the Unified Agent when it appears as well as future agents.

It would be a mistake however to alter Trove's standard method of
communication to please the new Unified Agent. In general, we should
try to keep Trove speaking to guest agents in Trove's terms alone to
prevent bloat.

If Trove's communication method is in fact superior to all others,
then perhaps we should discuss using that in the unified agent framework.

Also I'd certainly be interested in hearing about whether or not you
think the C++ agent could made generic enough for any project to use.
That would be a nice win.
If footprint is the concern, I am certain you would be better off with a C 
implementation.  C++ is full of fail in a variety of ways and offers no useful 
advantage for something as small as an agent ;-)

We wrote our heat agents in python because they are not long-lived processes, 
hence they can bleed into the guest OS memory on demand and are only typically 
used during bootstrapping, vs always taking a chunk off the top as a long-lived 
agent process would do.

Just for comparison sake, the basic footprint of an operation like
sleep(100) (eg the runtime)

python:
consumes 12.5MB of virt memory and 4.3MB of resident memory.

C:
4MB of virt memory and 328k of resident memory

C++:
12.5MB of virt memory and 784k of resident memory

C++ buys 3MB of ram, compared to all the losses of the great
functionality the infrastructure team has put into place.  But C is a real 
winner for footprint, especially since the C library (the virt
part) will almost always be demanded by some application via the dynamic loader.

Regards
-steve

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to