Hi Joe, thank you very much for the positive feedback,
I plan to spend a day during this week on the shedskin-compatibility
for rootwrap (I'll branch it, and tune/cut down as necessary) to make
it compile under shedskin [1] : nothing done yet.
It's a short-term alternative until we can have a rootwrap agent,
together with it's integration in neutron (for Juno). As, for the
compiled rootwrap, if it works, and if it does look good (security wise)
then we'd have a solution for Icehouse/Havana.
help in [1] is really welcome ;-) I'm available in #openstack-neutron
as 'ajo'.
Best regards,
Miguel Ángel.
[1] https://github.com/mangelajo/shedskin.rootwrap
On 03/18/2014 12:48 AM, Joe Gordon wrote:
On Tue, Mar 11, 2014 at 1:46 AM, Miguel Angel Ajo Pelayo
<mangel...@redhat.com <mailto:mangel...@redhat.com>> wrote:
I have included on the etherpad, the option to write a sudo
plugin (or several), specific for openstack.
And this is a test with shedskin, I suppose that in more complicated
dependecy scenarios it should perform better.
[majopela@redcylon tmp]$ cat <<EOF >test.py
> import sys
> print "hello world"
> sys.exit(0)
> EOF
[majopela@redcylon tmp]$ time python test.py
hello world
real 0m0.016s
user 0m0.015s
sys 0m0.001s
This looks very promising!
A few gotchas:
* Very limited library support
https://code.google.com/p/shedskin/wiki/docs#Library_Limitations
* no logging
* no six
* no subprocess
* no *args support
* https://code.google.com/p/shedskin/wiki/docs#Python_Subset_Restrictions
that being said I did a quick comparison with great results:
$ cat tmp.sh
#!/usr/bin/env bash
echo $0 $@
ip a
$ time ./tmp.sh foo bar> /dev/null
real 0m0.009s
user 0m0.003s
sys 0m0.006s
$ cat tmp.py
#!/usr/bin/env python
import os
import sys
print sys.argv
print os.system("ip a")
$ time ./tmp.py foo bar > /dev/null
min:
real 0m0.016s
user 0m0.004s
sys 0m0.012s
max:
real 0m0.038s
user 0m0.016s
sys 0m0.020s
shedskin tmp.py && make
$ time ./tmp foo bar > /dev/null
real 0m0.010s
user 0m0.007s
sys 0m0.002s
Based in these results I think a deeper dive into making rootwrap
supportshedskin is worthwhile.
[majopela@redcylon tmp]$ shedskin test.py
*** SHED SKIN Python-to-C++ Compiler 0.9.4 ***
Copyright 2005-2011 Mark Dufour; License GNU GPL version 3 (See LICENSE)
[analyzing types..]
********************************100%
[generating c++ code..]
[elapsed time: 1.59 seconds]
[majopela@redcylon tmp]$ make
g++ -O2 -march=native -Wno-deprecated -I.
-I/usr/lib/python2.7/site-packages/shedskin/lib /tmp/test.cpp
/usr/lib/python2.7/site-packages/shedskin/lib/sys.cpp
/usr/lib/python2.7/site-packages/shedskin/lib/re.cpp
/usr/lib/python2.7/site-packages/shedskin/lib/builtin.cpp -lgc
-lpcre -o test
[majopela@redcylon tmp]$ time ./test
hello world
real 0m0.003s
user 0m0.000s
sys 0m0.002s
----- Original Message -----
> We had this same issue with the dhcp-agent. Code was added that
paralleled
> the initial sync here: https://review.openstack.org/#/c/28914/
that made
> things a good bit faster if I remember correctly. Might be worth
doing
> something similar for the l3-agent.
>
> Best,
>
> Aaron
>
>
> On Mon, Mar 10, 2014 at 5:07 PM, Joe Gordon <
joe.gord...@gmail.com <mailto:joe.gord...@gmail.com> > wrote:
>
>
>
>
>
>
> On Mon, Mar 10, 2014 at 3:57 PM, Joe Gordon <
joe.gord...@gmail.com <mailto:joe.gord...@gmail.com> > wrote:
>
>
>
> I looked into the python to C options and haven't found anything
promising
> yet.
>
>
> I tried Cython, and RPython, on a trivial hello world app, but
git similar
> startup times to standard python.
>
> The one thing that did work was adding a '-S' when starting python.
>
> -S Disable the import of the module site and the site-dependent
manipulations
> of sys.path that it entails.
>
> Using 'python -S' didn't appear to help in devstack
>
> #!/usr/bin/python -S
> # PBR Generated from u'console_scripts'
>
> import sys
> import site
> site.addsitedir('/mnt/stack/oslo.rootwrap/oslo/rootwrap')
>
>
>
>
>
>
> I am not sure if we can do that for rootwrap.
>
>
> jogo@dev:~/tmp/pypy-2.2.1-src$ time ./tmp-c
> hello world
>
> real 0m0.021s
> user 0m0.000s
> sys 0m0.020s
> jogo@dev:~/tmp/pypy-2.2.1-src$ time ./tmp-c
> hello world
>
> real 0m0.021s
> user 0m0.000s
> sys 0m0.020s
> jogo@dev:~/tmp/pypy-2.2.1-src$ time python -S ./tmp.py
> hello world
>
> real 0m0.010s
> user 0m0.000s
> sys 0m0.008s
>
> jogo@dev:~/tmp/pypy-2.2.1-src$ time python -S ./tmp.py
> hello world
>
> real 0m0.010s
> user 0m0.000s
> sys 0m0.008s
>
>
>
> On Mon, Mar 10, 2014 at 3:26 PM, Miguel Angel Ajo Pelayo <
> mangel...@redhat.com <mailto:mangel...@redhat.com> > wrote:
>
>
> Hi Carl, thank you, good idea.
>
> I started reviewing it, but I will do it more carefully tomorrow
morning.
>
>
>
> ----- Original Message -----
> > All,
> >
> > I was writing down a summary of all of this and decided to just
do it
> > on an etherpad. Will you help me capture the big picture there? I'd
> > like to come up with some actions this week to try to address
at least
> > part of the problem before Icehouse releases.
> >
> > https://etherpad.openstack.org/p/neutron-agent-exec-performance
> >
> > Carl
> >
> > On Mon, Mar 10, 2014 at 5:26 AM, Miguel Angel Ajo <
majop...@redhat.com <mailto:majop...@redhat.com> >
> > wrote:
> > > Hi Yuri & Stephen, thanks a lot for the clarification.
> > >
> > > I'm not familiar with unix domain sockets at low level, but ,
I wonder
> > > if authentication could be achieved just with permissions
(only users in
> > > group "neutron" or group "rootwrap" accessing this service.
> > >
> > > I find it an interesting alternative, to the other proposed
solutions,
> > > but
> > > there are some challenges associated with this solution,
which could make
> > > it
> > > more complicated:
> > >
> > > 1) Access control, file system permission based or token based,
> > >
> > > 2) stdout/stderr/return encapsulation/forwarding to the caller,
> > > if we have a simple/fast RPC mechanism we can use, it's a matter
> > > of serializing a dictionary.
> > >
> > > 3) client side implementation for 1 + 2.
> > >
> > > 4) It would need to accept new domain socket connections in
green threads
> > > to
> > > avoid spawning a new process to handle a new connection.
> > >
> > > The advantages:
> > > * we wouldn't need to break the only-python-rule.
> > > * we don't need to rewrite/translate rootwrap.
> > >
> > > The disadvantages:
> > > * it needs changes on the client side (neutron + other projects),
> > >
> > >
> > > Cheers,
> > > Miguel Ángel.
> > >
> > >
> > >
> > > On 03/08/2014 07:09 AM, Yuriy Taraday wrote:
> > >>
> > >> On Fri, Mar 7, 2014 at 5:41 PM, Stephen Gran
> > >> < stephen.g...@theguardian.com
<mailto:stephen.g...@theguardian.com> <mailto:
stephen.g...@theguardian.com <mailto:stephen.g...@theguardian.com> >>
> > >> wrote:
> > >>
> > >> Hi,
> > >>
> > >> Given that Yuriy says explicitly 'unix socket', I dont think he
> > >> means 'MQ' when he says 'RPC'. I think he just means a daemon
> > >> listening on a unix socket for execution requests. This
seems like
> > >> a reasonably sensible idea to me.
> > >>
> > >>
> > >> Yes, you're right.
> > >>
> > >> On 07/03/14 12:52, Miguel Angel Ajo wrote:
> > >>
> > >>
> > >> I thought of this option, but didn't consider it, as It's
somehow
> > >> risky to expose an RPC end executing priviledged (even filtered)
> > >> commands.
> > >>
> > >>
> > >> subprocess module have some means to do RPC securely over
UNIX sockets.
> > >> I does this by passing some token along with messages. It
should be
> > >> secure because with UNIX sockets we don't need anything
stronger since
> > >> MITM attacks are not possible.
> > >>
> > >> If I'm not wrong, once you have credentials for messaging,
you can
> > >> send messages to any end, even filtered, I somehow see this as a
> > >> higher
> > >> risk option.
> > >>
> > >>
> > >> As Stephen noted, I'm not talking about using MQ for RPC.
Just some
> > >> local UNIX socket with very simple RPC over it.
> > >>
> > >> And btw, if we add RPC in the middle, it's possible that all
those
> > >> system call delays increase, or don't decrease all it'll be
> > >> desirable.
> > >>
> > >>
> > >> Every call to rootwrap would require the following.
> > >>
> > >> Client side:
> > >> - new client socket;
> > >> - one message sent;
> > >> - one message received.
> > >>
> > >> Server side:
> > >> - accepting new connection;
> > >> - one message received;
> > >> - one fork-exec;
> > >> - one message sent.
> > >>
> > >> This looks like way simpler than passing through sudo and
rootwrap that
> > >> requires three exec's and whole lot of configuration files
opened and
> > >> parsed.
> > >>
> > >> --
> > >>
> > >> Kind regards, Yuriy.
> > >>
> > >>
> > >> _______________________________________________
> > >> OpenStack-dev mailing list
> > >> OpenStack-dev@lists.openstack.org
<mailto:OpenStack-dev@lists.openstack.org>
> > >>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>
> > >
> > > _______________________________________________
> > > OpenStack-dev mailing list
> > > OpenStack-dev@lists.openstack.org
<mailto:OpenStack-dev@lists.openstack.org>
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
<mailto:OpenStack-dev@lists.openstack.org>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
<mailto:OpenStack-dev@lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
<mailto:OpenStack-dev@lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
<mailto:OpenStack-dev@lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
