Re: [Python-Dev] Cloning threading.py using proccesses
Josiah Carlson wrote: > Fredrik Lundh <[EMAIL PROTECTED]> wrote: >> Josiah Carlson wrote: >> >>> Presumably with this library you have created, you have also written a >>> fast object encoder/decoder (like marshal or pickle). If it isn't any >>> faster than cPickle or marshal, then users may bypass the module and opt >>> for fork/etc. + XML-RPC >> XML-RPC isn't close to marshal and cPickle in performance, though, so >> that statement is a bit misleading. > > You are correct, it is misleading, and relies on a few unstated > assumptions. > > In my own personal delving into process splitting, RPC, etc., I usually > end up with one of two cases; I need really fast call/return, or I need > not slow call/return. The not slow call/return is (in my opinion) > satisfactorally solved with XML-RPC. But I've personally not been > satisfied with the speed of any remote 'fast call/return' packages, as > they usually rely on cPickle or marshal, which are slow compared to > even moderately fast 100mbit network connections. When we are talking > about local connections, I have even seen cases where the > cPickle/marshal calls can make it so that forking the process is faster > than encoding the input to a called function. This is hard to believe. I've been in that business for a few years and so far have not found an OS/hardware/network combination with the mentioned features. Usually the worst part in performance breakdown for RPC is network latency, ie. time to connect, waiting for the packets to come through, etc. and this parameter doesn't really depend on the OS or hardware you're running the application on, but is more a factor of which network hardware, architecture and structure is being used. It also depends a lot on what you send as arguments, of course, but I assume that you're not pickling a gazillion objects :-) > I've had an idea for a fast object encoder/decoder (with limited support > for certain built-in Python objects), but I haven't gotten around to > actually implementing it as of yet. Would be interesting to look at. BTW, did you know about http://sourceforge.net/projects/py-xmlrpc/ ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 11 2006) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.4.4: backport classobject.c HAVE_WEAKREFS?
Martin v. Löwis wrote:
> Of course, if everybody would always recompile all extension modules
> for a new Python feature release, those flags weren't necessary.
a dynamic registration approach would be even better, with a single entry point
used to register all methods and hooks your C extension has implemented, and
code on the other side that builds a properly initialized type descriptor from
that
set, using fallback functions and error stubs where needed.
e.g. the impossible-to-write-from-scratch NoddyType struct initialization in
http://docs.python.org/ext/node24.html
would collapse to
static PyTypeObject NoddyType;
...
NoddyType = PyType_Setup("noddy.Noddy", sizeof(Noddy));
PyType_Register(NoddyType, PY_TP_DEALLOC, Noddy_dealloc);
PyType_Register(NoddyType, PY_TP_DOC, "Noddy objects");
PyType_Register(NoddyType, PY_TP_TRAVERSE, Noddy_traverse);
PyType_Register(NoddyType, PY_TP_CLEAR, Noddy_clear);
PyType_Register(NoddyType, PY_TP_METHODS, Noddy_methods);
PyType_Register(NoddyType, PY_TP_MEMBERS, Noddy_members);
PyType_Register(NoddyType, PY_TP_INIT, Noddy_init);
PyType_Register(NoddyType, PY_TP_NEW, Noddy_new);
if (PyType_Ready(&NoddyType) < 0)
return;
(a preprocessor that generated this based on suitable "macro decorators" could
be implemented in just over 8 lines of Python...)
with this in place, we could simply remove all those silly NULL checks from the
interpreter.
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.4.4: backport classobject.c HAVE_WEAKREFS?
I wrote: >PyType_Register(NoddyType, PY_TP_METHODS, Noddy_methods); methods and members could of course be registered to, so the implementation can chose how to store them (e.g. short lists for smaller method lists, dictionaries for others). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cloning threading.py using proccesses
"M.-A. Lemburg" <[EMAIL PROTECTED]> wrote:
>
> Josiah Carlson wrote:
> > Fredrik Lundh <[EMAIL PROTECTED]> wrote:
> >> Josiah Carlson wrote:
> >>
> >>> Presumably with this library you have created, you have also written a
> >>> fast object encoder/decoder (like marshal or pickle). If it isn't any
> >>> faster than cPickle or marshal, then users may bypass the module and opt
> >>> for fork/etc. + XML-RPC
> >> XML-RPC isn't close to marshal and cPickle in performance, though, so
> >> that statement is a bit misleading.
> >
> > You are correct, it is misleading, and relies on a few unstated
> > assumptions.
> >
> > In my own personal delving into process splitting, RPC, etc., I usually
> > end up with one of two cases; I need really fast call/return, or I need
> > not slow call/return. The not slow call/return is (in my opinion)
> > satisfactorally solved with XML-RPC. But I've personally not been
> > satisfied with the speed of any remote 'fast call/return' packages, as
> > they usually rely on cPickle or marshal, which are slow compared to
> > even moderately fast 100mbit network connections. When we are talking
> > about local connections, I have even seen cases where the
> > cPickle/marshal calls can make it so that forking the process is faster
> > than encoding the input to a called function.
>
> This is hard to believe. I've been in that business for a few
> years and so far have not found an OS/hardware/network combination
> with the mentioned features.
>
> Usually the worst part in performance breakdown for RPC is network
> latency, ie. time to connect, waiting for the packets to come through,
> etc. and this parameter doesn't really depend on the OS or hardware
> you're running the application on, but is more a factor of which
> network hardware, architecture and structure is being used.
I agree, that is usually the case. But for pre-existing connections
remote or local (whether via socket or unix domain socket), pickling
slows things down significantly. What do I mean? Set up a daemon that
reads and discards what is sent to it as fast as possible. Then start
sending it plain strings (constructed via something like 32768*'\0').
Compare it to a somewhat equivalently sized pickle-as-you-go sender.
Maybe I'm just not doing it right, but I always end up with a slowdown
that makes me want to write my own fast encoder/decoder.
> It also depends a lot on what you send as arguments, of course,
> but I assume that you're not pickling a gazillion objects :-)
According to tests on one of the few non-emulated linux machines I have
my hands on, forking to a child process runs on the order of
.0004-.00055 seconds. On that same machine, pickling...
128*['hello world', 18, {1:2}, 7.382]
...takes ~.0005 seconds. 512 somewhat mixed elements isn't a gazillion,
though in my case, I believe it was originally a list of tuples or
somesuch.
> > I've had an idea for a fast object encoder/decoder (with limited support
> > for certain built-in Python objects), but I haven't gotten around to
> > actually implementing it as of yet.
>
> Would be interesting to look at.
It would basically be something along the lines of cPickle, but would
only support the basic types of: int, long, float, str, unicode, tuple,
list, dictionary.
> BTW, did you know about http://sourceforge.net/projects/py-xmlrpc/ ?
I did not know about it. But it looks interesting. I'll have to
compile it for my (ancient) 2.3 installation and see how it does. Thank
you for the pointer.
- Josiah
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cloning threading.py using proccesses
"Richard Oudkerk" <[EMAIL PROTECTED]> wrote:
> On 10/10/06, Josiah Carlson <[EMAIL PROTECTED]> wrote:
> > > the really interesting thing here is a ready-made threading-style API, I
> > > think. reimplementing queues, locks, and semaphores can be a reasonable
> > > amount of work; might as well use an existing implementation.
> >
> > Really, it is a matter of asking what kind of API is desireable. Do we
> > want to have threading plus other stuff be the style of API that we want
> > to replicate? Do we want to have shared queue objects, or would an
> > XML-RPC-esque remote.queue_put('queue_X', value) and
> > remote.queue_get('queue_X', blocking=1) be better?
>
> Whatever the API is, I think it is useful if you can swap between
> threads and processes just by changing the import line. That way you
> can write applications without deciding upfront which to use.
It would be convenient, yes, but the question isn't always 'threads or
processes?' In my experience (not to say that it is more or better than
anyone else's), when going multi-process, the expense on some platforms
is significant enough to want to persist the process (this is counter to
my previous forking statement, but its all relative). And sometimes one
*wants* multiple threads running in a single process handling multiple
requests.
There's a recipe hanging out in the Python cookbook that adds a
threading mixin to the standard XML-RPC server in Python. For a set of
processes (perhaps on different machines) that are cooperating and
calling amongst each other, I've not seen a significantly better variant,
especially when the remote procedure call can take a long time to
complete. It does take a few tricks to make sure that sufficient
connections are available from process A to process B when A calls B
from multiple threads, but its not bad.
- Josiah
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cloning threading.py using proccesses
Josiah Carlson wrote: > It would basically be something along the lines of cPickle, but would > only support the basic types of: int, long, float, str, unicode, tuple, > list, dictionary. if you're aware of a way to do that faster than the current marshal implementation, maybe you could work on speeding up marshal instead? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cloning threading.py using proccesses
Josiah> It would basically be something along the lines of cPickle, but Josiah> would only support the basic types of: int, long, float, str, Josiah> unicode, tuple, list, dictionary. Isn't that approximately marshal's territory? If you can write a faster encoder/decoder, it might well be possible to apply the speedup ideas to marshal. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.4.4: backport classobject.c HAVE_WEAKREFS?
On 10/11/06, Fredrik Lundh <[EMAIL PROTECTED]> wrote:
Martin v. Löwis wrote:> Of course, if everybody would always recompile all extension modules> for a new Python feature release, those flags weren't necessary.a dynamic registration approach would be even better, with a single entry point
used to register all methods and hooks your C extension has implemented, andcode on the other side that builds a properly initialized type descriptor from thatset, using fallback functions and error stubs where needed.
e.g. the impossible-to-write-from-scratch NoddyType struct initialization inhttp://docs.python.org/ext/node24.htmlwould collapse tostatic PyTypeObject NoddyType;
...NoddyType = PyType_Setup("noddy.Noddy", sizeof(Noddy));PyType_Register(NoddyType, PY_TP_DEALLOC, Noddy_dealloc);PyType_Register(NoddyType, PY_TP_DOC, "Noddy objects");
PyType_Register(NoddyType, PY_TP_TRAVERSE, Noddy_traverse);PyType_Register(NoddyType, PY_TP_CLEAR, Noddy_clear);PyType_Register(NoddyType, PY_TP_METHODS, Noddy_methods);PyType_Register(NoddyType, PY_TP_MEMBERS, Noddy_members);
PyType_Register(NoddyType, PY_TP_INIT, Noddy_init);PyType_Register(NoddyType, PY_TP_NEW, Noddy_new);if (PyType_Ready(&NoddyType) < 0)return;(a preprocessor that generated this based on suitable "macro decorators" could
be implemented in just over 8 lines of Python...)with this in place, we could simply remove all those silly NULL checks from theinterpreter.This is also has the benefit of making it really easy to grep for the function used for the tp_init field since there is no guarantee someone will keep the traditional field comments in their file (I usually grep for PyTypeObject until I find the type I want).
If we went with C99 this wouldn't be an issue, but since I don't think that is necessarily in the cards I am very happy to go with this solution. It ends up feeling more like how Ruby does C extensions, and I have to admit I think they may have made it simpler than we have.
And of course with the change Raymond put in for checking the PyMethodDef slots can also easily allow people to name methods based on what the slots would be named had it been defined in Python (which we might want to do anyway with the C constants to make it more readable and less obtuse to new extension writers;
e.g. change PY_TP_NEW to PY__NEW__).And lastly, this approach makes sure that the basic requirement of what a type must have defined can be enforced in the PyType_Setup() method /F's proposing.+1 from me.
-Brett
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cloning threading.py using proccesses
Fredrik Lundh <[EMAIL PROTECTED]> wrote:
> Josiah Carlson wrote:
>
> > It would basically be something along the lines of cPickle, but would
> > only support the basic types of: int, long, float, str, unicode, tuple,
> > list, dictionary.
>
> if you're aware of a way to do that faster than the current marshal
> implementation, maybe you could work on speeding up marshal instead?
The current implementation uses a fixed resize semantic (1024 bytes at a
time) that makes large marshal operations slow. If we were to switch to
a list resize-like or cStringIO semantic (overallocate by ~size>>3,
or at least double, respectively), it would likely increase the speed
for large resize operations. (see the w_more definition) This should
make it significantly faster than cPickle in basically all cases.
w_object uses a giant if/else if block to handle all of the possible
cases, both for identity checks against None, False, True, etc., as well
as with the various Py*_Check(). This is necessary due to marshal
supporting subclasses (the Py*_Check() calls) and the dynamic layout of
memory during Python startup. The identity checks may be able to be
replaced with a small array-like thing if we were to statically allocate
them from a single array to guarantee that their addresses are a fixed
distance apart...
char base_objects[320];
PyObject* IDENTITY[8];
int cases[8];
/*
64 bytes per object is overkill, and we may want to allocate enough room
for 15 objects, to make sure that IDENTITY[0] = NULL;
*/
p = 0
for obj_init in objs_to_init:
init_object(base_objects+p, obj_init)
x = ((base_objects+p)>>6)&7
IDENTITY[x] = (PyObject*)(base_objects+p)
cases[x] = p//64
p += 64
Then we could use the following in w_object...
x = (v>>6)&7
if v == IDENTITY[x] {
switch (cases[x]) {
case 0: /* should be null */
...
case 1: /* based on the order of objs_to_init */
}
}
The Py*_Check() stuff isn't so amenable to potential speedup, but in a
custom no-subclasses only base types version, we ccould use a variant of
the above mechanism to look directly at types, then use a second
switch/case statement, which should be significantly faster than the
if/else if tests that it currently uses. An identity check, then a fast
type check, otherwise fail.
- Josiah
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cloning threading.py using proccesses
On 10/12/06, Josiah Carlson <[EMAIL PROTECTED]> wrote:
>
> It would basically be something along the lines of cPickle, but would
> only support the basic types of: int, long, float, str, unicode, tuple,
> list, dictionary.
>
Great idea! Check this thread for past efforts:
http://mail.python.org/pipermail/python-dev/2005-June/054313.html
The 'gherkin' module discussed there now lives in the cheeseshop as
part of the FibraNet package.
http://cheeseshop.python.org/pypi/FibraNet
I love benchmarks, especially when they come around for the second time.
I wrote a silly script which compares dumps performance between
different serialization modules for different simple objects using
Python 2.4.3. All figures are 'dumps per second'.
test: a tuple: ("a" * 1024,1.0,[1,2,3],{'1':2,'3':4})
gherkin: 10895.7762314
pickle: 6510.97245984
cPickle: 34218.5455317
marshal: 85562.2443672
xmlrpclib: 9468.0766772
test: a large string: 'a' * 10240
gherkin: 45955.4065455
pickle: 10209.0239868
cPickle: 13773.8138516
marshal: 24937.002069
xmlrpclib: Traceback
test: a small string: 'a' * 128
gherkin: 73453.0960495
pickle: 28357.0210654
cPickle: 122997.592425
marshal: 202428.776201
xmlrpclib: Traceback
test: a tupe of ints: tuple(range(64))
gherkin: 4522.06801154
pickle: 2273.12937965
cPickle: 23969.9306043
marshal: 143691.72582
xmlrpclib: 2781.3083894
Marshal is very quick for most cases, but still has this warning in
the documentation.
"""Warning: The marshal module is not intended to be secure against
erroneous or maliciously constructed data. Never unmarshal data
received from an untrusted or unauthenticated source."""
-Sw
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cloning threading.py using proccesses
Fredrik Lundh wrote: > if you're aware of a way to do that faster than the current marshal > implementation, maybe you could work on speeding up marshal instead? Even if it weren't faster than marshal, it could still be useful to have something nearly as fast that used a python-version-independent protocol. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cloning threading.py using proccesses
Greg Ewing wrote:
>> if you're aware of a way to do that faster than the current marshal
>> implementation, maybe you could work on speeding up marshal instead?
>
> Even if it weren't faster than marshal, it could still
> be useful to have something nearly as fast that used
> a python-version-independent protocol.
marshal hasn't changed in many years:
$ python1.5
>>> x = 1, 2.0, "three", [4, 5, 6, "seven"], {8: 9}, None
>>> import marshal
>>> marshal.dump(x, open("x.dat", "w"))
>>>
$ python2.5
>>> import marshal
>>> marshal.load(open("x.dat"))
(1, 2.0, 'three', [4, 5, 6, 'seven'], {8: 9}, None)
which is a good thing, because there are external non-Python tools that
generate marshalled data streams.
maybe you were thinking about marshalled code objects?
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
