[Python-Dev] Releasing 2.5.4
It seems r67740 shouldn't have been committed. Since this is a severe regression, I think I'll have to revert it, and release 2.5.4 with just that change. Unless I hear otherwise, I would release Python 2.5.4 (without a release candidate) tomorrow. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On 2008-12-20 23:16, Martin v. Löwis wrote: >>> I will try next week to see if I can come up with a smaller, >>> submittable example. Thanks. >> These long exit times are usually caused by the garbage collection >> of objects. This can be a very time consuming task. > > I doubt that. The long exit times are usually caused by a bad > malloc implementation. With "garbage collection" I meant the process of Py_DECREF'ing the objects in large containers or deeply nested structures, not the GC mechanism for breaking circular references in Python. This will usually also involve free() calls, so the malloc implementation affects this as well. However, I've seen such long exit times on Linux and Windows, which both have rather good malloc implementations. I don't think there's anything much we can do about it at the interpreter level. Deleting millions of objects takes time and that's not really surprising at all. It takes even longer if you have instances with .__del__() methods written in Python. Applications can choose other mechanisms for speeding up the exit process in various (less clean) ways, if they have a need for this. BTW: Rather than using a huge in-memory dict, I'd suggest to either use an on-disk dictionary such as the ones found in mxBeeBase or a database. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 22 2008) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2008-12-02: Released mxODBC.Connect 1.0.0 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [capi-sig] Exceptions with additional instance variables
On Mon, Dec 22, 2008 at 10:06 AM, wrote:
> On Mon, Dec 22, 2008 at 03:29, Guilherme Polo wrote:
>> On Sun, Dec 21, 2008 at 11:02 PM, wrote:
>>> Hello,
>>>
>>> I'm trying to implement custom exception that have to carry some
>>> useful info by means of instance members, to be used like:
>>>
>>> try:
>>>// some code
>>> except MyException, data:
>>>// use data.errorcode, data.errorcategory, data.errorlevel,
>>> data.errormessage and some others
>>>
>>> The question is - how to implement the instance variables with
>>> PyErr_NewException?
>>
>> Using PyErr_NewException is fine. You must understand that an
>> exception is a class, and thus PyErr_NewException creates one for you
>> and returns it.
>> Just like you would do with a class that has __dict__, set some
>> attributes to what you want. That is, use PyObject_SetAttrString or
>> something more appropriated for you.
>
> Ok so I did the following. In init function (forget refcounting and
> error checking for a moment ;-)
>
> PyObject *dict = PyDict_New();
> PyDict_SetItemString(dict, "errorcode", PyInt_FromLong(0));
> static PyObject *myexception =
> PyErr_NewException("module.MyException", NULL, dict);
You do not really have to create a dict here, one will be created for
you if you pass a NULL there.
> PyModule_AddObject(module, "MyException", myexception);
>
> It worked more or less as expected, the help shown:
>
> | --
> | Data and other attributes defined here:
> |
> | errorcode = 0
> |
> | --
>
> Then I did the following when raising the exception:
>
> PyObject_SetAttrString(myexception, "errorcode", PyInt_FromLong(111));
> PyErr_SetString(myexception, "Bad thing happened");
> return NULL;
>
> and the test code was:
> try:
>do_bad_thing();
> except MyException, data:
>
> and you surely already guessed it -- data.errorcode was 0 Not only
> that, module.MyException.errorcode was also 0...
>
> What I'm doing wrong? I certainly don't get the idea of exceptions in
> Python, especially what is being raised - a class or an instance?
There are two forms raise can take, both will end up involving a class
and a intsance.
> If
> the latter - how's the class instantiated?
You can call a class to instantiate it.
> If not - what about values
> in different threads? The docs are so vague about that...
>
>
> Thanks again in advance,
> Chojrak
>
Again, an exception is a class, so you could create a new type in C,
and do anything you wanted. But you probably don't want to create a
new type to achieve this, so there are two simple ways I'm going to
paste below:
#include "Python.h"
static PyObject *MyErr;
static PyMethodDef module_methods[] = {
{"raise_test", (PyCFunction)raise_test, METH_NOARGS, NULL},
{NULL},
};
PyMODINIT_FUNC
initfancy_exc(void)
{
PyObject *m;
m = Py_InitModule("fancy_exc", module_methods);
if (m == NULL)
return;
MyErr = PyErr_NewException("fancy_exc.err", NULL, NULL);
Py_INCREF(MyErr);
if (PyModule_AddObject(m, "err", MyErr) < 0)
return;
}
the raise_test function is missing, pick one of these:
static PyObject *
raise_test(PyObject *self)
{
PyObject_SetAttrString(MyErr, "code", PyInt_FromLong(42));
PyObject_SetAttrString(MyErr, "category", PyString_FromString("nice
one"));
PyErr_SetString(MyErr, "All is good, I hope");
return NULL;
}
or
static PyObject *
raise_test(PyObject *self)
{
PyObject *t = PyTuple_New(3);
PyTuple_SetItem(t, 0, PyString_FromString("error message"));
PyTuple_SetItem(t, 1, PyInt_FromLong(10));
PyTuple_SetItem(t, 2, PyString_FromString("category name here"));
PyErr_SetObject(MyErr, t);
Py_DECREF(t);
return NULL;
}
In this second form you check for the args attribute of the exception.
--
-- Guilherme H. Polo Goncalves
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [capi-sig] Exceptions with additional instance variables
On Mon, Dec 22, 2008 at 10:45 AM, Guilherme Polo wrote:
> On Mon, Dec 22, 2008 at 10:06 AM, wrote:
>> On Mon, Dec 22, 2008 at 03:29, Guilherme Polo wrote:
>>> On Sun, Dec 21, 2008 at 11:02 PM, wrote:
Hello,
I'm trying to implement custom exception that have to carry some
useful info by means of instance members, to be used like:
try:
// some code
except MyException, data:
// use data.errorcode, data.errorcategory, data.errorlevel,
data.errormessage and some others
The question is - how to implement the instance variables with
PyErr_NewException?
>>>
>>> Using PyErr_NewException is fine. You must understand that an
>>> exception is a class, and thus PyErr_NewException creates one for you
>>> and returns it.
>>> Just like you would do with a class that has __dict__, set some
>>> attributes to what you want. That is, use PyObject_SetAttrString or
>>> something more appropriated for you.
>>
>> Ok so I did the following. In init function (forget refcounting and
>> error checking for a moment ;-)
>>
>> PyObject *dict = PyDict_New();
>> PyDict_SetItemString(dict, "errorcode", PyInt_FromLong(0));
>> static PyObject *myexception =
>> PyErr_NewException("module.MyException", NULL, dict);
>
> You do not really have to create a dict here, one will be created for
> you if you pass a NULL there.
>
>> PyModule_AddObject(module, "MyException", myexception);
>>
>> It worked more or less as expected, the help shown:
>>
>> | --
>> | Data and other attributes defined here:
>> |
>> | errorcode = 0
>> |
>> | --
>>
>> Then I did the following when raising the exception:
>>
>> PyObject_SetAttrString(myexception, "errorcode", PyInt_FromLong(111));
>> PyErr_SetString(myexception, "Bad thing happened");
>> return NULL;
>>
>> and the test code was:
>> try:
>>do_bad_thing();
>> except MyException, data:
>>
>> and you surely already guessed it -- data.errorcode was 0 Not only
>> that, module.MyException.errorcode was also 0...
>>
>> What I'm doing wrong? I certainly don't get the idea of exceptions in
>> Python, especially what is being raised - a class or an instance?
>
> There are two forms raise can take, both will end up involving a class
> and a intsance.
>
>> If
>> the latter - how's the class instantiated?
>
> You can call a class to instantiate it.
>
>> If not - what about values
>> in different threads? The docs are so vague about that...
>>
>>
>> Thanks again in advance,
>> Chojrak
>>
>
> Again, an exception is a class, so you could create a new type in C,
> and do anything you wanted. But you probably don't want to create a
> new type to achieve this
By creating a type I mean one that involves defining a tp_init, and
everything else your type needs, not about the simple one created by
PyErr_NewException.
> , so there are two simple ways I'm going to
> paste below:
>
> #include "Python.h"
>
> static PyObject *MyErr;
>
> static PyMethodDef module_methods[] = {
>{"raise_test", (PyCFunction)raise_test, METH_NOARGS, NULL},
>{NULL},
> };
>
> PyMODINIT_FUNC
> initfancy_exc(void)
> {
>PyObject *m;
>
>m = Py_InitModule("fancy_exc", module_methods);
>if (m == NULL)
>return;
>
>MyErr = PyErr_NewException("fancy_exc.err", NULL, NULL);
>
>Py_INCREF(MyErr);
>if (PyModule_AddObject(m, "err", MyErr) < 0)
>return;
> }
>
> the raise_test function is missing, pick one of these:
>
> static PyObject *
> raise_test(PyObject *self)
> {
>PyObject_SetAttrString(MyErr, "code", PyInt_FromLong(42));
>PyObject_SetAttrString(MyErr, "category", PyString_FromString("nice
> one"));
>PyErr_SetString(MyErr, "All is good, I hope");
>return NULL;
> }
>
> or
>
> static PyObject *
> raise_test(PyObject *self)
> {
>
>PyObject *t = PyTuple_New(3);
>PyTuple_SetItem(t, 0, PyString_FromString("error message"));
>PyTuple_SetItem(t, 1, PyInt_FromLong(10));
>PyTuple_SetItem(t, 2, PyString_FromString("category name here"));
>PyErr_SetObject(MyErr, t);
>Py_DECREF(t);
>return NULL;
> }
>
> In this second form you check for the args attribute of the exception.
>
> --
> -- Guilherme H. Polo Goncalves
>
--
-- Guilherme H. Polo Goncalves
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Releasing 2.5.4
Martin> It seems r67740 shouldn't have been committed. Since this is a Martin> severe regression, I think I'll have to revert it, and release Martin> 2.5.4 with just that change. Martin> Unless I hear otherwise, I would release Python 2.5.4 (without a Martin> release candidate) tomorrow. I don't think there is a test case which fails with it applied and passes with it removed. If not, I think it might be worthwhile to write such a test even if it's used temporarily just to test the change. I wrote a trivial test case: Index: Lib/test/test_file.py === --- Lib/test/test_file.py (revision 67899) +++ Lib/test/test_file.py (working copy) @@ -116,6 +116,8 @@ except: self.assertEquals(self.f.__exit__(*sys.exc_info()), None) +def testReadWhenWriting(self): +self.assertRaises(IOError, self.f.read) class OtherFileTests(unittest.TestCase): which segfaults (on Solaris 10 at least) when run with the 2.5.3 released code and which passes after I undo r67740. Should we add this to the active branches (2.6, trunk, py3k, 3.0)? Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Releasing 2.5.4
> Should we add this to the active branches (2.6, trunk, py3k, 3.0)? Sure! Go ahead. For 2.5.3, I'd rather not add an additional test case, but merely revert the patch. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Releasing 2.5.4
On Dec 22, 2008, at 9:35 AM, [email protected] wrote: I don't think there is a test case which fails with it applied and passes with it removed. If not, I think it might be worthwhile to write such a test even if it's used temporarily just to test the change. I wrote a trivial test case: If this is sufficient to drive a release, then whatever test there is should be part of the release as well. -Fred -- Fred Drake ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.0.1
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Dec 19, 2008, at 9:44 PM, Martin v. Löwis wrote: Do you think we can get 3.0.1 out on December 24th? I won't have physical access to my build machine from December 24th to January 3rd. Okay. Let's just push it until after the new year then. In the mean time, please continue to work on fixes for 3.0.1. I'm thinking tentatively to do a release the week of January 5th. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSU+9IHEjvBPtnXfVAQL0vQQAmxcMDP1GUuhCOxCVHqnSGaywdG1mz3f0 iNCNs4lVsRLYV/AVdf/tbpWyLbcUvFL0hUyLDp8PCScOjZReKwe6VpnujL/BwU5E 4P7RtUn493QGqkFJDjHNJ2SIcxOfzk9Y7E3qyS0QHPmsqmNpSD6ZQQd0PkdCoqQo f08Z9HrKZZw= =ujaK -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.0.1
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Dec 21, 2008, at 6:56 AM, Dmitry Vasiliev wrote: Barry Warsaw wrote: Thanks. I've bumped that to release blocker for now. If there are any other 'high' bugs that you want considered for 3.0.1, please make the release blockers too, for now. I think wsgiref package needs to be fixed. For now it's totally broken. I've already found 4 issues about that: http://bugs.python.org/issue3348 http://bugs.python.org/issue3401 http://bugs.python.org/issue3795 http://bugs.python.org/issue4522 Please make sure these issues are release blockers. Fixes before January 5th would be able to make it into 3.0.1. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSU+9U3EjvBPtnXfVAQII5wP+M9tyL169XMIwoibqupyPErAjHNL+zWD1 wydak1MKc/gF6KvSFfs9t6uuI3p8GI42dNxeHXIXsCb1he16YfUgu7xG210ZJ9C3 YkDcr6vDDMYUvMI8XdVJGh9ASnQhrQRiyMI/TtiJTh16t3wnn78EH2F2IyrYcDrD 0xaKQjaK1+k= =t6EL -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.0.1
Barry Warsaw python.org> writes: > > Please make sure these issues are release blockers. Fixes before > January 5th would be able to make it into 3.0.1. Should http://bugs.python.org/issue4486 be a release blocker as well? (I don't think so, but...) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.0.1
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Dec 22, 2008, at 11:38 AM, Antoine Pitrou wrote: Barry Warsaw python.org> writes: Please make sure these issues are release blockers. Fixes before January 5th would be able to make it into 3.0.1. Should http://bugs.python.org/issue4486 be a release blocker as well? (I don't think so, but...) I don't think so either. It would be nice to have, but it needn't hold up the release. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSU/HgnEjvBPtnXfVAQKzcAP+NThqngryODxF/bKpeMs/EhpjfI9HV4eC Lul5LMocaxEe91ontMjhfnZQo6Tx/jJCGECzVLCLXVmrjKg7/d6/9TFEByc9OWFm zODpRvQ+4u+jd8c8DcBQmEwuFJF4MQZ5x6SUP8HxRTLmWq1KMcGM5WTNHCxMoOVw Gkg8JmknqjM= =6teE -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
Thanks for all of the useful suggestions. Here are some preliminary results.
With still gc.disable(), at the end of the program I first did a
gc.collect(), which took about five minutes. (So, reason enough not
to gc.enable(), at least without Antoine's patch.)
After that, I did a .clear() on the huge dict. That's where the time
is being spent. Doing the suggested "poor man's profiling" (repeated
backtraces via gdb), for 20 or so samples, one is within libc free,
but all of the rest are in the same place (same source line) within
PyObjectFree (see below), sometimes within list_dealloc and sometimes
within tuple_dealloc. So, apparently a lot of time is being spent in
this loop:
/* Case 3: We have to move the arena towards the end
* of the list, because it has more free pools than
* the arena to its right.
...
/* Locate the new insertion point by iterating over
* the list, using our nextarena pointer.
*/
while (ao->nextarena != NULL &&
nf > ao->nextarena->nfreepools) {
ao->prevarena = ao->nextarena;
ao->nextarena = ao->nextarena->nextarena;
}
Investigating further, from one stop, I used gdb to follow the chain
of pointers in the nextarena and prevarena directions. There were
5449 and 112765 links, respectively. maxarenas is 131072.
Sampling nf at different breaks gives values in the range(10,20).
This loop looks like an insertion sort. If it's the case that only a
"few" iterations are ever needed for any given free, this might be
okay--if not, it would seem that this must be quadratic.
I attempted to look further by setting a silent break with counter
within the loop and another break after the loop to inspect the
counter, but gdb's been buzzing away on that for 40 minutes without
coming back. That might mean that there are a lot of passes through
this loop per free (i.e., that gdb is taking a long time to process
100,000 silent breaks), or perhaps I've made a mistake, or gdb isn't
handling this well.
In any case, this looks like the problem locus.
It's tempting to say "don't do this arena ordering optimization if
we're doing final cleanup", but really the program could have done
this .clear() at any point. Maybe there needs to be a flag to disable
it altogether? Or perhaps there's a smarter way to manage the list of
arena/free pool info.
Mike
Program received signal SIGINT, Interrupt.
0x004461dc in PyObject_Free (p=0x5ec043db0) at Objects/obmalloc.c:1064
1064while (ao->nextarena != NULL &&
(gdb) bt
#0 0x004461dc in PyObject_Free (p=0x5ec043db0) at
Objects/obmalloc.c:1064
#1 0x00433478 in list_dealloc (op=0x5ec043dd0) at
Objects/listobject.c:281
#2 0x0044075b in PyDict_Clear (op=0x74c7cd0) at
Objects/dictobject.c:757
#3 0x004407b9 in dict_clear (mp=0x5ec043db0) at
Objects/dictobject.c:1776
#4 0x00485905 in PyEval_EvalFrameEx (f=0x746ca50,
throwflag=)
at Python/ceval.c:3557
#5 0x0048725f in PyEval_EvalCodeEx (co=0x72643f0,
globals=,
locals=, args=0x1, argcount=0, kws=0x72a5770,
kwcount=0, defs=0x743eba8,
defcount=1, closure=0x0) at Python/ceval.c:2836
#6 0x004855bc in PyEval_EvalFrameEx (f=0x72a55f0,
throwflag=)
at Python/ceval.c:3669
#7 0x0048725f in PyEval_EvalCodeEx (co=0x72644e0,
globals=,
locals=, args=0x0, argcount=0, kws=0x0,
kwcount=0, defs=0x0, defcount=0,
closure=0x0) at Python/ceval.c:2836
#8 0x004872a2 in PyEval_EvalCode (co=0x5ec043db0,
globals=0x543e41f10, locals=0x543b969c0)
at Python/ceval.c:494
#9 0x004a844e in PyRun_FileExFlags (fp=0x7171010,
filename=0x7af6b419
"/home/mkc/greylag/main/greylag_reannotate.py", start=,
globals=0x7194510, locals=0x7194510, closeit=1,
flags=0x7af69080) at Python/pythonrun.c:1273
#10 0x004a86e0 in PyRun_SimpleFileExFlags (fp=0x7171010,
filename=0x7af6b419
"/home/mkc/greylag/main/greylag_reannotate.py", closeit=1,
flags=0x7af69080) at Python/pythonrun.c:879
#11 0x00412275 in Py_Main (argc=,
argv=0x7af691a8) at Modules/main.c:523
#12 0x0030fea1d8b4 in __libc_start_main () from /lib64/libc.so.6
#13 0x00411799 in _start ()
On Sun, Dec 21, 2008 at 12:44 PM, Adam Olsen wrote:
> On Sat, Dec 20, 2008 at 6:09 PM, Mike Coleman wrote:
>> On Sat, Dec 20, 2008 at 5:40 PM, Alexandre Vassalotti
>>> Have you seen any significant difference in the exit time when the
>>> cyclic GC is disabled or enabled?
>>
>> Unfortunately, with GC enabled, the application is too slow to be
>> useful, because of the greatly increased time for dict creation. I
>> suppose it's theoretically possible that with this increased time, the
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On Mon, Dec 22, 2008 at 6:20 AM, M.-A. Lemburg wrote:
> BTW: Rather than using a huge in-memory dict, I'd suggest to either
> use an on-disk dictionary such as the ones found in mxBeeBase or
> a database.
I really want this to work in-memory. I have 64G RAM, and I'm only
trying to use 45G of it ("only" 45G :-), and I don't need the results
to persist after the program finishes.
Python should be able to do this. I don't want to hear "Just use Perl
instead" from my co-workers... ;-)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On Mon, Dec 22, 2008 at 11:01 AM, Mike Coleman wrote:
> Thanks for all of the useful suggestions. Here are some preliminary results.
>
> With still gc.disable(), at the end of the program I first did a
> gc.collect(), which took about five minutes. (So, reason enough not
> to gc.enable(), at least without Antoine's patch.)
>
> After that, I did a .clear() on the huge dict. That's where the time
> is being spent. Doing the suggested "poor man's profiling" (repeated
> backtraces via gdb), for 20 or so samples, one is within libc free,
> but all of the rest are in the same place (same source line) within
> PyObjectFree (see below), sometimes within list_dealloc and sometimes
> within tuple_dealloc. So, apparently a lot of time is being spent in
> this loop:
>
>
>/* Case 3: We have to move the arena towards the end
> * of the list, because it has more free pools than
> * the arena to its right.
>
> ...
>
>/* Locate the new insertion point by iterating over
> * the list, using our nextarena pointer.
> */
>while (ao->nextarena != NULL &&
>nf > ao->nextarena->nfreepools) {
>ao->prevarena = ao->nextarena;
>ao->nextarena = ao->nextarena->nextarena;
>}
>
> Investigating further, from one stop, I used gdb to follow the chain
> of pointers in the nextarena and prevarena directions. There were
> 5449 and 112765 links, respectively. maxarenas is 131072.
>
> Sampling nf at different breaks gives values in the range(10,20).
>
> This loop looks like an insertion sort. If it's the case that only a
> "few" iterations are ever needed for any given free, this might be
> okay--if not, it would seem that this must be quadratic.
>
> I attempted to look further by setting a silent break with counter
> within the loop and another break after the loop to inspect the
> counter, but gdb's been buzzing away on that for 40 minutes without
> coming back. That might mean that there are a lot of passes through
> this loop per free (i.e., that gdb is taking a long time to process
> 100,000 silent breaks), or perhaps I've made a mistake, or gdb isn't
> handling this well.
To make sure that's the correct line please recompile python without
optimizations. GCC happily reorders and merges different parts of a
function.
Adding a counter in C and recompiling would be a lot faster than using
a gdb hook.
--
Adam Olsen, aka Rhamphoryncus
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
> Or perhaps there's a smarter way to manage the list of > arena/free pool info. If that code is the real problem (in a reproducible test case), then this approach is the only acceptable solution. Disabling long-running code is not acceptable. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On Mon, Dec 22, 2008 at 2:38 PM, "Martin v. Löwis" wrote: >> Or perhaps there's a smarter way to manage the list of >> arena/free pool info. > > If that code is the real problem (in a reproducible test case), > then this approach is the only acceptable solution. Disabling > long-running code is not acceptable. By "disabling", I meant disabling the optimization that's trying to rearrange the arenas so that more memory can be returned to the OS. This presumably wouldn't be any worse than things were in Python 2.4, when memory was never returned to the OS. (I'm working on a test case.) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On Dec 22, 2008, at 1:13 PM, Mike Coleman wrote:
On Mon, Dec 22, 2008 at 6:20 AM, M.-A. Lemburg wrote:
BTW: Rather than using a huge in-memory dict, I'd suggest to either
use an on-disk dictionary such as the ones found in mxBeeBase or
a database.
I really want this to work in-memory. I have 64G RAM, and I'm only
trying to use 45G of it ("only" 45G :-), and I don't need the results
to persist after the program finishes.
It's still not clear to me, from reading the whole thread, precisely
what you're seeing. A self-contained test case, preferably with
generated random data, would be great, and save everyone a lot of
investigation time. In the meantime, can you 1) turn off all swap
files and partitions, and 2) confirm positively that your CPU cycles
are burning up in userland?
(In general, unless you know exactly why your workload needs swap, and
have written your program to take swapping into account, having _any_
swap on a machine with 64GB RAM is lunacy. The machine will grind to a
complete standstill long before filling up gigabytes of swap.)
--
Ivan Krstić | http://radian.org
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On 2008-12-22 19:13, Mike Coleman wrote:
> On Mon, Dec 22, 2008 at 6:20 AM, M.-A. Lemburg wrote:
>> BTW: Rather than using a huge in-memory dict, I'd suggest to either
>> use an on-disk dictionary such as the ones found in mxBeeBase or
>> a database.
>
> I really want this to work in-memory. I have 64G RAM, and I'm only
> trying to use 45G of it ("only" 45G :-), and I don't need the results
> to persist after the program finishes.
>
> Python should be able to do this. I don't want to hear "Just use Perl
> instead" from my co-workers... ;-)
What kinds of objects are you storing in your dictionary ? Python
instances, strings, integers ?
The time it takes to deallocate the objects in your dictionary
depends a lot on the types you are using.
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Dec 22 2008)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
2008-12-02: Released mxODBC.Connect 1.0.0 http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free !
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
>> If that code is the real problem (in a reproducible test case), >> then this approach is the only acceptable solution. Disabling >> long-running code is not acceptable. > > By "disabling", I meant disabling the optimization that's trying to > rearrange the arenas so that more memory can be returned to the OS. I meant the same thing - I'm opposed to giving up one feature or optimization in favor of a different feature or optimization. > This presumably wouldn't be any worse than things were in Python 2.4, > when memory was never returned to the OS. Going back to the state of Python 2.4 would not be acceptable. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [capi-sig] Exceptions with additional instance variables
2008/12/22 Guilherme Polo :
> On Mon, Dec 22, 2008 at 10:06 AM, wrote:
>
> #include "Python.h"
>
> static PyObject *MyErr;
>
> static PyMethodDef module_methods[] = {
>{"raise_test1", (PyCFunction)raise_test1, METH_NOARGS, NULL},
>{"raise_test2", (PyCFunction)raise_test2, METH_NOARGS, NULL},
>{"raise_test3", (PyCFunction)raise_test3, METH_NOARGS, NULL},
>{NULL},
> };
>
> PyMODINIT_FUNC
> initfancy_exc(void)
> {
>PyObject *m;
>
>m = Py_InitModule("fancy_exc", module_methods);
>if (m == NULL)
>return;
>
>MyErr = PyErr_NewException("fancy_exc.err", NULL, NULL);
>
>Py_INCREF(MyErr);
>if (PyModule_AddObject(m, "err", MyErr) < 0)
>return;
> }
>
> static PyObject *
> raise_test1(PyObject *self)
> {
>PyObject_SetAttrString(MyErr, "code", PyInt_FromLong(42));
>PyObject_SetAttrString(MyErr, "category", PyString_FromString("nice
> one"));
>PyErr_SetString(MyErr, "All is good, I hope");
>return NULL;
> }
>
> static PyObject *
> raise_test2(PyObject *self)
> {
>
>PyObject *t = PyTuple_New(3);
>PyTuple_SetItem(t, 0, PyString_FromString("error message"));
>PyTuple_SetItem(t, 1, PyInt_FromLong(10));
>PyTuple_SetItem(t, 2, PyString_FromString("category name here"));
>PyErr_SetObject(MyErr, t);
>Py_DECREF(t);
>return NULL;
> }
>
> In this second form you check for the args attribute of the exception.
static PyObject *
raise_test3(PyObject *self) {
PyObject *d = PyDict_New();
PyDict_SetItemString(d, "category", PyInt_FromLong(111));
PyDict_SetItemString(d, "message", PyString_FromString("error
message"));
PyErr_SetObject(MyErr, d);
Py_DECREF(d);
return NULL;
}
(Small changes in the above code to be able to call more variants of
raise_test methods simultaneously.)
Yes! I finally understood this (I think...) So to explain things for
people like me:
1) PyErr_NewException creates *the class* in the module, it's a simple
method of creating exception classes, but classes created that way are
limited in features (i.e. cannot be manipulated from the module in all
ways a 'full' type can). Third argument to PyErr_NewException can be
NULL, in which case API will create an empty dictionary. After
creating the class you need to add it to the module with
PyModule_AddObject. Side note: If you want to specify a help for the
class, you do PyObject_SetAttrString on the class with the key
'__doc__'.
2) there's no instantiation anywhere:
a. PyErr_SetString and PyErr_SetObject set the exception *class*
(exception type) and exception data -- see
http://docs.python.org/c-api/exceptions.html which notes that
exceptions are similar in concept to the global 'errno' variable, so
you just set what type of last error was and what error message (or
other data) you want to associate with it
b. the "code" and "category" variables from raise_test1() in the
above example inserted with PyObject_SetAttrString() are *class*
variables, not instance variables:
try:
fancy_exc.raise_test1()
except fancy_exc.err, e:
print e.code, fancy_exc.err.code
print fancy_exc.err.code
it prints:
42 42
42
c. the data is still present in the fancy_exc.err class after
exception handling is finished, which is ok for now but may be
problematic in case of multithreaded usage patterns (however I
probably don't understand how multithreading in Python works)
3) alternative to the above is to pass all required data to the
exception with PyErr_SetObject - you can prepare a dictionary or a
tuple earlier, which will be accessible with 'args' member:
try:
fancy_exc.raise_test2()
except fancy_exc.err, e:
print e.args[0]
If it's dictionary, the syntax is a bit weird because e.args is always a tuple:
try:
fancy_exc.raise_test3()
except fancy_exc.err, e:
print e.args[0]['category']
The 'args' values are unavailable outside of 'except' clause, however
you can still use the 'e' variable which retains the values. So it's
an instance variable.
4) creating the exception class using a new type in C (PyTypeObject
structure) would give the most robust solution because every nuance of
the class can be manipulated, but it's not worth the trouble now. I
can switch to it transparently at a later time. Transparently means
that nothing will need to be updated in Python solutions written by
the module users.
5) most of the projects I've inspected with Google Code Search use the
PyErr_NewException approach.
6) there's the option of using Cython which simplifies creating
extensions and hides many unnecessary internals.
Many thanks Guilherme and Stefan for your help and for the patience.
Kind regards,
Chojrak
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archiv
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On Dec 22, 2008, at 4:07 PM, M.-A. Lemburg wrote: What kinds of objects are you storing in your dictionary ? Python instances, strings, integers ? Answered in a previous message: On Dec 20, 2008, at 8:09 PM, Mike Coleman wrote: The dict keys were all uppercase alpha strings of length 7. I don't have access at the moment, but maybe something like 10-100M of them (not sure how redundant the set is). The values are all lists of pairs, where each pair is a (string, int). The pair strings are of length around 30, and drawn from a "small" fixed set of around 60K strings (). As mentioned previously, I think the ints are drawn pretty uniformly from something like range(1). The length of the lists depends on the redundancy of the key set, but I think there are around 100-200M pairs total, for the entire dict. (If you're curious about the application domain, see 'http://greylag.org '.) -- Ivan Krstić | http://radian.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [capi-sig] Exceptions with additional instance variables
Not this list, sorry ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Hello everyone + little question around Cpython/stackless
Hello snakemen and snakewomen I'm Pascal Chambon, a french engineer just leaving my Telecom School, blatantly fond of Python, of its miscellaneous offsprings and of all what's around dynamic languages and high level programming concepts. I'm currently studying all I can find on stackless python, PYPY and the concepts they've brought to Python, and so far I wonder : since stackless python claims to be 100% compatible with CPython's extensions, faster, and brings lots of fun stuffs (tasklets, coroutines and no C stack), how comes it hasn't been merged back, to become the standard 'fast' python implementation ? Would I have missed some crucial point around there ? Isn't that a pity to maintain two separate branches if they actually complete each other very well ? Waiting for your lights on this subject, regards, Pascal ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On Mon, 22 Dec 2008 11:20:59 pm M.-A. Lemburg wrote: > On 2008-12-20 23:16, Martin v. Löwis wrote: > >>> I will try next week to see if I can come up with a smaller, > >>> submittable example. Thanks. > >> > >> These long exit times are usually caused by the garbage collection > >> of objects. This can be a very time consuming task. > > > > I doubt that. The long exit times are usually caused by a bad > > malloc implementation. > > With "garbage collection" I meant the process of Py_DECREF'ing the > objects in large containers or deeply nested structures, not the GC > mechanism for breaking circular references in Python. > > This will usually also involve free() calls, so the malloc > implementation affects this as well. However, I've seen such long > exit times on Linux and Windows, which both have rather good > malloc implementations. > > I don't think there's anything much we can do about it at the > interpreter level. Deleting millions of objects takes time and that's > not really surprising at all. It takes even longer if you have > instances with .__del__() methods written in Python. This behaviour appears to be specific to deleting dicts, not deleting random objects. I haven't yet confirmed that the problem still exists in trunk (I hope to have time tonight or tomorrow), but in my previous tests deleting millions of items stored in a list of tuples completed in a minute or two, while deleting the same items stored as key:item pairs in a dict took 30+ minutes. I say plus because I never had the patience to let it run to completion, it could have been hours for all I know. > Applications can choose other mechanisms for speeding up the > exit process in various (less clean) ways, if they have a need for > this. > > BTW: Rather than using a huge in-memory dict, I'd suggest to either > use an on-disk dictionary such as the ones found in mxBeeBase or > a database. The original poster's application uses 45GB of data. In my earlier tests, I've experienced the problem with ~ 300 *megabytes* of data: hardly what I would call "huge". -- Steven D'Aprano ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
> Investigating further, from one stop, I used gdb to follow the chain > of pointers in the nextarena and prevarena directions. There were > 5449 and 112765 links, respectively. maxarenas is 131072. To reduce the time for keeping sorted lists of arenas, I was first thinking of a binheap. I had formulated it all, and don't want to waste that effort, so I attach it below in case my second idea (right below) is flawed. It then occurred that there are only 64 different values for nfreepools, as ARENA_SIZE is 256kiB, and POOL_SIZE is 4kiB. So rather than keeping the list sorted, I now propose to maintain 64 lists, accessible in an array double-linked lists indexed by nfreepools. Whenever nfreepools changes, the arena_object is unlinked from its current list, and linked into the new list. This should reduce the overhead for keeping the lists sorted down from O(n) to O(1), with a moderate overhead of 64 pointers (512 Bytes in your case). Allocation of a new pool would have to do a linear search in these pointers (finding the arena with the least number of pools); this could be sped up with a finger pointing to the last index where a pool was found (-1, since that pool will have moved). Regards, Martin a) usable_arenas becomes an arena_object**, pointing to an array of maxarenas+1 arena*. A second variable max_usable_arenas is added. arena_object loses the prevarena pointer, and gains a usable_index value of type size_t (which is 0 for unused or completely allocated arena_objects). usable_arenas should stay heap-sorted, with the arena_object with the smallest nfreepools at index 1. b) sink and swim operations are added, which keep usable_index intact whenever arena_object pointers get swapped. c) whenever a pool is allocated in an arena, nfreepools decreases, and swim is called for the arena. whenever a pool becomes free, sink is called. d) when the last pool was allocated in an arena, it is removed from the heap. likewise, when all pools are freed in an arena, it is removed from the heap and returned to the system. e) when the first pool gets freed in an arena, it is added to the heap. On each pool allocation/deallocation, this should get the O(n) complexity of keeping the arena list sorted down to O(log n). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] If I check something in ...
I have this trivial little test case for test_file.py: +def testReadWhenWriting(self): +self.assertRaises(IOError, self.f.read) I would like to add it to the 2.6 and 3.0 maintenance branch and the 2.x trunk and the py3k branch. What is the preferred way to do that? Do I really have to do the same task four times or can I check it in once (or twice) secure in the belief that someone will come along and do a monster merge? Thx, Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If I check something in ...
On Mon, Dec 22, 2008 at 4:02 PM, wrote: > > I have this trivial little test case for test_file.py: > >+def testReadWhenWriting(self): >+self.assertRaises(IOError, self.f.read) > > I would like to add it to the 2.6 and 3.0 maintenance branch and the 2.x > trunk and the py3k branch. What is the preferred way to do that? Do I > really have to do the same task four times or can I check it in once (or > twice) secure in the belief that someone will come along and do a monster > merge? If you check it into the trunk, it will find it's way into 2.6, 3.1, and 3.0. -- Regards, Benjamin Peterson ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hello everyone + little question around Cpython/stackless
> I'm currently studying all I can find on stackless python, PYPY and the > concepts they've brought to Python, and so far I wonder : since > stackless python claims to be 100% compatible with CPython's extensions, > faster, and brings lots of fun stuffs (tasklets, coroutines and no C > stack), how comes it hasn't been merged back, to become the standard > 'fast' python implementation ? There is a long history to it, and multiple reasons influenced that status. In summary, some of the reasons were: - Stackless Python was never officially proposed for inclusion into Python (it may be that parts of it were, and of those parts actually did get added). - Stackless Python originally was fairly unmaintainable; this prevented its inclusion. - in its current form, it has limited portability, as it needs to be ported to each microprocessor and operating system separately. CPython has so far avoided using assembler code, and is fairly portable. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If I check something in ...
> I would like to add it to the 2.6 and 3.0 maintenance branch and the 2.x > trunk and the py3k branch. What is the preferred way to do that? Do I > really have to do the same task four times or can I check it in once (or > twice) secure in the belief that someone will come along and do a monster > merge? You shouldn't check it in four times. But (IMO) you also shouldn't wait for somebody else to merge it (I know some people disagree with that recommendation). Instead, you should commit it into trunk, and then run svnmerge.py three times, namely: - in a release26-maint checkout, run svnmerge.py -r svn commit -F svnmerge-commit-something-press-tab - in a py3k checkout, run svnmerge.py -r svn commit -F svnmerge-commit-something-press-tab - in a release30-maint check, then run svnmerge.py -r svn revert . svn commit -F svnmerge-commit-something-press-tab Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
Martin v. Löwis v.loewis.de> writes: > > It then occurred that there are only 64 different values for nfreepools, > as ARENA_SIZE is 256kiB, and POOL_SIZE is 4kiB. So rather than keeping > the list sorted, I now propose to maintain 64 lists, accessible in > an array double-linked lists indexed by nfreepools. Whenever nfreepools > changes, the arena_object is unlinked from its current list, and linked > into the new list. This should reduce the overhead for keeping the lists > sorted down from O(n) to O(1), with a moderate overhead of 64 pointers > (512 Bytes in your case). > > Allocation of a new pool would have to do a linear search in these > pointers (finding the arena with the least number of pools); You mean the least number of free pools, right? IIUC, the heuristic is to favour a small number of busy arenas rather than a lot of sparse ones. And, by linear search in these pointers, do you mean just probe the 64 lists for the first non-NULL list head? If so, then it's likely fast enough for a rather infrequent operation. Now, we should find a way to benchmark this without having to steal Mike's machine and wait 30 minutes every time. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If I check something in ...
On Mon, Dec 22, 2008 at 4:27 PM, "Martin v. Löwis" wrote: > You shouldn't check it in four times. But (IMO) you also shouldn't wait > for somebody else to merge it (I know some people disagree with that > recommendation). I don't completely disagree. Certainly, if you want to make sure your change is merged correctly into every branches, then please do merge it yourself. It's also nice if platform-specific merges (ie Windows build files) are handled by the original committer. However, minor changes to the documentation or code formatting and even simple bug fixes are trivial to merge all at once between branches. In the end, I suppose it doesn't really matter; everyone can do what they are comfortable with. -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
>> Allocation of a new pool would have to do a linear search in these >> pointers (finding the arena with the least number of pools); > > You mean the least number of free pools, right? Correct. > IIUC, the heuristic is to favour > a small number of busy arenas rather than a lot of sparse ones. Correct. Or, more precisely, the hope is indeed to make most arenas sparse, so that they eventually see all their pools freed. > And, by linear search in these pointers, do you mean just probe the 64 lists > for > the first non-NULL list head? Correct. > If so, then it's likely fast enough for a rather infrequent operation. I would hope so, yes. However, the same hope applied to the current code (how much time can it take to sink an arena in a linear list?), so if we have the prospect of using larger arenas some day, this might change. > Now, we should find a way to benchmark this without having to steal Mike's > machine and wait 30 minutes every time. I think this can be simulated by using just arena objects, with no associated arenas, and just adjusting pool counters. Allocate 100,000 arena objects, and start out with them all being completely allocated. Then randomly chose one arena to deallocate a pool from; from time to time, also allocate a new pool. Unfortunately, this will require some hacking of the code to take the measurements. Alternatively, make the arena size 4k, and the pool size 32 bytes, and then come with a pattern to allocate and deallocate 8 byte blocks. Not sure whether the code works for these parameters, though (but it might be useful to fix it for non-standard sizes). This would require only 400MiB of memory to run the test. I think obmalloc is fairly independent from the rest of Python, so it should be possible to link it with a separate main() function, and nothing else of Python. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On Mon, Dec 22, 2008 at 2:54 PM, Ivan Krstić wrote: > It's still not clear to me, from reading the whole thread, precisely what > you're seeing. A self-contained test case, preferably with generated random > data, would be great, and save everyone a lot of investigation time. I'm still working on a test case. The first couple of attempts, using a half-hearted attempt to model the application behavior wrt this dict didn't demonstrate bad behavior. My impression is that no one's burning much time on this but me at the moment, aside from offering helpful advice. If you are, you might want to wait. I noticed just now that the original hardware was throwing some chipkills, so I'm retesting on something else. > In the > meantime, can you 1) turn off all swap files and partitions, and 2) confirm > positively that your CPU cycles are burning up in userland? For (1), I don't have that much control over the machine. Plus, based on watching with top, I seriously doubt the process is using swap in any way. For (2), yes, 100% CPU usage. > (In general, unless you know exactly why your workload needs swap, and have > written your program to take swapping into account, having _any_ swap on a > machine with 64GB RAM is lunacy. The machine will grind to a complete > standstill long before filling up gigabytes of swap.) The swap is not there to support my application per se. Clearly if you're swapping, generally you're crawling. This host is used by a reasonably large set of non- and novice programmers, who sometimes vacuum up VM without realizing it. If you have a nice, big swap space, you can 'kill -STOP' these offenders, and allow them to swap out while you have a leisurely discussion with the owner and possibly 'kill -CONT' later, as opposed to having to do a quick 'kill -KILL' to save the machine. That's my thinking, anyway. Mike ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On Mon, Dec 22, 2008 at 2:22 PM, Adam Olsen wrote:
> To make sure that's the correct line please recompile python without
> optimizations. GCC happily reorders and merges different parts of a
> function.
>
> Adding a counter in C and recompiling would be a lot faster than using
> a gdb hook.
Okay, I did this. The results are the same, except that now sampling
selects the different source statements within this loop, instead of
just the top of the loop (which makes sense).
I added a counter (static volatile long) as suggested, and a
breakpoint to sample it. Not every pass through PyObject_Free takes
case 3, but for those that do, this loop runs around 100-25000 times.
I didn't try to graph it, but based on a quick sample, it looks like
more than 5000 iterations on most occasions.
The total counter is 12.4 billion at the moment, and still growing.
That seems high, but I'm not sure what would be expected or hoped for.
I have a script that demonstrates the problem, but unfortunately the
behavior isn't clearly bad until large amounts of memory are used. I
don't think it shows at 2G, for example. (A 32G machine is
sufficient.) Here is a log of running the program at different sizes
($1):
1 4.04686999321 0.696660041809
2 8.1575551033 1.46393489838
3 12.6426320076 2.30558800697
4 16.471298933 3.80377006531
5 20.1461620331 4.96685886383
6 25.150053978 5.48230814934
7 28.9099609852 7.41244196892
8 32.283219099 6.31711483002
9 36.6974511147 7.40236377716
10 40.3126089573 9.01174497604
20 81.7559120655 20.3317198753
30 123.67071104 31.4815018177
40 161.935647011 61.4484620094
50 210.610441923 88.6161060333
60 248.89805007 118.821491003
70 288.944771051 194.166989088
80 329.93295002 262.14109993
90 396.209988832 454.317914009
100 435.610564947 564.191882133
If you plot this, it is clearly quadratic (or worse).
Here is the script:
#!/usr/bin/env python
"""
Try to trigger quadratic (?) behavior during .clear() of a large but simple
defaultdict.
"""
from collections import defaultdict
import time
import sys
import gc; gc.disable()
print >> sys.stderr, sys.version
h = defaultdict(list)
n = 0
lasttime = time.time()
megs = int(sys.argv[1])
print megs,
sys.stdout.flush()
# 100M iterations -> ~24GB? on my 64-bit host
for i in xrange(megs * 1024 * 1024):
s = '%0.7d' % i
h[s].append(('', 12345))
h[s].append(('', 12345))
h[s].append(('', 12345))
# if (i % 100) == 0:
# t = time.time()
# print >> sys.stderr, t-lasttime
# lasttime = t
t = time.time()
print t-lasttime,
sys.stdout.flush()
lasttime = t
h.clear()
t = time.time()
print t-lasttime,
sys.stdout.flush()
lasttime = t
print
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On Dec 22, 2008, at 6:28 PM, Mike Coleman wrote: For (2), yes, 100% CPU usage. 100% _user_ CPU usage? (I'm trying to make sure we're not chasing some particular degeneration of kmalloc/vmalloc and friends.) -- Ivan Krstić | http://radian.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
> Now, we should find a way to benchmark this without having to steal Mike's
> machine and wait 30 minutes every time.
So, I seem to reproduce it. The following script takes about 15 seconds to
run and allocates a 2 GB dict which it deletes at the end (gc disabled of
course).
With 2.4, deleting the dict takes ~1.2 seconds while with 2.5 and higher
(including 3.0), deleting the dict takes ~3.5 seconds. Nothing spectacular
but the difference is clear.
Also, after the dict is deleted and before the program exits, you can witness
(with `ps` or `top`) that 2.5 and higher has reclaimed 1GB, while 2.4 has
reclaimed nothing. There is a sleep() call at the end so that you have the
time :-)
You can tune memory occupation at the beginning of the script, but the lower
the more difficult it will be to witness a difference.
Regards
Antoine.
###
import random
import time
import gc
import itertools
# Adjust this parameter according to your system RAM!
target_size = int(2.0 * 1024**3) # 2.0 GB
pool_size = 4 * 1024
# This is a ballpark estimate: 60 bytes overhead for each
# { dict entry struct + float object + tuple object header },
# 1.3 overallocation factor for the dict.
target_length = int(target_size / (1.3 * (pool_size + 60)))
def make_dict():
print ("filling dict up to %d entries..." % target_length)
# 1. Initialize the dict from a set of pre-computed random keys.
keys = [random.random() for i in range(target_length)]
d = dict.fromkeys(keys)
# 2. Build the values that will constitute the dict. Each value will, as
#far as possible, span a contiguous `pool_size` memory area.
# Over 256 bytes per alloc, PyObject_Malloc defers to the system malloc()
# We avoid that by allocating tuples of smaller longs.
int_size = 200
# 24 roughly accounts for the long object overhead (YMMV)
int_start = 1 << ((int_size - 24) * 8 - 7)
int_range = range(1, 1 + pool_size // int_size)
values = [None] * target_length
# Maximize allocation locality by pre-allocating the values
for n in range(target_length):
values[n] = tuple(int_start + j for j in int_range)
if n % 1 == 0:
print (" %d iterations" % n)
# The keys are iterated over in their original order rather than in
# dict order, so as to randomly spread the values in the internal dict
# table wrt. allocation address.
for n, k in enumerate(keys):
d[k] = values[n]
print ("dict filled!")
return d
if __name__ == "__main__":
gc.disable()
t1 = time.time()
d = make_dict()
t2 = time.time()
print (" -> %.3f s." % (t2 - t1))
print ("deleting dict...")
t2 = time.time()
del d
t3 = time.time()
print (" -> %.3f s." % (t3 - t2))
print ("Finished, you can press Ctrl+C.")
time.sleep(10.0)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If I check something in ...
Benjamin> If you check it into the trunk, it will find it's way into Benjamin> 2.6, 3.1, and 3.0. Outstanding! Thx, Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
Steven D'Aprano wrote: > This behaviour appears to be specific to deleting dicts, not deleting > random objects. I haven't yet confirmed that the problem still exists > in trunk (I hope to have time tonight or tomorrow), but in my previous > tests deleting millions of items stored in a list of tuples completed > in a minute or two, while deleting the same items stored as key:item > pairs in a dict took 30+ minutes. I say plus because I never had the > patience to let it run to completion, it could have been hours for all > I know. There's actually an interesting comment in list_dealloc: /* Do it backwards, for Christian Tismer. There's a simple test case where somehow this reduces thrashing when a *very* large list is created and immediately deleted. */ The "backwards" the comment is referring to is the fact that it invokes DECREF on the last item in the list first and counts back down to the first item, instead of starting at the first item and incrementing the index each time around the loop. The revision number on that (13452) indicates that it predates the implementation of PyObject_Malloc and friends, so it was probably avoiding pathological behaviour in platform malloc() implementations by free'ing memory in the reverse order to which it was allocated (assuming the list was built initially starting with the first item). However, I'm now wondering it if also has the side effect of avoiding the quadratic behaviour Mike has found inside the more recent code to release arenas back to the OS. I'm working on a simple benchmark that looks for non-linear scaling of the deallocation times - I'll include a case of deallocation of a reversed list along with a normal list and a dictionary. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
2008/12/22 Ivan Krstić : > On Dec 22, 2008, at 6:28 PM, Mike Coleman wrote: >> >> For (2), yes, 100% CPU usage. > > 100% _user_ CPU usage? (I'm trying to make sure we're not chasing some > particular degeneration of kmalloc/vmalloc and friends.) Yes, user. No noticeable sys or wait CPU going on. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
I unfortunately don't have time to work out how obmalloc works myself, but I wonder if any of the constants in that file might need to scale somehow with memory size. That is, is it possible that some of them that work okay with 1G RAM won't work well with (say) 128G or 1024G (coming soon enough)? ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
Mike Coleman wrote:
> If you plot this, it is clearly quadratic (or worse).
Here's another comparison script that tries to probe the vagaries of the
obmalloc implementation. It looks at the proportional increases in
deallocation times for lists and dicts as the number of contained items
increases when using a variety of deallocation orders:
- in hash order (dict)
- in reverse order of allocation (list)
- in order of allocation (list, reversed in place)
- in random order (list, shuffled in place using the random module)
I've included the final output from a run on my own machine below [1],
but here are the main points I get out of it:
- at the sizes I can test (up to 20 million items in the containers),
this version of the script doesn't show any particularly horrible
non-linearity with deallocation of dicts, lists or reversed lists.
- when the items in a list are deallocated in *random* order, however,
the deallocation times are highly non-linear - by the time we get to 20
million items, deallocating in random order takes nearly twice as long
as deallocation in either order of allocation or in reverse order.
- after the list of items had been deallocated in random order,
subsequent deallocation of a dict and the list took significantly longer
than when those operations took place on a comparatively "clean"
obmalloc state.
I'm going to try making a new version of the script that uses random
integers with a consistent number of digits in place of the monotically
increasing values that are currently used and see what effect that has
on the dict scaling (that's where I expect to see the greatest effect,
since the hash ordering is the one which will be most affected by the
change to the item contents).
Cheers,
Nick.
[1] Full final results from local test run:
Dict: (Baseline=0.003135 seconds)
10=100.0%
100=1020.9%
200=2030.5%
500=5026.7%
1000=10039.7%
2000=20086.4%
List: (Baseline=0.005764 seconds)
10=100.0%
100=1043.7%
200=2090.1%
500=5227.2%
1000=10458.1%
2000=20942.7%
ReversedList: (Baseline=0.005879 seconds)
10=100.0%
100=1015.0%
200=2023.5%
500=5057.1%
1000=10114.0%
2000=20592.6%
ShuffledList: (Baseline=0.028241 seconds)
10=100.0%
100=1296.0%
200=2877.3%
500=7960.1%
1000=17216.9%
2000=37599.9%
PostShuffleDict: (Baseline=0.016229 seconds)
10=100.0%
100=1007.9%
200=2018.4%
500=5075.3%
1000=10217.5%
2000=20873.1%
PostShuffleList: (Baseline=0.020551 seconds)
10=100.0%
100=1021.9%
200=1978.2%
500=4953.6%
1000=10262.3%
2000=19854.0%
Baseline changes for Dict and List after deallocation of list in random
order:
Dict: 517.7%
List: 356.5%
from time import time
from random import shuffle
import gc
gc.disable()
def print_result(action, num_items, duration):
print "%s for %d items took %f seconds (%f minutes)" % (action, num_items, duration, duration / 60)
def make_list(num_items):
return [(x, str(x)) for x in xrange(num_items)]
def make_reversed_list(num_items):
seq = make_list(num_items)
seq.reverse()
return seq
def make_shuffled_list(num_items):
seq = make_list(num_items)
shuffle(seq)
return seq
def make_dict(num_items):
return dict((x, (x, str(x))) for x in xrange(num_items))
def run_test(name, factory, num_items, num_runs=3):
durations = []
for i in xrange(num_runs):
container = factory(num_items)
start = time()
del container
duration = time() - start;
print_result(name, num_items, duration)
durations.append(duration)
return min(durations)
TEST_NAMES = """Dict List ReversedList ShuffledList
PostShuffleDict PostShuffleList""".split()
TEST_FACTORIES = [make_dict, make_list, make_reversed_list, make_shuffled_list,
make_dict, make_list]
NUM_ITEMS = 10, 100, 200, 500, 1000, 2000
results = {}
try:
for t, f in zip(TEST_NAMES, TEST_FACTORIES):
durations = []
for n in NUM_ITEMS:
durations.append(run_test(t, f, n))
results[t] = durations
finally:
for t in TEST_NAMES:
durations = results[t]
baseline = durations[0]
percentages = [100 * d / baseline for d in durations]
print "%s: (Baseline=%f seconds)" % (t, baseline)
for n, p in zip(NUM_ITEMS, percentages):
print " %d=%.1f%%" % (n, p)
print
print "Baseline changes for Dict and List after deallocation of list in random order:"
print " Dict: %.1f%%" % (100 * results["PostShuffleDict"][0] / results["Dict"][0])
print " List: %.1f%%" % (100 * results["PostShuffleList"][0] / results["List"][0])
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/op
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
On Mon, Dec 22, 2008 at 7:34 PM, Antoine Pitrou wrote:
>
>> Now, we should find a way to benchmark this without having to steal Mike's
>> machine and wait 30 minutes every time.
>
> So, I seem to reproduce it. The following script takes about 15 seconds to
> run and allocates a 2 GB dict which it deletes at the end (gc disabled of
> course).
> With 2.4, deleting the dict takes ~1.2 seconds while with 2.5 and higher
> (including 3.0), deleting the dict takes ~3.5 seconds. Nothing spectacular
> but the difference is clear.
>
I modified your script to delete the dictionary without actually
deallocating the items in it. You can speed up a dictionary
deallocation significantly if you keep a reference to its items and
delete the dictionary before deleting its items. In Python 2.4, the
same behavior exists, but is not as strongly marked as in Python 2.6
with pymalloc enabled.
I can understand that deallocating the items in the order (or
actually, the reverse order) they were allocated is faster, than doing
so in a rather haphazard manner (i.e., like dict). However, I am not
sure why pymalloc accentuate this behavior.
-- Alexandre
Python 2.6 with pymalloc, without pydebug
a...@helios:~$ python2.6 dict_dealloc_test.py
creating 397476 items...
-> 6.613 s.
building dict...
-> 0.230 s.
deleting items...
-> 0.059 s.
deleting dict...
-> 2.299 s.
total deallocation time: 2.358 seconds.
a...@helios:~$ python2.6 dict_dealloc_test.py
creating 397476 items...
-> 6.530 s.
building dict...
-> 0.228 s.
deleting dict...
-> 0.089 s.
deleting items...
-> 0.971 s.
total deallocation time: 1.060 seconds.
Python 2.6 without pymalloc, without pydebug
a...@helios:release26-maint$ ./python /home/alex/dict_dealloc_test.py
creating 397476 items...
-> 5.921 s.
building dict...
-> 0.244 s.
deleting items...
-> 0.073 s.
deleting dict...
-> 1.502 s.
total deallocation time: 1.586 seconds.
a...@helios:release26-maint$ ./python /home/alex/dict_dealloc_test.py
creating 397476 items...
-> 6.122 s.
building dict...
-> 0.237 s.
deleting dict...
-> 0.092 s.
deleting items...
-> 1.238 s.
total deallocation time: 1.330 seconds.
a...@helios:~$ python2.4 dict_dealloc_test.py
creating 397476 items...
-> 6.164 s.
building dict...
-> 0.218 s.
deleting items...
-> 0.057 s.
deleting dict...
-> 1.185 s.
total deallocation time: 1.243 seconds.
a...@helios:~$ python2.4 dict_dealloc_test.py
creating 397476 items...
-> 6.202 s.
building dict...
-> 0.218 s.
deleting dict...
-> 0.090 s.
deleting items...
-> 0.852 s.
total deallocation time: 0.943 seconds.
##
import random
import time
import gc
# Adjust this parameter according to your system RAM!
target_size = int(2.0 * 1024**3) # 2.0 GB
pool_size = 4 * 1024
# This is a ballpark estimate: 60 bytes overhead for each
# { dict entry struct + float object + tuple object header },
# 1.3 overallocation factor for the dict.
target_length = int(target_size / (1.3 * (pool_size + 60)))
def make_items():
print ("creating %d items..." % target_length)
# 1. Initialize a set of pre-computed random keys.
keys = [random.random() for i in range(target_length)]
# 2. Build the values that will constitute the dict. Each value will, as
#far as possible, span a contiguous `pool_size` memory area.
# Over 256 bytes per alloc, PyObject_Malloc defers to the system malloc()
# We avoid that by allocating tuples of smaller longs.
int_size = 200
# 24 roughly accounts for the long object overhead (YMMV)
int_start = 1 << ((int_size - 24) * 8 - 7)
int_range = range(1, 1 + pool_size // int_size)
values = [None] * target_length
# Maximize allocation locality by pre-allocating the values
for n in range(target_length):
values[n] = tuple(int_start + j for j in int_range)
return list(zip(keys,values))
if __name__ == "__main__":
gc.disable()
t1 = time.time()
items = make_items()
t2 = time.time()
print " -> %.3f s." % (t2 - t1)
print "building dict..."
t1 = time.time()
testdict = dict(items)
t2 = time.time()
print " -> %.3f s." % (t2 - t1)
def delete_testdict():
global testdict
print "deleting dict..."
t1 = time.time()
del testdict
t2 = time.time()
print " -> %.3f s." % (t2 - t1)
def delete_items():
global items
print "deleting items..."
t1 = time.time()
del items
t2 = time.time()
print " -> %.3f s." % (t2 - t1)
t1 = time.time()
# Swap these, and look at the total time
delete_items()
delete_testdict()
t2 = time.time()
print "total deallocation time: %.3f seconds." % (t2 - t1)
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If I check something in ...
Martin> Instead, you should commit it into trunk, and then run svnmerge.py three Martin> times, namely: ... Thanks for that cheat sheet. I never would have figured that out on my own. Well, at least not in a timely fashion. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Releasing 2.5.4
Martin v. Löwis wrote: > It seems r67740 shouldn't have been committed. Since this > is a severe regression, I think I'll have to revert it, and > release 2.5.4 with just that change. My understanding of the problem is that clearerr() needs to be called before any FILE read operations on *some* platforms. The only platform I saw mentioned was OS X. Towards that end, I have attached a much simpler patch onto the tracker issue, which maybe somebody can verify solves the problem because I do not have access to a platform which fails the test that was originally given. -Scott -- Scott Dial [email protected] [email protected] ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Problems compiling 2.6.1 on Solaris 10
I am hoping someone can assist me. I normally don't care if the _ctypes module builds or not, but I now need to have it build. I am running Solaris 10 with Sun's C compiler under SunStudio 11. After running 'configure' and 'make', the _ctypes module fails with the following error: cc -xcode=pic32 -DNDEBUG -O -I. -I/data/python/Python-2.6.1/./Include -Ibuild/temp.solaris-2.10-sun4u-2.6/libffi/include -Ibuild/temp.solaris-2.10-sun4u-2.6/libffi -I/data/python/Python-2.6.1/Modules/_ctypes/libffi/src -I/usr/local/python/include -I. -IInclude -I./Include -I/usr/local/include -I/data/python/Python-2.6.1/Include -I/data/python/Python-2.6.1 -c /data/python/Python-2.6.1/Modules/_ctypes/_ctypes.c -o build/temp.solaris-2.10-sun4u-2.6/data/python/Python-2.6.1/Modules/_ctyp es/_ctypes.o "build/temp.solaris-2.10-sun4u-2.6/libffi/include/ffi.h", line 257: syntax error before or at: __attribute__ "build/temp.solaris-2.10-sun4u-2.6/libffi/include/ffi.h", line 257: warning: old-style declaration or incorrect type for: __attribute__ "build/temp.solaris-2.10-sun4u-2.6/libffi/include/ffi.h", line 257: warning: syntax error: empty declaration "/data/python/Python-2.6.1/Modules/_ctypes/_ctypes.c", line 187: cannot recover from previous errors cc: acomp failed for /data/python/Python-2.6.1/Modules/_ctypes/_ctypes.c Is there anything special I have to do to get it to compile under Solaris 10 and SunStudio 11? BTW: I cannot use GCC. Thank you very much, Lance ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
