On Jun 15, 1:04 am, [EMAIL PROTECTED] wrote: > However it seems that marshal.dumps() for large objects has a > quadratic performance issue which I'm assuming is that it grows its > memory buffer in constant increments.
Looking at the source in http://svn.python.org/projects/python/trunk/Python/marshal.c , it looks like the relevant fragment is in w_more(): . . . size = PyString_Size(p->str); newsize = size + size + 1024; if (newsize > 32*1024*1024) { newsize = size + 1024*1024; } if (_PyString_Resize(&p->str, newsize) != 0) { . . . When more space is needed, the resize operation over-allocates by double the previous need plus 1K. This should give amortized O(1) performance just like list.append(). However, when that strategy requests more than 32Mb, the resizing becomes less aggressive and grows only in 1MB blocks and giving your observed nasty quadratic behavior. Raymond -- http://mail.python.org/mailman/listinfo/python-list
