[issue26049] Poor performance when reading large xmlrpc data

Changes by Oscar Andres Alvarez Montero : -- nosy: +Oscar Andres Alvarez Montero ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue26049] Poor performance when reading large xmlrpc data

Cédric Krier added the comment: One advantage, I see, is when xmlrpclib is overridden to use an other marshaller which is can not be feed chunk by chunk. So reducing the number of call to feed will have a bigger impact. But I don't know if this is enough for Python. -- ___

[issue26049] Poor performance when reading large xmlrpc data

Serhiy Storchaka added the comment: Thanks Cédric. There is no improvement for the first request. But if repeat requests, there is small (about 5%) improvement. This is too small. -- ___ Python tracker __

[issue26049] Poor performance when reading large xmlrpc data

Cédric Krier added the comment: Here is the client/server scripts. I don't measure a big performance improvement with it. I think the improvement measured in msg257756 are linked to the way xmlrpclib is overriden in Tryton. -- Added file: http://bugs.python.org/file42044/server.py

[issue26049] Poor performance when reading large xmlrpc data

Changes by Cédric Krier : Added file: http://bugs.python.org/file42043/client.py ___ Python tracker ___ ___ Python-bugs-list mailing list Unsu

[issue26049] Poor performance when reading large xmlrpc data

Serhiy Storchaka added the comment: There is no special infrastructure. You can just provide two files. One script should implement simple server that produces large output. Other script should implement the client that makes a number of requests and measure the time. Of course you can impleme

[issue26049] Poor performance when reading large xmlrpc data

Cédric Krier added the comment: Is there an infrastructure already in place for such microbenchmark? -- ___ Python tracker ___ ___ Pyt

[issue26049] Poor performance when reading large xmlrpc data

Serhiy Storchaka added the comment: Could you please provide a microbenchmark that exposes the benefit of this optimization? -- ___ Python tracker ___ __

[issue26049] Poor performance when reading large xmlrpc data

Cédric Krier added the comment: ping -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org

[issue26049] Poor performance when reading large xmlrpc data

Changes by Raimon Esteve : -- nosy: +resteve ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue26049] Poor performance when reading large xmlrpc data

Changes by Serhiy Storchaka : -- stage: -> patch review ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:

[issue26049] Poor performance when reading large xmlrpc data

Cédric Krier added the comment: Answering to myself, it is better to read by chunk to feed the parser also by chunk. So here is a version of the patch which increases by 2 on every loop. -- Added file: http://bugs.python.org/file41545/default.patch _

[issue26049] Poor performance when reading large xmlrpc data

Cédric Krier added the comment: Will it not be better indeed to just stream.read() without any argument? Because HTTPResponse will call _safe_read with just the length of the header. -- ___ Python tracker _

[issue26049] Poor performance when reading large xmlrpc data

Serhiy Storchaka added the comment: Reading with 10 MB limit allocates 10 MB buffer. It may be better to start with 1024 bytes limit, and increase it by 2 times on every iteration. -- nosy: +serhiy.storchaka ___ Python tracker

[issue26049] Poor performance when reading large xmlrpc data

Terry J. Reedy added the comment: I believe performance enhancements are, by default, limited to 'default', but this one might be a good candidate for backport. -- nosy: +loewis, terry.reedy versions: +Python 3.6 ___ Python tracker

[issue26049] Poor performance when reading large xmlrpc data

Sergi Almacellas Abellana added the comment: I added a new patch which fixed comments -- Added file: http://bugs.python.org/file41536/default.patch ___ Python tracker ___ ___

[issue26049] Poor performance when reading large xmlrpc data

Changes by Sergi Almacellas Abellana : Removed file: http://bugs.python.org/file41534/default.patch ___ Python tracker ___ ___ Python-bugs-lis

[issue26049] Poor performance when reading large xmlrpc data

Sergi Almacellas Abellana added the comment: I attach a patch to fix on default series -- keywords: +patch Added file: http://bugs.python.org/file41534/default.patch ___ Python tracker _

[issue26049] Poor performance when reading large xmlrpc data

Sergi Almacellas Abellana added the comment: And Also another patch for 2.7 branches -- Added file: http://bugs.python.org/file41535/python27.patch ___ Python tracker ___ ___

[issue26049] Poor performance when reading large xmlrpc data

Cédric Krier added the comment: I don't think it is necessary to allow to customize the chunk size. Indeed Python should provide a good value by default that works for all platforms. -- nosy: +ced type: -> performance ___ Python tracker

[issue26049] Poor performance when reading large xmlrpc data

New submission from Sergi Almacellas Abellana: By default, python xmlrpclib parser reads data by chunks of 1024 bytes [1], which leads to a lot of data concatenations when reading large data, which is very slow in python. Increasing the chuck size from 1024 bytes to a higher value makes improv