Re: [Python-Dev] Add a new tracemalloc module to trace memory allocations
On Wed, Aug 28, 2013 at 8:16 PM, Victor Stinner wrote: > Hi, > > Thanks to the PEP 445, it becomes possible to trace easily memory > allocations. I propose to add a new tracemalloc module computing the > memory usage per file and per line number. It has also a private > method to retrieve the location (filename and line number) of a memory > allocation of an object. > > tracemalloc is different than Heapy or PySizer because it is focused > on the location of a memory allocation rather that the object type or > object content. > > I have an implementation of the module for Python 2.5-3.4, but it > requires to patch and recompile Python: > https://pypi.python.org/pypi/pytracemalloc > > > My proposed implementation for Python 3.4 is different: > > * reuse the PEP 445 to hook memory allocators > > * use a simple C implementation of an hash table called "cfuhash" > (coming from the libcfu project, BSD license) instead of depending on > the glib library. I simplified and adapted cfuhash for my usage > > * no enable() / disable() function: tracemalloc can only be enabled > before startup by setting PYTHONTRACEMALLOC=1 environment variable > > * traces (size of the memory block, Python filename, Python line > number) are stored directly in the memory block, not in a separated > hash table > > I chose PYTHONTRACEMALLOC env var instead of enable()/disable() > functions to be able to really trace *all* memory allocated by Python, > especially memory allocated at startup, during Python initialization. > > > The (high-level) API should be reviewed and discussed. The most > interesting part is to take "snapshots" and compare snapshots. The > module can load snapshots from disk and compare them later for deeper > analysis (ex: ignore some files). > > For the documentation, see the following page: > https://pypi.python.org/pypi/pytracemalloc > > I created the following issue to track the implementation: > http://bugs.python.org/issue18874 > > The implementation: > http://hg.python.org/features/tracemalloc Without looking at the code or docs I can the concept sounds very cool! > > > * * * > > I also created a "pyfailmalloc" project based on the PEP 445 to inject > MemoryError exceptions. I used this module to check if Python handles > correctly memory allocation failures. The answer is no, I fixed many > bugs (see issue #18408). > > Project homepage: > https://bitbucket.org/haypo/pyfailmalloc > > Charles-François Natali and Serhiy Storchaka asked me to add this > module somewhere in Python 3.4: "how about adding pyfailmalloc to the > main repo (maybe under Tools), with a script making it easy to run the > tests suite with it enabled?" > > What is the best place for such module? Add it to Modules/ directory > but as a private module: "_failmalloc"? > Would extension module authors find it useful? If so maybe we need a malloc package with trace and fail submodules? And if we add it we might want to add to running the tool as part of the devguide something people can work on. -Brett > > * * * > > Example of tracemalloc output (it is more readable with colors, try in > a terminal). The first top is sorted by total size, whereas the second > top is sorted (automatically) with the size difference. You can see > for example that the linecache module likes caching data (1.5 MB after > 10 seconds of tests). > > > $ PYTHONTRACEMALLOC=1 ./python -m test > ... > == CPython 3.4.0a1+ (default:2ce9e5f6b47c+, Aug 29 2013, 02:03:02) > [GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] > == Linux-3.9.4-200.fc18.x86_64-x86_64-with-fedora-18-Spherical_Cow > little-endian > == /home/haypo/prog/python/tracemalloc/build/test_python_11087 > ... > [ 1/380] test_grammar > [ 2/380] test_opcodes > [ 3/380] test_dict > [ 4/380] test_builtin > [ 5/380] test_exceptions > [ 6/380] test_types > [ 7/380] test_unittest > > 2013-08-29 02:06:22: Top 25 allocations per file and line > #1: :704: size=5 MiB, count=56227, > average=105 B > #2: .../tracemalloc/Lib/linecache.py:127: size=1004 KiB, count=8706, > average=118 B > #3: .../Lib/unittest/mock.py:1764: size=895 KiB, count=7841, average=116 B > #4: .../Lib/unittest/mock.py:1805: size=817 KiB, count=15101, average=55 B > #5: .../Lib/test/test_dict.py:35: size=768 KiB, count=8, average=96 KiB > #6: :274: size=703 KiB, count=4604, > average=156 B > #7: ???:?: size=511 KiB, count=4445, average=117 B > #8: .../Lib/unittest/mock.py:350: size=370 KiB, count=1227, average=308 B > #9: .../Lib/unittest/case.py:306: size=343 KiB, count=1390, average=253 B > #10: .../Lib/unittest/case.py:496: size=330 KiB, count=650, average=521 B > #11: .../Lib/unittest/case.py:327: size=291 KiB, count=717, average=416 B > #12: .../Lib/collections/__init__.py:368: size=239 KiB, count=2170, > average=113 B > #13: .../Lib/test/test_grammar.py:132: size=195 KiB, count=1250, > average=159 B > #14: .../Lib/unittest/mock.py:379: size=118 KiB, count=152, average=800 B > #15: .../tracemalloc/Lib/contextlib.py:37: size=102 KiB, count=
Re: [Python-Dev] Add a new tracemalloc module to trace memory allocations
2013/8/29 Brett Cannon : >> I also created a "pyfailmalloc" project based on the PEP 445 to inject >> MemoryError exceptions. (...) > > Would extension module authors find it useful? I don't know, I created two months ago and I didn't made a public annoucement. > If so maybe we need a malloc package with trace and fail submodules? I read somewhere "flat is better than nested". failmalloc and tracemalloc are not directly related. I guess that they can be used at the same time, but I didn't try. > And if we add it we might want to add to running the tool as part of the > devguide something people can work on. There are still tricky open issues related to failmalloc :-) - frame_fasttolocals.patch: fix for PyFrame_FastToLocals(), I didn't commit the patch because it is not atomic (it does not handle errors correctly). I should modify it to copy the locals before modifying the dict, so it can be restored in case of errors. - #18507: import_init() should not use Py_FatalError() but return an error - #18509: CJK decoders should return MBERR_EXCEPTION on PyUnicodeWriter error - fix tests hang when an exception occurs in a thread Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #16799: Switched from getopt to argparse style in regrtest's argument
On Thu, Aug 29, 2013 at 3:27 AM, serhiy.storchaka < [email protected]> wrote: > http://hg.python.org/cpython/rev/997de0edc5bd > changeset: 85444:997de0edc5bd > parent: 85442:676bbd5b0254 > user:Serhiy Storchaka > date:Thu Aug 29 12:26:23 2013 +0300 > summary: > Issue #16799: Switched from getopt to argparse style in regrtest's > argument > parsing. Added more tests for regrtest's argument parsing. > > files: > Lib/test/regrtest.py | 529 +++-- > Lib/test/test_regrtest.py | 328 --- > Misc/NEWS |3 + > 3 files changed, 500 insertions(+), 360 deletions(-) > > > diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py > --- a/Lib/test/regrtest.py > +++ b/Lib/test/regrtest.py > ... > diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py > --- a/Lib/test/test_regrtest.py > +++ b/Lib/test/test_regrtest.py > @@ -4,97 +4,281 @@ > > import argparse > import getopt > We aren't using getopt in this module anymore, are we? -eric > +import os.path > import unittest > from test import regrtest, support > > -def old_parse_args(args): > -"""Parse arguments as regrtest did strictly prior to 3.4. > - > -Raises getopt.GetoptError on bad arguments. > -""" > -return getopt.getopt(args, 'hvqxsoS:rf:lu:t:TD:NLR:FdwWM:nj:Gm:', > -['help', 'verbose', 'verbose2', 'verbose3', 'quiet', > - 'exclude', 'single', 'slow', 'randomize', 'fromfile=', > 'findleaks', > - 'use=', 'threshold=', 'coverdir=', 'nocoverdir', > - 'runleaks', 'huntrleaks=', 'memlimit=', 'randseed=', > - 'multiprocess=', 'coverage', 'slaveargs=', 'forever', 'debug', > - 'start=', 'nowindows', 'header', 'testdir=', 'timeout=', 'wait', > - 'failfast', 'match=']) > - > class ParseArgsTestCase(unittest.TestCase): > > -"""Test that regrtest's parsing code matches the prior getopt > behavior.""" > +"""Test regrtest's argument parsing.""" > > -def _parse_args(self, args): > -# This is the same logic as that used in regrtest.main() > -parser = regrtest._create_parser() > -ns = parser.parse_args(args=args) > -opts = regrtest._convert_namespace_to_getopt(ns) > -return opts, ns.args > +def checkError(self, args, msg): > +with support.captured_stderr() as err, > self.assertRaises(SystemExit): > +regrtest._parse_args(args) > +self.assertIn(msg, err.getvalue()) > > -def _check_args(self, args, expected=None): > -""" > -The expected parameter is for cases when the behavior of the new > -parse_args differs from the old (but deliberately so). > -""" > -if expected is None: > -try: > -expected = old_parse_args(args) > -except getopt.GetoptError: > -# Suppress usage string output when an > argparse.ArgumentError > -# error is raised. > -with support.captured_stderr(): > -self.assertRaises(SystemExit, self._parse_args, args) > -return > -# The new parse_args() sorts by long option string. > -expected[0].sort() > -actual = self._parse_args(args) > -self.assertEqual(actual, expected) > +def test_help(self): > +for opt in '-h', '--help': > +with self.subTest(opt=opt): > +with support.captured_stdout() as out, \ > + self.assertRaises(SystemExit): > +regrtest._parse_args([opt]) > +self.assertIn('Run Python regression tests.', > out.getvalue()) > + > +def test_timeout(self): > +ns = regrtest._parse_args(['--timeout', '4.2']) > +self.assertEqual(ns.timeout, 4.2) > +self.checkError(['--timeout'], 'expected one argument') > +self.checkError(['--timeout', 'foo'], 'invalid float value') > + > +def test_wait(self): > +ns = regrtest._parse_args(['--wait']) > +self.assertTrue(ns.wait) > + > +def test_slaveargs(self): > +ns = regrtest._parse_args(['--slaveargs', '[[], {}]']) > +self.assertEqual(ns.slaveargs, '[[], {}]') > +self.checkError(['--slaveargs'], 'expected one argument') > + > +def test_start(self): > +for opt in '-S', '--start': > +with self.subTest(opt=opt): > +ns = regrtest._parse_args([opt, 'foo']) > +self.assertEqual(ns.start, 'foo') > +self.checkError([opt], 'expected one argument') > + > +def test_verbose(self): > +ns = regrtest._parse_args(['-v']) > +self.assertEqual(ns.verbose, 1) > +ns = regrtest._parse_args(['-vvv']) > +self.assertEqual(ns.verbose, 3) > +ns = regrtest._parse_args(['--verbose']) > +self.assertEqual(ns.verbose, 1) > +ns = regrtest._parse_args(['--verbose'] * 3) > +self.assertEqual(ns.
[Python-Dev] Coverity Scan Spotlight Python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hello, Coverity has published its "Coverity Scan Spotlight Python" a couple of hours ago. It features a summary of Python's ecosystem, an interview with me about Python core development and a defect report. The report is awesome. We have reached a defect density of .005 defects per 1,000 lines of code. In 2012 the average defect density of Open Source Software was 0.69. http://www.coverity.com/company/press-releases/read/coverity-finds-python-sets-new-level-of-quality-for-open-source-software http://wpcme.coverity.com/wp-content/uploads/2013-Coverity-Scan-Spotlight-Python.pdf The internet likes it, too. http://www.prnewswire.com/news-releases/coverity-finds-python-sets-new-level-of-quality-for-open-source-software-221629931.html http://www.securityweek.com/python-gets-high-marks-open-source-software-security-report Thank you very much to Kristin Brennan and Dakshesh Vyas from Coverity as well as everybody who has helped to fix the remaining issues! Christian -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQIcBAEBCgAGBQJSH8bEAAoJEMeIxMHUVQ1FFQcQAL1/Tb5PFMdLXwWsMt9D06aP A2qQPunEnfDBMdQz4GTEeDmHPdjs/EgAtUz4sLI48HlAmpdWEtoVPCdg1GvKSvMi IRVHR5LAtxe5p8M42+8DnSFyIOtEsbtv06W5cHvRxr6RuIkY3bTy0SVhtP9JW+N7 wQKsp2cOIOz/FHDWWQWjxwlZmUWEGkvSSggzbYxcdsaJeGHoJgkuzoChQ3mCtUCo w231OTKBZhGQp/VpMK+Q7OXWm78BZdB6d4GcSR3meCU9GpRMfPBxPF7v4IWvDPv9 4l/y922hmLLoOchJG+PDqcDhX1dnFm1t3Q199iqS5c0c+ttgaMRdSJEXZpZrubxe k+frJiOivG4G7BuzgQ39yF01rRHpjs57FW9FBbt4pp2c+4iOEkgARH+L/e2ZwOnk puXE45AfKwJwHLc4RDOhxdaPy/ovOh53HY68UxXoKjeZKWK5ShRopk0muvYG0y5O +8PbAKOYgJbe//NC3ac89V/1eu4rrFhN7xsK2Wc8i+kcbTB2XIVFElLHuV5wjmLd MMXFlm9LDJFOw12E4sF3MPaHyXQYpNJHvbnuxCkcHRQoLKzrcRJ2Y0Jj4HPSUCsj JhfmHX7Zu+/akmT4haqXUdtRrn4wji0OYqGydEqi4aLy7ELrC1EVNZY4OkbUhJO8 gGbpseJXtVThXQ7fymMS =++g9 -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan Spotlight Python
Great work, Christian! On Thu, Aug 29, 2013 at 3:10 PM, Christian Heimes wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > Hello, > > Coverity has published its "Coverity Scan Spotlight Python" a couple > of hours ago. It features a summary of Python's ecosystem, an > interview with me about Python core development and a defect report. > The report is awesome. We have reached a defect density of .005 > defects per 1,000 lines of code. In 2012 the average defect density of > Open Source Software was 0.69. > > > http://www.coverity.com/company/press-releases/read/coverity-finds-python-sets-new-level-of-quality-for-open-source-software > > > http://wpcme.coverity.com/wp-content/uploads/2013-Coverity-Scan-Spotlight-Python.pdf > > The internet likes it, too. > > > http://www.prnewswire.com/news-releases/coverity-finds-python-sets-new-level-of-quality-for-open-source-software-221629931.html > > > http://www.securityweek.com/python-gets-high-marks-open-source-software-security-report > > > Thank you very much to Kristin Brennan and Dakshesh Vyas from Coverity > as well as everybody who has helped to fix the remaining issues! > > Christian > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.12 (GNU/Linux) > Comment: Using GnuPG with undefined - http://www.enigmail.net/ > > iQIcBAEBCgAGBQJSH8bEAAoJEMeIxMHUVQ1FFQcQAL1/Tb5PFMdLXwWsMt9D06aP > A2qQPunEnfDBMdQz4GTEeDmHPdjs/EgAtUz4sLI48HlAmpdWEtoVPCdg1GvKSvMi > IRVHR5LAtxe5p8M42+8DnSFyIOtEsbtv06W5cHvRxr6RuIkY3bTy0SVhtP9JW+N7 > wQKsp2cOIOz/FHDWWQWjxwlZmUWEGkvSSggzbYxcdsaJeGHoJgkuzoChQ3mCtUCo > w231OTKBZhGQp/VpMK+Q7OXWm78BZdB6d4GcSR3meCU9GpRMfPBxPF7v4IWvDPv9 > 4l/y922hmLLoOchJG+PDqcDhX1dnFm1t3Q199iqS5c0c+ttgaMRdSJEXZpZrubxe > k+frJiOivG4G7BuzgQ39yF01rRHpjs57FW9FBbt4pp2c+4iOEkgARH+L/e2ZwOnk > puXE45AfKwJwHLc4RDOhxdaPy/ovOh53HY68UxXoKjeZKWK5ShRopk0muvYG0y5O > +8PbAKOYgJbe//NC3ac89V/1eu4rrFhN7xsK2Wc8i+kcbTB2XIVFElLHuV5wjmLd > MMXFlm9LDJFOw12E4sF3MPaHyXQYpNJHvbnuxCkcHRQoLKzrcRJ2Y0Jj4HPSUCsj > JhfmHX7Zu+/akmT4haqXUdtRrn4wji0OYqGydEqi4aLy7ELrC1EVNZY4OkbUhJO8 > gGbpseJXtVThXQ7fymMS > =++g9 > -END PGP SIGNATURE- > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/eliben%40gmail.com > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan Spotlight Python
On Fri, 30 Aug 2013 00:10:27 +0200 Christian Heimes wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > Hello, > > Coverity has published its "Coverity Scan Spotlight Python" a couple > of hours ago. It features a summary of Python's ecosystem, an > interview with me about Python core development and a defect report. > The report is awesome. We have reached a defect density of .005 > defects per 1,000 lines of code. What is a defect? Isn't it a bit weird to keep having a non-zero defect density, if those defects are identified? (or, if those defects are not bugs, what is the metric supposed to measure?) Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan Spotlight Python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Am 30.08.2013 00:46, schrieb Antoine Pitrou: > On Fri, 30 Aug 2013 00:10:27 +0200 Christian Heimes > wrote: >> -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 >> >> Hello, >> >> Coverity has published its "Coverity Scan Spotlight Python" a >> couple of hours ago. It features a summary of Python's ecosystem, >> an interview with me about Python core development and a defect >> report. The report is awesome. We have reached a defect density >> of .005 defects per 1,000 lines of code. > > What is a defect? Isn't it a bit weird to keep having a non-zero > defect density, if those defects are identified? > > (or, if those defects are not bugs, what is the metric supposed to > measure?) The last defect is http://bugs.python.org/issue18550 "internal_setblocking() doesn't check return value of fcntl()". It's unlikely that the missing check is going to cause trouble. It's tedious to fix it, too. At least one affected function can't signal an error because it is defined as void. Christian -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iQIcBAEBCgAGBQJSH9adAAoJEMeIxMHUVQ1FU+wQAKEQcZbCrOgD1vIzOdfZXgGV qRHRqhhSoxfhApQ+zhCem/qPGNYBhQyZ4ReXVdCtlvd15p28oa5thFDO7wFfbaBm iQ9mV6nUn3vWgKr2PueEtUrQFd80t4t97AHyU04KblBJjesq8tv5l26i2SGl5YtS QWAJMi3zCbv2iZ2DlyjSs3zpGMzk2mj85dKYtU6ql+mKXH7utR3HUpFiHiL7sjCw D6Q5leORscqoqRxSwNtaT+vAWold5cmWHaH2nGOKj6vaBGKQbFEXRuMAj0sKyPj/ h3N/o+8DAdWH4J3eP8RcIKsai65vmXnzc77s8V2t9kFbuqZn/6CyMwkhsGxsl86h DyN24LhwcB+pK45KFBX92JEhYWQ8OumcfE3Hb/2wIHNFClEvMNSbh7N+5GzjXE0u xpsPjQpT9cldhWOcbPpVFx77zDVvsQczGSiqeH90zKCT7T9AIwUOYrjA0GiO/Nm/ wDMbmyL2/EMkDrnZ+X1YIwWaZOBEQlQofSSVnd1/g0fMm+5kJrW44W1D4grt0hpK TB2uApUCls4qdh3Juu630rMZNKm5/Tvfmtjr/mKHtRCcQvMmhRs2x901/I8ZdwQ+ AoL+yM2qPmsriSTkANGwZHJw2yzTJOv2PXG41ohitE2GdS10i5aRhySVepcjZx/k Gn/FRAsP/AVKReqOVooF =AyxK -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan Spotlight Python
Do the numbers add up? .005 defects in 1,000 lines of code is one defect in every 200,000 lines of code. However they also claim that "to date, the Coverity Scan service has analyzed nearly 400,000 lines of Python code and identified 996 new defects – 860 of which have been fixed by the Python community." Sturla Sendt fra min iPad Den 30. aug. 2013 kl. 00:10 skrev Christian Heimes : > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > Hello, > > Coverity has published its "Coverity Scan Spotlight Python" a couple > of hours ago. It features a summary of Python's ecosystem, an > interview with me about Python core development and a defect report. > The report is awesome. We have reached a defect density of .005 > defects per 1,000 lines of code. In 2012 the average defect density of > Open Source Software was 0.69. > > http://www.coverity.com/company/press-releases/read/coverity-finds-python-sets-new-level-of-quality-for-open-source-software > > http://wpcme.coverity.com/wp-content/uploads/2013-Coverity-Scan-Spotlight-Python.pdf > > The internet likes it, too. > > http://www.prnewswire.com/news-releases/coverity-finds-python-sets-new-level-of-quality-for-open-source-software-221629931.html > > http://www.securityweek.com/python-gets-high-marks-open-source-software-security-report > > > Thank you very much to Kristin Brennan and Dakshesh Vyas from Coverity > as well as everybody who has helped to fix the remaining issues! > > Christian > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.12 (GNU/Linux) > Comment: Using GnuPG with undefined - http://www.enigmail.net/ > > iQIcBAEBCgAGBQJSH8bEAAoJEMeIxMHUVQ1FFQcQAL1/Tb5PFMdLXwWsMt9D06aP > A2qQPunEnfDBMdQz4GTEeDmHPdjs/EgAtUz4sLI48HlAmpdWEtoVPCdg1GvKSvMi > IRVHR5LAtxe5p8M42+8DnSFyIOtEsbtv06W5cHvRxr6RuIkY3bTy0SVhtP9JW+N7 > wQKsp2cOIOz/FHDWWQWjxwlZmUWEGkvSSggzbYxcdsaJeGHoJgkuzoChQ3mCtUCo > w231OTKBZhGQp/VpMK+Q7OXWm78BZdB6d4GcSR3meCU9GpRMfPBxPF7v4IWvDPv9 > 4l/y922hmLLoOchJG+PDqcDhX1dnFm1t3Q199iqS5c0c+ttgaMRdSJEXZpZrubxe > k+frJiOivG4G7BuzgQ39yF01rRHpjs57FW9FBbt4pp2c+4iOEkgARH+L/e2ZwOnk > puXE45AfKwJwHLc4RDOhxdaPy/ovOh53HY68UxXoKjeZKWK5ShRopk0muvYG0y5O > +8PbAKOYgJbe//NC3ac89V/1eu4rrFhN7xsK2Wc8i+kcbTB2XIVFElLHuV5wjmLd > MMXFlm9LDJFOw12E4sF3MPaHyXQYpNJHvbnuxCkcHRQoLKzrcRJ2Y0Jj4HPSUCsj > JhfmHX7Zu+/akmT4haqXUdtRrn4wji0OYqGydEqi4aLy7ELrC1EVNZY4OkbUhJO8 > gGbpseJXtVThXQ7fymMS > =++g9 > -END PGP SIGNATURE- > > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/sturla%40molden.no ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Can someone try to duplicate corruption on Gentoo?
In http://bugs.python.org/issue18843 a user reported a debug PyMalloc "bad leading pad byte" memory corruption death while running their code. After some thrashing, they decided to rebuild Python, and got the same kind of error while rebuilding Python. See http://bugs.python.org/msg196481 in that bug report: """ # emerge dev-lang/python:2.7 * IMPORTANT: 11 news items need reading for repository 'gentoo'. * Use eselect news to read news items. Calculating dependencies... done! Debug memory block at address p=0xa7f5900: API 'o' 80 bytes originally requested The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfb): at p-7: 0xfb at p-6: 0xfb at p-5: 0xfa *** OUCH at p-4: 0xfb at p-3: 0xfb at p-2: 0xfb at p-1: 0xfb Because memory is corrupted at the start, the count of bytes requested may be bogus, and checking the trailing pad bytes may segfault. The 8 pad bytes at tail=0xa7f5950 are FORBIDDENBYTE, as expected. The block was made by call #21242094 to debug malloc/realloc. Data at p: 73 00 00 00 79 00 00 00 ... 67 00 00 00 00 00 00 00 Fatal Python error: bad leading pad byte Aborted (core dumped) # """ I don't have access to Gentoo, and don't know squat about its `emerge`, but if someone else can do this it might help ;-) The Python used to run `emerge` here was a --with-pydebug Python the bug reporter built earlier. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan Spotlight Python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/29/2013 07:24 PM, Sturla Molden wrote: > > Do the numbers add up? > > .005 defects in 1,000 lines of code is one defect in every 200,000 > lines of code. > > However they also claim that "to date, the Coverity Scan service has > analyzed nearly 400,000 lines of Python code and identified 996 new > defects – 860 of which have been fixed by the Python community." FWIW: David Wheeler's 'sloccount' reports 800,489 lines of code in the Python 3.3.1 tarball, of which 403,266 lines are Python code, and 368,474 are ANSI C. That defect rate would imply 4 open defects in Python itself. Tres. - -- === Tres Seaver +1 540-429-0999 [email protected] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlIf6e0ACgkQ+gerLs4ltQ6X6wCgosAIUJyGjcBqbeAMLwMH24TJ j3cAoNKPEuKEbVmke2IZuSdtl2nMAFL4 =MoZm -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can someone try to duplicate corruption on Gentoo?
On Thu, 29 Aug 2013 18:43:13 -0500, Tim Peters wrote: > In > > http://bugs.python.org/issue18843 > > a user reported a debug PyMalloc "bad leading pad byte" memory > corruption death while running their code. After some thrashing, they > decided to rebuild Python, and got the same kind of error while > rebuilding Python. See > > http://bugs.python.org/msg196481 > > in that bug report: > > """ > # emerge dev-lang/python:2.7 > > * IMPORTANT: 11 news items need reading for repository 'gentoo'. > * Use eselect news to read news items. > > Calculating dependencies... done! > Debug memory block at address p=0xa7f5900: API 'o' > 80 bytes originally requested > The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfb): > at p-7: 0xfb > at p-6: 0xfb > at p-5: 0xfa *** OUCH > at p-4: 0xfb > at p-3: 0xfb > at p-2: 0xfb > at p-1: 0xfb > Because memory is corrupted at the start, the count of bytes requested >may be bogus, and checking the trailing pad bytes may segfault. > The 8 pad bytes at tail=0xa7f5950 are FORBIDDENBYTE, as expected. > The block was made by call #21242094 to debug malloc/realloc. > Data at p: 73 00 00 00 79 00 00 00 ... 67 00 00 00 00 00 00 00 > Fatal Python error: bad leading pad byte > Aborted (core dumped) Emerge uses Python, and 2.7 is the default system python on Gentoo, so unless he changed his default, that error almost certainly came from the existing Python he was having trouble with. Just for fun I re-emerged 2.7.3-r3 on my system, and that worked fine. --David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can someone try to duplicate corruption on Gentoo?
[R. David Murray ] > Emerge uses Python, and 2.7 is the default system python on Gentoo, > so unless he changed his default, that error almost certainly came from > the existing Python he was having trouble with. Yes, "The Python used to run `emerge` here was a --with-pydebug Python the bug reporter built earlier". > Just for fun I re-emerged 2.7.3-r3 on my system, and that worked fine. Thanks for trying! Note that only a debug-build (or at least with PYMALLOC_DEBUG defined) Python can generate a "bad leading pad byte" error, so if you didn't use a debug-build Python to run `emerge`, you could not have seen the same error. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Coverity Scan Spotlight Python
On 8/29/2013 7:24 PM, Sturla Molden wrote: Do the numbers add up? .005 defects in 1,000 lines of code is one defect in every 200,000 lines of code. However they also claim that "to date, the Coverity Scan service has analyzed nearly 400,000 lines of Python code and identified 996 new defects – 860 of which have been fixed by the Python community." Some marked as 'false positive', some as 'intentional'. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
