[Python-Dev] Re: Detect memory leaks in unit tests
> Py_DECREF calls in the C code I think this part specifically is already covered through refleak checks: https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py Since it can involve the repetition of tests many times, these aren't run on the CI though, they do get run on the refleak buildbots so issues get caught eventually: https://buildbot.python.org/all/#/builders?tags=%2Brefleak But again this is for PyObjects only. Were you able to find any memory leaks with your proof-of-concept? I don't think there's a lot of chances of someone missing a PyMem_Free call and there's not a lot of other manual memory management but I could be wrong. Anything found there could help motivate adding this a bit more. On Tue, May 12, 2020 at 4:59 PM Giampaolo Rodola' wrote: > > Hello there, > I would like to discuss a proposal regarding one aspect which AFAIK is > currently missing from cPython's test suite: the ability to detect memory > leaks of functions implemented in the C extension modules. > In psutil I use a test class/framework which calls a function many times, and > fails if the process memory increased after doing so. I do this in order to > quickly detect missing free() or Py_DECREF calls in the C code, but I suppose > there may be other use cases. Here's the class: > https://github.com/giampaolo/psutil/blob/913d4b1d6dcce88dea6ef9382b93883a04a66cd7/psutil/tests/__init__.py#L901 > > Detecting a memory leak is no easy task, and that's because the process > memory fluctuates. Sometimes it may increase (or even decrease!) even if > there's no leak, I suppose because of how the OS handles memory, the Python's > garbage collector, the fact that RSS is an approximation, and who knows what > else. In order to compensate fluctuations I did the following: in case of > failure (mem > 0 after calling fun() N times) I retry the test for up to 5 > times, increasing N (repetitions) each time, so I consider the test a failure > only if the memory keeps increasing across all runs. So for instance, here's > a legitimate failure: > > > psutil.tests.test_memory_leaks.TestModuleFunctionsLeaks.test_disk_partitions > ... > Run #1: extra-mem=696.0K, per-call=3.5K, calls=200 > Run #2: extra-mem=1.4M, per-call=3.5K, calls=400 > Run #3: extra-mem=2.1M, per-call=3.5K, calls=600 > Run #4: extra-mem=2.7M, per-call=3.5K, calls=800 > Run #5: extra-mem=3.4M, per-call=3.5K, calls=1000 > FAIL > > If, on the other hand, the memory increased on one run (say 200 calls) but > decreased on the next run (say 400 calls), then it clearly means it's a false > positive, because memory consumption may be > 0 on the second run, but if > it's lower than the previous run with less repetitions, then it cannot > possibly represent a leak (just a fluctuation): > > > psutil.tests.test_memory_leaks.TestModuleFunctionsLeaks.test_net_connections > ... > Run #1: extra-mem=568.0K, per-call=2.8K, calls=200 > Run #2: extra-mem=24.0K, per-call=61.4B, calls=400 > OK > > This is the best I could come up with as a simple leak detection mechanism to > integrate with CI services, and keep more advanced tools like Valgrind out of > the picture (I just wanted to know if there's a leak, not to debug the leak > itself). In addition, since psutil is able to get the number of fds (UNIX) > and handles (Windows) opened by a process, I also run a separate set of tests > to make sure I didn't forget to call close(2) or CloseHandle() in C. > > Would something like this make sense to have in cPython? Here's a quick PoC I > put together just to show how this thing would look like in practice: > https://github.com/giampaolo/cpython/pull/2/files > A proper work in terms of API coverage would result being quite huge (test > all C modules), and ideally should also include cases where functions raise > an exception when being fed with an improper input. The biggest stopper here > is, of course, psutil, since it's a third party dep, but before getting to > that I wanted to see how this idea is perceived in general. > > Cheers, > > -- > Giampaolo - http://grodola.blogspot.com > > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/NFHW4TP3ALY3CVRBVKRI4SRG7BOLZLJH/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/DFITPQJHVHFXVODGNBCJOJUZM7YG2WCI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Detect memory leaks in unit tests
On Wed, May 13, 2020 at 9:17 AM Ammar Askar wrote:
> > Py_DECREF calls in the C code
>
> I think this part specifically is already covered through refleak
> checks:
> https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py
>
> Since it can involve the repetition of tests many times, these aren't
> run on the CI though, they do get run on the refleak buildbots so
> issues get caught eventually:
> https://buildbot.python.org/all/#/builders?tags=%2Brefleak
>
> But again this is for PyObjects only. Were you able to find any memory
> leaks with your proof-of-concept? I don't think there's a lot of
> chances of someone missing a PyMem_Free call and there's not a lot of
> other manual memory management but I could be wrong. Anything found
> there could help motivate adding this a bit more.
Yeah, I agree it depends on how many PyMem_* occurrences are there, and it
probably makes more sense to cover those ones only. Under Modules/* I found:
- 24 occurrences for PyMem_RawMalloc
- 2 for PyMem_RawCalloc
- 106 for PyMem_Malloc
- 12 for PyMem_Calloc
- 39 for PyMem_New
- 5 for " = malloc("
I spent an hour covering around 20 of those and didn't find any leak. It's
a boring work. I will try to work on it over the next few weeks.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/C7ZZRDPGIUS7Q6Q4AS4YFPD2OOF56JBO/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Detect memory leaks in unit tests
> But again this is for PyObjects only.
Not really, we check also memory blocks:
https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py#L72
as long as you don't directly call malloc and use one of the Python
specific APIs like PyMem_Malloc
then the reflect code should catch that.
On Wed, 13 May 2020 at 12:29, Giampaolo Rodola' wrote:
> On Wed, May 13, 2020 at 9:17 AM Ammar Askar wrote:
>
>> > Py_DECREF calls in the C code
>>
>> I think this part specifically is already covered through refleak
>> checks:
>> https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py
>>
>> Since it can involve the repetition of tests many times, these aren't
>> run on the CI though, they do get run on the refleak buildbots so
>> issues get caught eventually:
>> https://buildbot.python.org/all/#/builders?tags=%2Brefleak
>>
>> But again this is for PyObjects only. Were you able to find any memory
>> leaks with your proof-of-concept? I don't think there's a lot of
>> chances of someone missing a PyMem_Free call and there's not a lot of
>> other manual memory management but I could be wrong. Anything found
>> there could help motivate adding this a bit more.
>
>
> Yeah, I agree it depends on how many PyMem_* occurrences are there, and it
> probably makes more sense to cover those ones only. Under Modules/* I found:
>
> - 24 occurrences for PyMem_RawMalloc
> - 2 for PyMem_RawCalloc
> - 106 for PyMem_Malloc
> - 12 for PyMem_Calloc
> - 39 for PyMem_New
> - 5 for " = malloc("
>
> I spent an hour covering around 20 of those and didn't find any leak. It's
> a boring work. I will try to work on it over the next few weeks.
>
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/C7ZZRDPGIUS7Q6Q4AS4YFPD2OOF56JBO/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/YNBGAWNCCHF3XK3MBHJWZI4NKTNEPKV6/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Detect memory leaks in unit tests
Hi, Le mer. 13 mai 2020 à 02:04, Giampaolo Rodola' a écrit : > I would like to discuss a proposal regarding one aspect which AFAIK is > currently missing from cPython's test suite: the ability to detect memory > leaks of functions implemented in the C extension modules. test.regrtest can be used to detect 3 kinds of leaks: * reference leaks: use sys.gettotalrefcount() * memory block leaks: use sys.getallocatedblocks() * file descriptor leaks: use test.support.fd_count() See Lib/test/libregrtest/refleak.py. > Detecting a memory leak is no easy task, and that's because the process > memory fluctuates. Sometimes it may increase (or even decrease!) even if > there's no leak, I suppose because of how the OS handles memory, the Python's > garbage collector, the fact that RSS is an approximation, and who knows what > else. In order to compensate fluctuations I did the following: in case of > failure (mem > 0 after calling fun() N times) I retry the test for up to 5 > times, increasing N (repetitions) each time, so I consider the test a failure > only if the memory keeps increasing across all runs. So for instance, here's > a legitimate failure: > > > psutil.tests.test_memory_leaks.TestModuleFunctionsLeaks.test_disk_partitions > ... > Run #1: extra-mem=696.0K, per-call=3.5K, calls=200 > Run #2: extra-mem=1.4M, per-call=3.5K, calls=400 > Run #3: extra-mem=2.1M, per-call=3.5K, calls=600 > Run #4: extra-mem=2.7M, per-call=3.5K, calls=800 > Run #5: extra-mem=3.4M, per-call=3.5K, calls=1000 > FAIL regrtest usually uses 3 test runs to "warmup" Python: fill caches. Then it runs the test 3 more times and check for differences. For references and memory blocks, it only consider that there is a leak if each test run increased the counter difference by at least one. For file descriptor, it considers that there is a leak if any test run changed a counter. Before reading "counters", regrtest tries to clear "all" caches that it knows in the stdlib. Examples: path importer cache, re module cache, type method cache, etc. See dash_R_cleanup() function of test.libregrtest.refleak. Sadly, there are still a few false alarms time to time, like: "test_functools leaked [1, 2, 1] memory blocks, sum=4" https://bugs.python.org/issue36560 > This is the best I could come up with as a simple leak detection mechanism to > integrate with CI services, and keep more advanced tools like Valgrind out of > the picture (I just wanted to know if there's a leak, not to debug the leak > itself). In addition, since psutil is able to get the number of fds (UNIX) > and handles (Windows) opened by a process, I also run a separate set of tests > to make sure I didn't forget to call close(2) or CloseHandle() in C. I tried to modify regrtest to check for leak of Windows handles: https://bugs.python.org/issue18174 But many stdlib modules leak handles in various cases. I gave up when I failed to fix a race condition in multiprocessing: https://bugs.python.org/issue33966 The parent expects the child process to "steal" a handle, but sometimes the child process is killed before it steals the handle... -- I also tried to check for PyMem_RawMalloc() memory leaks, but it made regrtest not reliable at all: https://bugs.python.org/issue26850 I understood that CPython has many internal caches and regrtest fails to clear them all, or it was something different. I never investigated. > Would something like this make sense to have in cPython? Here's a quick PoC I > put together just to show how this thing would look like in practice: > https://github.com/giampaolo/cpython/pull/2/files > A proper work in terms of API coverage would result being quite huge (test > all C modules), and ideally should also include cases where functions raise > an exception when being fed with an improper input. The biggest stopper here > is, of course, psutil, since it's a third party dep, but before getting to > that I wanted to see how this idea is perceived in general. regrtest has many features, sadly it's not an officialy API. It would be nice if someone could try to move some of its features into unittest. Sadly, regrtest refleak feature also rely a lot on CPython internals. Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/X5COX25VKMH455Y7OBHJA2FFJOGNOLKC/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Detect memory leaks in unit tests
C extensions "should" use PyMem_Malloc/PyMem_Free to benefit of PEP
445 hooks like debug hooks and tracemalloc.
* Debug hooks (enabled by -X dev) helps to debug buffer overflow:
https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks
* tracemalloc: https://docs.python.org/dev/library/tracemalloc.html
* PEP 445, API for memory allocators: https://www.python.org/dev/peps/pep-0445/
tracemalloc is used more and more often in CPython internals to
provide more information when an issue is detected. Example with
ResourceWarning:
https://docs.python.org/dev/library/devmode.html#resourcewarning-example
Buffer overflow:
"On error, the debug hooks use the tracemalloc module to get the
traceback where a memory block was allocated. The traceback is only
displayed if tracemalloc is tracing Python memory allocations and the
memory block was traced."
Also, PyMem_Malloc/PyMem_Free now use pymalloc memory allocator which
is more efficient than libc malloc/free for memory blocks smaller than
or equal to 512 bytes. When I modified them to use pymalloc, I saw a
global speedup around 10% faster. But these functions require the
caller to hold the GIL. Use PyMem_RawMalloc/PyMem_RawFree if you don't
hold the GIL.
Note: The Python ssl module doesn't use PyMem_Malloc/PyMem_Free for
OpenSSL. Last time I tried, the OpenSSL function to set a custom
memory allocator didn't work at all. Maybe it has been fixed in the
meanwhile.
https://bugs.python.org/issue18227#msg191610
See https://bugs.python.org/issue18227#msg191250 for the list of C
extensions which may be modified to use Python memory allocators.
Victor
Le mer. 13 mai 2020 à 14:23, Pablo Galindo Salgado
a écrit :
>
> > But again this is for PyObjects only.
>
> Not really, we check also memory blocks:
>
> https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py#L72
>
> as long as you don't directly call malloc and use one of the Python specific
> APIs like PyMem_Malloc
> then the reflect code should catch that.
>
> On Wed, 13 May 2020 at 12:29, Giampaolo Rodola' wrote:
>>
>> On Wed, May 13, 2020 at 9:17 AM Ammar Askar wrote:
>>>
>>> > Py_DECREF calls in the C code
>>>
>>> I think this part specifically is already covered through refleak
>>> checks:
>>> https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py
>>>
>>> Since it can involve the repetition of tests many times, these aren't
>>> run on the CI though, they do get run on the refleak buildbots so
>>> issues get caught eventually:
>>> https://buildbot.python.org/all/#/builders?tags=%2Brefleak
>>>
>>> But again this is for PyObjects only. Were you able to find any memory
>>> leaks with your proof-of-concept? I don't think there's a lot of
>>> chances of someone missing a PyMem_Free call and there's not a lot of
>>> other manual memory management but I could be wrong. Anything found
>>> there could help motivate adding this a bit more.
>>
>>
>> Yeah, I agree it depends on how many PyMem_* occurrences are there, and it
>> probably makes more sense to cover those ones only. Under Modules/* I found:
>>
>> - 24 occurrences for PyMem_RawMalloc
>> - 2 for PyMem_RawCalloc
>> - 106 for PyMem_Malloc
>> - 12 for PyMem_Calloc
>> - 39 for PyMem_New
>> - 5 for " = malloc("
>>
>> I spent an hour covering around 20 of those and didn't find any leak. It's a
>> boring work. I will try to work on it over the next few weeks.
>>
>> ___
>> Python-Dev mailing list -- [email protected]
>> To unsubscribe send an email to [email protected]
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/[email protected]/message/C7ZZRDPGIUS7Q6Q4AS4YFPD2OOF56JBO/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/YNBGAWNCCHF3XK3MBHJWZI4NKTNEPKV6/
> Code of Conduct: http://python.org/psf/codeofconduct/
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/APHU4VQMD2IRUNI4YMOQ7IWTMXYN6AIY/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Detect memory leaks in unit tests
Aah, thanks for pointing this out. As a quick side note: are these cool
regrtest features documented anywhere? I remember seeing some mention of
the -R 3:3 argument in the dev guide but nothing else.
Probably useful to have this documented somewhere in the dev guide.
On Wed, May 13, 2020, 5:14 AM Pablo Galindo Salgado
wrote:
> > But again this is for PyObjects only.
>
> Not really, we check also memory blocks:
>
>
> https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py#L72
>
> as long as you don't directly call malloc and use one of the Python
> specific APIs like PyMem_Malloc
> then the reflect code should catch that.
>
> On Wed, 13 May 2020 at 12:29, Giampaolo Rodola'
> wrote:
>
>> On Wed, May 13, 2020 at 9:17 AM Ammar Askar wrote:
>>
>>> > Py_DECREF calls in the C code
>>>
>>> I think this part specifically is already covered through refleak
>>> checks:
>>> https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py
>>>
>>> Since it can involve the repetition of tests many times, these aren't
>>> run on the CI though, they do get run on the refleak buildbots so
>>> issues get caught eventually:
>>> https://buildbot.python.org/all/#/builders?tags=%2Brefleak
>>>
>>> But again this is for PyObjects only. Were you able to find any memory
>>> leaks with your proof-of-concept? I don't think there's a lot of
>>> chances of someone missing a PyMem_Free call and there's not a lot of
>>> other manual memory management but I could be wrong. Anything found
>>> there could help motivate adding this a bit more.
>>
>>
>> Yeah, I agree it depends on how many PyMem_* occurrences are there, and
>> it probably makes more sense to cover those ones only. Under Modules/* I
>> found:
>>
>> - 24 occurrences for PyMem_RawMalloc
>> - 2 for PyMem_RawCalloc
>> - 106 for PyMem_Malloc
>> - 12 for PyMem_Calloc
>> - 39 for PyMem_New
>> - 5 for " = malloc("
>>
>> I spent an hour covering around 20 of those and didn't find any leak.
>> It's a boring work. I will try to work on it over the next few weeks.
>>
>> ___
>> Python-Dev mailing list -- [email protected]
>> To unsubscribe send an email to [email protected]
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/[email protected]/message/C7ZZRDPGIUS7Q6Q4AS4YFPD2OOF56JBO/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/GFY67CXZIJVGWITWXPJ3Y2OMKMAIPJ7Z/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 618: Add Optional Length-Checking To zip
On Tue, 12 May 2020 11:36:38 -0700 Ethan Furman wrote: > > Also, if a flag is used, won't that slow down every call to zip even when the > flag is False? Why would it? Is that based on a rational analysis of how argument parsing works? Regards Antoine. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/ICEW7BIU43EPR26AXF4NT25ZY6WH7BIA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Detect memory leaks in unit tests
On Wed, 2020-05-13 at 13:14 +0100, Pablo Galindo Salgado wrote:
> > But again this is for PyObjects only.
>
> Not really, we check also memory blocks:
>
> https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py#L72
>
> as long as you don't directly call malloc and use one of the Python
> specific APIs like PyMem_Malloc
> then the reflect code should catch that.
Maybe worth briefly mentioning in this discussion that pytest-leaks
exists:
https://github.com/abalkin/pytest-leaks
which serves much the same purpose as refleak.py for pytest users. (I
run it semi-regularly on NumPy and it helped me get the tests de-facto
free of leaks.) I have used valgrind for similar purpose, but
refleaks/pytest-leaks is better for leaks.
Cheers,
Sebastian
>
> On Wed, 13 May 2020 at 12:29, Giampaolo Rodola'
> wrote:
>
> > On Wed, May 13, 2020 at 9:17 AM Ammar Askar
> > wrote:
> >
> > > > Py_DECREF calls in the C code
> > >
> > > I think this part specifically is already covered through refleak
> > > checks:
> > > https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py
> > >
> > > Since it can involve the repetition of tests many times, these
> > > aren't
> > > run on the CI though, they do get run on the refleak buildbots so
> > > issues get caught eventually:
> > > https://buildbot.python.org/all/#/builders?tags=%2Brefleak
> > >
> > > But again this is for PyObjects only. Were you able to find any
> > > memory
> > > leaks with your proof-of-concept? I don't think there's a lot of
> > > chances of someone missing a PyMem_Free call and there's not a
> > > lot of
> > > other manual memory management but I could be wrong. Anything
> > > found
> > > there could help motivate adding this a bit more.
> >
> > Yeah, I agree it depends on how many PyMem_* occurrences are there,
> > and it
> > probably makes more sense to cover those ones only. Under Modules/*
> > I found:
> >
> > - 24 occurrences for PyMem_RawMalloc
> > - 2 for PyMem_RawCalloc
> > - 106 for PyMem_Malloc
> > - 12 for PyMem_Calloc
> > - 39 for PyMem_New
> > - 5 for " = malloc("
> >
> > I spent an hour covering around 20 of those and didn't find any
> > leak. It's
> > a boring work. I will try to work on it over the next few weeks.
> >
> > ___
> > Python-Dev mailing list -- [email protected]
> > To unsubscribe send an email to [email protected]
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> > https://mail.python.org/archives/list/[email protected]/message/C7ZZRDPGIUS7Q6Q4AS4YFPD2OOF56JBO/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
>
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/YNBGAWNCCHF3XK3MBHJWZI4NKTNEPKV6/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/TOPMBKEH7VVKMDZM7QE6XQ6GUU64WKCW/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 618: Add Optional Length-Checking To zip
On Wed, May 13, 2020 at 03:04:02PM +0200, Antoine Pitrou wrote: > On Tue, 12 May 2020 11:36:38 -0700 > Ethan Furman wrote: > > > > Also, if a flag is used, won't that slow down every call to zip even when > > the flag is False? > > Why would it? Is that based on a rational analysis of how argument > parsing works? I'm not an expert on Python's argument passing code in fine detail, but I'm reasonable sure that it takes longer to pass arguments by keyword than by position, and it takes time to fill in missing arguments with the default (they have to be read from the function defaults, which takes time) and these things don't happen for free. But I do think that's probably a spurious point based on a micro- or even nano-optimization. I don't think that the extra nanosecond or whatever it takes for zip to receive an extra argument, fill in the default if missing, and branch is significant. I don't think that `zip(*args, strict=flag)` is the wrong choice because it's a nanosecond slower, I think it's the wrong choice because it makes for a poorer API when there are better, more future-proof choices available. -- Steven ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/HRU3M6ZZ7QRC7QVSCA3EQMHGXJAIGN6G/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 618: Add Optional Length-Checking To zip
On 05/13/2020 06:04 AM, Antoine Pitrou wrote: On Tue, 12 May 2020 11:36:38 -0700 Ethan Furman wrote: Also, if a flag is used, won't that slow down every call to zip even when the flag is False? Why would it? Is that based on a rational analysis of how argument parsing works? Not the call itself, but the running of zip. Absent some clever programming it seems to me that there are two choices if we have a flag: - have two independent branches (basically `zip` and a `zip_strict` functions inside `zip` itself) and have the flag select the branch; or - have one branch which has extra logic that checks for at least one StopIteration and at least one item in each iteration to see if it should raise. If the first method is chosen then we may as well have two different functions; if the second method is chosen that would seem to be a performance hit, even when the flag is False. -- ~Ethan~ ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/BXSBYTYSRNK3ZTWBDKPIGYDOMV2EPCDA/ Code of Conduct: http://python.org/psf/codeofconduct/
