[Python-Dev] Documenting METH_FASTCALL
Hello, has the time come to document METH_FASTCALL? This was introduced in Python 3.6 for positional arguments only and extended in Python 3.7 to support also keyword arguments. No changes have been made since then. The vectorcall protocol (PEP 590) uses a calling convention based on METH_FASTCALL, so I expect METH_FASTCALL to stay as it is. PEP 590 also means that CPython is now even more than before optimized for METH_FASTCALL instead of METH_VARARGS. Whether or not METH_FASTCALL should be added to the limited API is another question. I'm only talking about adding it to the documentation. Jeroen. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/RDSZ3JUQ6RK3CYVWXA2ESW7KF3HND3B7/
[Python-Dev] Re: Parser module in the stdlib
Le 16/05/2019 à 23:12, Pablo Galindo Salgado a écrit : > I propose to remove the current parser module and expose pgen2 as a > standard library module. As a first step, I created https://bugs.python.org/issue37268 to propose to deprecate the parser module already in Python 3.8. Victor ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/T3IEJQ5QPLCEWZ6BNUF33FNRZRY3J2AC/
[Python-Dev] Re: Documenting METH_FASTCALL
Hi Jeroen, My initial plan was to hide it to ensure that nobody uses it, wait until the code and the API is battle-tested, and then propose a public API. I was a good idea to not make it public in the first place, since Python 3.7 modified FASTCALL deeply to not include keyword parameters by default (need METH_FASTCALL | METH_KEYWORDS). But Cython decided to use it, it was not well hidden enough :-) Then the PEP 590 did what I planned to do: polish the API and make it public (good job! it optimizes more cases than FASTCALL, especially methods). Well, yes, it should now be documented, since it's already in used in the wild. We may deprecate it and document that VECTORCALL should be preferred. Victor Le 13/06/2019 à 09:29, Jeroen Demeyer a écrit : Hello, has the time come to document METH_FASTCALL? This was introduced in Python 3.6 for positional arguments only and extended in Python 3.7 to support also keyword arguments. No changes have been made since then. The vectorcall protocol (PEP 590) uses a calling convention based on METH_FASTCALL, so I expect METH_FASTCALL to stay as it is. PEP 590 also means that CPython is now even more than before optimized for METH_FASTCALL instead of METH_VARARGS. Whether or not METH_FASTCALL should be added to the limited API is another question. I'm only talking about adding it to the documentation. Jeroen. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/RDSZ3JUQ6RK3CYVWXA2ESW7KF3HND3B7/ ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/5DX6SMA3QAG4367S4KVAH3JT4V4W2W45/
[Python-Dev] Re: Documenting METH_FASTCALL
On 2019-06-13 14:21, Victor Stinner wrote: We may deprecate it and document that VECTORCALL should be preferred. Not really. Vectorcall and METH_FASTCALL solve different problems on different levels. METH_FASTCALL is used specifically in PythonMethodDef, in other words for instances of "method_descriptor" or "builtin_function_or_method". It has no meaning outside of that. Vectorcall is meant for classes implementing callables. It is used for example to implement the *classes* "function", "method", "method_descriptor" and "builtin_function_or_method". The expectation is that most manually written C extensions will be happy with the functionality exposed by "method_descriptor" and "builtin_function_or_method" and they can use METH_FASTCALL. Cython on the other hand already has a custom class for callables and the expectation is that this class will implement vectorcall. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/X3KXJVCOLGISBFETCBWMOKLI4DLM4PD3/
[Python-Dev] PyAPI_FUNC() is needed to private APIs?
Hi, all. Some functions in cpython are private. These APIs are called from only python executable or DLL. They are not called even from extension modules in stdlib. In this time, I want to keep _PyObject_GetMethod private. https://github.com/python/cpython/pull/14015#discussion_r293351734 As far as I know, it is used to expose the function from DLL user on Windows. So I think when the private function is not intended to be called from outside of DLL, it should not use PyAPI_FUNC. Am I collect? Currently, many private APIs uses `PyAPI_FUNC()`. Is there any downside about having much unnecessary exported functions? (e.g. slower calling invention is used, bothering link time optimization, LoadLibrary get slower, etc...) Regards, -- Inada Naoki ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/PC6PWMPO47Q7S5SQQYOSJMAIMCOYVKWK/
[Python-Dev] Re: PyAPI_FUNC() is needed to private APIs?
Hi, Le 13/06/2019 à 15:05, Inada Naoki a écrit : Some functions in cpython are private. These APIs are called from only python executable or DLL. They are not called even from extension modules in stdlib. Python 3.8 now has a better separation between "private" and "internal" APIs: * private are "_Py" symbols which are exported in the python DLL: they should not be used, but can be used techically * internal are symbols defined in internal header files (Include/internal/). Some symbols use PyAPI_FUNC()/PyAPI_DATA(), some only use "extern". I'm in favor of moving towards "extern" for new internal APIs. I'm trying to keep PyAPI_FUNC/PyAPI_DATA to export symbols in DLL for things which might be useful for 3rd party tools like debuggers or profilers. Currently, many private APIs uses `PyAPI_FUNC()`. Well, that's mostly for historical reasons :-) Is there any downside about having much unnecessary exported functions? The main risk is that people start to use it, expect these APIs to be there forever, and might be surprised that their code fail with the newer Python. My plan for http://pythoncapi.readthedocs.io/ is to *reduce* the size of the C API and hide as much as possible implementation details. Export a new function that doesn't make sense outside Python internals goes against this plan. -- IMHO private (ex: Include/cpython/ headers) must continue to use PyAPI_FUNC/PyAPI_DATA. I converted some *private* functions to internal functions, but I didn't always replace PyAPI_FUNC with extern. For private functions, the contract is that they can change whenever: there is no warranty (and they must not be used ;-)). Victor ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/KNICIXJ2ZYYGGDTSC4IFP76GCRIYHLRS/
[Python-Dev] Re: PyAPI_FUNC() is needed to private APIs?
On 2019-06-13 15:36, Victor Stinner wrote: The main risk is that people start to use it If people use it, it should be taken as a sign that the function is useful and deserves to be public API. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/YRXR5TYMKCH2E3GSCJEEZ6AT4J3L4WDL/
[Python-Dev] Re: PyAPI_FUNC() is needed to private APIs?
On Thu, Jun 13, 2019 at 10:37 PM Victor Stinner wrote: > > Python 3.8 now has a better separation between "private" and "internal" > APIs: > > * private are "_Py" symbols which are exported in the python DLL: they > should not be used, but can be used techically > > * internal are symbols defined in internal header files > (Include/internal/). Some symbols use PyAPI_FUNC()/PyAPI_DATA(), some > only use "extern". > Thank you for clarifying. I confused always about it. > I'm in favor of moving towards "extern" for new internal APIs. I'm > trying to keep PyAPI_FUNC/PyAPI_DATA to export symbols in DLL for things > which might be useful for 3rd party tools like debuggers or profilers. Hmm, debugger or profiler need this? I thought only loaders and DLL explorers use this. > > > Currently, many private APIs uses `PyAPI_FUNC()`. > > Well, that's mostly for historical reasons :-) OK, I see. > > > Is there any downside about having much unnecessary exported functions? > > The main risk is that people start to use it, expect these APIs to be > there forever, and might be surprised that their code fail with the > newer Python. I know, and I agree with both of you and Jeroen. But I concern about performance, stack memory usage, and binary size. By quick googling, I find some answers. https://docs.microsoft.com/en-us/cpp/cpp/dllexport-dllimport?view=vs-2019 It seems dllexport doesn't affect to calling convention. https://clang.llvm.org/docs/LTOVisibility.html https://devblogs.microsoft.com/oldnewthing/?p=2123 It seems dllexport affects linker. At least, linker can not remove dllexport-ed function even if the function is not called anywhere in the DLL. Regards, -- Inada Naoki ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/VMFW7Y4LHHNYMQRR4UMCMV4X54CBMSQI/
[Python-Dev] Re: PyAPI_FUNC() is needed to private APIs?
On 13Jun2019 0643, Jeroen Demeyer wrote: On 2019-06-13 15:36, Victor Stinner wrote: The main risk is that people start to use it If people use it, it should be taken as a sign that the function is useful and deserves to be public API. If it's useful, then someone can write a justification (possibly a PEP) and we can agree to add it. More likely it's just convenient. The cost of that convenience is that we can never optimise internals because they are now public API. Better off never to have leaked the details. Cheers, Steve ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/437UCQH3R4ZPGX4S23HQ74RGEKFPX35U/
[Python-Dev] Re: PyAPI_FUNC() is needed to private APIs?
On 2019-06-13 17:11, Steve Dower wrote: The cost of that convenience is that we can never optimise internals because they are now public API. I think that the opposite is true actually: the reason that people access internals is because there is no public API doing what they want. Having more public API should *reduce* the need for accessing internals. For example, _PyObject_GetMethod is not public API but it's useful functionality. So Cython is forced to reinvent _PyObject_GetMethod (i.e. copy verbatim that function from the CPython sources), which requires accessing internals. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/BQKHK2S4YFHKZT5XNIYCZ6WKWO2UA4WZ/
[Python-Dev] Re: PyAPI_FUNC() is needed to private APIs?
On 13Jun2019 0816, Jeroen Demeyer wrote: On 2019-06-13 17:11, Steve Dower wrote: The cost of that convenience is that we can never optimise internals because they are now public API. I think that the opposite is true actually: the reason that people access internals is because there is no public API doing what they want. Having more public API should *reduce* the need for accessing internals. Right, but we need to know what API that is. We can't just make everything public by default. For example, _PyObject_GetMethod is not public API but it's useful functionality. So Cython is forced to reinvent _PyObject_GetMethod (i.e. copy verbatim that function from the CPython sources), which requires accessing internals. What's wrong with using PyObject_GetAttr() and then doing PyType_HasFeature(Py_TYPE(descr), Py_TPFLAGS_METHOD_DESCRIPTOR) on the result? More importantly, why do you need to know that it's a method descriptor and not just a callable object that can be accessed via an attribute? And is this something that's generally needed, or is Cython just special (I expect Cython to be special in many situations, which is why we have a tiered API and expect Cython-generated code to be regenerated for different CPython versions). Once we start discussing actual API needs, we can get to real designs. Simply making everything public by default does not improve any designs. Cheers, Steve ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/HC7BTCWMFQAY3OHNRBPNKVKBXJSDH453/
[Python-Dev] Re: PyAPI_FUNC() is needed to private APIs?
On Fri, Jun 14, 2019 at 12:29 AM Jeroen Demeyer wrote: > > > I think that the opposite is true actually: the reason that people > access internals is because there is no public API doing what they want. > Having more public API should *reduce* the need for accessing internals. > > For example, _PyObject_GetMethod is not public API but it's useful > functionality. So Cython is forced to reinvent _PyObject_GetMethod (i.e. > copy verbatim that function from the CPython sources), which requires > accessing internals. This became off topic... but. In case of _PyObject_GetMethod, I agree that it means we don't provide *some* useful API. But it doesn't mean _PyObject_GetMethod is the missing useful API. We don't provide method calling API which uses optimization same to LOAD_METHOD. Which may be like this: /* methname is Unicode, nargs > 0, and args[0] is self. */ PyObject_VectorCallMethod(PyObject *methname, PyObject **args, Py_ssize_t nargs, PyObject *kwds) (Would you try adding this? Or may I?) Anyway, do you think _PyObject_GetMethod is useful even after we provide PyObject_VectorCallMethod ? I'd like to put _PyObject_GetMethod in private/ (with PyAPI_FUNC) in 3.9, for users like Cython. If you want to make it public in 3.9, please create new thread. Let's discuss about how it is useful, and current name and signature are good enough to make it public. Regards, -- Inada Naoki ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/Y5AG7N7YKO3GQ6H223JTIBMERGKSV7W5/
[Python-Dev] EuroPython 2019 sprints
Hi all Just letting everyone know that I signed up any core devs who are going to be at EuroPython to run sprints :) (including myself, since I'll be there) Like PyCon US, the sprints at EuroPython will attract many first-time contributors, so it will be helpful to have as many people who understand our workflow around to help out. We might also be able to get some of our own contributions done. For those who didn't hear, the EuroPython Society is offering a grant for core developers to get free attendance at EuroPython (see https://www.europython-society.org/core-grant). If you take this, it would be a nice gesture of thanks to help support the sprints! And in the meantime, if you see an easy bug on the tracker, please mark it as "Easy" or "Easy (C)" and post an explanation of how to fix it. That way we will have a good supply of tasks for new contributors. Cheers, Steve ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/CQNQCPXNWFZS6FEIAX3PDP6ASL27AI72/
[Python-Dev] Re: EuroPython 2019 sprints
There were some easy issues assigned to me, that were set aside for the mentored sprint. They can be worked on by new contributors during the Euro Python sprint. ᐧ On Thu, Jun 13, 2019 at 9:37 AM Steve Dower wrote: > Hi all > > Just letting everyone know that I signed up any core devs who are going > to be at EuroPython to run sprints :) (including myself, since I'll be > there) > > Like PyCon US, the sprints at EuroPython will attract many first-time > contributors, so it will be helpful to have as many people who > understand our workflow around to help out. We might also be able to get > some of our own contributions done. > > For those who didn't hear, the EuroPython Society is offering a grant > for core developers to get free attendance at EuroPython (see > https://www.europython-society.org/core-grant). If you take this, it > would be a nice gesture of thanks to help support the sprints! > > And in the meantime, if you see an easy bug on the tracker, please mark > it as "Easy" or "Easy (C)" and post an explanation of how to fix it. > That way we will have a good supply of tasks for new contributors. > > Cheers, > Steve > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/CQNQCPXNWFZS6FEIAX3PDP6ASL27AI72/ > ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/P22YJKCWK2DHG3P4VME5GZVOBYDUTL26/
[Python-Dev] Re: Expected stability of PyCode_New() and types.CodeType() signatures
On Thu, 13 Jun 2019 02:07:54 +0200 Victor Stinner wrote: > > For PyCodeOptions: does it contain posonlyargcount? If not, how do you > pass posonlyargcount. I'm not sure if you plan to handle the backward > compatibility. The idiom to use it could be: PyCodeOptions opts = PyCodeOptions_INIT; // set attributes here PyObject* code_obj = PyCode_NewEx(opts); So you can change the PyCodeOptions_INIT macro when some new fields are added. Regards Antoine. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/XLXKUDDEQLCAAJL366CSXIGWK34TED5G/
[Python-Dev] Re: PyAPI_FUNC() is needed to private APIs?
On Thu, 13 Jun 2019 22:05:01 +0900 Inada Naoki wrote: > > Currently, many private APIs uses `PyAPI_FUNC()`. It's easier to use PyAPI_FUNC() everywhere than forget it in some places and then bother Windows users. Private APIs are sometimes used in third-party modules if they want access to higher-performance or specific APIs. Regards Antoine. ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/P7ICOGW4ZHZ76R4RYIDSTQPDABSLIPUY/
