Re: [Python-Dev] Support of the Android platform
On 12/13/2017 10:56 PM, Victor Stinner wrote: > I looked at your scripts to build Android but I failed to use them. You failed because you did not read the README and tried to improvise. I will change the documentation and the build process to make it simpler for those that do not have the time to RTFM :-) I have documented the correct steps in the PR especially for you (they are just a summary of what is in the README) after you reported that failure. So you can follow these steps any time now and use correctly the build system and report the results here. This would be only fair, I think. BTW You keep saying they are scripts when the build is actually driven by Makefiles (with the '.mk' extension). > Anyway, I'm not sure why these scripts have to be part of the CPython > git repository. > > Technically, is there a reason to put it aside the source code and > Unix build scripts (configure/Makefile/setup)? > Your https://github.com/python/cpython/pull/1629 only adds new files > without touching existing files. > > I suggest to create new Git project. It may be in the python > organization, or you may start with your GitHub account. The 'Mac' build system has its own subdirectory in the source tree and it makes sense as it is the reference build system for this platform. I do not see why this should be different for Android. > Cross-compilation is hard, and I'm not sure that it's possible to > build a single recipe for all Android API versions, all configuration, > any set of libraries, etc. For Android, it seems like each developer > might want a subtle different configuration which might not be easy to > support. You are mistaken, this proposal does not suggest that we are going to support "all Android API versions, all configuration, any set of libraries, etc.", quite the opposite actually. The proposal is an Android build system for a specific API, a set of architectures using the NDK r14 toolchain and a set of optional external libraries. The build system enforces the use of NDK r14 for example (as you have painfuly experienced). Python is tested on emulators so that there is no interference with vendor specific additions found on the Android devices or with installed PlayStore applications. > Having a separated Git project would allow people to contribute more > easily, experiment their fork, etc. > > What do you think? Certainly not. We, core-devs, are very happy that no one is experimenting with our build system, it is complex enough as it is. The same goes for this Android build system. Your suggestion seems to be driven by the failure you have experienced with this new build system and the fact that a user is also reporting a failure. The origin of this other failure is unclear because I cannot reproduce it even though all the components used for the build are well defined and identical for everyone: the NDK includes the clang compiler, the libraries and the headers, the external libraries are downloaded by the build system, all the users use identical tools (same versions) and the same source code and the only difference may be with some utility tools such as sed, awk, etc... This is a bad start for this proposal and it would have been fair to inform me that you were working in irc collaboration with this other user in testing the build system. On the other hand, these problems may have some positive consequences since it allows us to be aware of the fact that the bpo audience may change if we support Android and that this may be a problem. Android attracts all kind of developers that do not have the average expertise of unix developers and more importantly that do not have the same motivations and the same etiquette. I am now concerned by the fact that the quality of the bug reports on bpo may dramatically decrease if we adopt this proposal. Xavier ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 489: module m_traverse called with NULL module state
On Thu, 14 Dec 2017 17:00:10 +1000 Nick Coghlan wrote: > On 14 Dec. 2017 9:19 am, "Antoine Pitrou" wrote: > > > Hello, > > After debugging a crash on AppVeyor for a submitter's PR > (see https://github.com/python/cpython/pull/4611 ), I came to the > following diagnosis: converting the "atexit" module (which is a > built-in C extension) to PEP 489 multiphase initialization can lead to > its m_traverse function (and presumably also m_clear and m_free) to be > called while not module state is yet registered: that is, > `PyModule_GetState(self)` when called from m_traverse returns NULL! > > Is that an expected or known subtlety? > > > Not that I'm aware of, so I'd be inclined to classify it as a bug in the > way we're handling multi-phase initialisation unless/until we determine > there's no way to preserve the existing invariant from the single phase > case. Speaking of which, the doc is not very clear: is PEP 489 required for multi-interpreter support or is PyModule_GetState() sufficient? Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] sqlite3 module needs maintainer
Hello, After noticing that many issues were opened for it and it was lacking maintenance, I contacted the sqlite3 module's historical author and maintainer, Gerhard Häring (for the record, Gerhard didn't make any changes to the sqlite3 module since 2011... and the Python 2-only, third-party "pysqlite" module, which he maintains as well, did not receive many changes lately). He answered me that he was ok to declare the sqlite3 module as officially unmaintained. Since sqlite3 is such a useful and widely-used standard library module, it probably deserves someone competent and motivated to maintain it. Berker Peksağ is also interested in sqlite3 and is interested in helping maintain it (I'm trying to channel his private words here... I hope I don't misrepresent his position). Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 489: module m_traverse called with NULL module state
On 12/14/2017 12:00 PM, Antoine Pitrou wrote: On Thu, 14 Dec 2017 17:00:10 +1000 Nick Coghlan wrote: On 14 Dec. 2017 9:19 am, "Antoine Pitrou" wrote: Hello, After debugging a crash on AppVeyor for a submitter's PR (see https://github.com/python/cpython/pull/4611 ), I came to the following diagnosis: converting the "atexit" module (which is a built-in C extension) to PEP 489 multiphase initialization can lead to its m_traverse function (and presumably also m_clear and m_free) to be called while not module state is yet registered: that is, `PyModule_GetState(self)` when called from m_traverse returns NULL! Is that an expected or known subtlety? Thank you for looking into this, Antoine! Not that I'm aware of, so I'd be inclined to classify it as a bug in the way we're handling multi-phase initialisation unless/until we determine there's no way to preserve the existing invariant from the single phase case. Yes, it's a bug – at least in documentation. From initial investigation, the problem is that between the two phases of multi-phase init, module state is NULL, and Python code can run. This is expected, so I'm thinking m_traverse for all modules using multi-phase init should have a check for NULL. And this should be documented. Let's have Marcel run with this a bit further. Speaking of which, the doc is not very clear: is PEP 489 required for multi-interpreter support or is PyModule_GetState() sufficient? I'm not exactly sure what you're asking; which doc are you referring to? PEP 489 gives you good defaults, if you use it and avoid global state (roughly: C-level mutable static variables), then you should get multi-interpreter support for free in simple cases. It's also possible to use PyModule_GetState() and other APIs directly. However, I'd like to avoid solving subinterpreter support separately (and slightly differently) in each module. For a slightly bigger picture: as a part-time internship, Marcel is identifying where PEP 489 is inadequate, and solving the problems for the complex cases. This is part of better support for subinterpreter support in general. Going with a PEP 489-based solution with atexit would help us in that effort. I'm assuming fixing the atexit bug from 2009 [0] can be delayed a bit as issues with PEP 489 are investigated & solved. Does that sound fair? [0] https://bugs.python.org/issue6531 ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support of the Android platform
2017-12-14 10:26 GMT+01:00 Xavier de Gaye : > The 'Mac' build system has its own subdirectory in the source tree and it > makes sense as it is the reference build system for this platform. I do not > see why this should be different for Android. Hum, Mac/ is mostly the recipe to build the installer and a .dmg image, no? macOS uses the same configure/Makefile than Linux, no? Building Python on macOS is quite simple, there is nothing special about macOS. And macOS is used on a *very* limited set of hardware, only sold by a single vendor, Apple. Android is quite the opposite. The propose Android directory is complex recipe for cross-compiling for a wide range of hardware, and likely different usages of Python. Some people may want a REPL, some people may only be interested by a GUI (like Kivy or Panda3D?), some people probably want to do something else. All Python modules work on macOS, since macOS is just one flavor of POSIX. While Android is compatible with POSIX, from the bug reports that I saw, many modules don't work and will not work on Android. So supporting Android is much more complex than supporting macOS. > You are mistaken, this proposal does not suggest that we are going to > support "all Android API versions, all configuration, any set of libraries, > etc.", quite the opposite actually. Since we are talking about the future, I would like to remain open to support wider Android API versions and any configuration. As I wrote, I don't require to support all configurations and all API versions, but just provide best effort support, and only fully support one specific API version and one specific config. > The build system enforces the use of NDK r14 for example (as you > have painfuly experienced). It seems like Android is evolving quickly, I would say quicker than Python releases. I'm asking if it's a good idea to put a recipe aside the Python source code for one specific Android API version? Would it still make sense to build for NDK v14 in 2 or 5 years? > This is a bad start for this proposal and it would have been fair to inform > me that you were working in irc collaboration with this other user in > testing the build system. Right, I'm working with Paul Peny (pmpp on IRC). He is helping me to test your PR. Paul understand Android much better than me. For me, it's still a huge black box. Basically, I don't understand what I'm doing :-) Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 489: module m_traverse called with NULL module state
On Thu, 14 Dec 2017 15:05:02 +0100 Petr Viktorin wrote: > > PEP 489 gives you good defaults, if you use it and avoid global state > (roughly: C-level mutable static variables), then you should get > multi-interpreter support for free in simple cases. > It's also possible to use PyModule_GetState() and other APIs directly. > However, I'd like to avoid solving subinterpreter support separately > (and slightly differently) in each module. My question is: can you get multi-interpreter support *without* PEP 489? That is, using single-phase initialization and PyModule_GetState(). > For a slightly bigger picture: as a part-time internship, Marcel is > identifying where PEP 489 is inadequate, and solving the problems for > the complex cases. Is Marcel mentored by anyone in particular? > I'm assuming fixing the atexit bug from 2009 [0] can be delayed a bit as > issues with PEP 489 are investigated & solved. > Does that sound fair? Probably, but the atexit bug deserves fixing in itself. If a fix is ready, it would be a pity not to let it in. Regards Antoine. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 432 progress: Python initalization
Hi, Serhiy Storchaka seems to be worried by the high numbers of commits in https://bugs.python.org/issue32030 "PEP 432: Rewrite Py_Main()", so let me explain the context of this work :-) To prepare CPython to implement my UTF-8 Mode PEP (PEP 540), I worked on the implementation of Nick Coghlan's PEP 432: PEP 432 -- Restructuring the CPython startup sequence https://www.python.org/dev/peps/pep-0432/ The startup sequence is a big pile of code made of multiple functions: main(), Py_Main(), Py_Initialize(), Py_Finalize()... and a lot of tiny "configuration" functions like Py_SetPath(). Over the years, many configuration options were added in the middle of the code. The priority of configuration options is not always correct between command line options, envrionment variables, configuration files (like "pyenv.cfg"), etc. For technical reasons, it's hard to impement properly the -E option (ignore PYTHON* environment variables). For example, the new PYTHONCOERCECLOCALE environment variable (of PEP 538) doesn't handle properly -E (it ignores -E), because it was too complex to support -E. -- I'm working on fixing this. Last weeks, I mostly worked on the Py_Main() function, Modules/getpath.c and PC/getpathp.c, to "refactor" the code: * Split big functions (300 to 500 lines) into multiple small functions (50 lines or less), to make it easily to follow the control flow and to allow to more easily move code * Replace static and global variables with memory allocated on the heap. * Reorganize how the configuration is read: populate a first temporary structure (_PyMain using wchar_t*), then create Python objects (_PyMainInterpreterConfig) to finish with the real configuration (like setting attributes of the sys module). The goal is to centralize all code reading configuration to fix the priority and to simplify the code. My motivation was to write a correct implementation of the UTF-8 Mode (PEP 540). Nick's motivation is to make CPython easily to embed. His plan for Python 3.8 is to give access to the new _PyCoreConfig and _PyMainInterpreterConfig structures to: * easily give access to most (if not all?) configuration options to "embedders" * allow to configure Python without environment variables, command line options, configuration files, but only using these structures * allow to configure Python using Python objects (PyObject*) rather than C types (like wchar_t*) (I'm not sure that I understood correctly, so please read the PEP 432 ;-)) IMHO the most visible change of the PEP 432 is to split Python initialization in two parts: * Core: strict minimum to use the Python C API * Main: everything else The goal is to introduce the opportunity to configure Python between Core and Main. The implementation is currently a work-in-progress. Nick will not have the bandwidth, neither do I, to update his PEP and finish the implementation, before Python 3.7. So this work remains private until at least Python 3.8. Another part of the work is to enhance the documentation. You can for example now find an explicit list of C functions which can be called before Py_Initialize(): https://docs.python.org/dev/c-api/init.html#before-python-initialization And also a list of functions that must not be called before Py_Initialize(), whereas you might want to call them :-) Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 432 progress: Python initalization
Currently, we have the following configuration options:
typedef struct {
int ignore_environment; /* -E */
int use_hash_seed; /* PYTHONHASHSEED=x */
unsigned long hash_seed;
int _disable_importlib; /* Needed by freeze_importlib */
const char *allocator; /* Memory allocator: _PyMem_SetupAllocators() */
int dev_mode; /* -X dev */
int faulthandler; /* -X faulthandler */
int tracemalloc;/* -X tracemalloc=N */
int import_time;/* -X importtime */
int show_ref_count; /* -X showrefcount */
int show_alloc_count; /* -X showalloccount */
int dump_refs; /* PYTHONDUMPREFS */
int malloc_stats; /* PYTHONMALLOCSTATS */
int utf8_mode; /* -X utf8 or PYTHONUTF8 environment variable */
wchar_t *module_search_path_env; /* PYTHONPATH environment variable */
wchar_t *home; /* PYTHONHOME environment variable,
see also Py_SetPythonHome(). */
wchar_t *program_name; /* Program name, see also Py_GetProgramName() */
} _PyCoreConfig;
and
typedef struct {
int install_signal_handlers;
PyObject *argv; /* sys.argv list, can be NULL */
PyObject *module_search_path; /* sys.path list */
PyObject *warnoptions; /* sys.warnoptions list, can be NULL */
PyObject *xoptions; /* sys._xoptions dict, can be NULL */
} _PyMainInterpreterConfig;
Victor
2017-12-14 16:16 GMT+01:00 Victor Stinner :
> Hi,
>
> Serhiy Storchaka seems to be worried by the high numbers of commits in
> https://bugs.python.org/issue32030 "PEP 432: Rewrite Py_Main()", so
> let me explain the context of this work :-)
>
> To prepare CPython to implement my UTF-8 Mode PEP (PEP 540), I worked
> on the implementation of Nick Coghlan's PEP 432:
>
> PEP 432 -- Restructuring the CPython startup sequence
> https://www.python.org/dev/peps/pep-0432/
>
> The startup sequence is a big pile of code made of multiple functions:
> main(), Py_Main(), Py_Initialize(), Py_Finalize()... and a lot of tiny
> "configuration" functions like Py_SetPath().
>
> Over the years, many configuration options were added in the middle of
> the code. The priority of configuration options is not always correct
> between command line options, envrionment variables, configuration
> files (like "pyenv.cfg"), etc. For technical reasons, it's hard to
> impement properly the -E option (ignore PYTHON* environment
> variables).
>
> For example, the new PYTHONCOERCECLOCALE environment variable (of PEP
> 538) doesn't handle properly -E (it ignores -E), because it was too
> complex to support -E. -- I'm working on fixing this.
>
> Last weeks, I mostly worked on the Py_Main() function,
> Modules/getpath.c and PC/getpathp.c, to "refactor" the code:
>
> * Split big functions (300 to 500 lines) into multiple small functions
> (50 lines or less), to make it easily to follow the control flow and
> to allow to more easily move code
>
> * Replace static and global variables with memory allocated on the heap.
>
> * Reorganize how the configuration is read: populate a first temporary
> structure (_PyMain using wchar_t*), then create Python objects
> (_PyMainInterpreterConfig) to finish with the real configuration (like
> setting attributes of the sys module). The goal is to centralize all
> code reading configuration to fix the priority and to simplify the
> code.
>
> My motivation was to write a correct implementation of the UTF-8 Mode (PEP
> 540).
>
> Nick's motivation is to make CPython easily to embed. His plan for
> Python 3.8 is to give access to the new _PyCoreConfig and
> _PyMainInterpreterConfig structures to:
>
> * easily give access to most (if not all?) configuration options to
> "embedders"
> * allow to configure Python without environment variables, command
> line options, configuration files, but only using these structures
> * allow to configure Python using Python objects (PyObject*) rather
> than C types (like wchar_t*)
>
> (I'm not sure that I understood correctly, so please read the PEP 432 ;-))
>
> IMHO the most visible change of the PEP 432 is to split Python
> initialization in two parts:
>
> * Core: strict minimum to use the Python C API
> * Main: everything else
>
> The goal is to introduce the opportunity to configure Python between
> Core and Main.
>
> The implementation is currently a work-in-progress. Nick will not have
> the bandwidth, neither do I, to update his PEP and finish the
> implementation, before Python 3.7. So this work remains private until
> at least Python 3.8.
>
> Another part of the work is to enhance the documentation. You can for
> example now find an explicit list of C functions which can be called
> before Py_Initialize():
>
> https://docs.python.org/dev/c-api/init.html#before-python-initialization
>
> And also a list of functions that must not be called before
> Py_Initialize(), whereas you might want to call them :-)
>
> Victor
__
Re: [Python-Dev] sqlite3 module needs maintainer
SGTM. It's one of my favorite stdlib modules wrapping an external library -- I use it for a variety of tasks to which it is well-suited. Go Berker! On Thu, Dec 14, 2017 at 4:42 AM, Antoine Pitrou wrote: > > Hello, > > After noticing that many issues were opened for it and it was lacking > maintenance, I contacted the sqlite3 module's historical author and > maintainer, Gerhard Häring (for the record, Gerhard didn't make any > changes to the sqlite3 module since 2011... and the Python 2-only, > third-party "pysqlite" module, which he maintains as well, did not > receive many changes lately). He answered me that he was ok to declare > the sqlite3 module as officially unmaintained. > > Since sqlite3 is such a useful and widely-used standard library module, > it probably deserves someone competent and motivated to maintain it. > Berker Peksağ is also interested in sqlite3 and is interested in > helping maintain it (I'm trying to channel his private words here... I > hope I don't misrepresent his position). > > Regards > > Antoine. > > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 432 progress: Python initalization
On 12/14/2017 10:16 AM, Victor Stinner wrote: Hi, Serhiy Storchaka seems to be worried by the high numbers of commits in https://bugs.python.org/issue32030 "PEP 432: Rewrite Py_Main()", so let me explain the context of this work :-) You could have (and still could) made that a master issue with multiple dependencies. Last summer, I merged at least 20 patches for one idlelib file. I split them up among 1 master issue and about 6 dependency issues. That was essential because most of the patches were written by one of 3 new contributors and needed separate discussions about the strategy for a particular patch. I completely agree with keeping PRs to a reviewable size. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 432 progress: Python initalization
2017-12-14 22:54 GMT+01:00 Terry Reedy : > You could have (and still could) made that a master issue with multiple > dependencies. Last summer, I merged at least 20 patches for one idlelib > file. I split them up among 1 master issue and about 6 dependency issues. > That was essential because most of the patches were written by one of 3 new > contributors and needed separate discussions about the strategy for a > particular patch. > > I completely agree with keeping PRs to a reviewable size. I'm not sure that multiple issues are needed since all these changes are related to Py_Main() or are very close to Py_Main(), and they implement what is defined in the PEP 432. Technically, I could push a single giant commit, but it would be impossible to review it, even for myself, whereas I'm reading each change multiple times. I'm testing each change on Windows, macOS, Linux and FreeBSD to make sure that everything is fine. Py_Main() has a few functions specific to one platform like Windows or macOS. I also had to "iterate" on the code to move slowly the code, step by step. I'm not really proud of all these refactoring changes :-( But I hope that "at the end", the code will be much easier to understand and to maintain. Moreover, as I wrote, my intent is also to fix all the code handling configuration. For example, I just fixed the code to define sys.argv earlier. Now, sys.argv is defined very soon in Python initialization. Previously, sys.argv was only defined after Py_Initialize() completed. For example, the site module cannot access sys.argv: Traceback (most recent call last): File "/home/vstinner/prog/python/3.6/Lib/site.py", line 600, in print(sys.argv) AttributeError: module 'sys' has no attribute 'argv' I'm not sure that it's useful, but I was surprised that sys was only partially initialized before the site moduel was loaded. Victor ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepting PEP 560 -- Core support for typing module and generic types
Ivan, Guido,
Would it be possible to add a slot so that types defined in C can
implement __class_getitem__?
static PyClassMethodDef class_methods = {
foo_class_getitem /* cm_class_getitem */
}
static PyTypeObject Foo = {
.tp_class_methods = class_methods
}
Yury
On Mon, Dec 4, 2017 at 5:18 PM, Ivan Levkivskyi wrote:
> Thank you! It looks like we have a bunch of accepted PEPs today.
> It is great to see all this! Thanks everyone who participated in discussions
> here, on python-ideas and
> on typing tracker. Special thanks to Mark who started this discussion.
>
> --
> Ivan
>
>
>
> ___
> Python-Dev mailing list
> [email protected]
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com
>
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepting PEP 560 -- Core support for typing module and generic types
A slot is pretty expensive, as *every* class in existence will be another 8
bytes larger (and possibly more due to malloc rounding). So unless we find
that there's a significant performance benefit I think we should hold back
on this. IIRC Ivan has already measured an order of magnitude's speedup
(well, 7x), so we may not need it. :-)
On Thu, Dec 14, 2017 at 3:49 PM, Yury Selivanov
wrote:
> Ivan, Guido,
>
> Would it be possible to add a slot so that types defined in C can
> implement __class_getitem__?
>
> static PyClassMethodDef class_methods = {
> foo_class_getitem /* cm_class_getitem */
> }
>
> static PyTypeObject Foo = {
> .tp_class_methods = class_methods
> }
>
> Yury
>
> On Mon, Dec 4, 2017 at 5:18 PM, Ivan Levkivskyi
> wrote:
> > Thank you! It looks like we have a bunch of accepted PEPs today.
> > It is great to see all this! Thanks everyone who participated in
> discussions
> > here, on python-ideas and
> > on typing tracker. Special thanks to Mark who started this discussion.
> >
> > --
> > Ivan
> >
> >
> >
> > ___
> > Python-Dev mailing list
> > [email protected]
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/
> yselivanov.ml%40gmail.com
> >
>
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepting PEP 560 -- Core support for typing module and generic types
On Thu, Dec 14, 2017 at 7:03 PM, Guido van Rossum wrote: > A slot is pretty expensive, as *every* class in existence will be another 8 > bytes larger (and possibly more due to malloc rounding). So unless we find > that there's a significant performance benefit I think we should hold back > on this. IIRC Ivan has already measured an order of magnitude's speedup > (well, 7x), so we may not need it. :-) My motivation to add the slot wasn't the performance: it's just not possible to have a class-level __getitem__ on types defined in C. The only way is to define a base class in C and then extend it in pure-Python. This isn't too hard usually, though. BTW that slot could also host the new __mro_entries__ method, and, potentially, other magic methods like __subclasscheck__ and __instancecheck__. Yury ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepting PEP 560 -- Core support for typing module and generic types
On Thu, 14 Dec 2017 16:03:48 -0800 Guido van Rossum wrote: > A slot is pretty expensive, as *every* class in existence will be another 8 > bytes larger (and possibly more due to malloc rounding). I'm always surprised by the discussions about class object size. Even imagining you have 1 classes in memory (a pretty large number, though I'm sure you can reach that number with a lot of dependencies), we're talking about a total 800 kB memory growth (let's recall that each of those classes will probably have code objects, docstrings and what not attached to it -- i.e. you don't often create empty classes). Is it really an important concern? Regards Antoine. PS: simple experiment at an IPython prompt, trying to load every large third-party package I have lying around (I may be forgetting some). >>> def count_classes(): ...:types = [object] ...:seen = set(types) ...:while types: ...:types = [c for c in itertools.chain.from_iterable(type.__subclasses__(c) for c in types) ...: if c not in seen] ...:seen.update(types) ...:return len(seen) ...: >>> import numpy, asyncio, cython, requests, pandas, curio >>> import django.apps, django.contrib, django.db, django.forms, django.http, >>> django.middleware, django.views >>> import twisted.internet.reactor, twisted.web >>> import tornado.ioloop, tornado.gen, tornado.locks >>> len(sys.modules) 1668 >>> count_classes() 6130 At this point, the IPython process uses 113 MB RSS, which adding a 8-byte slot to each of those 6130 classes would increase by a mere 49 kB. And I'm not even doing anything useful (no user data) with all those modules, so an actual application using those modules would weigh much more. (and for the curious, the actual list of classes: https://gist.github.com/pitrou/8bd03dbb480f5acbc3abbe6782df5ebd) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepting PEP 560 -- Core support for typing module and generic types
On Thu, Dec 14, 2017 at 4:29 PM Yury Selivanov wrote: > On Thu, Dec 14, 2017 at 7:03 PM, Guido van Rossum > wrote: > My motivation to add the slot wasn't the performance: it's just not > possible to have a class-level __getitem__ on types defined in C. The > only way is to define a base class in C and then extend it in > pure-Python. This isn't too hard usually, though. This could potentially make it much more complicated to adding typing support to NumPy. numpy.ndarray is defined in C. ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepting PEP 560 -- Core support for typing module and generic types
In the light of Antoine's and Stephan's feedback I think this can be reconsidered -- while I want to take a cautious stance about resource consumption I don't want to stand in the way of progress. On Thu, Dec 14, 2017 at 4:36 PM, Stephan Hoyer wrote: > On Thu, Dec 14, 2017 at 4:29 PM Yury Selivanov > wrote: > >> On Thu, Dec 14, 2017 at 7:03 PM, Guido van Rossum >> wrote: >> My motivation to add the slot wasn't the performance: it's just not >> possible to have a class-level __getitem__ on types defined in C. The >> only way is to define a base class in C and then extend it in >> pure-Python. This isn't too hard usually, though. > > > This could potentially make it much more complicated to adding typing > support to NumPy. numpy.ndarray is defined in C. > > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
Hi, folks. TLDR, was the final decision made already? If "dict keeps insertion order" is not language spec and we continue to recommend people to use OrderedDict to keep order, I want to optimize OrderedDict for creation/iteration and memory usage. (See https://bugs.python.org/issue31265#msg301942 ) If dict ordering is language spec, I'll stop the effort and use remaining time to another optimizations. My thought is, +1 to make it language spec. * PHP (PHP 7.2 interpreter is faster than Python) keeps insertion order. So even we make it language spec, I think we have enough room to optimize. * It can make stop discussion like "Does X keeps insertion order? It's language spec?", "What about Y? Z?". Everything on top of dict keeps insertion order. It's simple to learn and explain. Regards, INADA Naoki On Sun, Nov 5, 2017 at 3:35 AM, Guido van Rossum wrote: > This sounds reasonable -- I think when we introduced this in 3.6 we were > worried that other implementations (e.g. Jython) would have a problem with > this, but AFAIK they've reported back that they can do this just fine. So > let's just document this as a language guarantee. > > On Sat, Nov 4, 2017 at 10:30 AM, Stefan Krah wrote: >> >> >> Hello, >> >> would it be possible to guarantee that dict literals are ordered in v3.7? >> >> >> The issue is well-known and the workarounds are tedious, example: >> >> >> https://mail.python.org/pipermail/python-ideas/2015-December/037423.html >> >> >> If the feature is guaranteed now, people can rely on it around v3.9. >> >> >> >> Stefan Krah >> >> >> >> ___ >> Python-Dev mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido) > > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
I'm in favor of stating that dict keeps order as part of the language spec. However re-reading https://mail.python.org/pipermail/python-dev/2017-November/150381.html there's still a point of debate: should it be allowed if dict reorders after deletion (presumably as a result of a rehash)? I'm in favor of prescribing that the order should be preserved even in that case, but I realize there's additional implementation work to be done. Inada-san, what do you think of this? --Guido On Thu, Dec 14, 2017 at 6:03 PM, INADA Naoki wrote: > Hi, folks. > > TLDR, was the final decision made already? > > If "dict keeps insertion order" is not language spec and we > continue to recommend people to use OrderedDict to keep > order, I want to optimize OrderedDict for creation/iteration > and memory usage. (See https://bugs.python.org/issue31265#msg301942 ) > > If dict ordering is language spec, I'll stop the effort and > use remaining time to another optimizations. > > My thought is, +1 to make it language spec. > > * PHP (PHP 7.2 interpreter is faster than Python) keeps insertion order. > So even we make it language spec, I think we have enough room > to optimize. > > * It can make stop discussion like "Does X keeps insertion order? > It's language spec?", "What about Y? Z?". Everything on top of dict > keeps insertion order. It's simple to learn and explain. > > Regards, > INADA Naoki > > > On Sun, Nov 5, 2017 at 3:35 AM, Guido van Rossum wrote: > > This sounds reasonable -- I think when we introduced this in 3.6 we were > > worried that other implementations (e.g. Jython) would have a problem > with > > this, but AFAIK they've reported back that they can do this just fine. So > > let's just document this as a language guarantee. > > > > On Sat, Nov 4, 2017 at 10:30 AM, Stefan Krah > wrote: > >> > >> > >> Hello, > >> > >> would it be possible to guarantee that dict literals are ordered in > v3.7? > >> > >> > >> The issue is well-known and the workarounds are tedious, example: > >> > >> > >> https://mail.python.org/pipermail/python-ideas/2015- > December/037423.html > >> > >> > >> If the feature is guaranteed now, people can rely on it around v3.9. > >> > >> > >> > >> Stefan Krah > >> > >> > >> > >> ___ > >> Python-Dev mailing list > >> [email protected] > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > > > > > -- > > --Guido van Rossum (python.org/~guido) > > > > ___ > > Python-Dev mailing list > > [email protected] > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/ > songofacandy%40gmail.com > > > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
Oh, I just found https://mail.python.org/pipermail/python-dev/2017-November/150323.html so I already know what you think: we should go with "dicts preserve insertion order" rather than "dicts preserve insertion order until the first deletion". I guess we should wait for Serhiy to confirm that he's okay with this. On Thu, Dec 14, 2017 at 6:20 PM, Guido van Rossum wrote: > I'm in favor of stating that dict keeps order as part of the language spec. > > However re-reading https://mail.python.org/pipermail/python-dev/2017- > November/150381.html there's still a point of debate: should it be > allowed if dict reorders after deletion (presumably as a result of a > rehash)? I'm in favor of prescribing that the order should be preserved > even in that case, but I realize there's additional implementation work to > be done. Inada-san, what do you think of this? > > --Guido > > On Thu, Dec 14, 2017 at 6:03 PM, INADA Naoki > wrote: > >> Hi, folks. >> >> TLDR, was the final decision made already? >> >> If "dict keeps insertion order" is not language spec and we >> continue to recommend people to use OrderedDict to keep >> order, I want to optimize OrderedDict for creation/iteration >> and memory usage. (See https://bugs.python.org/issue31265#msg301942 ) >> >> If dict ordering is language spec, I'll stop the effort and >> use remaining time to another optimizations. >> >> My thought is, +1 to make it language spec. >> >> * PHP (PHP 7.2 interpreter is faster than Python) keeps insertion order. >> So even we make it language spec, I think we have enough room >> to optimize. >> >> * It can make stop discussion like "Does X keeps insertion order? >> It's language spec?", "What about Y? Z?". Everything on top of dict >> keeps insertion order. It's simple to learn and explain. >> >> Regards, >> INADA Naoki >> >> >> On Sun, Nov 5, 2017 at 3:35 AM, Guido van Rossum >> wrote: >> > This sounds reasonable -- I think when we introduced this in 3.6 we were >> > worried that other implementations (e.g. Jython) would have a problem >> with >> > this, but AFAIK they've reported back that they can do this just fine. >> So >> > let's just document this as a language guarantee. >> > >> > On Sat, Nov 4, 2017 at 10:30 AM, Stefan Krah >> wrote: >> >> >> >> >> >> Hello, >> >> >> >> would it be possible to guarantee that dict literals are ordered in >> v3.7? >> >> >> >> >> >> The issue is well-known and the workarounds are tedious, example: >> >> >> >> >> >> https://mail.python.org/pipermail/python-ideas/2015-December >> /037423.html >> >> >> >> >> >> If the feature is guaranteed now, people can rely on it around v3.9. >> >> >> >> >> >> >> >> Stefan Krah >> >> >> >> >> >> >> >> ___ >> >> Python-Dev mailing list >> >> [email protected] >> >> https://mail.python.org/mailman/listinfo/python-dev >> >> Unsubscribe: >> >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > >> > >> > >> > >> > -- >> > --Guido van Rossum (python.org/~guido) >> > >> > ___ >> > Python-Dev mailing list >> > [email protected] >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> > https://mail.python.org/mailman/options/python-dev/songofaca >> ndy%40gmail.com >> > >> > > > > -- > --Guido van Rossum (python.org/~guido) > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
> On Dec 14, 2017, at 6:03 PM, INADA Naoki wrote: > > If "dict keeps insertion order" is not language spec and we > continue to recommend people to use OrderedDict to keep > order, I want to optimize OrderedDict for creation/iteration > and memory usage. (See https://bugs.python.org/issue31265#msg301942 ) I support having regular dicts maintain insertion order but am opposed to Inada changing the implementation of collections.OrderedDict We can have the first without having the second. Over the holidays, I hope to have time to do further analysis and create convincing demonstrations of why we want to keep the doubly-linked list implementation for collections.OrderedDict(). The current regular dictionary is based on the design I proposed several years ago. The primary goals of that design were compactness and faster iteration over the dense arrays of keys and values. Maintaining order was an artifact rather than a design goal. The design can maintain order but that is not its specialty. In contrast, I gave collections.OrderedDict a different design (later coded in C by Eric Snow). The primary goal was to have efficient maintenance of order even for severe workloads such at that imposed by the lru_cache which frequently alters order without touching the underlying dict. Intentionally, the OrderedDict has a design that prioritizes ordering capabilities at the expense of additional memory overhead and a constant factor worse insertion time. It is still my goal to have collections.OrderedDict have a different design with different performance characteristics than regular dicts. It has some order specific methods that regular dicts don't have (such as a move_to_end() and a popitem() that pops efficiently from either end). The OrderedDict needs to be good at those operations because that is what differentiates it from regular dicts. The tracker issue https://bugs.python.org/issue31265 is assigned to me and I currently do not approve of it going forward. The sentiment is nice but it undoes very intentional design decisions. In the upcoming months, I will give it additional study and will be open minded but it is not cool to use a python-dev post as a way to do an end-run around my objections. Back to the original topic of ordering, it is my feeling that it was inevitable that sooner or later we would guarantee ordering for regular dicts. Once we had a performant implementation, the decision would be dominated by how convenient it is users. Also, a single guarantee is simpler for everyone and is better than having a hodgepodge of rules stating that X and Y are guaranteed while Z is not. I think an ordering guarantee for regular dicts would be a nice Christmas present for our users and developers. Cheers, Raymond ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
On Dec 14, 2017 21:30, "Raymond Hettinger" wrote: > On Dec 14, 2017, at 6:03 PM, INADA Naoki wrote: > > If "dict keeps insertion order" is not language spec and we > continue to recommend people to use OrderedDict to keep > order, I want to optimize OrderedDict for creation/iteration > and memory usage. (See https://bugs.python.org/issue31265#msg301942 ) I support having regular dicts maintain insertion order but am opposed to Inada changing the implementation of collections.OrderedDict We can have the first without having the second. It seems like the two quoted paragraphs are in vociferous agreement. -n ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
> I support having regular dicts maintain insertion order but am opposed to > Inada changing the implementation of collections.OrderedDict We can have > the first without having the second. > > It seems like the two quoted paragraphs are in vociferous agreement. The referenced tracker entry proposes, "Issue31265: Remove doubly-linked list from C OrderedDict". I don't think that should go forward regardless of whether regular dict order is guaranteed. Inada presented a compound proposition: either guarantee regular dict order or let him rip out the core design of OrderedDicts against my wishes. Raymond ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepting PEP 560 -- Core support for typing module and generic types
On Thu, Dec 14, 2017 at 9:00 PM, Guido van Rossum wrote: > In the light of Antoine's and Stephan's feedback I think this can be > reconsidered -- while I want to take a cautious stance about resource > consumption I don't want to stand in the way of progress. I've created an issue to discuss this further: https://bugs.python.org/issue32332 Yury ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepting PEP 560 -- Core support for typing module and generic types
15.12.17 02:25, Yury Selivanov пише: My motivation to add the slot wasn't the performance: it's just not possible to have a class-level __getitem__ on types defined in C. The only way is to define a base class in C and then extend it in pure-Python. This isn't too hard usually, though. What are problems? How this differs from __sizeof__ and __getstate__? ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
